System

Abstract

A system is provided.SOLUTION: A system comprising: means for measuring environmental data and health data of a user by a sensor device; server means for receiving the measurement data from the sensor device and analyzing the measurement data to detect an abnormality; means for creating an emergency notification and transmitting the emergency notification to an emergency contact when the abnormality is detected; and means for the emergency contact who has received the emergency notification to perform a corresponding action.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For elderly people to maintain their independence and live safely, they need a system that monitors their health condition and living environment in real time and responds quickly if an abnormality occurs. However, existing systems have many issues. For example, sensors function independently and data is not managed in real time, abnormality detection is inaccurate, and emergency response is slow. A comprehensive system is needed to solve these issues and improve the quality of life for elderly people. [Means for solving the problem]

[0005] This invention provides a system that includes a means for measuring environmental data and user health data using sensor devices, a server means for receiving the measurement data from the sensor devices and analyzing it to detect abnormalities, a means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, and a means for the emergency contact to take action upon receiving the emergency notification. The system also includes a means for using an AI model to detect abnormalities. By providing a system that receives data from sensor devices in real time, detects abnormalities with high accuracy using the AI ​​model, and can respond quickly, the system effectively supports the independence and safety of the elderly.

[0006] A "sensor device" is a device used to measure a user's environmental and health data.

[0007] "Environmental data" refers to information such as ambient temperature, humidity, sound, and movement.

[0008] "Health data" refers to physiological data such as a user's heart rate, blood pressure, and activity level.

[0009] A "server" is a computer system that analyzes data received from sensor devices and detects abnormalities.

[0010] "Abnormal" refers to data or behavior that deviates from the normal range and requires emergency response.

[0011] An "AI model" is an artificial intelligence algorithm that analyzes collected data and detects anomalies.

[0012] "Emergency notification" refers to the transmission of information to emergency contacts that is generated when an abnormality is detected.

[0013] "Emergency contacts" are contact details for family members, care services, etc. who should be contacted when an abnormality is detected.

[0014] A "response action" is an action taken by an emergency contact upon receiving an emergency notification to ensure the user's safety. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a comprehensive system to support the independence and safety of the elderly, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts.

[0037] System configuration and operation overview

[0038] The role of sensor devices

[0039] Terminal

[0040] The terminal includes various sensor devices for monitoring the elderly's living environment and health condition, including temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and transmit it to a server at regular intervals.

[0041] Sensor devices are placed in various rooms in the home to measure temperature, humidity, sound, and movement, while wearable devices worn by users measure health data such as heart rate, blood pressure, and activity levels.

[0042] Server Roles

[0043] server

[0044] The server centrally receives and analyzes the data sent from the sensor devices. The server is equipped with an AI model that analyzes the received data and detects anomalies.

[0045] The received data is first validated to ensure it is normal. Next, the data is stored in a database and analyzed using an AI model. If an abnormality is detected as a result of the analysis, an emergency notification is generated and sent to emergency contacts.

[0046] Emergency notification and response actions

[0047] User

[0048] Users, i.e., elderly people and their families, receive emergency notifications from the server. The emergency notifications are sent to pre-registered emergency contacts, who then receive the notifications via SMS, email, push notifications, etc.

[0049] For example, if a user falls at home, a motion detection sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family. The family will receive the notification and take prompt action.

[0050] Specific examples

[0051] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[0052] Terminal

[0053] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[0054] server

[0055] The server receives this data and uses an AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[0056] User

[0057] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[0058] In this way, the system is designed to ensure the independence and safety of the elderly and to respond quickly in the event of an emergency.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] Terminal

[0062] Sensor devices measure the user's environmental and health data in real time. For example, a temperature sensor measures the room temperature, and a heart rate sensor measures the user's heart rate. The measured data is collected into data packets at regular intervals.

[0063] Step 2:

[0064] Terminal

[0065] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and is sent to the server via an HTTP POST request.

[0066] Step 3:

[0067] server

[0068] The server receives the data sent from the sensor device, and the received data is first validated to ensure that the data structure and format are correct.

[0069] Step 4:

[0070] server

[0071] Data that passes validation is stored in a time-series database, allowing past and present data to be properly managed and used for subsequent analysis.

[0072] Step 5:

[0073] server

[0074] After the data is saved, it is analyzed by an AI model, which detects anomalies based on the received data and determines whether or not there are any anomalies.

[0075] Step 6:

[0076] server

[0077] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, and location information.

[0078] Step 7:

[0079] server

[0080] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[0081] Step 8:

[0082] User

[0083] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[0084] Step 9:

[0085] User

[0086] For example, if a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe.

[0087] In this way, the system works by linking the sensor device, server, and user (emergency contact) sections to detect abnormalities and facilitate rapid response.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] To effectively support the safety and independence of elderly people, a system that monitors environmental and health data in real time and responds quickly when an abnormality occurs is required. However, conventional systems have issues with data validation, the accuracy of abnormality detection, and methods for sending emergency notifications, and there is a lack of a comprehensive solution to these issues.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for receiving and validating measurement data from the sensor devices, means for storing the measurement data in a database, and means for using a generative AI model to analyze the measurement data, which enables real-time data collection and validation, enabling highly accurate anomaly detection and rapid emergency notification.

[0093] A "sensor device" is a device for measuring environmental data and user health data, and specifically includes a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, etc.

[0094] "Measurement data" refers to data such as temperature, humidity, sound, movement, heart rate, and blood pressure measured by sensor devices.

[0095] "Validation" is the process of checking whether the received measurement data is normal, and includes checking the data format, checking for missing values, verifying the consistency of timestamps, etc.

[0096] A "database" is a system for electronically storing and managing measurement data, and specifically, a database system such as PostgreSQL is used.

[0097] "Generative AI model" refers to technology that uses artificial intelligence models to analyze received measurement data and detect anomalies. Specific examples include TensorFlow and PyTorch.

[0098] "Abnormal" refers to data that deviates from the normal range in the user's environment or health condition, such as a sudden increase in heart rate or a fall.

[0099] An "emergency notification" is a notification that is generated when an abnormality is detected, and is a message that is sent to an emergency contact along with specific information about the abnormality.

[0100] "Emergency Contact" refers to a person or organization that has been pre-registered to receive notification when an abnormality is detected.

[0101] "Real-time" refers to data being collected, transmitted, and analyzed almost immediately, requiring the system to respond immediately.

[0102] "Response actions" refer to the actions taken by emergency contacts who receive an emergency notification to ensure the safety of the elderly person, and may include making inquiries, visiting, or contacting emergency services.

[0103] "SMS" is an abbreviation for Short Message Service, which refers to a service that sends short text messages using mobile phones.

[0104] "Mail" is an abbreviation for electronic mail and refers to a means of communication for exchanging text and files over the Internet.

[0105] "Push notification" refers to the ability of a mobile application to notify users of information in real time, typically using the smartphone's notification system.

[0106] This invention is a comprehensive system for supporting the independence and safety of elderly people, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts. The system consists of the following components:

[0107] Sensor device configuration and operation

[0108] Terminal

[0109] The terminal includes sensor devices for monitoring the living environment and health condition of the elderly, such as temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices collect data in real time and transmit it to a server at regular intervals.

[0110] Specifically, temperature and humidity sensors are placed in each room of the home, and wearable devices measure heart rate, blood pressure, and activity levels.Moreover, motion detection sensors detect sudden movements by the user, for example, to detect falls.

[0111] Server configuration and operation

[0112] server

[0113] The server receives and analyzes the data sent from the sensor devices. The server is equipped with a generative AI model (e.g., TensorFlow or PyTorch) that analyzes the received data and detects anomalies.

[0114] The received data first passes validation to ensure it is normal. It is then stored in a database (e.g., PostgreSQL) and analyzed by a generative AI model. If an anomaly is detected, an emergency notification is generated and sent to emergency contacts.

[0115] Emergency notification and response actions

[0116] User

[0117] Users, i.e., elderly people and their families, receive emergency notifications from the server. The notifications are sent to pre-registered emergency contacts via SMS, email, push notification, etc. The emergency contacts receive these notifications and can take prompt action.

[0118] For example, if a user falls at home, a motion sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family, who will then receive the notification and take immediate action.

[0119] Specific examples

[0120] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[0121] Terminal

[0122] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[0123] server

[0124] The server receives this data and uses the generative AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[0125] User

[0126] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[0127] Example prompts for the generative AI model to use

[0128] "Describe a system that uses sensor data to monitor the safety of elderly people and send emergency notifications if an abnormality occurs. The following is an outline of a specific system. The sensor device..."

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1: Sensor devices collect data

[0131] Terminal

[0132] The sensor devices collect real-time data about the elderly's living environment and health status. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. The collected data is stored with a timestamp.

[0133] Input: Environmental and health data such as temperature, humidity, and heart rate

[0134] Output: Timestamped sensor data

[0135] Step 2: The sensor device sends the collected data to the server

[0136] Terminal

[0137] The sensor device sends the collected data to the server at regular intervals, for example, every minute, and uploads the data to the server. The data includes the sensor's identification information and the measured value.

[0138] Input: Timestamped sensor data

[0139] Output: Sensor data sent to the server

[0140] Step 3: The server receives and validates the data

[0141] server

[0142] The server receives data sent from the sensor device. The received data is first validated to check for any irregularities or defects. Specifically, the data format is checked, missing values ​​are checked, and the consistency of timestamps is verified. Invalid data is recorded in an error log.

[0143] Input: Sensor data sent to the server

[0144] Output: Correct sensor data that passes validation (or invalid data recorded in the error log)

[0145] Step 4: The server saves the data to the database

[0146] server

[0147] The data that passes validation is stored in a database. For example, a database system such as PostgreSQL is used, and the data is written to a sensor data table. The data is used for later analysis.

[0148] Input: Correct sensor data that has passed validation

[0149] Output: Sensor data stored in a database

[0150] Step 5: The server analyzes the data using the generative AI model

[0151] server

[0152] The server performs data analysis using a generative AI model (e.g., TensorFlow or PyTorch). To detect anomalies in the received data, the AI ​​model analyzes environmental and health data to find data or patterns that deviate from normal ranges.

[0153] Input: Sensor data stored in a database

[0154] Output: Analysis results (presence or absence of abnormalities)

[0155] Step 6: Generate an emergency notification if the server detects an anomaly

[0156] server

[0157] If an abnormality is detected based on the analysis results, an emergency notification is generated. The notification content includes the type of abnormality detected, a timestamp, and location information. For example, a specific message such as "A fall has been detected in the living room. Please take action."

[0158] Input: Analysis results (presence or absence of abnormalities)

[0159] Output: Urgent notification message

[0160] Step 7: Server sends emergency notification to emergency contacts

[0161] server

[0162] The generated emergency notification is sent to pre-registered emergency contacts via SMS, email, push notification, etc. Specifically, SMS is sent using the Twilio API, and email is sent using the SendGrid API.

[0163] Input: Emergency notification message

[0164] Output: Notification sent to emergency contacts

[0165] Step 8: User receives notification and takes action

[0166] User

[0167] The emergency contact (such as the elderly person's family member) who receives the emergency notification can check the notification via a smartphone app or email. Based on the notification content, they can take appropriate action to ensure the safety of the elderly person. For example, a family member can call the elderly person to check on their safety and contact emergency services if necessary.

[0168] Input: Notification sent to emergency contacts

[0169] Output: Immediate response actions to ensure the safety of the elderly

[0170] (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] There is a need to provide an environment where elderly people and customers with health concerns can enjoy shopping safely in physical stores. However, current technology lacks a system that can detect elderly people's health conditions and abnormal situations in real time and respond quickly. This makes it difficult for customers to enjoy shopping with peace of mind, and there is a lack of means for store staff and family members to respond quickly. A system to solve this problem is needed.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes a means for measuring environmental data and user health data using a sensor device, a means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, a means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, a means for monitoring the safety of elderly customers and customers with health concerns within the store environment, and a means for notifying store staff when an abnormality in a customer is detected. This allows elderly customers and customers with health concerns to spend time safely within the physical store, and enables a quick response when an abnormality occurs.

[0175] A "sensor device" is a hardware device that measures environmental data and user health data.

[0176] "Environmental data" refers to data relating to ambient environmental conditions such as temperature, humidity, sound, and motion.

[0177] "Health data" refers to data related to the user's physical condition, such as heart rate, blood pressure, and activity level.

[0178] A "server" is a centralized computing system that analyzes data sent from sensor devices, detects abnormalities, and sends emergency notifications.

[0179] "Anomaly detection" is the process of identifying unusual conditions or changes based on environmental and health data.

[0180] "Emergency notification" is a system that sends warnings and information to pre-designated emergency contacts when an abnormality is detected.

[0181] An "emergency contact" is a person or organization that should receive notifications in the event of an emergency, such as a relative or person in charge of an elderly or health-conscious user who has been registered in advance.

[0182] The "store environment" is the sum of the physical space and environmental conditions within a physical store.

[0183] "Customer safety monitoring" is the process of using sensor devices to monitor the health status and abnormal situations of customers in physical stores in real time and take necessary measures.

[0184] "Store staff notification" is the process of quickly sending warnings and information to store staff when an abnormality is detected among customers.

[0185] This invention is a comprehensive system for supporting elderly people and customers with health concerns to spend time safely in physical stores. The detailed implementation method of this system will be described below.

[0186] Configuration and operation overview

[0187] The role of sensor devices

[0188] Sensor Device

[0189] The sensor devices are used to measure environmental data and user health data. These devices include temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and send it to a server at regular intervals. The sensor devices are placed in various locations throughout the store and constantly monitor health data such as customer heart rate, body temperature, and activity level.

[0190] Server Roles

[0191] server

[0192] The server centrally receives and analyzes data sent from the sensor devices. First, it validates the data to ensure it is normal. Next, it stores the data in a database and analyzes it using an AI model. This AI model is used to detect anomalies, and if an anomaly is detected, it immediately generates an emergency notification and sends it to store staff and registered emergency contacts.

[0193] Emergency notification and response actions

[0194] User

[0195] Users (customers) and emergency contacts (store staff and family members) receive emergency notifications from the server via smartphone or tablet applications and take appropriate action.

[0196] Hardware and software used

[0197] This system uses the following hardware and software:

[0198] Hardware: Various sensor devices (heart rate sensors, motion sensors, etc.), smartphones or tablets with internet connectivity

[0199] Software: Programmed using Python 3. Data analysis uses a simple threshold-based anomaly detection algorithm, and notifications are sent via an HTTP request to an external SMS sending API.

[0200] Specific examples

[0201] If 75-year-old Customer A falls while shopping in a store, the motion detection sensor in the store will detect this movement and send the data to a server. An AI model installed on the server will detect this as an abnormality and immediately generate an emergency notification. This notification will be sent to the smartphone or tablet of a store staff member, saying, "Customer A has fallen. Please check." A similar notification will also be sent to the family, allowing them to respond quickly.

[0202] Prompt Sentence Examples

[0203] To help elderly and health-conscious customers feel safe while shopping in physical stores, you will help design and implement a system that uses sensor devices to monitor heart rate and movement, and notify store staff and family members if any abnormalities occur.

[0204] In this way, a system is realized that provides a safe environment for elderly people and customers with health concerns, and that can respond quickly and appropriately in the event of an emergency.

[0205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0206] Step 1:

[0207] Sensor devices measure environmental data and user health data, including temperature, humidity, sound, movement, heart rate, blood pressure, etc. The inputs to a sensor device include data measured by various sensors. The sensor device collects this data in real time and outputs it as measurement data.

[0208] Step 2:

[0209] The terminal (sensor device) sends measurement data to the server at regular intervals. The terminal receives measurement data obtained from the sensor device as input and sends the data to the server as output.

[0210] Step 3:

[0211] The server validates the measurement data received from the sensor device. As input, it receives the measurement data sent from the sensor device and identifies valid and invalid data as output. Specific operations include checking the data format and verifying the numerical range.

[0212] Step 4:

[0213] The server saves the validated data to the database. It receives valid data that has passed validation as input and saves it to the database as output. Specific operations include database write operations.

[0214] Step 5:

[0215] The server inputs the stored data into the AI ​​model to detect anomalies. It receives the measurement data stored in the database as input and obtains the anomaly detection results as output. Specific operations include data analysis using the AI ​​model.

[0216] Step 6:

[0217] The server creates an emergency notification when an anomaly is detected. It receives the anomaly detection result as input and generates an emergency notification message as output. Specific operations include creating the message content and specifying the destination.

[0218] Step 7:

[0219] The server sends emergency notifications to store staff and emergency contacts. As input, it receives the generated emergency notification message and as output, it sends notifications to registered contacts. Specific operations include sending SMS, email, and push notifications.

[0220] Step 8:

[0221] The user (store staff) receives the emergency notification and takes appropriate action. The input is the emergency notification received on a smartphone or tablet, and the output is rushing to the scene or contacting an emergency service. Specific actions include checking the scene and taking emergency action.

[0222] In this way, the entire system is processed sequentially, realizing a system that supports elderly customers and customers with health concerns to stay safe in stores.

[0223] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0224] This invention combines an emotion engine with a comprehensive system that supports the independence and safety of the elderly. It analyzes user emotion data in addition to environmental and health data, enabling more comprehensive and rapid anomaly detection and response.

[0225] System configuration and operation overview

[0226] The role of sensor devices

[0227] Terminal

[0228] The sensor devices include sensors that monitor the elderly's living environment and health status, and an emotion engine that recognizes the user's emotions. Specifically, the devices include temperature sensors, humidity sensors, voice detection sensors, motion detection sensors, heart rate sensors, blood pressure monitors, and emotion recognition devices using cameras and microphones.

[0229] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to recognize the user's emotions, making it possible to determine in real time whether the user is sad, angry, excited, etc.

[0230] Server Roles

[0231] server

[0232] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. It is then stored in a database and analyzed by the AI ​​model or emotion engine.

[0233] The server integrates and analyzes health and emotional data, and if an abnormality is detected, it determines the appropriate response depending on the type and severity of the abnormality. If an abnormality is detected, an emergency notification is generated and sent to emergency contacts.

[0234] Emergency notification and response actions

[0235] User

[0236] The user (emergency contact) receives an emergency notification from the server. The emergency notification includes the abnormality detection result, emotional state, and required response action. The emergency contact receives these notifications via SMS, email, push notification, etc., and responds promptly.

[0237] Specific examples

[0238] For example, consider the case where a user falls at home and expresses emotions of fear and confusion. Here is how the system works:

[0239] Terminal

[0240] The fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[0241] server

[0242] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[0243] User

[0244] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the elderly person's safety and, if necessary, call emergency services.

[0245] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of an emotion engine enables more comprehensive anomaly detection that also takes the user's emotional state into account, further improving the quality of life for the elderly.

[0246] The processing flow will be explained below.

[0247] Step 1:

[0248] Terminal

[0249] The sensor devices measure the user's environmental and health data in real time. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. In addition, the emotion engine uses the camera and microphone to capture the user's facial expressions and voice and analyze their emotional state.

[0250] Step 2:

[0251] Terminal

[0252] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and sent via an HTTP POST request. Emotion data is also collected and sent to the server.

[0253] Step 3:

[0254] server

[0255] The server receives data sent from the sensor devices and emotion engine. The received data is first validated to ensure that the data structure and format are correct.

[0256] Step 4:

[0257] server

[0258] Data that passes validation is stored in a time series database, which ensures that past and present data are properly managed.

[0259] Step 5:

[0260] server

[0261] After the data is saved, it is analyzed by an AI model and an emotion engine. The AI ​​model uses health and environmental data to detect abnormalities, and the emotion engine analyzes the user's emotional data to detect abnormal emotional states.

[0262] Step 6:

[0263] server

[0264] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, location information, and emotional state, for example, "Your heart rate has increased sharply, and you have detected an emotion of fear."

[0265] Step 7:

[0266] server

[0267] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[0268] Step 8:

[0269] User

[0270] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[0271] Step 9:

[0272] User

[0273] For example, when a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe. It also takes into account the user's emotional state and provides any necessary psychological support.

[0274] In this way, the system detects abnormalities through collaboration between the sensor device, server, and user (emergency contact) sections, and promotes prompt and appropriate responses.The introduction of an emotion engine also analyzes the user's emotional state, achieving more comprehensive safety management.

[0275] Example 2

[0276] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0277] In systems designed to support the independence and safety of elderly people, conventional technologies primarily monitor environmental and health data, without taking into account the user's emotional state when detecting anomalies. Therefore, comprehensive anomaly detection and rapid response, including not only falls and deterioration in health status but also emotional anomalies, are required. Furthermore, real-time data acquisition and analysis are incomplete, sometimes leading to delays in emergency response.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0279] In this invention, the server includes a measuring means for monitoring the living environment and health status, a means for transmitting data obtained by the measuring means to the server, a means for the server to receive the data and validate its contents, a means for storing the validated data in a database, a means for analyzing the stored data and using an AI model to detect anomalies, a means for generating an emergency notification when an anomaly is detected and sending it to an emergency contact, and a means for the emergency contact to take action upon receiving the emergency notification. This enables real-time data acquisition and analysis, as well as comprehensive anomaly detection and rapid response that takes into account emotional states.

[0280] "Measurement means" refers to a device that acquires data using a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone, etc., to monitor living environment and health condition.

[0281] The "server" is a central management system that receives and validates data sent from the measurement means, stores it in a database, and analyzes the data using an AI model.

[0282] "Validation" is the process by which the server verifies that the data it receives is correct and valid.

[0283] A "database" is a system for systematically storing and managing validated data.

[0284] An "AI model" is a machine learning algorithm or artificial intelligence system used to analyze stored data and detect anomalies.

[0285] An "emergency notification" is a message or alert that the server generates and sends to emergency contacts when an abnormality is detected.

[0286] An "emergency contact" is a person or organization designated to receive emergency notifications when an anomaly is detected.

[0287] "Response actions" are actions that an emergency contact must take upon receiving an emergency notification.

[0288] The "living environment" refers to the conditions such as temperature, humidity, sound, and movement in the place where the user lives their daily life.

[0289] "Health status" refers to physiological data such as heart rate and blood pressure that indicate the user's physical health.

[0290] "Emotional state" refers to the psychological state of the user that can be analyzed from facial expressions and tone of voice.

[0291] This invention is a comprehensive system for supporting the independence and safety of elderly people. By incorporating an emotion engine, the system analyzes the user's emotional state and integrates it with environmental and health data to achieve comprehensive anomaly detection and rapid response.

[0292] System configuration and operation overview

[0293] The role of sensor devices

[0294] Terminal

[0295] The device is equipped with a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera, and microphone. These devices monitor the user's living environment and health. For example, the heart rate sensor captures the user's heart rate every second, and the camera and microphone capture the user's facial expressions and tone of voice every second.

[0296] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to determine the user's emotional state in real time, identifying whether the user is sad, angry, excited, etc.

[0297] Server Roles

[0298] server

[0299] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. The data is then saved in a database. Database management systems such as MySQL and PostgreSQL are used.

[0300] The server analyzes the stored data using an AI model. For example, facial expression data and tone of voice data are input into an emotion analysis algorithm to extract the user's emotional state. Then, health data and emotional data are integrated and considered to detect abnormalities. If an abnormality is detected, an emergency notification is generated according to the type and severity of the abnormality. The notification includes the type of abnormality, the user's emotional state, and any necessary response actions.

[0301] Emergency notification and response actions

[0302] User

[0303] The user (emergency contact) receives an emergency notification from the server. Notifications are sent via SMS, email, push notification, and other methods, allowing the emergency contact to respond quickly. For example, a message may be sent stating, "A fall has been detected in the living room and fear has been recognized. Please check immediately." Based on this notification, the emergency contact can quickly confirm the user's safety and arrange for emergency services if necessary.

[0304] Specific examples

[0305] As a concrete example, consider the case where a user falls at home and expresses feelings of fear and confusion.

[0306] Terminal

[0307] The device's fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[0308] server

[0309] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[0310] User

[0311] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the user's safety and arrange for emergency services if necessary.

[0312] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of the emotion engine enables more comprehensive anomaly detection that takes emotional state into account, further improving the quality of life for the elderly.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1: Acquiring Sensor Data

[0315] Terminal

[0316] The device collects data from a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone. Specifically, the heart rate sensor measures the heart rate every second, the camera captures the user's face image every minute, and the sound detection sensor constantly monitors the user's voice.

[0317] Input: Environmental data (e.g., temperature, humidity), health data (e.g., heart rate, blood pressure), emotional data (e.g., facial expression, voice)

[0318] Output: Acquired raw data (e.g., heart rate data, facial image data)

[0319] Step 2: Sending data

[0320] Terminal

[0321] The data acquired by the device is sent to a server via Wi-Fi or Bluetooth. For example, heart rate data is sent to the server in batches every minute, and emergency data such as falls is sent in real time.

[0322] Input: Acquired raw data (e.g., heart rate data, facial image data)

[0323] Output: Data sent to the server

[0324] Step 3: Validate the data

[0325] server

[0326] The server validates the data received from the device, for example, checking that the heart rate is not too extreme (0 or over 200) and filtering out invalid data.

[0327] Input: Transmitted data (e.g. heart rate data)

[0328] Output: Normal data (e.g. filtered heart rate data)

[0329] Step 4: Save your data

[0330] server

[0331] The server saves the validated data in a database. Specifically, it uses a database management system (e.g., MySQL, PostgreSQL) and saves each data with a timestamp.

[0332] Input: Normal data (e.g. filtered heart rate data)

[0333] Output: Data stored in the database

[0334] Step 5: Analyze the sentiment data

[0335] server

[0336] The server analyzes the stored facial image data and voice data using an AI model to determine the user's emotional state, specifically using facial expression analysis algorithms and voice tone analysis algorithms.

[0337] Input: Facial image data, audio data

[0338] Output: Parsed emotion data (e.g., fear, joy)

[0339] Step 6: Detect anomalies

[0340] server

[0341] The server integrates and analyzes health data and emotional data to detect abnormalities. For example, if data on a fall and emotional fear occur simultaneously, it will be detected as an abnormality.

[0342] Input: Health data (e.g., fall data), emotion data (e.g., fear)

[0343] Output: Anomaly detection result (e.g., fall + fear = abnormal)

[0344] Step 7: Generate and send emergency notifications

[0345] server

[0346] If the server detects an anomaly, it generates an emergency notification and sends it to emergency contacts. The notification includes the type of anomaly, the emotional state, and a recommended response action. For example, it can generate a message like, "A fall has been detected in the living room and the emotion of fear has been recognized. Please take immediate action."

[0347] Input: Anomaly detection result (e.g., fall + fear = abnormal)

[0348] Output: Emergency notification message

[0349] Step 8: Receive emergency notifications and take action

[0350] User

[0351] The user (emergency contact) receives the emergency notification from the server, checks the content, and then promptly takes action. Specifically, the notification is received via a smartphone app or SMS, and the user's safety is promptly confirmed and emergency services are called if necessary.

[0352] Input: Emergency notification message

[0353] Output: Response actions (e.g., phone call to check on safety or arrange for emergency services)

[0354] (Application example 2)

[0355] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] When elderly people and other passengers use self-driving vehicles, they need to be able to comprehensively monitor their health and emotional states in real time and respond quickly if an abnormality occurs. However, current systems only monitor health data and do not consider emotional state information, which can result in incomplete anomaly detection. The present invention aims to solve these problems and improve the safety and security of self-driving vehicles by comprehensively monitoring the user's health and emotional state.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0358] In this invention, the server includes means for measuring environmental data and user health data using a sensor device, means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, means for analyzing emotional data obtained from the sensor device and recognizing the user's emotional state, means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, and means for the emergency contact who receives the emergency notification to take response action. This allows for comprehensive monitoring of the user's health and emotional state, enabling a prompt and appropriate response when an abnormality is detected.

[0359] A "sensor device" is a device that measures environmental data and user health data.

[0360] A "server" is a computer system that analyzes measurement data received from sensor devices and detects abnormalities.

[0361] An "emergency notification" is a warning message that the server creates and sends to emergency contacts when an abnormality is detected.

[0362] An "emergency contact" is a contact who is responsible for taking action in response to a user's abnormality.

[0363] An "AI model" is an algorithm that uses machine learning to analyze data and detect anomalies.

[0364] "Real-time" means that data is processed immediately at the moment it is generated.

[0365] "Emotion data" is information about the emotional state of the user analyzed from their facial expressions and voice.

[0366] A "library for voice analysis" is a software component for analyzing voice data and recognizing its content.

[0367] An "on-board device" is a device that is installed in a vehicle and integrates and analyzes data acquired from sensor devices.

[0368] A "portable device" is a communication terminal that a user can carry around with them.

[0369] This invention is a comprehensive system that comprehensively monitors the health and emotional state of a user and responds quickly if an abnormality occurs. The system of the present invention is installed in an autonomous vehicle and includes the following components:

[0370] System Configuration

[0371] 1. Sensor device:

[0372] These devices include cameras, microphones, heart rate sensors, and pressure sensors installed in the vehicle. These devices measure the user's environmental and health data in real time. Emotion data is obtained by analyzing facial expressions and tone of voice using the cameras and microphones.

[0373] 2. Server:

[0374] The system receives and analyzes measurement and emotion data sent from the sensor devices. Specific software used includes OpenCV, Dlib, and DeepFace for face tracking and facial expression recognition, Google Cloud Speech-to-Text API for voice analysis, and TensorFlow for health data analysis. The server integrates this data and generates emergency notifications if an abnormality is detected.

[0375] 3. Onboard equipment:

[0376] This device sends emergency notifications to a display in the vehicle or to the user's mobile device. When an emergency notification is sent, the information is also sent to registered emergency contacts.

[0377] Program processing and data calculation

[0378] Data Acquisition:

[0379] The sensor device collects the user's health data (heart rate, pressure, etc.) and emotional data (facial expressions, voice) in real time and sends this to the server.

[0380] Data Analysis:

[0381] The server analyzes this data using AI models such as TensorFlow and DeepFace. First, it validates the data to ensure it is normal. Then, the AI ​​model detects any anomalies.

[0382] Emergency notification:

[0383] If an abnormality is detected, the system generates an emergency notification and sends it to the vehicle's display and the user's mobile device, as well as to registered emergency contacts via social media and email.

[0384] Specific examples

[0385] For example, consider a case where an elderly person's heart rate suddenly rises while using an autonomous vehicle, and the camera analysis reveals a fearful expression. In this case, the system operates as follows:

[0386] The camera and heart rate sensor collect data in real time and send it to a server.

[0387] The server uses the acquired data to analyze the user's emotions and health status and detect any abnormalities.

[0388] If an abnormality is detected, an emergency notification is generated and displayed on the vehicle's display and on the user's mobile device, stating "An abnormality has been detected, please stop the vehicle at a safe location."

[0389] In addition, an emergency notification will be sent to pre-registered emergency contacts stating, "An abnormality has been detected. Please check immediately."

[0390] Prompt Sentence Examples

[0391] "If an elderly person sitting in the passenger seat of a self-driving vehicle suddenly experiences a rise in heart rate and facial expressions indicate abnormalities, send that data and generate an immediate emergency notification."

[0392] Prompt sentence for prompt generation AI model:

[0393] "Input data: Heart rate data: 120 bpm, Facial expression data: scared, Audio data: speaking with trembling. An abnormality has been detected, so please generate an emergency notification."

[0394] This system will ensure the safety and security of elderly and other passengers, and will also enable a prompt and appropriate response in the event of an abnormality.

[0395] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0396] Step 1:

[0397] Data Acquisition

[0398] The terminal (sensor device) uses devices installed inside the vehicle, such as a camera, microphone, heart rate sensor, and pressure sensor, to measure the user's environmental data and health data in real time. The camera and microphone are used to capture the user's facial expression and voice data, which are then recognized as emotional data. The heart rate sensor and pressure sensor are also used to capture the user's heart rate and seat pressure data. This data is sent from the terminal to a server.

[0399] Input: Real-time environmental data, heart rate data, pressure data, facial expression data, voice data

[0400] Output: Sending the retrieved data

[0401] Step 2:

[0402] Data Receipt and Validation

[0403] The server receives the environmental data, health data, and emotion data sent from the device. The server first validates the received data to ensure that it is correct, thereby ensuring the reliability of the data.

[0404] Input: Data sent from the terminal

[0405] Output: Validated data

[0406] Step 3:

[0407] Data analysis

[0408] The server then uses the validated data to perform analysis. Specifically, it proceeds as follows:

[0409] OpenCV, Dlib, and DeepFace are used for face tracking and facial expression recognition, and the user's emotions are analyzed based on image data obtained from the camera.

[0410] For voice analysis, the Google Cloud Speech-to-Text API is used to convert voice data obtained from the microphone into text and analyze emotions.

[0411] TensorFlow is used for health data analysis, analyzing heart rate and pressure data.

[0412] This data is analyzed comprehensively to determine whether any abnormalities are detected.

[0413] Input: Validated environmental data, health data, and emotion data

[0414] Output: Analysis results (presence or absence of abnormalities)

[0415] Step 4:

[0416] Anomaly detection

[0417] If the server detects an anomaly based on the analysis results, it generates an emergency notification according to the type and severity of the anomaly, using an anomaly detection algorithm based on an analytical model (TensorFlow or DeepFace).

[0418] Input: Analysis results

[0419] Output: Urgent Notification

[0420] Step 5:

[0421] Sending emergency notifications

[0422] If an abnormality is detected, the server generates an emergency notification and sends it to the vehicle's display and the user's mobile device.The server also sends the emergency notification to pre-registered emergency contacts via email and social media.

[0423] Input: Emergency Notification

[0424] Output: Notification to emergency contacts and mobile devices

[0425] Step 6:

[0426] Response actions

[0427] The user and emergency contacts can take immediate action based on the emergency notification they receive, such as checking the user's health and providing first aid or contacting emergency services if necessary, or even stopping the self-driving vehicle at a safe location based on the notification displayed on the vehicle's display.

[0428] Input: Emergency Notification

[0429] Output: Execute response action

[0430] In this way, the system monitors the user's health and emotional state in real time, enabling rapid and appropriate response when an abnormality is detected.

[0431] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0432] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0433] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0434] [Second embodiment]

[0435] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0436] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0437] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0438] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0439] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0440] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0441] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0442] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0443] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0444] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0445] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0446] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0447] This invention is a comprehensive system to support the independence and safety of the elderly, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts.

[0448] System configuration and operation overview

[0449] The role of sensor devices

[0450] Terminal

[0451] The terminal includes various sensor devices for monitoring the elderly's living environment and health condition, including temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and transmit it to a server at regular intervals.

[0452] Sensor devices are placed in various rooms in the home to measure temperature, humidity, sound, and movement, while wearable devices worn by users measure health data such as heart rate, blood pressure, and activity levels.

[0453] Server Roles

[0454] server

[0455] The server centrally receives and analyzes the data sent from the sensor devices. The server is equipped with an AI model that analyzes the received data and detects anomalies.

[0456] The received data is first validated to ensure it is normal. Next, the data is stored in a database and analyzed using an AI model. If an abnormality is detected as a result of the analysis, an emergency notification is generated and sent to emergency contacts.

[0457] Emergency notification and response actions

[0458] User

[0459] Users, i.e., elderly people and their families, receive emergency notifications from the server. The emergency notifications are sent to pre-registered emergency contacts, who then receive the notifications via SMS, email, push notifications, etc.

[0460] For example, if a user falls at home, a motion detection sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family. The family will receive the notification and take prompt action.

[0461] Specific examples

[0462] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[0463] Terminal

[0464] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[0465] server

[0466] The server receives this data and uses an AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[0467] User

[0468] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[0469] In this way, the system is designed to ensure the independence and safety of the elderly and to respond quickly in the event of an emergency.

[0470] The processing flow will be explained below.

[0471] Step 1:

[0472] Terminal

[0473] Sensor devices measure the user's environmental and health data in real time. For example, a temperature sensor measures the room temperature, and a heart rate sensor measures the user's heart rate. The measured data is collected into data packets at regular intervals.

[0474] Step 2:

[0475] Terminal

[0476] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and is sent to the server via an HTTP POST request.

[0477] Step 3:

[0478] server

[0479] The server receives the data sent from the sensor device, and the received data is first validated to ensure that the data structure and format are correct.

[0480] Step 4:

[0481] server

[0482] Data that passes validation is stored in a time-series database, allowing past and present data to be properly managed and used for subsequent analysis.

[0483] Step 5:

[0484] server

[0485] After the data is saved, it is analyzed by an AI model, which detects anomalies based on the received data and determines whether or not there are any anomalies.

[0486] Step 6:

[0487] server

[0488] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, and location information.

[0489] Step 7:

[0490] server

[0491] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[0492] Step 8:

[0493] User

[0494] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[0495] Step 9:

[0496] User

[0497] For example, if a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe.

[0498] In this way, the system works by linking the sensor device, server, and user (emergency contact) sections to detect abnormalities and facilitate rapid response.

[0499] Example 1

[0500] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0501] To effectively support the safety and independence of elderly people, a system that monitors environmental and health data in real time and responds quickly when an abnormality occurs is required. However, conventional systems have issues with data validation, the accuracy of abnormality detection, and methods for sending emergency notifications, and there is a lack of a comprehensive solution to these issues.

[0502] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0503] In this invention, the server includes means for receiving and validating measurement data from the sensor devices, means for storing the measurement data in a database, and means for using a generative AI model to analyze the measurement data, which enables real-time data collection and validation, enabling highly accurate anomaly detection and rapid emergency notification.

[0504] A "sensor device" is a device for measuring environmental data and user health data, and specifically includes a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, etc.

[0505] "Measurement data" refers to data such as temperature, humidity, sound, movement, heart rate, and blood pressure measured by sensor devices.

[0506] "Validation" is the process of checking whether the received measurement data is normal, and includes checking the data format, checking for missing values, verifying the consistency of timestamps, etc.

[0507] A "database" is a system for electronically storing and managing measurement data, and specifically, a database system such as PostgreSQL is used.

[0508] "Generative AI model" refers to technology that uses artificial intelligence models to analyze received measurement data and detect anomalies. Specific examples include TensorFlow and PyTorch.

[0509] "Abnormal" refers to data that deviates from the normal range in the user's environment or health condition, such as a sudden increase in heart rate or a fall.

[0510] An "emergency notification" is a notification that is generated when an abnormality is detected, and is a message that is sent to an emergency contact along with specific information about the abnormality.

[0511] "Emergency Contact" refers to a person or organization that has been pre-registered to receive notification when an abnormality is detected.

[0512] "Real-time" refers to data being collected, transmitted, and analyzed almost immediately, requiring the system to respond immediately.

[0513] "Response actions" refer to the actions taken by emergency contacts who receive an emergency notification to ensure the safety of the elderly person, and may include making inquiries, visiting, or contacting emergency services.

[0514] "SMS" is an abbreviation for Short Message Service, which refers to a service that sends short text messages using mobile phones.

[0515] "Mail" is an abbreviation for electronic mail and refers to a means of communication for exchanging text and files over the Internet.

[0516] "Push notification" refers to the ability of a mobile application to notify users of information in real time, typically using the smartphone's notification system.

[0517] This invention is a comprehensive system for supporting the independence and safety of elderly people, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts. The system consists of the following components:

[0518] Sensor device configuration and operation

[0519] Terminal

[0520] The terminal includes sensor devices for monitoring the living environment and health condition of the elderly, such as temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices collect data in real time and transmit it to a server at regular intervals.

[0521] Specifically, temperature and humidity sensors are placed in each room of the home, and wearable devices measure heart rate, blood pressure, and activity levels.Moreover, motion detection sensors detect sudden movements by the user, for example, to detect falls.

[0522] Server configuration and operation

[0523] server

[0524] The server receives and analyzes the data sent from the sensor devices. The server is equipped with a generative AI model (e.g., TensorFlow or PyTorch) that analyzes the received data and detects anomalies.

[0525] The received data first passes validation to ensure it is normal. It is then stored in a database (e.g., PostgreSQL) and analyzed by a generative AI model. If an anomaly is detected, an emergency notification is generated and sent to emergency contacts.

[0526] Emergency notification and response actions

[0527] User

[0528] Users, i.e., elderly people and their families, receive emergency notifications from the server. The notifications are sent to pre-registered emergency contacts via SMS, email, push notification, etc. The emergency contacts receive these notifications and can take prompt action.

[0529] For example, if a user falls at home, a motion sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family, who will then receive the notification and take immediate action.

[0530] Specific examples

[0531] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[0532] Terminal

[0533] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[0534] server

[0535] The server receives this data and uses the generative AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[0536] User

[0537] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[0538] Example prompts for the generative AI model to use

[0539] "Describe a system that uses sensor data to monitor the safety of elderly people and send emergency notifications if an abnormality occurs. The following is an outline of a specific system. The sensor device..."

[0540] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0541] Step 1: Sensor devices collect data

[0542] Terminal

[0543] The sensor devices collect real-time data about the elderly's living environment and health status. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. The collected data is stored with a timestamp.

[0544] Input: Environmental and health data such as temperature, humidity, and heart rate

[0545] Output: Timestamped sensor data

[0546] Step 2: The sensor device sends the collected data to the server

[0547] Terminal

[0548] The sensor device sends the collected data to the server at regular intervals, for example, every minute, and uploads the data to the server. The data includes the sensor's identification information and the measured value.

[0549] Input: Timestamped sensor data

[0550] Output: Sensor data sent to the server

[0551] Step 3: The server receives and validates the data

[0552] server

[0553] The server receives data sent from the sensor device. The received data is first validated to check for any irregularities or defects. Specifically, the data format is checked, missing values ​​are checked, and the consistency of timestamps is verified. Invalid data is recorded in an error log.

[0554] Input: Sensor data sent to the server

[0555] Output: Correct sensor data that passes validation (or invalid data recorded in the error log)

[0556] Step 4: The server saves the data to the database

[0557] server

[0558] The data that passes validation is stored in a database. For example, a database system such as PostgreSQL is used, and the data is written to a sensor data table. The data is used for later analysis.

[0559] Input: Correct sensor data that has passed validation

[0560] Output: Sensor data stored in a database

[0561] Step 5: The server analyzes the data using the generative AI model

[0562] server

[0563] The server performs data analysis using a generative AI model (e.g., TensorFlow or PyTorch). To detect anomalies in the received data, the AI ​​model analyzes environmental and health data to find data or patterns that deviate from normal ranges.

[0564] Input: Sensor data stored in a database

[0565] Output: Analysis results (presence or absence of abnormalities)

[0566] Step 6: Generate an emergency notification if the server detects an anomaly

[0567] server

[0568] If an abnormality is detected based on the analysis results, an emergency notification is generated. The notification content includes the type of abnormality detected, a timestamp, and location information. For example, a specific message such as "A fall has been detected in the living room. Please take action."

[0569] Input: Analysis results (presence or absence of abnormalities)

[0570] Output: Urgent notification message

[0571] Step 7: Server sends emergency notification to emergency contacts

[0572] server

[0573] The generated emergency notification is sent to pre-registered emergency contacts via SMS, email, push notification, etc. Specifically, SMS is sent using the Twilio API, and email is sent using the SendGrid API.

[0574] Input: Emergency notification message

[0575] Output: Notification sent to emergency contacts

[0576] Step 8: User receives notification and takes action

[0577] User

[0578] The emergency contact (such as the elderly person's family member) who receives the emergency notification can check the notification via a smartphone app or email. Based on the notification content, they can take appropriate action to ensure the safety of the elderly person. For example, a family member can call the elderly person to check on their safety and contact emergency services if necessary.

[0579] Input: Notification sent to emergency contacts

[0580] Output: Immediate response actions to ensure the safety of the elderly

[0581] (Application example 1)

[0582] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0583] There is a need to provide an environment where elderly people and customers with health concerns can enjoy shopping safely in physical stores. However, current technology lacks a system that can detect elderly people's health conditions and abnormal situations in real time and respond quickly. This makes it difficult for customers to enjoy shopping with peace of mind, and there is a lack of means for store staff and family members to respond quickly. A system to solve this problem is needed.

[0584] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0585] In this invention, the server includes a means for measuring environmental data and user health data using a sensor device, a means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, a means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, a means for monitoring the safety of elderly customers and customers with health concerns within the store environment, and a means for notifying store staff when an abnormality in a customer is detected. This allows elderly customers and customers with health concerns to spend time safely within the physical store, and enables a quick response when an abnormality occurs.

[0586] A "sensor device" is a hardware device that measures environmental data and user health data.

[0587] "Environmental data" refers to data relating to ambient environmental conditions such as temperature, humidity, sound, and motion.

[0588] "Health data" refers to data related to the user's physical condition, such as heart rate, blood pressure, and activity level.

[0589] A "server" is a centralized computing system that analyzes data sent from sensor devices, detects abnormalities, and sends emergency notifications.

[0590] "Anomaly detection" is the process of identifying unusual conditions or changes based on environmental and health data.

[0591] "Emergency notification" is a system that sends warnings and information to pre-designated emergency contacts when an abnormality is detected.

[0592] An "emergency contact" is a person or organization that should receive notifications in the event of an emergency, such as a relative or person in charge of an elderly or health-conscious user who has been registered in advance.

[0593] The "store environment" is the sum of the physical space and environmental conditions within a physical store.

[0594] "Customer safety monitoring" is the process of using sensor devices to monitor the health status and abnormal situations of customers in physical stores in real time and take necessary measures.

[0595] "Store staff notification" is the process of quickly sending warnings and information to store staff when an abnormality is detected among customers.

[0596] This invention is a comprehensive system for supporting elderly people and customers with health concerns to spend time safely in physical stores. The detailed implementation method of this system will be described below.

[0597] Configuration and operation overview

[0598] The role of sensor devices

[0599] Sensor Device

[0600] The sensor devices are used to measure environmental data and user health data. These devices include temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and send it to a server at regular intervals. The sensor devices are placed in various locations throughout the store and constantly monitor health data such as customer heart rate, body temperature, and activity level.

[0601] Server Roles

[0602] server

[0603] The server centrally receives and analyzes data sent from the sensor devices. First, it validates the data to ensure it is normal. Next, it stores the data in a database and analyzes it using an AI model. This AI model is used to detect anomalies, and if an anomaly is detected, it immediately generates an emergency notification and sends it to store staff and registered emergency contacts.

[0604] Emergency notification and response actions

[0605] User

[0606] Users (customers) and emergency contacts (store staff and family members) receive emergency notifications from the server via smartphone or tablet applications and take appropriate action.

[0607] Hardware and software used

[0608] This system uses the following hardware and software:

[0609] Hardware: Various sensor devices (heart rate sensors, motion sensors, etc.), smartphones or tablets with internet connectivity

[0610] Software: Programmed using Python 3. Data analysis uses a simple threshold-based anomaly detection algorithm, and notifications are sent via an HTTP request to an external SMS sending API.

[0611] Specific examples

[0612] If 75-year-old Customer A falls while shopping in a store, the motion detection sensor in the store will detect this movement and send the data to a server. An AI model installed on the server will detect this as an abnormality and immediately generate an emergency notification. This notification will be sent to the smartphone or tablet of a store staff member, saying, "Customer A has fallen. Please check." A similar notification will also be sent to the family, allowing them to respond quickly.

[0613] Prompt Sentence Examples

[0614] To help elderly and health-conscious customers feel safe while shopping in physical stores, you will help design and implement a system that uses sensor devices to monitor heart rate and movement, and notify store staff and family members if any abnormalities occur.

[0615] In this way, a system is realized that provides a safe environment for elderly people and customers with health concerns, and that can respond quickly and appropriately in the event of an emergency.

[0616] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0617] Step 1:

[0618] Sensor devices measure environmental data and user health data, including temperature, humidity, sound, movement, heart rate, blood pressure, etc. The inputs to a sensor device include data measured by various sensors. The sensor device collects this data in real time and outputs it as measurement data.

[0619] Step 2:

[0620] The terminal (sensor device) sends measurement data to the server at regular intervals. The terminal receives measurement data obtained from the sensor device as input and sends the data to the server as output.

[0621] Step 3:

[0622] The server validates the measurement data received from the sensor device. As input, it receives the measurement data sent from the sensor device and identifies valid and invalid data as output. Specific operations include checking the data format and verifying the numerical range.

[0623] Step 4:

[0624] The server saves the validated data to the database. It receives valid data that has passed validation as input and saves it to the database as output. Specific operations include database write operations.

[0625] Step 5:

[0626] The server inputs the stored data into the AI ​​model to detect anomalies. It receives the measurement data stored in the database as input and obtains the anomaly detection results as output. Specific operations include data analysis using the AI ​​model.

[0627] Step 6:

[0628] The server creates an emergency notification when an anomaly is detected. It receives the anomaly detection result as input and generates an emergency notification message as output. Specific operations include creating the message content and specifying the destination.

[0629] Step 7:

[0630] The server sends emergency notifications to store staff and emergency contacts. As input, it receives the generated emergency notification message and as output, it sends notifications to registered contacts. Specific operations include sending SMS, email, and push notifications.

[0631] Step 8:

[0632] The user (store staff) receives the emergency notification and takes appropriate action. The input is the emergency notification received on a smartphone or tablet, and the output is rushing to the scene or contacting an emergency service. Specific actions include checking the scene and taking emergency action.

[0633] In this way, the entire system is processed sequentially, realizing a system that supports elderly customers and customers with health concerns to stay safe in stores.

[0634] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0635] This invention combines an emotion engine with a comprehensive system that supports the independence and safety of the elderly. It analyzes user emotion data in addition to environmental and health data, enabling more comprehensive and rapid anomaly detection and response.

[0636] System configuration and operation overview

[0637] The role of sensor devices

[0638] Terminal

[0639] The sensor devices include sensors that monitor the elderly's living environment and health status, and an emotion engine that recognizes the user's emotions. Specifically, the devices include temperature sensors, humidity sensors, voice detection sensors, motion detection sensors, heart rate sensors, blood pressure monitors, and emotion recognition devices using cameras and microphones.

[0640] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to recognize the user's emotions, making it possible to determine in real time whether the user is sad, angry, excited, etc.

[0641] Server Roles

[0642] server

[0643] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. It is then stored in a database and analyzed by the AI ​​model or emotion engine.

[0644] The server integrates and analyzes health and emotional data, and if an abnormality is detected, it determines the appropriate response depending on the type and severity of the abnormality. If an abnormality is detected, an emergency notification is generated and sent to emergency contacts.

[0645] Emergency notification and response actions

[0646] User

[0647] The user (emergency contact) receives an emergency notification from the server. The emergency notification includes the abnormality detection result, emotional state, and required response action. The emergency contact receives these notifications via SMS, email, push notification, etc., and responds promptly.

[0648] Specific examples

[0649] For example, consider the case where a user falls at home and expresses emotions of fear and confusion. Here is how the system works:

[0650] Terminal

[0651] The fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[0652] server

[0653] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[0654] User

[0655] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the elderly person's safety and, if necessary, call emergency services.

[0656] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of an emotion engine enables more comprehensive anomaly detection that also takes the user's emotional state into account, further improving the quality of life for the elderly.

[0657] The processing flow will be explained below.

[0658] Step 1:

[0659] Terminal

[0660] The sensor devices measure the user's environmental and health data in real time. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. In addition, the emotion engine uses the camera and microphone to capture the user's facial expressions and voice and analyze their emotional state.

[0661] Step 2:

[0662] Terminal

[0663] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and sent via an HTTP POST request. Emotion data is also collected and sent to the server.

[0664] Step 3:

[0665] server

[0666] The server receives data sent from the sensor devices and emotion engine. The received data is first validated to ensure that the data structure and format are correct.

[0667] Step 4:

[0668] server

[0669] Data that passes validation is stored in a time series database, which ensures that past and present data are properly managed.

[0670] Step 5:

[0671] server

[0672] After the data is saved, it is analyzed by an AI model and an emotion engine. The AI ​​model uses health and environmental data to detect abnormalities, and the emotion engine analyzes the user's emotional data to detect abnormal emotional states.

[0673] Step 6:

[0674] server

[0675] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, location information, and emotional state, for example, "Your heart rate has increased sharply, and you have detected an emotion of fear."

[0676] Step 7:

[0677] server

[0678] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[0679] Step 8:

[0680] User

[0681] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[0682] Step 9:

[0683] User

[0684] For example, when a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe. It also takes into account the user's emotional state and provides any necessary psychological support.

[0685] In this way, the system detects abnormalities through collaboration between the sensor device, server, and user (emergency contact) sections, and promotes prompt and appropriate responses.The introduction of an emotion engine also analyzes the user's emotional state, achieving more comprehensive safety management.

[0686] Example 2

[0687] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] In systems designed to support the independence and safety of elderly people, conventional technologies primarily monitor environmental and health data, without taking into account the user's emotional state when detecting anomalies. Therefore, comprehensive anomaly detection and rapid response, including not only falls and deterioration in health status but also emotional anomalies, are required. Furthermore, real-time data acquisition and analysis are incomplete, sometimes leading to delays in emergency response.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0690] In this invention, the server includes a measuring means for monitoring the living environment and health status, a means for transmitting data obtained by the measuring means to the server, a means for the server to receive the data and validate its contents, a means for storing the validated data in a database, a means for analyzing the stored data and using an AI model to detect anomalies, a means for generating an emergency notification when an anomaly is detected and sending it to an emergency contact, and a means for the emergency contact to take action upon receiving the emergency notification. This enables real-time data acquisition and analysis, as well as comprehensive anomaly detection and rapid response that takes into account emotional states.

[0691] "Measurement means" refers to a device that acquires data using a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone, etc., to monitor living environment and health condition.

[0692] The "server" is a central management system that receives and validates data sent from the measurement means, stores it in a database, and analyzes the data using an AI model.

[0693] "Validation" is the process by which the server verifies that the data it receives is correct and valid.

[0694] A "database" is a system for systematically storing and managing validated data.

[0695] An "AI model" is a machine learning algorithm or artificial intelligence system used to analyze stored data and detect anomalies.

[0696] An "emergency notification" is a message or alert that the server generates and sends to emergency contacts when an abnormality is detected.

[0697] An "emergency contact" is a person or organization designated to receive emergency notifications when an anomaly is detected.

[0698] "Response actions" are actions that an emergency contact must take upon receiving an emergency notification.

[0699] The "living environment" refers to the conditions such as temperature, humidity, sound, and movement in the place where the user lives their daily life.

[0700] "Health status" refers to physiological data such as heart rate and blood pressure that indicate the user's physical health.

[0701] "Emotional state" refers to the psychological state of the user that can be analyzed from facial expressions and tone of voice.

[0702] This invention is a comprehensive system for supporting the independence and safety of elderly people. By incorporating an emotion engine, the system analyzes the user's emotional state and integrates it with environmental and health data to achieve comprehensive anomaly detection and rapid response.

[0703] System configuration and operation overview

[0704] The role of sensor devices

[0705] Terminal

[0706] The device is equipped with a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera, and microphone. These devices monitor the user's living environment and health. For example, the heart rate sensor captures the user's heart rate every second, and the camera and microphone capture the user's facial expressions and tone of voice every second.

[0707] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to determine the user's emotional state in real time, identifying whether the user is sad, angry, excited, etc.

[0708] Server Roles

[0709] server

[0710] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. The data is then saved in a database. Database management systems such as MySQL and PostgreSQL are used.

[0711] The server analyzes the stored data using an AI model. For example, facial expression data and tone of voice data are input into an emotion analysis algorithm to extract the user's emotional state. Then, health data and emotional data are integrated and considered to detect abnormalities. If an abnormality is detected, an emergency notification is generated according to the type and severity of the abnormality. The notification includes the type of abnormality, the user's emotional state, and any necessary response actions.

[0712] Emergency notification and response actions

[0713] User

[0714] The user (emergency contact) receives an emergency notification from the server. Notifications are sent via SMS, email, push notification, and other methods, allowing the emergency contact to respond quickly. For example, a message may be sent stating, "A fall has been detected in the living room and fear has been recognized. Please check immediately." Based on this notification, the emergency contact can quickly confirm the user's safety and arrange for emergency services if necessary.

[0715] Specific examples

[0716] As a concrete example, consider the case where a user falls at home and expresses feelings of fear and confusion.

[0717] Terminal

[0718] The device's fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[0719] server

[0720] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[0721] User

[0722] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the user's safety and arrange for emergency services if necessary.

[0723] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of the emotion engine enables more comprehensive anomaly detection that takes emotional state into account, further improving the quality of life for the elderly.

[0724] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0725] Step 1: Acquiring Sensor Data

[0726] Terminal

[0727] The device collects data from a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone. Specifically, the heart rate sensor measures the heart rate every second, the camera captures the user's face image every minute, and the sound detection sensor constantly monitors the user's voice.

[0728] Input: Environmental data (e.g., temperature, humidity), health data (e.g., heart rate, blood pressure), emotional data (e.g., facial expression, voice)

[0729] Output: Acquired raw data (e.g., heart rate data, facial image data)

[0730] Step 2: Sending data

[0731] Terminal

[0732] The data acquired by the device is sent to a server via Wi-Fi or Bluetooth. For example, heart rate data is sent to the server in batches every minute, and emergency data such as falls is sent in real time.

[0733] Input: Acquired raw data (e.g., heart rate data, facial image data)

[0734] Output: Data sent to the server

[0735] Step 3: Validate the data

[0736] server

[0737] The server validates the data received from the device, for example, checking that the heart rate is not too extreme (0 or over 200) and filtering out invalid data.

[0738] Input: Transmitted data (e.g. heart rate data)

[0739] Output: Normal data (e.g. filtered heart rate data)

[0740] Step 4: Save your data

[0741] server

[0742] The server saves the validated data in a database. Specifically, it uses a database management system (e.g., MySQL, PostgreSQL) and saves each data with a timestamp.

[0743] Input: Normal data (e.g. filtered heart rate data)

[0744] Output: Data stored in the database

[0745] Step 5: Analyze the sentiment data

[0746] server

[0747] The server analyzes the stored facial image data and voice data using an AI model to determine the user's emotional state, specifically using facial expression analysis algorithms and voice tone analysis algorithms.

[0748] Input: Facial image data, audio data

[0749] Output: Parsed emotion data (e.g., fear, joy)

[0750] Step 6: Detect anomalies

[0751] server

[0752] The server integrates and analyzes health data and emotional data to detect abnormalities. For example, if data on a fall and emotional fear occur simultaneously, it will be detected as an abnormality.

[0753] Input: Health data (e.g., fall data), emotion data (e.g., fear)

[0754] Output: Anomaly detection result (e.g., fall + fear = abnormal)

[0755] Step 7: Generate and send emergency notifications

[0756] server

[0757] If the server detects an anomaly, it generates an emergency notification and sends it to emergency contacts. The notification includes the type of anomaly, the emotional state, and a recommended response action. For example, it can generate a message like, "A fall has been detected in the living room and the emotion of fear has been recognized. Please take immediate action."

[0758] Input: Anomaly detection result (e.g., fall + fear = abnormal)

[0759] Output: Emergency notification message

[0760] Step 8: Receive emergency notifications and take action

[0761] User

[0762] The user (emergency contact) receives the emergency notification from the server, checks the content, and then promptly takes action. Specifically, the notification is received via a smartphone app or SMS, and the user's safety is promptly confirmed and emergency services are called if necessary.

[0763] Input: Emergency notification message

[0764] Output: Response actions (e.g., phone call to check on safety or arrange for emergency services)

[0765] (Application example 2)

[0766] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0767] When elderly people and other passengers use self-driving vehicles, they need to be able to comprehensively monitor their health and emotional states in real time and respond quickly if an abnormality occurs. However, current systems only monitor health data and do not consider emotional state information, which can result in incomplete anomaly detection. The present invention aims to solve these problems and improve the safety and security of self-driving vehicles by comprehensively monitoring the user's health and emotional state.

[0768] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0769] In this invention, the server includes means for measuring environmental data and user health data using a sensor device, means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, means for analyzing emotional data obtained from the sensor device and recognizing the user's emotional state, means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, and means for the emergency contact who receives the emergency notification to take response action. This allows for comprehensive monitoring of the user's health and emotional state, enabling a prompt and appropriate response when an abnormality is detected.

[0770] A "sensor device" is a device that measures environmental data and user health data.

[0771] A "server" is a computer system that analyzes measurement data received from sensor devices and detects abnormalities.

[0772] An "emergency notification" is a warning message that the server creates and sends to emergency contacts when an abnormality is detected.

[0773] An "emergency contact" is a contact who is responsible for taking action in response to a user's abnormality.

[0774] An "AI model" is an algorithm that uses machine learning to analyze data and detect anomalies.

[0775] "Real-time" means that data is processed immediately at the moment it is generated.

[0776] "Emotion data" is information about the emotional state of the user analyzed from their facial expressions and voice.

[0777] A "library for voice analysis" is a software component for analyzing voice data and recognizing its content.

[0778] An "on-board device" is a device that is installed in a vehicle and integrates and analyzes data acquired from sensor devices.

[0779] A "portable device" is a communication terminal that a user can carry around with them.

[0780] This invention is a comprehensive system that comprehensively monitors the health and emotional state of a user and responds quickly if an abnormality occurs. The system of the present invention is installed in an autonomous vehicle and includes the following components:

[0781] System Configuration

[0782] 1. Sensor device:

[0783] These devices include cameras, microphones, heart rate sensors, and pressure sensors installed in the vehicle. These devices measure the user's environmental and health data in real time. Emotion data is obtained by analyzing facial expressions and tone of voice using the cameras and microphones.

[0784] 2. Server:

[0785] The system receives and analyzes measurement and emotion data sent from the sensor devices. Specific software used includes OpenCV, Dlib, and DeepFace for face tracking and facial expression recognition, Google Cloud Speech-to-Text API for voice analysis, and TensorFlow for health data analysis. The server integrates this data and generates emergency notifications if an abnormality is detected.

[0786] 3. Onboard equipment:

[0787] This device sends emergency notifications to a display in the vehicle or to the user's mobile device. When an emergency notification is sent, the information is also sent to registered emergency contacts.

[0788] Program processing and data calculation

[0789] Data Acquisition:

[0790] The sensor device collects the user's health data (heart rate, pressure, etc.) and emotional data (facial expressions, voice) in real time and sends this to the server.

[0791] Data Analysis:

[0792] The server analyzes this data using AI models such as TensorFlow and DeepFace. First, it validates the data to ensure it is normal. Then, the AI ​​model detects any anomalies.

[0793] Emergency notification:

[0794] If an abnormality is detected, the system generates an emergency notification and sends it to the vehicle's display and the user's mobile device, as well as to registered emergency contacts via social media and email.

[0795] Specific examples

[0796] For example, consider a case where an elderly person's heart rate suddenly rises while using an autonomous vehicle, and the camera analysis reveals a fearful expression. In this case, the system operates as follows:

[0797] The camera and heart rate sensor collect data in real time and send it to a server.

[0798] The server uses the acquired data to analyze the user's emotions and health status and detect any abnormalities.

[0799] If an abnormality is detected, an emergency notification is generated and displayed on the vehicle's display and on the user's mobile device, stating "An abnormality has been detected, please stop the vehicle at a safe location."

[0800] In addition, an emergency notification will be sent to pre-registered emergency contacts stating, "An abnormality has been detected. Please check immediately."

[0801] Prompt Sentence Examples

[0802] "If an elderly person sitting in the passenger seat of a self-driving vehicle suddenly experiences a rise in heart rate and facial expressions indicate abnormalities, send that data and generate an immediate emergency notification."

[0803] Prompt sentence for prompt generation AI model:

[0804] "Input data: Heart rate data: 120 bpm, Facial expression data: scared, Audio data: speaking with trembling. An abnormality has been detected, so please generate an emergency notification."

[0805] This system will ensure the safety and security of elderly and other passengers, and will also enable a prompt and appropriate response in the event of an abnormality.

[0806] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0807] Step 1:

[0808] Data Acquisition

[0809] The terminal (sensor device) uses devices installed inside the vehicle, such as a camera, microphone, heart rate sensor, and pressure sensor, to measure the user's environmental data and health data in real time. The camera and microphone are used to capture the user's facial expression and voice data, which are then recognized as emotional data. The heart rate sensor and pressure sensor are also used to capture the user's heart rate and seat pressure data. This data is sent from the terminal to a server.

[0810] Input: Real-time environmental data, heart rate data, pressure data, facial expression data, voice data

[0811] Output: Sending the retrieved data

[0812] Step 2:

[0813] Data Receipt and Validation

[0814] The server receives the environmental data, health data, and emotion data sent from the device. The server first validates the received data to ensure that it is correct, thereby ensuring the reliability of the data.

[0815] Input: Data sent from the terminal

[0816] Output: Validated data

[0817] Step 3:

[0818] Data analysis

[0819] The server then uses the validated data to perform analysis. Specifically, it proceeds as follows:

[0820] OpenCV, Dlib, and DeepFace are used for face tracking and facial expression recognition, and the user's emotions are analyzed based on image data obtained from the camera.

[0821] For voice analysis, the Google Cloud Speech-to-Text API is used to convert voice data obtained from the microphone into text and analyze emotions.

[0822] TensorFlow is used for health data analysis, analyzing heart rate and pressure data.

[0823] This data is analyzed comprehensively to determine whether any abnormalities are detected.

[0824] Input: Validated environmental data, health data, and emotion data

[0825] Output: Analysis results (presence or absence of abnormalities)

[0826] Step 4:

[0827] Anomaly detection

[0828] If the server detects an anomaly based on the analysis results, it generates an emergency notification according to the type and severity of the anomaly, using an anomaly detection algorithm based on an analytical model (TensorFlow or DeepFace).

[0829] Input: Analysis results

[0830] Output: Urgent Notification

[0831] Step 5:

[0832] Sending emergency notifications

[0833] If an abnormality is detected, the server generates an emergency notification and sends it to the vehicle's display and the user's mobile device.The server also sends the emergency notification to pre-registered emergency contacts via email and social media.

[0834] Input: Emergency Notification

[0835] Output: Notification to emergency contacts and mobile devices

[0836] Step 6:

[0837] Response actions

[0838] The user and emergency contacts can take immediate action based on the emergency notification they receive, such as checking the user's health and providing first aid or contacting emergency services if necessary, or even stopping the self-driving vehicle at a safe location based on the notification displayed on the vehicle's display.

[0839] Input: Emergency Notification

[0840] Output: Execute response action

[0841] In this way, the system monitors the user's health and emotional state in real time, enabling rapid and appropriate response when an abnormality is detected.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0843] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0845] [Third embodiment]

[0846] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0847] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0848] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0849] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0850] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0851] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0852] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0853] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0854] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0855] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0856] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0857] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0858] This invention is a comprehensive system to support the independence and safety of the elderly, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts.

[0859] System configuration and operation overview

[0860] The role of sensor devices

[0861] Terminal

[0862] The terminal includes various sensor devices for monitoring the elderly's living environment and health condition, including temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and transmit it to a server at regular intervals.

[0863] Sensor devices are placed in various rooms in the home to measure temperature, humidity, sound, and movement, while wearable devices worn by users measure health data such as heart rate, blood pressure, and activity levels.

[0864] Server Roles

[0865] server

[0866] The server centrally receives and analyzes the data sent from the sensor devices. The server is equipped with an AI model that analyzes the received data and detects anomalies.

[0867] The received data is first validated to ensure it is normal. Next, the data is stored in a database and analyzed using an AI model. If an abnormality is detected as a result of the analysis, an emergency notification is generated and sent to emergency contacts.

[0868] Emergency notification and response actions

[0869] User

[0870] Users, i.e., elderly people and their families, receive emergency notifications from the server. The emergency notifications are sent to pre-registered emergency contacts, who then receive the notifications via SMS, email, push notifications, etc.

[0871] For example, if a user falls at home, a motion detection sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family. The family will receive the notification and take prompt action.

[0872] Specific examples

[0873] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[0874] Terminal

[0875] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[0876] server

[0877] The server receives this data and uses an AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[0878] User

[0879] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[0880] In this way, the system is designed to ensure the independence and safety of the elderly and to respond quickly in the event of an emergency.

[0881] The processing flow will be explained below.

[0882] Step 1:

[0883] Terminal

[0884] Sensor devices measure the user's environmental and health data in real time. For example, a temperature sensor measures the room temperature, and a heart rate sensor measures the user's heart rate. The measured data is collected into data packets at regular intervals.

[0885] Step 2:

[0886] Terminal

[0887] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and is sent to the server via an HTTP POST request.

[0888] Step 3:

[0889] server

[0890] The server receives the data sent from the sensor device, and the received data is first validated to ensure that the data structure and format are correct.

[0891] Step 4:

[0892] server

[0893] Data that passes validation is stored in a time-series database, allowing past and present data to be properly managed and used for subsequent analysis.

[0894] Step 5:

[0895] server

[0896] After the data is saved, it is analyzed by an AI model, which detects anomalies based on the received data and determines whether or not there are any anomalies.

[0897] Step 6:

[0898] server

[0899] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, and location information.

[0900] Step 7:

[0901] server

[0902] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[0903] Step 8:

[0904] User

[0905] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[0906] Step 9:

[0907] User

[0908] For example, if a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe.

[0909] In this way, the system works by linking the sensor device, server, and user (emergency contact) sections to detect abnormalities and facilitate rapid response.

[0910] Example 1

[0911] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0912] To effectively support the safety and independence of elderly people, a system that monitors environmental and health data in real time and responds quickly when an abnormality occurs is required. However, conventional systems have issues with data validation, the accuracy of abnormality detection, and methods for sending emergency notifications, and there is a lack of a comprehensive solution to these issues.

[0913] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0914] In this invention, the server includes means for receiving and validating measurement data from the sensor devices, means for storing the measurement data in a database, and means for using a generative AI model to analyze the measurement data, which enables real-time data collection and validation, enabling highly accurate anomaly detection and rapid emergency notification.

[0915] A "sensor device" is a device for measuring environmental data and user health data, and specifically includes a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, etc.

[0916] "Measurement data" refers to data such as temperature, humidity, sound, movement, heart rate, and blood pressure measured by sensor devices.

[0917] "Validation" is the process of checking whether the received measurement data is normal, and includes checking the data format, checking for missing values, verifying the consistency of timestamps, etc.

[0918] A "database" is a system for electronically storing and managing measurement data, and specifically, a database system such as PostgreSQL is used.

[0919] "Generative AI model" refers to technology that uses artificial intelligence models to analyze received measurement data and detect anomalies. Specific examples include TensorFlow and PyTorch.

[0920] "Abnormal" refers to data that deviates from the normal range in the user's environment or health condition, such as a sudden increase in heart rate or a fall.

[0921] An "emergency notification" is a notification that is generated when an abnormality is detected, and is a message that is sent to an emergency contact along with specific information about the abnormality.

[0922] "Emergency Contact" refers to a person or organization that has been pre-registered to receive notification when an abnormality is detected.

[0923] "Real-time" refers to data being collected, transmitted, and analyzed almost immediately, requiring the system to respond immediately.

[0924] "Response actions" refer to the actions taken by emergency contacts who receive an emergency notification to ensure the safety of the elderly person, and may include making inquiries, visiting, or contacting emergency services.

[0925] "SMS" is an abbreviation for Short Message Service, which refers to a service that sends short text messages using mobile phones.

[0926] "Mail" is an abbreviation for electronic mail and refers to a means of communication for exchanging text and files over the Internet.

[0927] "Push notification" refers to the ability of a mobile application to notify users of information in real time, typically using the smartphone's notification system.

[0928] This invention is a comprehensive system for supporting the independence and safety of elderly people, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts. The system consists of the following components:

[0929] Sensor device configuration and operation

[0930] Terminal

[0931] The terminal includes sensor devices for monitoring the living environment and health condition of the elderly, such as temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices collect data in real time and transmit it to a server at regular intervals.

[0932] Specifically, temperature and humidity sensors are placed in each room of the home, and wearable devices measure heart rate, blood pressure, and activity levels.Moreover, motion detection sensors detect sudden movements by the user, for example, to detect falls.

[0933] Server configuration and operation

[0934] server

[0935] The server receives and analyzes the data sent from the sensor devices. The server is equipped with a generative AI model (e.g., TensorFlow or PyTorch) that analyzes the received data and detects anomalies.

[0936] The received data first passes validation to ensure it is normal. It is then stored in a database (e.g., PostgreSQL) and analyzed by a generative AI model. If an anomaly is detected, an emergency notification is generated and sent to emergency contacts.

[0937] Emergency notification and response actions

[0938] User

[0939] Users, i.e., elderly people and their families, receive emergency notifications from the server. The notifications are sent to pre-registered emergency contacts via SMS, email, push notification, etc. The emergency contacts receive these notifications and can take prompt action.

[0940] For example, if a user falls at home, a motion sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family, who will then receive the notification and take immediate action.

[0941] Specific examples

[0942] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[0943] Terminal

[0944] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[0945] server

[0946] The server receives this data and uses the generative AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[0947] User

[0948] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[0949] Example prompts for the generative AI model to use

[0950] "Describe a system that uses sensor data to monitor the safety of elderly people and send emergency notifications if an abnormality occurs. The following is an outline of a specific system. The sensor device..."

[0951] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0952] Step 1: Sensor devices collect data

[0953] Terminal

[0954] The sensor devices collect real-time data about the elderly's living environment and health status. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. The collected data is stored with a timestamp.

[0955] Input: Environmental and health data such as temperature, humidity, and heart rate

[0956] Output: Timestamped sensor data

[0957] Step 2: The sensor device sends the collected data to the server

[0958] Terminal

[0959] The sensor device sends the collected data to the server at regular intervals, for example, every minute, and uploads the data to the server. The data includes the sensor's identification information and the measured value.

[0960] Input: Timestamped sensor data

[0961] Output: Sensor data sent to the server

[0962] Step 3: The server receives and validates the data

[0963] server

[0964] The server receives data sent from the sensor device. The received data is first validated to check for any irregularities or defects. Specifically, the data format is checked, missing values ​​are checked, and the consistency of timestamps is verified. Invalid data is recorded in an error log.

[0965] Input: Sensor data sent to the server

[0966] Output: Correct sensor data that passes validation (or invalid data recorded in the error log)

[0967] Step 4: The server saves the data to the database

[0968] server

[0969] The data that passes validation is stored in a database. For example, a database system such as PostgreSQL is used, and the data is written to a sensor data table. The data is used for later analysis.

[0970] Input: Correct sensor data that has passed validation

[0971] Output: Sensor data stored in a database

[0972] Step 5: The server analyzes the data using the generative AI model

[0973] server

[0974] The server performs data analysis using a generative AI model (e.g., TensorFlow or PyTorch). To detect anomalies in the received data, the AI ​​model analyzes environmental and health data to find data or patterns that deviate from normal ranges.

[0975] Input: Sensor data stored in a database

[0976] Output: Analysis results (presence or absence of abnormalities)

[0977] Step 6: Generate an emergency notification if the server detects an anomaly

[0978] server

[0979] If an abnormality is detected based on the analysis results, an emergency notification is generated. The notification content includes the type of abnormality detected, a timestamp, and location information. For example, a specific message such as "A fall has been detected in the living room. Please take action."

[0980] Input: Analysis results (presence or absence of abnormalities)

[0981] Output: Urgent notification message

[0982] Step 7: Server sends emergency notification to emergency contacts

[0983] server

[0984] The generated emergency notification is sent to pre-registered emergency contacts via SMS, email, push notification, etc. Specifically, SMS is sent using the Twilio API, and email is sent using the SendGrid API.

[0985] Input: Emergency notification message

[0986] Output: Notification sent to emergency contacts

[0987] Step 8: User receives notification and takes action

[0988] User

[0989] The emergency contact (such as the elderly person's family member) who receives the emergency notification can check the notification via a smartphone app or email. Based on the notification content, they can take appropriate action to ensure the safety of the elderly person. For example, a family member can call the elderly person to check on their safety and contact emergency services if necessary.

[0990] Input: Notification sent to emergency contacts

[0991] Output: Immediate response actions to ensure the safety of the elderly

[0992] (Application example 1)

[0993] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0994] There is a need to provide an environment where elderly people and customers with health concerns can enjoy shopping safely in physical stores. However, current technology lacks a system that can detect elderly people's health conditions and abnormal situations in real time and respond quickly. This makes it difficult for customers to enjoy shopping with peace of mind, and there is a lack of means for store staff and family members to respond quickly. A system to solve this problem is needed.

[0995] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0996] In this invention, the server includes a means for measuring environmental data and user health data using a sensor device, a means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, a means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, a means for monitoring the safety of elderly customers and customers with health concerns within the store environment, and a means for notifying store staff when an abnormality in a customer is detected. This allows elderly customers and customers with health concerns to spend time safely within the physical store, and enables a quick response when an abnormality occurs.

[0997] A "sensor device" is a hardware device that measures environmental data and user health data.

[0998] "Environmental data" refers to data relating to ambient environmental conditions such as temperature, humidity, sound, and motion.

[0999] "Health data" refers to data related to the user's physical condition, such as heart rate, blood pressure, and activity level.

[1000] A "server" is a centralized computing system that analyzes data sent from sensor devices, detects abnormalities, and sends emergency notifications.

[1001] "Anomaly detection" is the process of identifying unusual conditions or changes based on environmental and health data.

[1002] "Emergency notification" is a system that sends warnings and information to pre-designated emergency contacts when an abnormality is detected.

[1003] An "emergency contact" is a person or organization that should receive notifications in the event of an emergency, such as a relative or person in charge of an elderly or health-conscious user who has been registered in advance.

[1004] The "store environment" is the sum of the physical space and environmental conditions within a physical store.

[1005] "Customer safety monitoring" is the process of using sensor devices to monitor the health status and abnormal situations of customers in physical stores in real time and take necessary measures.

[1006] "Store staff notification" is the process of quickly sending warnings and information to store staff when an abnormality is detected among customers.

[1007] This invention is a comprehensive system for supporting elderly people and customers with health concerns to spend time safely in physical stores. The detailed implementation method of this system will be described below.

[1008] Configuration and operation overview

[1009] The role of sensor devices

[1010] Sensor Device

[1011] The sensor devices are used to measure environmental data and user health data. These devices include temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and send it to a server at regular intervals. The sensor devices are placed in various locations throughout the store and constantly monitor health data such as customer heart rate, body temperature, and activity level.

[1012] Server Roles

[1013] server

[1014] The server centrally receives and analyzes data sent from the sensor devices. First, it validates the data to ensure it is normal. Next, it stores the data in a database and analyzes it using an AI model. This AI model is used to detect anomalies, and if an anomaly is detected, it immediately generates an emergency notification and sends it to store staff and registered emergency contacts.

[1015] Emergency notification and response actions

[1016] User

[1017] Users (customers) and emergency contacts (store staff and family members) receive emergency notifications from the server via smartphone or tablet applications and take appropriate action.

[1018] Hardware and software used

[1019] This system uses the following hardware and software:

[1020] Hardware: Various sensor devices (heart rate sensors, motion sensors, etc.), smartphones or tablets with internet connectivity

[1021] Software: Programmed using Python 3. Data analysis uses a simple threshold-based anomaly detection algorithm, and notifications are sent via an HTTP request to an external SMS sending API.

[1022] Specific examples

[1023] If 75-year-old Customer A falls while shopping in a store, the motion detection sensor in the store will detect this movement and send the data to a server. An AI model installed on the server will detect this as an abnormality and immediately generate an emergency notification. This notification will be sent to the smartphone or tablet of a store staff member, saying, "Customer A has fallen. Please check." A similar notification will also be sent to the family, allowing them to respond quickly.

[1024] Prompt Sentence Examples

[1025] To help elderly and health-conscious customers feel safe while shopping in physical stores, you will help design and implement a system that uses sensor devices to monitor heart rate and movement, and notify store staff and family members if any abnormalities occur.

[1026] In this way, a system is realized that provides a safe environment for elderly people and customers with health concerns, and that can respond quickly and appropriately in the event of an emergency.

[1027] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1028] Step 1:

[1029] Sensor devices measure environmental data and user health data, including temperature, humidity, sound, movement, heart rate, blood pressure, etc. The inputs to a sensor device include data measured by various sensors. The sensor device collects this data in real time and outputs it as measurement data.

[1030] Step 2:

[1031] The terminal (sensor device) sends measurement data to the server at regular intervals. The terminal receives measurement data obtained from the sensor device as input and sends the data to the server as output.

[1032] Step 3:

[1033] The server validates the measurement data received from the sensor device. As input, it receives the measurement data sent from the sensor device and identifies valid and invalid data as output. Specific operations include checking the data format and verifying the numerical range.

[1034] Step 4:

[1035] The server saves the validated data to the database. It receives valid data that has passed validation as input and saves it to the database as output. Specific operations include database write operations.

[1036] Step 5:

[1037] The server inputs the stored data into the AI ​​model to detect anomalies. It receives the measurement data stored in the database as input and obtains the anomaly detection results as output. Specific operations include data analysis using the AI ​​model.

[1038] Step 6:

[1039] The server creates an emergency notification when an anomaly is detected. It receives the anomaly detection result as input and generates an emergency notification message as output. Specific operations include creating the message content and specifying the destination.

[1040] Step 7:

[1041] The server sends emergency notifications to store staff and emergency contacts. As input, it receives the generated emergency notification message and as output, it sends notifications to registered contacts. Specific operations include sending SMS, email, and push notifications.

[1042] Step 8:

[1043] The user (store staff) receives the emergency notification and takes appropriate action. The input is the emergency notification received on a smartphone or tablet, and the output is rushing to the scene or contacting an emergency service. Specific actions include checking the scene and taking emergency action.

[1044] In this way, the entire system is processed sequentially, realizing a system that supports elderly customers and customers with health concerns to stay safe in stores.

[1045] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1046] This invention combines an emotion engine with a comprehensive system that supports the independence and safety of the elderly. It analyzes user emotion data in addition to environmental and health data, enabling more comprehensive and rapid anomaly detection and response.

[1047] System configuration and operation overview

[1048] The role of sensor devices

[1049] Terminal

[1050] The sensor devices include sensors that monitor the elderly's living environment and health status, and an emotion engine that recognizes the user's emotions. Specifically, the devices include temperature sensors, humidity sensors, voice detection sensors, motion detection sensors, heart rate sensors, blood pressure monitors, and emotion recognition devices using cameras and microphones.

[1051] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to recognize the user's emotions, making it possible to determine in real time whether the user is sad, angry, excited, etc.

[1052] Server Roles

[1053] server

[1054] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. It is then stored in a database and analyzed by the AI ​​model or emotion engine.

[1055] The server integrates and analyzes health and emotional data, and if an abnormality is detected, it determines the appropriate response depending on the type and severity of the abnormality. If an abnormality is detected, an emergency notification is generated and sent to emergency contacts.

[1056] Emergency notification and response actions

[1057] User

[1058] The user (emergency contact) receives an emergency notification from the server. The emergency notification includes the abnormality detection result, emotional state, and required response action. The emergency contact receives these notifications via SMS, email, push notification, etc., and responds promptly.

[1059] Specific examples

[1060] For example, consider the case where a user falls at home and expresses emotions of fear and confusion. Here is how the system works:

[1061] Terminal

[1062] The fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[1063] server

[1064] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[1065] User

[1066] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the elderly person's safety and, if necessary, call emergency services.

[1067] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of an emotion engine enables more comprehensive anomaly detection that also takes the user's emotional state into account, further improving the quality of life for the elderly.

[1068] The processing flow will be explained below.

[1069] Step 1:

[1070] Terminal

[1071] The sensor devices measure the user's environmental and health data in real time. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. In addition, the emotion engine uses the camera and microphone to capture the user's facial expressions and voice and analyze their emotional state.

[1072] Step 2:

[1073] Terminal

[1074] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and sent via an HTTP POST request. Emotion data is also collected and sent to the server.

[1075] Step 3:

[1076] server

[1077] The server receives data sent from the sensor devices and emotion engine. The received data is first validated to ensure that the data structure and format are correct.

[1078] Step 4:

[1079] server

[1080] Data that passes validation is stored in a time series database, which ensures that past and present data are properly managed.

[1081] Step 5:

[1082] server

[1083] After the data is saved, it is analyzed by an AI model and an emotion engine. The AI ​​model uses health and environmental data to detect abnormalities, and the emotion engine analyzes the user's emotional data to detect abnormal emotional states.

[1084] Step 6:

[1085] server

[1086] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, location information, and emotional state, for example, "Your heart rate has increased sharply, and you have detected an emotion of fear."

[1087] Step 7:

[1088] server

[1089] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[1090] Step 8:

[1091] User

[1092] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[1093] Step 9:

[1094] User

[1095] For example, when a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe. It also takes into account the user's emotional state and provides any necessary psychological support.

[1096] In this way, the system detects abnormalities through collaboration between the sensor device, server, and user (emergency contact) sections, and promotes prompt and appropriate responses.The introduction of an emotion engine also analyzes the user's emotional state, achieving more comprehensive safety management.

[1097] Example 2

[1098] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1099] In systems designed to support the independence and safety of elderly people, conventional technologies primarily monitor environmental and health data, without taking into account the user's emotional state when detecting anomalies. Therefore, comprehensive anomaly detection and rapid response, including not only falls and deterioration in health status but also emotional anomalies, are required. Furthermore, real-time data acquisition and analysis are incomplete, sometimes leading to delays in emergency response.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1101] In this invention, the server includes a measuring means for monitoring the living environment and health status, a means for transmitting data obtained by the measuring means to the server, a means for the server to receive the data and validate its contents, a means for storing the validated data in a database, a means for analyzing the stored data and using an AI model to detect anomalies, a means for generating an emergency notification when an anomaly is detected and sending it to an emergency contact, and a means for the emergency contact to take action upon receiving the emergency notification. This enables real-time data acquisition and analysis, as well as comprehensive anomaly detection and rapid response that takes into account emotional states.

[1102] "Measurement means" refers to a device that acquires data using a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone, etc., to monitor living environment and health condition.

[1103] The "server" is a central management system that receives and validates data sent from the measurement means, stores it in a database, and analyzes the data using an AI model.

[1104] "Validation" is the process by which the server verifies that the data it receives is correct and valid.

[1105] A "database" is a system for systematically storing and managing validated data.

[1106] An "AI model" is a machine learning algorithm or artificial intelligence system used to analyze stored data and detect anomalies.

[1107] An "emergency notification" is a message or alert that the server generates and sends to emergency contacts when an abnormality is detected.

[1108] An "emergency contact" is a person or organization designated to receive emergency notifications when an anomaly is detected.

[1109] "Response actions" are actions that an emergency contact must take upon receiving an emergency notification.

[1110] The "living environment" refers to the conditions such as temperature, humidity, sound, and movement in the place where the user lives their daily life.

[1111] "Health status" refers to physiological data such as heart rate and blood pressure that indicate the user's physical health.

[1112] "Emotional state" refers to the psychological state of the user that can be analyzed from facial expressions and tone of voice.

[1113] This invention is a comprehensive system for supporting the independence and safety of elderly people. By incorporating an emotion engine, the system analyzes the user's emotional state and integrates it with environmental and health data to achieve comprehensive anomaly detection and rapid response.

[1114] System configuration and operation overview

[1115] The role of sensor devices

[1116] Terminal

[1117] The device is equipped with a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera, and microphone. These devices monitor the user's living environment and health. For example, the heart rate sensor captures the user's heart rate every second, and the camera and microphone capture the user's facial expressions and tone of voice every second.

[1118] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to determine the user's emotional state in real time, identifying whether the user is sad, angry, excited, etc.

[1119] Server Roles

[1120] server

[1121] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. The data is then saved in a database. Database management systems such as MySQL and PostgreSQL are used.

[1122] The server analyzes the stored data using an AI model. For example, facial expression data and tone of voice data are input into an emotion analysis algorithm to extract the user's emotional state. Then, health data and emotional data are integrated and considered to detect abnormalities. If an abnormality is detected, an emergency notification is generated according to the type and severity of the abnormality. The notification includes the type of abnormality, the user's emotional state, and any necessary response actions.

[1123] Emergency notification and response actions

[1124] User

[1125] The user (emergency contact) receives an emergency notification from the server. Notifications are sent via SMS, email, push notification, and other methods, allowing the emergency contact to respond quickly. For example, a message may be sent stating, "A fall has been detected in the living room and fear has been recognized. Please check immediately." Based on this notification, the emergency contact can quickly confirm the user's safety and arrange for emergency services if necessary.

[1126] Specific examples

[1127] As a concrete example, consider the case where a user falls at home and expresses feelings of fear and confusion.

[1128] Terminal

[1129] The device's fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[1130] server

[1131] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[1132] User

[1133] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the user's safety and arrange for emergency services if necessary.

[1134] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of the emotion engine enables more comprehensive anomaly detection that takes emotional state into account, further improving the quality of life for the elderly.

[1135] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1136] Step 1: Acquiring Sensor Data

[1137] Terminal

[1138] The device collects data from a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone. Specifically, the heart rate sensor measures the heart rate every second, the camera captures the user's face image every minute, and the sound detection sensor constantly monitors the user's voice.

[1139] Input: Environmental data (e.g., temperature, humidity), health data (e.g., heart rate, blood pressure), emotional data (e.g., facial expression, voice)

[1140] Output: Acquired raw data (e.g., heart rate data, facial image data)

[1141] Step 2: Sending data

[1142] Terminal

[1143] The data acquired by the device is sent to a server via Wi-Fi or Bluetooth. For example, heart rate data is sent to the server in batches every minute, and emergency data such as falls is sent in real time.

[1144] Input: Acquired raw data (e.g., heart rate data, facial image data)

[1145] Output: Data sent to the server

[1146] Step 3: Validate the data

[1147] server

[1148] The server validates the data received from the device, for example, checking that the heart rate is not too extreme (0 or over 200) and filtering out invalid data.

[1149] Input: Transmitted data (e.g. heart rate data)

[1150] Output: Normal data (e.g. filtered heart rate data)

[1151] Step 4: Save your data

[1152] server

[1153] The server saves the validated data in a database. Specifically, it uses a database management system (e.g., MySQL, PostgreSQL) and saves each data with a timestamp.

[1154] Input: Normal data (e.g. filtered heart rate data)

[1155] Output: Data stored in the database

[1156] Step 5: Analyze the sentiment data

[1157] server

[1158] The server analyzes the stored facial image data and voice data using an AI model to determine the user's emotional state, specifically using facial expression analysis algorithms and voice tone analysis algorithms.

[1159] Input: Facial image data, audio data

[1160] Output: Parsed emotion data (e.g., fear, joy)

[1161] Step 6: Detect anomalies

[1162] server

[1163] The server integrates and analyzes health data and emotional data to detect abnormalities. For example, if data on a fall and emotional fear occur simultaneously, it will be detected as an abnormality.

[1164] Input: Health data (e.g., fall data), emotion data (e.g., fear)

[1165] Output: Anomaly detection result (e.g., fall + fear = abnormal)

[1166] Step 7: Generate and send emergency notifications

[1167] server

[1168] If the server detects an anomaly, it generates an emergency notification and sends it to emergency contacts. The notification includes the type of anomaly, the emotional state, and a recommended response action. For example, it can generate a message like, "A fall has been detected in the living room and the emotion of fear has been recognized. Please take immediate action."

[1169] Input: Anomaly detection result (e.g., fall + fear = abnormal)

[1170] Output: Emergency notification message

[1171] Step 8: Receive emergency notifications and take action

[1172] User

[1173] The user (emergency contact) receives the emergency notification from the server, checks the content, and then promptly takes action. Specifically, the notification is received via a smartphone app or SMS, and the user's safety is promptly confirmed and emergency services are called if necessary.

[1174] Input: Emergency notification message

[1175] Output: Response actions (e.g., phone call to check on safety or arrange for emergency services)

[1176] (Application example 2)

[1177] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1178] When elderly people and other passengers use self-driving vehicles, they need to be able to comprehensively monitor their health and emotional states in real time and respond quickly if an abnormality occurs. However, current systems only monitor health data and do not consider emotional state information, which can result in incomplete anomaly detection. The present invention aims to solve these problems and improve the safety and security of self-driving vehicles by comprehensively monitoring the user's health and emotional state.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1180] In this invention, the server includes means for measuring environmental data and user health data using a sensor device, means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, means for analyzing emotional data obtained from the sensor device and recognizing the user's emotional state, means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, and means for the emergency contact who receives the emergency notification to take response action. This allows for comprehensive monitoring of the user's health and emotional state, enabling a prompt and appropriate response when an abnormality is detected.

[1181] A "sensor device" is a device that measures environmental data and user health data.

[1182] A "server" is a computer system that analyzes measurement data received from sensor devices and detects abnormalities.

[1183] An "emergency notification" is a warning message that the server creates and sends to emergency contacts when an abnormality is detected.

[1184] An "emergency contact" is a contact who is responsible for taking action in response to a user's abnormality.

[1185] An "AI model" is an algorithm that uses machine learning to analyze data and detect anomalies.

[1186] "Real-time" means that data is processed immediately at the moment it is generated.

[1187] "Emotion data" is information about the emotional state of the user analyzed from their facial expressions and voice.

[1188] A "library for voice analysis" is a software component for analyzing voice data and recognizing its content.

[1189] An "on-board device" is a device that is installed in a vehicle and integrates and analyzes data acquired from sensor devices.

[1190] A "portable device" is a communication terminal that a user can carry around with them.

[1191] This invention is a comprehensive system that comprehensively monitors the health and emotional state of a user and responds quickly if an abnormality occurs. The system of the present invention is installed in an autonomous vehicle and includes the following components:

[1192] System Configuration

[1193] 1. Sensor device:

[1194] These devices include cameras, microphones, heart rate sensors, and pressure sensors installed in the vehicle. These devices measure the user's environmental and health data in real time. Emotion data is obtained by analyzing facial expressions and tone of voice using the cameras and microphones.

[1195] 2. Server:

[1196] The system receives and analyzes measurement and emotion data sent from the sensor devices. Specific software used includes OpenCV, Dlib, and DeepFace for face tracking and facial expression recognition, Google Cloud Speech-to-Text API for voice analysis, and TensorFlow for health data analysis. The server integrates this data and generates emergency notifications if an abnormality is detected.

[1197] 3. Onboard equipment:

[1198] This device sends emergency notifications to a display in the vehicle or to the user's mobile device. When an emergency notification is sent, the information is also sent to registered emergency contacts.

[1199] Program processing and data calculation

[1200] Data Acquisition:

[1201] The sensor device collects the user's health data (heart rate, pressure, etc.) and emotional data (facial expressions, voice) in real time and sends this to the server.

[1202] Data Analysis:

[1203] The server analyzes this data using AI models such as TensorFlow and DeepFace. First, it validates the data to ensure it is normal. Then, the AI ​​model detects any anomalies.

[1204] Emergency notification:

[1205] If an abnormality is detected, the system generates an emergency notification and sends it to the vehicle's display and the user's mobile device, as well as to registered emergency contacts via social media and email.

[1206] Specific examples

[1207] For example, consider a case where an elderly person's heart rate suddenly rises while using an autonomous vehicle, and the camera analysis reveals a fearful expression. In this case, the system operates as follows:

[1208] The camera and heart rate sensor collect data in real time and send it to a server.

[1209] The server uses the acquired data to analyze the user's emotions and health status and detect any abnormalities.

[1210] If an abnormality is detected, an emergency notification is generated and displayed on the vehicle's display and on the user's mobile device, stating "An abnormality has been detected, please stop the vehicle at a safe location."

[1211] In addition, an emergency notification will be sent to pre-registered emergency contacts stating, "An abnormality has been detected. Please check immediately."

[1212] Prompt Sentence Examples

[1213] "If an elderly person sitting in the passenger seat of a self-driving vehicle suddenly experiences a rise in heart rate and facial expressions indicate abnormalities, send that data and generate an immediate emergency notification."

[1214] Prompt sentence for prompt generation AI model:

[1215] "Input data: Heart rate data: 120 bpm, Facial expression data: scared, Audio data: speaking with trembling. An abnormality has been detected, so please generate an emergency notification."

[1216] This system will ensure the safety and security of elderly and other passengers, and will also enable a prompt and appropriate response in the event of an abnormality.

[1217] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1218] Step 1:

[1219] Data Acquisition

[1220] The terminal (sensor device) uses devices installed inside the vehicle, such as a camera, microphone, heart rate sensor, and pressure sensor, to measure the user's environmental data and health data in real time. The camera and microphone are used to capture the user's facial expression and voice data, which are then recognized as emotional data. The heart rate sensor and pressure sensor are also used to capture the user's heart rate and seat pressure data. This data is sent from the terminal to a server.

[1221] Input: Real-time environmental data, heart rate data, pressure data, facial expression data, voice data

[1222] Output: Sending the retrieved data

[1223] Step 2:

[1224] Data Receipt and Validation

[1225] The server receives the environmental data, health data, and emotion data sent from the device. The server first validates the received data to ensure that it is correct, thereby ensuring the reliability of the data.

[1226] Input: Data sent from the terminal

[1227] Output: Validated data

[1228] Step 3:

[1229] Data analysis

[1230] The server then uses the validated data to perform analysis. Specifically, it proceeds as follows:

[1231] OpenCV, Dlib, and DeepFace are used for face tracking and facial expression recognition, and the user's emotions are analyzed based on image data obtained from the camera.

[1232] For voice analysis, the Google Cloud Speech-to-Text API is used to convert voice data obtained from the microphone into text and analyze emotions.

[1233] TensorFlow is used for health data analysis, analyzing heart rate and pressure data.

[1234] This data is analyzed comprehensively to determine whether any abnormalities are detected.

[1235] Input: Validated environmental data, health data, and emotion data

[1236] Output: Analysis results (presence or absence of abnormalities)

[1237] Step 4:

[1238] Anomaly detection

[1239] If the server detects an anomaly based on the analysis results, it generates an emergency notification according to the type and severity of the anomaly, using an anomaly detection algorithm based on an analytical model (TensorFlow or DeepFace).

[1240] Input: Analysis results

[1241] Output: Urgent Notification

[1242] Step 5:

[1243] Sending emergency notifications

[1244] If an abnormality is detected, the server generates an emergency notification and sends it to the vehicle's display and the user's mobile device.The server also sends the emergency notification to pre-registered emergency contacts via email and social media.

[1245] Input: Emergency Notification

[1246] Output: Notification to emergency contacts and mobile devices

[1247] Step 6:

[1248] Response actions

[1249] The user and emergency contacts can take immediate action based on the emergency notification they receive, such as checking the user's health and providing first aid or contacting emergency services if necessary, or even stopping the self-driving vehicle at a safe location based on the notification displayed on the vehicle's display.

[1250] Input: Emergency Notification

[1251] Output: Execute response action

[1252] In this way, the system monitors the user's health and emotional state in real time, enabling rapid and appropriate response when an abnormality is detected.

[1253] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1254] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1255] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1256] [Fourth embodiment]

[1257] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1258] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1259] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1260] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1261] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1262] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1263] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1264] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1265] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1266] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1267] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1268] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1269] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1270] This invention is a comprehensive system to support the independence and safety of the elderly, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts.

[1271] System configuration and operation overview

[1272] The role of sensor devices

[1273] Terminal

[1274] The terminal includes various sensor devices for monitoring the elderly's living environment and health condition, including temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and transmit it to a server at regular intervals.

[1275] Sensor devices are placed in various rooms in the home to measure temperature, humidity, sound, and movement, while wearable devices worn by users measure health data such as heart rate, blood pressure, and activity levels.

[1276] Server Roles

[1277] server

[1278] The server centrally receives and analyzes the data sent from the sensor devices. The server is equipped with an AI model that analyzes the received data and detects anomalies.

[1279] The received data is first validated to ensure it is normal. Next, the data is stored in a database and analyzed using an AI model. If an abnormality is detected as a result of the analysis, an emergency notification is generated and sent to emergency contacts.

[1280] Emergency notification and response actions

[1281] User

[1282] Users, i.e., elderly people and their families, receive emergency notifications from the server. The emergency notifications are sent to pre-registered emergency contacts, who then receive the notifications via SMS, email, push notifications, etc.

[1283] For example, if a user falls at home, a motion detection sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family. The family will receive the notification and take prompt action.

[1284] Specific examples

[1285] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[1286] Terminal

[1287] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[1288] server

[1289] The server receives this data and uses an AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[1290] User

[1291] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[1292] In this way, the system is designed to ensure the independence and safety of the elderly and to respond quickly in the event of an emergency.

[1293] The processing flow will be explained below.

[1294] Step 1:

[1295] Terminal

[1296] Sensor devices measure the user's environmental and health data in real time. For example, a temperature sensor measures the room temperature, and a heart rate sensor measures the user's heart rate. The measured data is collected into data packets at regular intervals.

[1297] Step 2:

[1298] Terminal

[1299] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and is sent to the server via an HTTP POST request.

[1300] Step 3:

[1301] server

[1302] The server receives the data sent from the sensor device, and the received data is first validated to ensure that the data structure and format are correct.

[1303] Step 4:

[1304] server

[1305] Data that passes validation is stored in a time-series database, allowing past and present data to be properly managed and used for subsequent analysis.

[1306] Step 5:

[1307] server

[1308] After the data is saved, it is analyzed by an AI model, which detects anomalies based on the received data and determines whether or not there are any anomalies.

[1309] Step 6:

[1310] server

[1311] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, and location information.

[1312] Step 7:

[1313] server

[1314] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[1315] Step 8:

[1316] User

[1317] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[1318] Step 9:

[1319] User

[1320] For example, if a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe.

[1321] In this way, the system works by linking the sensor device, server, and user (emergency contact) sections to detect abnormalities and facilitate rapid response.

[1322] Example 1

[1323] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1324] To effectively support the safety and independence of elderly people, a system that monitors environmental and health data in real time and responds quickly when an abnormality occurs is required. However, conventional systems have issues with data validation, the accuracy of abnormality detection, and methods for sending emergency notifications, and there is a lack of a comprehensive solution to these issues.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1326] In this invention, the server includes means for receiving and validating measurement data from the sensor devices, means for storing the measurement data in a database, and means for using a generative AI model to analyze the measurement data, which enables real-time data collection and validation, enabling highly accurate anomaly detection and rapid emergency notification.

[1327] A "sensor device" is a device for measuring environmental data and user health data, and specifically includes a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, etc.

[1328] "Measurement data" refers to data such as temperature, humidity, sound, movement, heart rate, and blood pressure measured by sensor devices.

[1329] "Validation" is the process of checking whether the received measurement data is normal, and includes checking the data format, checking for missing values, verifying the consistency of timestamps, etc.

[1330] A "database" is a system for electronically storing and managing measurement data, and specifically, a database system such as PostgreSQL is used.

[1331] "Generative AI model" refers to technology that uses artificial intelligence models to analyze received measurement data and detect anomalies. Specific examples include TensorFlow and PyTorch.

[1332] "Abnormal" refers to data that deviates from the normal range in the user's environment or health condition, such as a sudden increase in heart rate or a fall.

[1333] An "emergency notification" is a notification that is generated when an abnormality is detected, and is a message that is sent to an emergency contact along with specific information about the abnormality.

[1334] "Emergency Contact" refers to a person or organization that has been pre-registered to receive notification when an abnormality is detected.

[1335] "Real-time" refers to data being collected, transmitted, and analyzed almost immediately, requiring the system to respond immediately.

[1336] "Response actions" refer to the actions taken by emergency contacts who receive an emergency notification to ensure the safety of the elderly person, and may include making inquiries, visiting, or contacting emergency services.

[1337] "SMS" is an abbreviation for Short Message Service, which refers to a service that sends short text messages using mobile phones.

[1338] "Mail" is an abbreviation for electronic mail and refers to a means of communication for exchanging text and files over the Internet.

[1339] "Push notification" refers to the ability of a mobile application to notify users of information in real time, typically using the smartphone's notification system.

[1340] This invention is a comprehensive system for supporting the independence and safety of elderly people, and aims to detect abnormalities and respond quickly through communication with sensor devices, servers, and emergency contacts. The system consists of the following components:

[1341] Sensor device configuration and operation

[1342] Terminal

[1343] The terminal includes sensor devices for monitoring the living environment and health condition of the elderly, such as temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices collect data in real time and transmit it to a server at regular intervals.

[1344] Specifically, temperature and humidity sensors are placed in each room of the home, and wearable devices measure heart rate, blood pressure, and activity levels.Moreover, motion detection sensors detect sudden movements by the user, for example, to detect falls.

[1345] Server configuration and operation

[1346] server

[1347] The server receives and analyzes the data sent from the sensor devices. The server is equipped with a generative AI model (e.g., TensorFlow or PyTorch) that analyzes the received data and detects anomalies.

[1348] The received data first passes validation to ensure it is normal. It is then stored in a database (e.g., PostgreSQL) and analyzed by a generative AI model. If an anomaly is detected, an emergency notification is generated and sent to emergency contacts.

[1349] Emergency notification and response actions

[1350] User

[1351] Users, i.e., elderly people and their families, receive emergency notifications from the server. The notifications are sent to pre-registered emergency contacts via SMS, email, push notification, etc. The emergency contacts receive these notifications and can take prompt action.

[1352] For example, if a user falls at home, a motion sensor will detect the sudden movement and send the data to the server. The server will detect this as an abnormality and immediately send an emergency notification to the family, who will then receive the notification and take immediate action.

[1353] Specific examples

[1354] For example, if a user falls in their living room, a fall sensor detects the abnormal movement and sends the data to the server, including the timestamp of the fall, the sensor location, and the strength of the movement.

[1355] Terminal

[1356] The fall sensor sends data such as "a sudden fall was detected in the living room" to the server.

[1357] server

[1358] The server receives this data and uses the generative AI model to check for any abnormalities. If an abnormality is detected, it sends an emergency notification to pre-registered emergency contacts saying, "A fall has been detected in the living room. Please check."

[1359] User

[1360] Emergency contacts (family members) will receive this notification via a smartphone app and take immediate action to confirm the user's safety.

[1361] Example prompts for the generative AI model to use

[1362] "Describe a system that uses sensor data to monitor the safety of elderly people and send emergency notifications if an abnormality occurs. The following is an outline of a specific system. The sensor device..."

[1363] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1364] Step 1: Sensor devices collect data

[1365] Terminal

[1366] The sensor devices collect real-time data about the elderly's living environment and health status. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. The collected data is stored with a timestamp.

[1367] Input: Environmental and health data such as temperature, humidity, and heart rate

[1368] Output: Timestamped sensor data

[1369] Step 2: The sensor device sends the collected data to the server

[1370] Terminal

[1371] The sensor device sends the collected data to the server at regular intervals, for example, every minute, and uploads the data to the server. The data includes the sensor's identification information and the measured value.

[1372] Input: Timestamped sensor data

[1373] Output: Sensor data sent to the server

[1374] Step 3: The server receives and validates the data

[1375] server

[1376] The server receives data sent from the sensor device. The received data is first validated to check for any irregularities or defects. Specifically, the data format is checked, missing values ​​are checked, and the consistency of timestamps is verified. Invalid data is recorded in an error log.

[1377] Input: Sensor data sent to the server

[1378] Output: Correct sensor data that passes validation (or invalid data recorded in the error log)

[1379] Step 4: The server saves the data to the database

[1380] server

[1381] The data that passes validation is stored in a database. For example, a database system such as PostgreSQL is used, and the data is written to a sensor data table. The data is used for later analysis.

[1382] Input: Correct sensor data that has passed validation

[1383] Output: Sensor data stored in a database

[1384] Step 5: The server analyzes the data using the generative AI model

[1385] server

[1386] The server performs data analysis using a generative AI model (e.g., TensorFlow or PyTorch). To detect anomalies in the received data, the AI ​​model analyzes environmental and health data to find data or patterns that deviate from normal ranges.

[1387] Input: Sensor data stored in a database

[1388] Output: Analysis results (presence or absence of abnormalities)

[1389] Step 6: Generate an emergency notification if the server detects an anomaly

[1390] server

[1391] If an abnormality is detected based on the analysis results, an emergency notification is generated. The notification content includes the type of abnormality detected, a timestamp, and location information. For example, a specific message such as "A fall has been detected in the living room. Please take action."

[1392] Input: Analysis results (presence or absence of abnormalities)

[1393] Output: Urgent notification message

[1394] Step 7: Server sends emergency notification to emergency contacts

[1395] server

[1396] The generated emergency notification is sent to pre-registered emergency contacts via SMS, email, push notification, etc. Specifically, SMS is sent using the Twilio API, and email is sent using the SendGrid API.

[1397] Input: Emergency notification message

[1398] Output: Notification sent to emergency contacts

[1399] Step 8: User receives notification and takes action

[1400] User

[1401] The emergency contact (such as the elderly person's family member) who receives the emergency notification can check the notification via a smartphone app or email. Based on the notification content, they can take appropriate action to ensure the safety of the elderly person. For example, a family member can call the elderly person to check on their safety and contact emergency services if necessary.

[1402] Input: Notification sent to emergency contacts

[1403] Output: Immediate response actions to ensure the safety of the elderly

[1404] (Application example 1)

[1405] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1406] There is a need to provide an environment where elderly people and customers with health concerns can enjoy shopping safely in physical stores. However, current technology lacks a system that can detect elderly people's health conditions and abnormal situations in real time and respond quickly. This makes it difficult for customers to enjoy shopping with peace of mind, and there is a lack of means for store staff and family members to respond quickly. A system to solve this problem is needed.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1408] In this invention, the server includes a means for measuring environmental data and user health data using a sensor device, a means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, a means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, a means for monitoring the safety of elderly customers and customers with health concerns within the store environment, and a means for notifying store staff when an abnormality in a customer is detected. This allows elderly customers and customers with health concerns to spend time safely within the physical store, and enables a quick response when an abnormality occurs.

[1409] A "sensor device" is a hardware device that measures environmental data and user health data.

[1410] "Environmental data" refers to data relating to ambient environmental conditions such as temperature, humidity, sound, and motion.

[1411] "Health data" refers to data related to the user's physical condition, such as heart rate, blood pressure, and activity level.

[1412] A "server" is a centralized computing system that analyzes data sent from sensor devices, detects abnormalities, and sends emergency notifications.

[1413] "Anomaly detection" is the process of identifying unusual conditions or changes based on environmental and health data.

[1414] "Emergency notification" is a system that sends warnings and information to pre-designated emergency contacts when an abnormality is detected.

[1415] An "emergency contact" is a person or organization that should receive notifications in the event of an emergency, such as a relative or person in charge of an elderly or health-conscious user who has been registered in advance.

[1416] The "store environment" is the sum of the physical space and environmental conditions within a physical store.

[1417] "Customer safety monitoring" is the process of using sensor devices to monitor the health status and abnormal situations of customers in physical stores in real time and take necessary measures.

[1418] "Store staff notification" is the process of quickly sending warnings and information to store staff when an abnormality is detected among customers.

[1419] This invention is a comprehensive system for supporting elderly people and customers with health concerns to spend time safely in physical stores. The detailed implementation method of this system will be described below.

[1420] Configuration and operation overview

[1421] The role of sensor devices

[1422] Sensor Device

[1423] The sensor devices are used to measure environmental data and user health data. These devices include temperature sensors, humidity sensors, sound detection sensors, motion detection sensors, heart rate sensors, and blood pressure monitors. These devices measure data in real time and send it to a server at regular intervals. The sensor devices are placed in various locations throughout the store and constantly monitor health data such as customer heart rate, body temperature, and activity level.

[1424] Server Roles

[1425] server

[1426] The server centrally receives and analyzes data sent from the sensor devices. First, it validates the data to ensure it is normal. Next, it stores the data in a database and analyzes it using an AI model. This AI model is used to detect anomalies, and if an anomaly is detected, it immediately generates an emergency notification and sends it to store staff and registered emergency contacts.

[1427] Emergency notification and response actions

[1428] User

[1429] Users (customers) and emergency contacts (store staff and family members) receive emergency notifications from the server via smartphone or tablet applications and take appropriate action.

[1430] Hardware and software used

[1431] This system uses the following hardware and software:

[1432] Hardware: Various sensor devices (heart rate sensors, motion sensors, etc.), smartphones or tablets with internet connectivity

[1433] Software: Programmed using Python 3. Data analysis uses a simple threshold-based anomaly detection algorithm, and notifications are sent via an HTTP request to an external SMS sending API.

[1434] Specific examples

[1435] If 75-year-old Customer A falls while shopping in a store, the motion detection sensor in the store will detect this movement and send the data to a server. An AI model installed on the server will detect this as an abnormality and immediately generate an emergency notification. This notification will be sent to the smartphone or tablet of a store staff member, saying, "Customer A has fallen. Please check." A similar notification will also be sent to the family, allowing them to respond quickly.

[1436] Prompt Sentence Examples

[1437] To help elderly and health-conscious customers feel safe while shopping in physical stores, you will help design and implement a system that uses sensor devices to monitor heart rate and movement, and notify store staff and family members if any abnormalities occur.

[1438] In this way, a system is realized that provides a safe environment for elderly people and customers with health concerns, and that can respond quickly and appropriately in the event of an emergency.

[1439] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1440] Step 1:

[1441] Sensor devices measure environmental data and user health data, including temperature, humidity, sound, movement, heart rate, blood pressure, etc. The inputs to a sensor device include data measured by various sensors. The sensor device collects this data in real time and outputs it as measurement data.

[1442] Step 2:

[1443] The terminal (sensor device) sends measurement data to the server at regular intervals. The terminal receives measurement data obtained from the sensor device as input and sends the data to the server as output.

[1444] Step 3:

[1445] The server validates the measurement data received from the sensor device. As input, it receives the measurement data sent from the sensor device and identifies valid and invalid data as output. Specific operations include checking the data format and verifying the numerical range.

[1446] Step 4:

[1447] The server saves the validated data to the database. It receives valid data that has passed validation as input and saves it to the database as output. Specific operations include database write operations.

[1448] Step 5:

[1449] The server inputs the stored data into the AI ​​model to detect anomalies. It receives the measurement data stored in the database as input and obtains the anomaly detection results as output. Specific operations include data analysis using the AI ​​model.

[1450] Step 6:

[1451] The server creates an emergency notification when an anomaly is detected. It receives the anomaly detection result as input and generates an emergency notification message as output. Specific operations include creating the message content and specifying the destination.

[1452] Step 7:

[1453] The server sends emergency notifications to store staff and emergency contacts. As input, it receives the generated emergency notification message and as output, it sends notifications to registered contacts. Specific operations include sending SMS, email, and push notifications.

[1454] Step 8:

[1455] The user (store staff) receives the emergency notification and takes appropriate action. The input is the emergency notification received on a smartphone or tablet, and the output is rushing to the scene or contacting an emergency service. Specific actions include checking the scene and taking emergency action.

[1456] In this way, the entire system is processed sequentially, realizing a system that supports elderly customers and customers with health concerns to stay safe in stores.

[1457] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1458] This invention combines an emotion engine with a comprehensive system that supports the independence and safety of the elderly. It analyzes user emotion data in addition to environmental and health data, enabling more comprehensive and rapid anomaly detection and response.

[1459] System configuration and operation overview

[1460] The role of sensor devices

[1461] Terminal

[1462] The sensor devices include sensors that monitor the elderly's living environment and health status, and an emotion engine that recognizes the user's emotions. Specifically, the devices include temperature sensors, humidity sensors, voice detection sensors, motion detection sensors, heart rate sensors, blood pressure monitors, and emotion recognition devices using cameras and microphones.

[1463] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to recognize the user's emotions, making it possible to determine in real time whether the user is sad, angry, excited, etc.

[1464] Server Roles

[1465] server

[1466] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. It is then stored in a database and analyzed by the AI ​​model or emotion engine.

[1467] The server integrates and analyzes health and emotional data, and if an abnormality is detected, it determines the appropriate response depending on the type and severity of the abnormality. If an abnormality is detected, an emergency notification is generated and sent to emergency contacts.

[1468] Emergency notification and response actions

[1469] User

[1470] The user (emergency contact) receives an emergency notification from the server. The emergency notification includes the abnormality detection result, emotional state, and required response action. The emergency contact receives these notifications via SMS, email, push notification, etc., and responds promptly.

[1471] Specific examples

[1472] For example, consider the case where a user falls at home and expresses emotions of fear and confusion. Here is how the system works:

[1473] Terminal

[1474] The fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[1475] server

[1476] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[1477] User

[1478] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the elderly person's safety and, if necessary, call emergency services.

[1479] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of an emotion engine enables more comprehensive anomaly detection that also takes the user's emotional state into account, further improving the quality of life for the elderly.

[1480] The processing flow will be explained below.

[1481] Step 1:

[1482] Terminal

[1483] The sensor devices measure the user's environmental and health data in real time. For example, the temperature sensor measures the room temperature, and the heart rate sensor measures the user's heart rate. In addition, the emotion engine uses the camera and microphone to capture the user's facial expressions and voice and analyze their emotional state.

[1484] Step 2:

[1485] Terminal

[1486] The measurement data is collected and sent as a data packet to the server via Wi-Fi or Bluetooth, and sent via an HTTP POST request. Emotion data is also collected and sent to the server.

[1487] Step 3:

[1488] server

[1489] The server receives data sent from the sensor devices and emotion engine. The received data is first validated to ensure that the data structure and format are correct.

[1490] Step 4:

[1491] server

[1492] Data that passes validation is stored in a time series database, which ensures that past and present data are properly managed.

[1493] Step 5:

[1494] server

[1495] After the data is saved, it is analyzed by an AI model and an emotion engine. The AI ​​model uses health and environmental data to detect abnormalities, and the emotion engine analyzes the user's emotional data to detect abnormal emotional states.

[1496] Step 6:

[1497] server

[1498] If an anomaly is detected, the server immediately generates an emergency notification, which includes the type of anomaly, the time of occurrence, location information, and emotional state, for example, "Your heart rate has increased sharply, and you have detected an emotion of fear."

[1499] Step 7:

[1500] server

[1501] The generated emergency notification is sent to pre-registered emergency contacts (family members or care services) via SMS, email, push notification, etc.

[1502] Step 8:

[1503] User

[1504] The emergency contact receives the emergency notification, and upon receiving the notification, the emergency contact promptly checks the user's safety and takes any necessary action.

[1505] Step 9:

[1506] User

[1507] For example, when a family member receives a notification, they can contact the user's location and make a personal visit or arrange for emergency services to ensure the elderly person is safe. It also takes into account the user's emotional state and provides any necessary psychological support.

[1508] In this way, the system detects abnormalities through collaboration between the sensor device, server, and user (emergency contact) sections, and promotes prompt and appropriate responses.The introduction of an emotion engine also analyzes the user's emotional state, achieving more comprehensive safety management.

[1509] Example 2

[1510] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1511] In systems designed to support the independence and safety of elderly people, conventional technologies primarily monitor environmental and health data, without taking into account the user's emotional state when detecting anomalies. Therefore, comprehensive anomaly detection and rapid response, including not only falls and deterioration in health status but also emotional anomalies, are required. Furthermore, real-time data acquisition and analysis are incomplete, sometimes leading to delays in emergency response.

[1512] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1513] In this invention, the server includes a measuring means for monitoring the living environment and health status, a means for transmitting data obtained by the measuring means to the server, a means for the server to receive the data and validate its contents, a means for storing the validated data in a database, a means for analyzing the stored data and using an AI model to detect anomalies, a means for generating an emergency notification when an anomaly is detected and sending it to an emergency contact, and a means for the emergency contact to take action upon receiving the emergency notification. This enables real-time data acquisition and analysis, as well as comprehensive anomaly detection and rapid response that takes into account emotional states.

[1514] "Measurement means" refers to a device that acquires data using a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone, etc., to monitor living environment and health condition.

[1515] The "server" is a central management system that receives and validates data sent from the measurement means, stores it in a database, and analyzes the data using an AI model.

[1516] "Validation" is the process by which the server verifies that the data it receives is correct and valid.

[1517] A "database" is a system for systematically storing and managing validated data.

[1518] An "AI model" is a machine learning algorithm or artificial intelligence system used to analyze stored data and detect anomalies.

[1519] An "emergency notification" is a message or alert that the server generates and sends to emergency contacts when an abnormality is detected.

[1520] An "emergency contact" is a person or organization designated to receive emergency notifications when an anomaly is detected.

[1521] "Response actions" are actions that an emergency contact must take upon receiving an emergency notification.

[1522] The "living environment" refers to the conditions such as temperature, humidity, sound, and movement in the place where the user lives their daily life.

[1523] "Health status" refers to physiological data such as heart rate and blood pressure that indicate the user's physical health.

[1524] "Emotional state" refers to the psychological state of the user that can be analyzed from facial expressions and tone of voice.

[1525] This invention is a comprehensive system for supporting the independence and safety of elderly people. By incorporating an emotion engine, the system analyzes the user's emotional state and integrates it with environmental and health data to achieve comprehensive anomaly detection and rapid response.

[1526] System configuration and operation overview

[1527] The role of sensor devices

[1528] Terminal

[1529] The device is equipped with a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera, and microphone. These devices monitor the user's living environment and health. For example, the heart rate sensor captures the user's heart rate every second, and the camera and microphone capture the user's facial expressions and tone of voice every second.

[1530] The emotion engine analyzes facial image data and voice data acquired from the camera and microphone, and uses AI algorithms to determine the user's emotional state in real time, identifying whether the user is sad, angry, excited, etc.

[1531] Server Roles

[1532] server

[1533] The server receives and analyzes data sent from the sensor devices and emotion engine. The received data is first validated to ensure it is correct. The data is then saved in a database. Database management systems such as MySQL and PostgreSQL are used.

[1534] The server analyzes the stored data using an AI model. For example, facial expression data and tone of voice data are input into an emotion analysis algorithm to extract the user's emotional state. Then, health data and emotional data are integrated and considered to detect abnormalities. If an abnormality is detected, an emergency notification is generated according to the type and severity of the abnormality. The notification includes the type of abnormality, the user's emotional state, and any necessary response actions.

[1535] Emergency notification and response actions

[1536] User

[1537] The user (emergency contact) receives an emergency notification from the server. Notifications are sent via SMS, email, push notification, and other methods, allowing the emergency contact to respond quickly. For example, a message may be sent stating, "A fall has been detected in the living room and fear has been recognized. Please check immediately." Based on this notification, the emergency contact can quickly confirm the user's safety and arrange for emergency services if necessary.

[1538] Specific examples

[1539] As a concrete example, consider the case where a user falls at home and expresses feelings of fear and confusion.

[1540] Terminal

[1541] The device's fall sensor detects the user's sudden fall and sends the information to the server, while the emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions such as fear and confusion.

[1542] server

[1543] The server integrates and analyzes the fall data received from the sensor device and the emotion data received from the emotion engine. The AI ​​model detects the simultaneous occurrence of a fall and the emotion of fear as an anomaly and immediately generates an emergency notification.

[1544] User

[1545] The emergency contact (family member) receives a notification on their smartphone app saying, "A fall has been detected in the living room and a fearful emotion has been recognized. Please check immediately." Based on this notification, the family member can quickly check on the user's safety and arrange for emergency services if necessary.

[1546] In this way, the system comprehensively monitors the living environment, health, and emotional state of the elderly, and can respond quickly when an abnormality occurs. The introduction of the emotion engine enables more comprehensive anomaly detection that takes emotional state into account, further improving the quality of life for the elderly.

[1547] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1548] Step 1: Acquiring Sensor Data

[1549] Terminal

[1550] The device collects data from a temperature sensor, humidity sensor, sound detection sensor, motion detection sensor, heart rate sensor, blood pressure monitor, camera and microphone. Specifically, the heart rate sensor measures the heart rate every second, the camera captures the user's face image every minute, and the sound detection sensor constantly monitors the user's voice.

[1551] Input: Environmental data (e.g., temperature, humidity), health data (e.g., heart rate, blood pressure), emotional data (e.g., facial expression, voice)

[1552] Output: Acquired raw data (e.g., heart rate data, facial image data)

[1553] Step 2: Sending data

[1554] Terminal

[1555] The data acquired by the device is sent to a server via Wi-Fi or Bluetooth. For example, heart rate data is sent to the server in batches every minute, and emergency data such as falls is sent in real time.

[1556] Input: Acquired raw data (e.g., heart rate data, facial image data)

[1557] Output: Data sent to the server

[1558] Step 3: Validate the data

[1559] server

[1560] The server validates the data received from the device, for example, checking that the heart rate is not too extreme (0 or over 200) and filtering out invalid data.

[1561] Input: Transmitted data (e.g. heart rate data)

[1562] Output: Normal data (e.g. filtered heart rate data)

[1563] Step 4: Save your data

[1564] server

[1565] The server saves the validated data in a database. Specifically, it uses a database management system (e.g., MySQL, PostgreSQL) and saves each data with a timestamp.

[1566] Input: Normal data (e.g. filtered heart rate data)

[1567] Output: Data stored in the database

[1568] Step 5: Analyze the sentiment data

[1569] server

[1570] The server analyzes the stored facial image data and voice data using an AI model to determine the user's emotional state, specifically using facial expression analysis algorithms and voice tone analysis algorithms.

[1571] Input: Facial image data, audio data

[1572] Output: Parsed emotion data (e.g., fear, joy)

[1573] Step 6: Detect anomalies

[1574] server

[1575] The server integrates and analyzes health data and emotional data to detect abnormalities. For example, if data on a fall and emotional fear occur simultaneously, it will be detected as an abnormality.

[1576] Input: Health data (e.g., fall data), emotion data (e.g., fear)

[1577] Output: Anomaly detection result (e.g., fall + fear = abnormal)

[1578] Step 7: Generate and send emergency notifications

[1579] server

[1580] If the server detects an anomaly, it generates an emergency notification and sends it to emergency contacts. The notification includes the type of anomaly, the emotional state, and a recommended response action. For example, it can generate a message like, "A fall has been detected in the living room and the emotion of fear has been recognized. Please take immediate action."

[1581] Input: Anomaly detection result (e.g., fall + fear = abnormal)

[1582] Output: Emergency notification message

[1583] Step 8: Receive emergency notifications and take action

[1584] User

[1585] The user (emergency contact) receives the emergency notification from the server, checks the content, and then promptly takes action. Specifically, the notification is received via a smartphone app or SMS, and the user's safety is promptly confirmed and emergency services are called if necessary.

[1586] Input: Emergency notification message

[1587] Output: Response actions (e.g., phone call to check on safety or arrange for emergency services)

[1588] (Application example 2)

[1589] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1590] When elderly people and other passengers use self-driving vehicles, they need to be able to comprehensively monitor their health and emotional states in real time and respond quickly if an abnormality occurs. However, current systems only monitor health data and do not consider emotional state information, which can result in incomplete anomaly detection. The present invention aims to solve these problems and improve the safety and security of self-driving vehicles by comprehensively monitoring the user's health and emotional state.

[1591] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1592] In this invention, the server includes means for measuring environmental data and user health data using a sensor device, means for receiving the measurement data from the sensor device and analyzing it to detect abnormalities, means for analyzing emotional data obtained from the sensor device and recognizing the user's emotional state, means for creating an emergency notification and sending it to an emergency contact when an abnormality is detected, and means for the emergency contact who receives the emergency notification to take response action. This allows for comprehensive monitoring of the user's health and emotional state, enabling a prompt and appropriate response when an abnormality is detected.

[1593] A "sensor device" is a device that measures environmental data and user health data.

[1594] A "server" is a computer system that analyzes measurement data received from sensor devices and detects abnormalities.

[1595] An "emergency notification" is a warning message that the server creates and sends to emergency contacts when an abnormality is detected.

[1596] An "emergency contact" is a contact who is responsible for taking action in response to a user's abnormality.

[1597] An "AI model" is an algorithm that uses machine learning to analyze data and detect anomalies.

[1598] "Real-time" means that data is processed immediately at the moment it is generated.

[1599] "Emotion data" is information about the emotional state of the user analyzed from their facial expressions and voice.

[1600] A "library for voice analysis" is a software component for analyzing voice data and recognizing its content.

[1601] An "on-board device" is a device that is installed in a vehicle and integrates and analyzes data acquired from sensor devices.

[1602] A "portable device" is a communication terminal that a user can carry around with them.

[1603] This invention is a comprehensive system that comprehensively monitors the health and emotional state of a user and responds quickly if an abnormality occurs. The system of the present invention is installed in an autonomous vehicle and includes the following components:

[1604] System Configuration

[1605] 1. Sensor device:

[1606] These devices include cameras, microphones, heart rate sensors, and pressure sensors installed in the vehicle. These devices measure the user's environmental and health data in real time. Emotion data is obtained by analyzing facial expressions and tone of voice using the cameras and microphones.

[1607] 2. Server:

[1608] The system receives and analyzes measurement and emotion data sent from the sensor devices. Specific software used includes OpenCV, Dlib, and DeepFace for face tracking and facial expression recognition, Google Cloud Speech-to-Text API for voice analysis, and TensorFlow for health data analysis. The server integrates this data and generates emergency notifications if an abnormality is detected.

[1609] 3. Onboard equipment:

[1610] This device sends emergency notifications to a display in the vehicle or to the user's mobile device. When an emergency notification is sent, the information is also sent to registered emergency contacts.

[1611] Program processing and data calculation

[1612] Data Acquisition:

[1613] The sensor device collects the user's health data (heart rate, pressure, etc.) and emotional data (facial expressions, voice) in real time and sends this to the server.

[1614] Data Analysis:

[1615] The server analyzes this data using AI models such as TensorFlow and DeepFace. First, it validates the data to ensure it is normal. Then, the AI ​​model detects any anomalies.

[1616] Emergency notification:

[1617] If an abnormality is detected, the system generates an emergency notification and sends it to the vehicle's display and the user's mobile device, as well as to registered emergency contacts via social media and email.

[1618] Specific examples

[1619] For example, consider a case where an elderly person's heart rate suddenly rises while using an autonomous vehicle, and the camera analysis reveals a fearful expression. In this case, the system operates as follows:

[1620] The camera and heart rate sensor collect data in real time and send it to a server.

[1621] The server uses the acquired data to analyze the user's emotions and health status and detect any abnormalities.

[1622] If an abnormality is detected, an emergency notification is generated and displayed on the vehicle's display and on the user's mobile device, stating "An abnormality has been detected, please stop the vehicle at a safe location."

[1623] In addition, an emergency notification will be sent to pre-registered emergency contacts stating, "An abnormality has been detected. Please check immediately."

[1624] Prompt Sentence Examples

[1625] "If an elderly person sitting in the passenger seat of a self-driving vehicle suddenly experiences a rise in heart rate and facial expressions indicate abnormalities, send that data and generate an immediate emergency notification."

[1626] Prompt sentence for prompt generation AI model:

[1627] "Input data: Heart rate data: 120 bpm, Facial expression data: scared, Audio data: speaking with trembling. An abnormality has been detected, so please generate an emergency notification."

[1628] This system will ensure the safety and security of elderly and other passengers, and will also enable a prompt and appropriate response in the event of an abnormality.

[1629] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1630] Step 1:

[1631] Data Acquisition

[1632] The terminal (sensor device) uses devices installed inside the vehicle, such as a camera, microphone, heart rate sensor, and pressure sensor, to measure the user's environmental data and health data in real time. The camera and microphone are used to capture the user's facial expression and voice data, which are then recognized as emotional data. The heart rate sensor and pressure sensor are also used to capture the user's heart rate and seat pressure data. This data is sent from the terminal to a server.

[1633] Input: Real-time environmental data, heart rate data, pressure data, facial expression data, voice data

[1634] Output: Sending the retrieved data

[1635] Step 2:

[1636] Data Receipt and Validation

[1637] The server receives the environmental data, health data, and emotion data sent from the device. The server first validates the received data to ensure that it is correct, thereby ensuring the reliability of the data.

[1638] Input: Data sent from the terminal

[1639] Output: Validated data

[1640] Step 3:

[1641] Data analysis

[1642] The server then uses the validated data to perform analysis. Specifically, it proceeds as follows:

[1643] OpenCV, Dlib, and DeepFace are used for face tracking and facial expression recognition, and the user's emotions are analyzed based on image data obtained from the camera.

[1644] For voice analysis, the Google Cloud Speech-to-Text API is used to convert voice data obtained from the microphone into text and analyze emotions.

[1645] TensorFlow is used for health data analysis, analyzing heart rate and pressure data.

[1646] This data is analyzed comprehensively to determine whether any abnormalities are detected.

[1647] Input: Validated environmental data, health data, and emotion data

[1648] Output: Analysis results (presence or absence of abnormalities)

[1649] Step 4:

[1650] Anomaly detection

[1651] If the server detects an anomaly based on the analysis results, it generates an emergency notification according to the type and severity of the anomaly, using an anomaly detection algorithm based on an analytical model (TensorFlow or DeepFace).

[1652] Input: Analysis results

[1653] Output: Urgent Notification

[1654] Step 5:

[1655] Sending emergency notifications

[1656] If an abnormality is detected, the server generates an emergency notification and sends it to the vehicle's display and the user's mobile device.The server also sends the emergency notification to pre-registered emergency contacts via email and social media.

[1657] Input: Emergency Notification

[1658] Output: Notification to emergency contacts and mobile devices

[1659] Step 6:

[1660] Response actions

[1661] The user and emergency contacts can take immediate action based on the emergency notification they receive, such as checking the user's health and providing first aid or contacting emergency services if necessary, or even stopping the self-driving vehicle at a safe location based on the notification displayed on the vehicle's display.

[1662] Input: Emergency Notification

[1663] Output: Execute response action

[1664] In this way, the system monitors the user's health and emotional state in real time, enabling rapid and appropriate response when an abnormality is detected.

[1665] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1666] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1667] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1668] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1669] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1670] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1671] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1672] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1673] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1674] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1675] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1676] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1677] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1678] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1679] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1680] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1681] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1682] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1683] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1684] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1685] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1686] The following is further disclosed regarding the above embodiment.

[1687] (Claim 1)

[1688] A means for measuring environmental data and user health data by a sensor device;

[1689] a server means for receiving measurement data from the sensor device and analyzing the data to detect anomalies;

[1690] a means of generating and sending emergency notifications to emergency contacts when an anomaly is detected;

[1691] a means for an emergency contact to take action upon receiving the emergency notification;

[1692] A system including:

[1693] (Claim 2)

[1694] 10. The system of claim 1, further comprising means for using an AI model to detect anomalies.

[1695] (Claim 3)

[1696] 10. The system of claim 1, further comprising means for receiving data from the sensor device in real time.

[1697] "Example 1"

[1698] (Claim 1)

[1699] A means for measuring environmental data and user health data by a sensor device;

[1700] A means for receiving and validating measurement data from the sensor device;

[1701] a means for storing the measurement data in a database;

[1702] a means for using a generative AI model to analyze the measurement data;

[1703] a means of generating and sending emergency notifications to emergency contacts when an anomaly is detected;

[1704] a means for an emergency contact to take action upon receiving the emergency notification;

[1705] A system including:

[1706] (Claim 2)

[1707] 10. The system of claim 1, further comprising means for receiving data from the sensor device in real time.

[1708] (Claim 3)

[1709] 10. The system of claim 1, further comprising means for generating and sending an emergency notification to an emergency contact when an anomaly is detected, such as via SMS, email, or push notification.

[1710] "Application Example 1"

[1711] (Claim 1)

[1712] A means for measuring environmental data and user health data by a sensor device;

[1713] a server means for receiving measurement data from the sensor device and analyzing the data to detect anomalies;

[1714] a means of generating and sending emergency notifications to emergency contacts when an anomaly is detected;

[1715] a means for an emergency contact to take action upon receiving the emergency notification;

[1716] A means of monitoring the safety of elderly and health-conscious customers within the store environment; and

[1717] A means for notifying store staff when an abnormality in a customer is detected;

[1718] A system including:

[1719] (Claim 2)

[1720] 10. The system of claim 1, further comprising means for using an AI model to detect anomalies.

[1721] (Claim 3)

[1722] 10. The system of claim 1, further comprising means for receiving data from the sensor device in real time.

[1723] "Example 2: Combining Emotion Engines"

[1724] (Claim 1)

[1725] Instruments for monitoring living conditions and health conditions;

[1726] means for transmitting data obtained by the measuring means to a server;

[1727] A means by which the server receives the data and validates the content;

[1728] A means of saving the validated data to a database; and

[1729] A means of analyzing the stored data and using AI models to detect anomalies; and

[1730] means for generating and sending an emergency notification to an emergency contact when an anomaly is detected;

[1731] a means for an emergency contact to take action upon receiving the emergency notification;

[1732] A system including:

[1733] (Claim 2)

[1734] 2. The system according to claim 1, further comprising an analysis means for analyzing the emotional state in real time and performing anomaly detection on the integrated data.

[1735] (Claim 3)

[1736] 10. The system of claim 1, further comprising means for receiving data from the measurement means in real time.

[1737] "Application example 2 when combining emotion engines"

[1738] (Claim 1)

[1739] A means for measuring environmental data and user health data by a sensor device;

[1740] a server means for receiving measurement data from the sensor device and analyzing the data to detect anomalies;

[1741] means for analyzing emotion data acquired from the sensor device and recognizing the user's emotional state;

[1742] a means of generating and sending emergency notifications to emergency contacts when an anomaly is detected;

[1743] a means for an emergency contact to take action upon receiving the emergency notification;

[1744] A system including:

[1745] (Claim 2)

[1746] 10. The system of claim 1, further comprising means for using an AI model to detect anomalies.

[1747] (Claim 3)

[1748] 10. The system of claim 1, further comprising means for receiving data from the sensor device in real time.

[1749] (Claim 4)

[1750] 10. The system of claim 1, further comprising means for using a library for audio analysis to analyze the emotional state of the user.

[1751] (Claim 5)

[1752] 10. The system of claim 1, further comprising means for integrating and analyzing data acquired from the sensor devices using an on-board device.

[1753] (Claim 6)

[1754] 10. The system of claim 1, further comprising means for transmitting an emergency notification to a display in the vehicle and to a user's portable device when an abnormality is detected. [Explanation of symbols]

[1755] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for measuring environmental data and user health data by a sensor device; a server means for receiving measurement data from the sensor device and analyzing the data to detect anomalies; a means of generating and sending emergency notifications to emergency contacts when an anomaly is detected; a means for an emergency contact to take action upon receiving the emergency notification; A system including:

2. 10. The system of claim 1, further comprising means for using an AI model to detect anomalies.

3. 10. The system of claim 1, further comprising means for receiving data in real time from the sensor device.

Images