system

Abstract

We provide the system. [Solution] Acoustic signal processing means for detecting abnormal sounds, A temperature sensing means for measuring the temperature distribution of the floor surface, A pressure sensing means for measuring pressure changes on the floor surface, An artificial intelligence model means for analyzing data collected from the aforementioned acoustic signal processing means, temperature detection means, and pressure detection means to determine anomalies, A communication method for sending notifications to external terminals when an anomaly is detected, A means for identifying lifestyle patterns to learn activity patterns within the home, detect changes therein, and notify, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When an elderly person lives alone and an emergency such as a fall or poor physical condition occurs, the means to seek help from the surroundings are limited, and it is difficult to respond quickly. In such a situation, there is a need for a mechanism that allows the elderly person to live safely and enables the family members who are in a remote location to respond quickly.

Means for Solving the Problems

[0005] The present invention solves the problem by providing a system that includes acoustic signal processing means for detecting abnormal sounds, temperature detection means for measuring the temperature of the floor surface, pressure detection means for measuring changes in pressure on the floor surface, artificial intelligence modeling means for analyzing this data, and communication means for transmitting notifications to an external terminal when an abnormality is detected. This system can quickly detect falls and abnormalities in elderly people and send real-time alerts to family members in remote locations, thereby ensuring the safety of the elderly and encouraging a quick response.

[0006] "Acoustic signal processing means" refers to a device or system that has the function of detecting sounds in the environment and analyzing their volume and frequency.

[0007] A "temperature detection means" is a sensor or system that measures the temperature of the floor surface or surrounding area and acquires that data.

[0008] A "pressure detection means" is a sensor or system that has the ability to detect changes in pressure on the floor surface and quantify them.

[0009] An "artificial intelligence model means" is a PC or system that executes algorithms or programs to analyze collected data and determine whether or not there are anomalies.

[0010] "Communication means" refers to a network device or interface that has the function of transmitting data or notifications to an external terminal or system.

[0011] "External terminal" refers to a device such as a PC or mobile phone used by the user to receive abnormal notifications. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention relates to an anomaly detection system installed in the homes of elderly people living alone. This system is composed of acoustic signal processing means, temperature detection means, pressure detection means, artificial intelligence modeling means, and communication means.

[0034] Terminal embodiment

[0035] The device is equipped with multiple sensors to continuously monitor ambient noise, floor temperature, and pressure within the home. Specifically, acoustic sensors collect sounds within the home and process their intensity and frequency as digital signals. Temperature sensors accurately measure floor temperature and collect this data. Pressure sensors have the ability to detect changes in pressure on the floor. Data from these sensors is aggregated in real time into a data processing unit within the device. The device immediately analyzes this data and, if it identifies any anomalies exceeding specified values, sends the information to a server.

[0036] Server Embodiment

[0037] The server constantly receives data transmitted from the terminal and uses an AI model to analyze for anomalies. Specifically, the server compares acoustic, temperature, and pressure data to detect abnormal deviations from past patterns. If an anomaly is detected, the server prepares a notification as the next step and sends a push notification to the user's external terminal via communication means.

[0038] User Embodiment

[0039] Users receive real-time anomaly notifications on their smartphones or PCs. These notifications include details such as the location and time of the anomaly, and the type of anomaly detected (e.g., unusual sounds, elevated body temperature). This allows users to quickly verify and respond to the issue.

[0040] Specific example

[0041] Consider a scenario where an elderly person falls in their room one night. At this time, a loud noise is detected by an acoustic sensor, the temperature of the part of the body that came into contact with the floor rises sharply, and a pressure sensor also records a significant change. The terminal immediately transmits this data to a server. The server uses an AI model to analyze this data and determines that it is an anomaly indicating a fall. As a result, a notification is sent to the user's smartphone stating, "Fall detected. Please check immediately." Upon receiving this notification, the user can quickly check for any abnormalities in their home. In this way, the present invention enables elderly people to live in a safer environment.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The device continuously collects sound, temperature, and pressure data from within the home using its respective sensors. Acoustic sensors convert sound into digital signals, temperature sensors measure floor surface temperature, and pressure sensors record the pressure on the floor. All of this data is aggregated in the device's data processing unit.

[0045] Step 2:

[0046] The terminal immediately analyzes the collected data and compares it to a pre-set normal range. If changes in volume, temperature, or pressure exceed a set threshold, the data is flagged as abnormal and prepared to be sent to the server as abnormal detection data.

[0047] Step 3:

[0048] The terminal sends data with an abnormality flag set to the server. At this time, it generates packet data containing detailed information such as the type, time, and location of the abnormality, and sends this data to the server via the communication means.

[0049] Step 4:

[0050] The server passes the abnormal data received from the terminal to an AI model for processing and analysis. The AI ​​model compares the data with previously accumulated data to determine whether the anomaly is temporary or indicates a serious problem.

[0051] Step 5:

[0052] Based on the AI ​​analysis results, if the server determines that an anomaly is serious, it will use communication means to generate a push notification to the user's external device. The notification will include the details of the anomaly and the recommended next action.

[0053] Step 6:

[0054] Users receive notifications from the server via their smartphones or PCs. Upon receiving a notification, users review the content and, if necessary, directly check on the situation in their home or arrange for necessary assistance.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] In modern society, the number of elderly people living alone is increasing, but effective anomaly detection systems to ensure their safety remain insufficient. In particular, the challenge lies in quickly detecting abnormal situations such as falls or sudden changes in health and prompting appropriate responses. In response to this, the present invention aims to provide a system that enables real-time anomaly detection and rapid external notification.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes means for processing acoustic information and detecting anomalies, means for measuring the temperature of the floor surface, and means for measuring changes in pressure on the floor surface. This makes it possible to quickly and accurately detect abnormal situations and immediately notify users in remote locations.

[0060] "Acoustic information" refers to digital data about the intensity and frequency of sounds collected from the environment.

[0061] "Temperature information" refers to measurement data that indicates the temperature state of an object.

[0062] "Pressure information" refers to data showing measurement results regarding the forces applied to floors and other surfaces.

[0063] A "generative AI model" is an artificial intelligence technique that learns patterns from large amounts of data and makes judgments and predictions for specific tasks.

[0064] "Communication methods" refer to technologies used to transmit data and information obtained from sensors and servers to external terminals.

[0065] "Means for collecting data from sensors in real time and performing initial analysis" refers to technologies that have the function of analyzing data acquired from various sensors immediately and quickly detecting signs of anomalies.

[0066] This invention relates to an anomaly detection system for ensuring the safety of the elderly. This system continuously collects acoustic, temperature, and pressure information and analyzes it using a generative AI model to quickly detect abnormal situations and enable notification to external terminals.

[0067] Terminal embodiment

[0068] The terminal is installed in the home and collects acoustic, temperature, and pressure information from the environment in real time through multiple sensors. These environmental sensors include acoustic sensors, temperature sensors, and pressure sensors. For example, the acoustic sensor detects noise levels and unusual sounds in the home and converts this into digital data. The temperature sensor measures changes in floor temperature at regular intervals and provides this as temperature data. The pressure sensor detects changes in pressure on the floor and collects this as pressure data. The terminal aggregates the data from these sensors into a data processing unit and analyzes early signs of anomalies.

[0069] Server Embodiment

[0070] The server receives information transmitted from the terminal and analyzes this data in detail using a generating AI model. Acoustic, temperature, and pressure data are compared to past normal patterns to check for abnormal fluctuations. For example, if an unusual sound occurs outside of normal activity hours, the data is identified as an anomaly. If the server detects an anomaly, a push notification is sent to the external user's device via communication.

[0071] User Embodiment

[0072] Users can receive anomaly notifications on their smartphones or PCs. The notifications include detailed information such as the location, time, and type of anomaly detected, allowing users to quickly check the situation and take necessary action.

[0073] Specific example

[0074] If an elderly person falls at home one night, an acoustic sensor detects a loud noise, a temperature sensor records a rapid rise in floor temperature, and a pressure sensor detects an abnormal pressure change. This data is transmitted from the terminal to a server, where an AI model recognizes it as a "fall." As a result, the user's smartphone receives a notification stating, "Fall detected. Please check immediately," allowing the user to quickly check the scene. In this way, the present invention can contribute to the safety of the elderly.

[0075] An example of a prompt message is: "Please explain in detail how the anomaly detection system for the elderly analyzes sound, temperature, and pressure data in real time to detect anomalies."

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The device collects data in real time through acoustic, temperature, and pressure sensors placed throughout the home. The inputs from these sensors are raw data regarding environmental acoustics, floor temperature, and pressure changes, respectively. The device processes this raw data and organizes it as digital signals. For example, acoustic data is converted into sound intensity and frequency, and temperature data is the measurement itself from the sensor. Pressure data is output as the amount of pressure change per unit time.

[0079] Step 2:

[0080] The terminal integrates digital signals from each sensor and performs initial analysis in the data processing unit. The input data is compared to past normal values ​​to determine if there are any abnormalities. This process analyzes, for example, whether the volume exceeds a threshold, whether there is a sudden temperature rise, or whether there is a pressure change that exceeds the prediction. The output is represented as a flag indicating normal or abnormal.

[0081] Step 3:

[0082] If an anomaly flag is set, the terminal sends the data to the server. At this time, the terminal encrypts the data and transmits it through a secure communication channel. The transmitted data includes acoustic data, temperature data, and pressure data related to the anomaly, and a secure data stream is generated as the communication output.

[0083] Step 4:

[0084] The server receives anomalous data transmitted from the terminal and performs a detailed analysis using a generating AI model. The input to the server is an anomalous data stream from the terminal. The server feeds this data into an analysis algorithm and further examines the anomaly by comparing it with past data. The output is information that describes the anomaly and its occurrence in detail.

[0085] Step 5:

[0086] When the server detects an anomaly, it sends a notification to an external user terminal using a communication method. The server's input is the analysis results from a generative AI model. Based on these results, the server creates a notification message and sends it to the user as a push notification. The output is a notification message that can be received on the user's terminal.

[0087] Step 6:

[0088] Users receive push notifications on their smartphones or PCs to check the details of an anomaly. User input is the push notification data sent from the server. Output is detailed information displayed on the screen, including the location, time, and type of the anomaly. Based on this information, users can quickly take corrective action.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In environments where elderly people live alone, there is a need to quickly detect unexpected physical abnormalities or changes in daily routines and accurately notify caregivers who are located remotely. However, conventional systems have challenges in detecting abnormalities accurately and identifying changes in activity patterns, making it difficult to respond quickly. Therefore, a system capable of more accurate and real-time monitoring and notification is needed.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes artificial intelligence model means for analyzing acoustic signals, temperature data, and pressure data; lifestyle pattern identification means for learning and detecting changes in activity patterns within the home; and communication means for notifying abnormalities and changes in activity patterns. This enables real-time detection of sudden accidents or changes in health conditions within the home, allowing caregivers to respond quickly.

[0094] An "acoustic signal processing device" is a device that collects ambient sounds and analyzes their magnitude and frequency as digital signals.

[0095] A "temperature detection means" is a device that measures the temperature of the floor surface and obtains accurate temperature data.

[0096] A "pressure detection device" is a device that measures changes in the pressure applied to the floor surface and collects that data.

[0097] An "artificial intelligence modeling device" is a device equipped with an algorithm for analyzing acoustic signals, temperature data, and pressure data to determine whether or not there is an anomaly.

[0098] A "communication device" is a device used to send notifications to external terminals when an anomaly is detected.

[0099] A "lifestyle pattern identification device" is a device that learns activity patterns within a household and detects changes in those patterns.

[0100] The server receives acoustic signals, temperature data, and pressure data collected through an anomaly detection system. The system's terminals are installed within the home and continuously monitor this data using multiple sensors. The terminals are equipped with the ability to acquire sounds within the home using acoustic sensors and analyze the characteristics of their digital signals. They also acquire floor temperature using temperature sensors and detect temperature changes in real time. Furthermore, pressure sensors detect changes in pressure on the floor, enabling the system to sense situations such as falls.

[0101] Data from the terminal is aggregated and analyzed on a server based on cloud infrastructure. The server processes this data using an artificial intelligence model based on TENSORFLOW® to determine whether or not anomalies are present. In particular, the lifestyle pattern recognition function learns patterns of daily activities and identifies sudden changes. When an anomaly or activity change is detected, a push notification is sent via communication to the user's external terminal, such as a smartphone. This communication function allows the user to respond immediately to anomalies, even from a remote location.

[0102] Users can receive notifications on their smartphones or computers, allowing them to stay informed about the situation in their homes. For example, if an elderly person falls at home, the system will simultaneously detect a loud noise, a change in floor temperature, and an abnormality in the pressure sensor, sending the user a message saying, "Fall detected. Please check immediately." This allows users to respond quickly, further enhancing the safety of the elderly.

[0103] An example of a prompt for a generated AI model is, "Analyze the following household sensor data to detect changes in the health status of elderly individuals and identify any abnormalities." This prompt prompts the model to begin analyzing the sensor data and assist in detecting anomalies.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The device collects data from within the home through acoustic, temperature, and pressure sensors. Sound data acquired by the acoustic sensor is directly converted into a digital signal, and the loudness and frequency of the sound are measured. The temperature sensor accurately measures the floor temperature, and this data is updated in real time. The pressure sensor detects changes in pressure on the floor and prepares the data to be sent to the server. The input is raw data from each sensor, and the output is processed digital data.

[0107] Step 2:

[0108] The server receives acoustic, temperature, and pressure data transmitted from the terminal. The server receives this data and performs preprocessing to reduce noise and standardize the data. The input is digital data from the terminal, and the output is clean data for analysis.

[0109] Step 3:

[0110] The server analyzes the received data using an artificial intelligence model based on TensorFlow. Acoustic, temperature, and pressure data are input to the AI ​​model, and the probability of detecting anomalies is calculated. Here, prompts based on the generated AI model are used to set the analysis conditions for identifying anomalies. The input is a clean dataset, and the output is the result of the anomaly detection and inferences about changes in activity patterns.

[0111] Step 4:

[0112] When the server detects an anomaly using the AI ​​model, it pushes that information to an external terminal. The notification contains detailed information, including the type of anomaly, its location, and the time it occurred. The push notification is sent via a communication method, with the input being the anomaly detection information and the output being a notification message to the user's external terminal.

[0113] Step 5:

[0114] Users receive notifications on their smartphones or PCs and check the details of the anomaly. Based on the notification details, users can quickly check the situation within their home. The input is a push notification message, and the output is the initiation of user verification action.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention relates to a system for detecting abnormalities and recognizing user emotions in the home of an elderly person living alone. This system is comprised of an acoustic signal processing means, a temperature detection means, a pressure detection means, an artificial intelligence model means for analyzing this data, and an emotion engine for recognizing user emotions.

[0117] Terminal embodiment

[0118] The device is equipped with acoustic, temperature, and pressure sensors to constantly monitor environmental factors within the home. Each sensor collects data in real time, which is processed immediately within the device. If abnormal data deviating from the normal range is detected during this processing, the information is sent to the server. In addition, the device collects emotional data from the user's voice and facial expressions and analyzes it using an emotion engine. This emotion engine identifies the user's current emotional state and helps customize notification content.

[0119] Server Embodiment

[0120] The server comprehensively analyzes the data transmitted from the terminal and performs anomaly detection. When an anomaly is detected, it considers the emotional data from the emotion engine and generates an appropriate notification for the user. For example, if a sense of urgency is recognized, the urgency of the notification is increased to encourage the user to take a quick action.

[0121] User Embodiment

[0122] Users can receive notifications sent from the server through their external devices, such as smartphones. These notifications are customized to the user's current emotional state, allowing for more appropriate responses. For example, when a user is feeling anxious, a message designed to provide reassurance is sent.

[0123] Specific example

[0124] Consider a scenario where an elderly person suddenly loses consciousness in their room at night. In this case, the terminal's sensors detect abnormal noise levels, temperature changes, and pressure drops. This data obtained from the sensors is immediately sent from the terminal to the server. The server's AI model recognizes this as a serious anomaly. Simultaneously, if the emotion engine detects an emotion indicating urgency from the elderly person's facial expressions and voice, the server promptly sends an emergency notification to the family's smartphone. Upon receiving this notification, the family can quickly check on the elderly person's well-being and take necessary action. Thus, this invention provides an environment in which elderly people can live with greater peace of mind.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The device collects real-time data within the home using acoustic, temperature, and pressure sensors. The acoustic sensor monitors ambient sounds, the temperature sensor observes changes in floor temperature, and the pressure sensor detects the weight on the floor. This data is quickly aggregated within the device.

[0128] Step 2:

[0129] The device immediately evaluates the aggregated data by comparing it to a set threshold. If an anomaly is detected, the information is flagged and marked as data to be sent to the server. At this point, the device captures voice and facial expression changes, and collects emotional data in parallel.

[0130] Step 3:

[0131] The device sends emotional data collected along with anomaly data to the server. The transmitted information includes the type of anomaly, the exact time it occurred, and the acquired emotional indicators.

[0132] Step 4:

[0133] When the server receives data from a terminal, it uses an AI model to analyze the severity of the anomaly. At the same time, it evaluates the user's emotional state, as provided by the emotion engine, and adjusts the urgency level accordingly.

[0134] Step 5:

[0135] The server generates an anomaly notification based on the analysis results. A message tailored to the user's emotional state is created. For example, if the user is feeling urgent, the message will include a prompt action.

[0136] Step 6:

[0137] Users receive notifications sent from the server on their smartphones or PCs. These notifications include details of detected anomalies and recommended actions, allowing users to take appropriate action based on this information. Personalized messages based on detected emotions enable users to quickly take the most appropriate response to the situation.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0140] There is a need to provide a system that enables a swift and appropriate response in the event of an emergency in households where elderly people live alone. In particular, a challenge is to determine if an abnormality has occurred while taking into account the emotional state of the elderly person and to notify them of appropriate countermeasures.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes means for processing acoustic information, means for detecting temperature changes, and means for detecting pressure fluctuations. This makes it possible to immediately determine abnormalities in the home environment of an elderly person living alone and, based on the results, transmit appropriate notifications tailored to the emotional state of the elderly person to an external information processing device.

[0143] "Means for processing acoustic information" refers to technologies for detecting sound, analyzing its characteristics, and detecting anomalies.

[0144] A "means for detecting temperature changes" refers to a device that measures the ambient temperature in real time and observes its fluctuations.

[0145] "Means for detecting pressure fluctuations" refers to sensor technology that continuously monitors the pressure acting on an object and captures changes in it.

[0146] An "artificial intelligence tool" is a program that analyzes collected data and recognizes specific patterns or anomalies.

[0147] "Emotional analysis methods" refer to technologies that analyze an individual's voice and facial expression data to determine their emotional state.

[0148] "Communication infrastructure" refers to the infrastructure used to send and receive information with other devices and systems.

[0149] An "external information processing device" is an electronic device capable of processing data outside of a system, such as a mobile phone or computer.

[0150] This system is designed for anomaly detection and emotion recognition in the homes of elderly people. The terminal is equipped with sensors to detect sound, temperature, and pressure. These sensors collect data in real time and process it immediately within the terminal. When the terminal identifies data deemed abnormal, it sends the information to the server. The server analyzes this data using a generative AI model to determine if it is an anomaly. Furthermore, it uses speech recognition and facial expression analysis technologies to analyze the user's emotions and adjust notifications according to their emotional state.

[0151] When generating notifications, the server considers the user's emotional state to determine appropriate content and urgency. For example, if the server determines that the user is in a state of stress, it will deliver a more urgent notification. This allows family members to quickly check on the elderly person's situation. In this way, the entire system works efficiently together to support the safety and well-being of the elderly.

[0152] A concrete example is a situation where an elderly person loses consciousness at night. In this case, the device's sensors acquire data such as abnormal volume, sudden temperature changes, and pressure drops. When this data is recognized as a serious anomaly by the server, the emotion engine detects emotions indicating the elderly person's urgency, and an emergency notification is promptly sent to the family's smartphone.

[0153] An example of a prompt message would be: "When an elderly person loses consciousness at night, what data should the device's sensors detect? Also, what notification should the server generate based on that data?" This prompt message clearly indicates the system's operation and the content of the notification.

[0154] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0155] Step 1:

[0156] The device collects environmental data in real time using acoustic, temperature, and pressure sensors. Specifically, it records sound with a microphone, measures room temperature with a temperature sensor, and measures floor load with a pressure sensor. The input is raw data obtained from the sensors, which is recorded as time-series data. The output is the sequenced data described above, which is supplied to the next processing step.

[0157] Step 2:

[0158] The terminal immediately analyzes the collected environmental data and detects anomalies by comparing it to known standards. Data processing includes calculating the average and peak volume, temperature fluctuation range, and pressure change rate. The input is the time-series data acquired in step 1, and the output is flag data indicating an anomaly. When an anomaly is detected, this flag is sent to the server.

[0159] Step 3:

[0160] The device collects user voice and facial expression data and performs emotion analysis. Specifically, it uses a camera and microphone to record the user's facial expressions and voice, and analyzes them using an emotion engine. It analyzes tone and intonation from the voice, and subtle changes from the facial expressions. The input is raw video and audio data, and the output is data indicating the user's emotional state.

[0161] Step 4:

[0162] The server receives anomaly and emotion data transmitted from the terminal and uses a generative AI model to perform anomaly detection and emotion analysis. Here, a series of data is used to determine the severity of the anomaly and the necessary response. The input is the output data from steps 2 and 3, and the output is a suggestion of appropriate countermeasures according to the type of anomaly and the emotional state.

[0163] Step 5:

[0164] The server generates a notification based on the analysis results and sends it to the user. The notification content will be text or voice message depending on the situation. For example, if an anomaly is detected and the user expresses anxiety, a message such as "The room temperature has dropped rapidly. Are you feeling alright?" will be generated. The input is the analysis results from step 4, and the output is notification information for family members or caregivers.

[0165] (Application Example 2)

[0166] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0167] In modern society, the number of elderly people living alone is increasing, making their safety a major challenge. In particular, a system that can respond immediately to sudden abnormalities or emergencies is required. However, elderly people living alone may have difficulty recognizing abnormalities themselves and notifying others. Furthermore, if the elderly person's emotional state is unstable, there is a risk of delays in appropriate responses, thus necessitating the development of more effective support systems.

[0168] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0169] In this invention, the server includes acoustic signal processing means, temperature detection means, and pressure detection means. This enables rapid detection of environmental abnormalities in the living space of elderly people, analysis of these abnormalities to determine their nature, and recognition of the elderly person's emotional state using an emotion engine. Furthermore, if an abnormality or change in emotion is detected, an automatic notification can be sent to an external device to prompt immediate action.

[0170] "Acoustic signal processing means" refers to means for receiving acoustic data and detecting abnormal sounds from that data.

[0171] A "temperature detection means" is a means for measuring the temperature in the environment and collecting that data.

[0172] A "pressure detection means" is a means for measuring changes in the pressure applied to the environment and collecting that data.

[0173] An "information processing model means" is a means for analyzing collected acoustic, temperature, and pressure data to determine whether or not there is an anomaly.

[0174] An "emotion engine" is a means of recognizing a user's emotional state by analyzing their voice and facial expression data.

[0175] "Information transmission means" refers to means for notifying an external device of the detected abnormal information.

[0176] An "external device" is a device capable of receiving data, such as a portable device or computer, used to receive abnormality notifications.

[0177] The system for realizing this invention is comprised of a combination of acoustic signal processing means, temperature detection means, pressure detection means, information processing model means, emotion engine, and information transmission means. The system uses multiple sensors within the home to constantly monitor environmental data such as acoustics, temperature, and pressure. Each sensor is managed by a microcontroller, such as a Raspberry Pi, which collects data in real time and transmits it to a server for analysis.

[0178] The server receives data using a web framework such as Flask and analyzes it using an information processing model based on Scikit-learn. If an anomaly is detected, the user's emotional state is also evaluated by an emotion engine utilizing Google Cloud's Speech-to-Text and Face Recognition. Based on the results, push notifications are sent to mobile devices via the Twilio API to quickly inform family members and caregivers of the urgency of the elderly person's situation.

[0179] For example, if there is a sound such as an elderly person falling, the acoustic signal processing means detects it, and the server determines it to be an anomaly. At this point, if the emotion engine detects the elderly person's anxious voice, the information transmission means sends a more urgent message to the family member's smartphone. This procedure enables prompt and appropriate support for the elderly.

[0180] An example of a prompt statement is, "Please propose a system for detecting abnormalities or emotional changes in the environment where elderly people live alone."

[0181] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0182] Step 1:

[0183] The device collects environmental data in real time from acoustic, temperature, and pressure sensors. Each sensor receives acoustic signals, temperature data, and pressure data as input, and outputs this data in a digital format. This allows for the recording of changes in the surrounding environment.

[0184] Step 2:

[0185] The device sends the collected data to the Raspberry Pi, where it is temporarily stored. At this stage, the data is still raw and requires analysis. The input is raw data from the device, which is stored on the Raspberry Pi as received data.

[0186] Step 3:

[0187] The server receives data from the Raspberry Pi using Flask. The Flask server prepares to begin analysis using the data as input and outputs data in a format that can be processed. At this stage, data formatting is performed as a preprocessing step.

[0188] Step 4:

[0189] The server uses Scikit-learn to analyze data using an information processing model. The input consists of pre-processed acoustic, temperature, and pressure data, and the model determines whether anomalies exist. The output is the result of the determination regarding the presence or absence of anomalies. If an anomaly is detected, the server proceeds to the next step.

[0190] Step 5:

[0191] The server uses an emotion engine powered by Google Cloud's Speech-to-Text and Face Recognition to analyze user emotions. Input is audio or image data, and after analysis, it provides output that identifies the user's emotional state. This allows for the determination of the user's emotions.

[0192] Step 6:

[0193] If an anomaly or emergency is detected, the server uses the Twilio API to send a notification to an external device. The input consists of the anomaly detection result and the emotional state, and a customized message is generated and output as the notification. This allows remote users to understand the situation immediately.

[0194] Step 7:

[0195] Users receive notifications sent from the server on their mobile devices. The input is the notification message from the server, and the output is the user confirming the information and taking immediate action. This creates a system that allows for the rapid confirmation of the safety of elderly people.

[0196] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0197] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0198] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0199] [Second Embodiment]

[0200] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0201] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0202] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0203] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0204] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0205] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0206] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0207] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0208] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0209] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0210] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0211] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0212] This invention relates to an anomaly detection system installed in the homes of elderly people living alone. This system is composed of acoustic signal processing means, temperature detection means, pressure detection means, artificial intelligence modeling means, and communication means.

[0213] Terminal embodiment

[0214] The device is equipped with multiple sensors to continuously monitor ambient noise, floor temperature, and pressure within the home. Specifically, acoustic sensors collect sounds within the home and process their intensity and frequency as digital signals. Temperature sensors accurately measure floor temperature and collect this data. Pressure sensors have the ability to detect changes in pressure on the floor. Data from these sensors is aggregated in real time into a data processing unit within the device. The device immediately analyzes this data and, if it identifies any anomalies exceeding specified values, sends the information to a server.

[0215] Server Embodiment

[0216] The server constantly receives data transmitted from the terminal and uses an AI model to analyze for anomalies. Specifically, the server compares acoustic, temperature, and pressure data to detect abnormal deviations from past patterns. If an anomaly is detected, the server prepares a notification as the next step and sends a push notification to the user's external terminal via communication means.

[0217] User Embodiment

[0218] Users receive real-time anomaly notifications on their smartphones or PCs. These notifications include details such as the location and time of the anomaly, and the type of anomaly detected (e.g., unusual sounds, elevated body temperature). This allows users to quickly verify and respond to the issue.

[0219] Specific example

[0220] Consider a scenario where an elderly person falls in their room one night. At this time, a loud noise is detected by an acoustic sensor, the temperature of the part of the body that came into contact with the floor rises sharply, and a pressure sensor also records a significant change. The terminal immediately transmits this data to a server. The server uses an AI model to analyze this data and determines that it is an anomaly indicating a fall. As a result, a notification is sent to the user's smartphone stating, "Fall detected. Please check immediately." Upon receiving this notification, the user can quickly check for any abnormalities in their home. In this way, the present invention enables elderly people to live in a safer environment.

[0221] The following describes the processing flow.

[0222] Step 1:

[0223] The device continuously collects sound, temperature, and pressure data from within the home using its respective sensors. Acoustic sensors convert sound into digital signals, temperature sensors measure floor surface temperature, and pressure sensors record the pressure on the floor. All of this data is aggregated in the device's data processing unit.

[0224] Step 2:

[0225] The terminal immediately analyzes the collected data and compares it to a pre-set normal range. If changes in volume, temperature, or pressure exceed a set threshold, the data is flagged as abnormal and prepared to be sent to the server as abnormal detection data.

[0226] Step 3:

[0227] The terminal sends data with an abnormality flag set to the server. At this time, it generates packet data containing detailed information such as the type, time, and location of the abnormality, and sends this data to the server via the communication means.

[0228] Step 4:

[0229] The server passes the abnormal data received from the terminal to an AI model for processing and analysis. The AI ​​model compares the data with previously accumulated data to determine whether the anomaly is temporary or indicates a serious problem.

[0230] Step 5:

[0231] Based on the AI ​​analysis results, if the server determines that an anomaly is serious, it will use communication means to generate a push notification to the user's external device. The notification will include the details of the anomaly and the recommended next action.

[0232] Step 6:

[0233] Users receive notifications from the server via their smartphones or PCs. Upon receiving a notification, users review the content and, if necessary, directly check on the situation in their home or arrange for necessary assistance.

[0234] (Example 1)

[0235] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] In modern society, the number of elderly people living alone is increasing, but effective anomaly detection systems to ensure their safety remain insufficient. In particular, the challenge lies in quickly detecting abnormal situations such as falls or sudden changes in health and prompting appropriate responses. In response to this, the present invention aims to provide a system that enables real-time anomaly detection and rapid external notification.

[0237] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0238] In this invention, the server includes means for processing acoustic information and detecting anomalies, means for measuring the temperature of the floor surface, and means for measuring changes in pressure on the floor surface. This makes it possible to quickly and accurately detect abnormal situations and immediately notify users in remote locations.

[0239] "Acoustic information" refers to digital data about the intensity and frequency of sounds collected from the environment.

[0240] "Temperature information" refers to measurement data that indicates the temperature state of an object.

[0241] "Pressure information" refers to data showing measurement results regarding the forces applied to floors and other surfaces.

[0242] A "generative AI model" is an artificial intelligence technique that learns patterns from large amounts of data and makes judgments and predictions for specific tasks.

[0243] "Communication methods" refer to technologies used to transmit data and information obtained from sensors and servers to external terminals.

[0244] "Means for collecting data from sensors in real time and performing initial analysis" refers to technologies that have the function of analyzing data acquired from various sensors immediately and quickly detecting signs of anomalies.

[0245] This invention relates to an anomaly detection system for ensuring the safety of the elderly. This system continuously collects acoustic, temperature, and pressure information and analyzes it using a generative AI model to quickly detect abnormal situations and enable notification to external terminals.

[0246] Terminal embodiment

[0247] The terminal is installed in the home and collects acoustic, temperature, and pressure information from the environment in real time through multiple sensors. These environmental sensors include acoustic sensors, temperature sensors, and pressure sensors. For example, the acoustic sensor detects noise levels and unusual sounds in the home and converts this into digital data. The temperature sensor measures changes in floor temperature at regular intervals and provides this as temperature data. The pressure sensor detects changes in pressure on the floor and collects this as pressure data. The terminal aggregates the data from these sensors into a data processing unit and analyzes early signs of anomalies.

[0248] Server Embodiment

[0249] The server receives information transmitted from the terminal and analyzes this data in detail using a generating AI model. Acoustic, temperature, and pressure data are compared to past normal patterns to check for abnormal fluctuations. For example, if an unusual sound occurs outside of normal activity hours, the data is identified as an anomaly. If the server detects an anomaly, a push notification is sent to the external user's device via communication.

[0250] User Embodiment

[0251] Users can receive anomaly notifications on their smartphones or PCs. The notifications include detailed information such as the location, time, and type of anomaly detected, allowing users to quickly check the situation and take necessary action.

[0252] Specific example

[0253] If an elderly person falls at home one night, an acoustic sensor detects a loud noise, a temperature sensor records a rapid rise in floor temperature, and a pressure sensor detects an abnormal pressure change. This data is transmitted from the terminal to a server, where an AI model recognizes it as a "fall." As a result, the user's smartphone receives a notification stating, "Fall detected. Please check immediately," allowing the user to quickly check the scene. In this way, the present invention can contribute to the safety of the elderly.

[0254] An example of a prompt message is: "Please explain in detail how the anomaly detection system for the elderly analyzes sound, temperature, and pressure data in real time to detect anomalies."

[0255] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0256] Step 1:

[0257] The device collects data in real time through acoustic, temperature, and pressure sensors placed throughout the home. The inputs from these sensors are raw data regarding environmental acoustics, floor temperature, and pressure changes, respectively. The device processes this raw data and organizes it as digital signals. For example, acoustic data is converted into sound intensity and frequency, and temperature data is the measurement itself from the sensor. Pressure data is output as the amount of pressure change per unit time.

[0258] Step 2:

[0259] The terminal integrates digital signals from each sensor and performs initial analysis in the data processing unit. The input data is compared to past normal values ​​to determine if there are any abnormalities. This process analyzes, for example, whether the volume exceeds a threshold, whether there is a sudden temperature rise, or whether there is a pressure change that exceeds the prediction. The output is represented as a flag indicating normal or abnormal.

[0260] Step 3:

[0261] If an anomaly flag is set, the terminal sends the data to the server. At this time, the terminal encrypts the data and transmits it through a secure communication channel. The transmitted data includes acoustic data, temperature data, and pressure data related to the anomaly, and a secure data stream is generated as the communication output.

[0262] Step 4:

[0263] The server receives anomalous data transmitted from the terminal and performs a detailed analysis using a generating AI model. The input to the server is an anomalous data stream from the terminal. The server feeds this data into an analysis algorithm and further examines the anomaly by comparing it with past data. The output is information that describes the anomaly and its occurrence in detail.

[0264] Step 5:

[0265] When the server detects an anomaly, it sends a notification to an external user terminal using a communication method. The server's input is the analysis results from a generative AI model. Based on these results, the server creates a notification message and sends it to the user as a push notification. The output is a notification message that can be received on the user's terminal.

[0266] Step 6:

[0267] Users receive push notifications on their smartphones or PCs to check the details of an anomaly. User input is the push notification data sent from the server. Output is detailed information displayed on the screen, including the location, time, and type of the anomaly. Based on this information, users can quickly take corrective action.

[0268] (Application Example 1)

[0269] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] In environments where elderly people live alone, there is a need to quickly detect unexpected physical abnormalities or changes in daily routines and accurately notify caregivers who are located remotely. However, conventional systems have challenges in detecting abnormalities accurately and identifying changes in activity patterns, making it difficult to respond quickly. Therefore, a system capable of more accurate and real-time monitoring and notification is needed.

[0271] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0272] In this invention, the server includes artificial intelligence model means for analyzing acoustic signals, temperature data, and pressure data; lifestyle pattern identification means for learning and detecting changes in activity patterns within the home; and communication means for notifying abnormalities and changes in activity patterns. This enables real-time detection of sudden accidents or changes in health conditions within the home, allowing caregivers to respond quickly.

[0273] An "acoustic signal processing device" is a device that collects ambient sounds and analyzes their magnitude and frequency as digital signals.

[0274] A "temperature detection means" is a device that measures the temperature of the floor surface and obtains accurate temperature data.

[0275] A "pressure detection device" is a device that measures changes in the pressure applied to the floor surface and collects that data.

[0276] An "artificial intelligence modeling device" is a device equipped with an algorithm for analyzing acoustic signals, temperature data, and pressure data to determine whether or not there is an anomaly.

[0277] A "communication device" is a device used to send notifications to external terminals when an anomaly is detected.

[0278] A "lifestyle pattern identification device" is a device that learns activity patterns within a household and detects changes in those patterns.

[0279] The server receives acoustic signals, temperature data, and pressure data collected through an anomaly detection system. The system's terminals are installed within the home and continuously monitor this data using multiple sensors. The terminals are equipped with the ability to acquire sounds within the home using acoustic sensors and analyze the characteristics of their digital signals. They also acquire floor temperature using temperature sensors and detect temperature changes in real time. Furthermore, pressure sensors detect changes in pressure on the floor, enabling the system to sense situations such as falls.

[0280] Data from the device is aggregated and analyzed on a server based on cloud infrastructure. The server processes this data using an artificial intelligence model based on TensorFlow to determine whether or not anomalies are present. In particular, the lifestyle pattern recognition function learns patterns of daily activities and identifies sudden changes. When an anomaly or activity change is detected, a push notification is sent via communication to the user's external device, such as a smartphone. This communication function allows the user to respond immediately to anomalies, even from a remote location.

[0281] Users can receive notifications on their smartphones or computers, allowing them to stay informed about the situation in their homes. For example, if an elderly person falls at home, the system will simultaneously detect a loud noise, a change in floor temperature, and an abnormality in the pressure sensor, sending the user a message saying, "Fall detected. Please check immediately." This allows users to respond quickly, further enhancing the safety of the elderly.

[0282] An example of a prompt for a generated AI model is, "Analyze the following household sensor data to detect changes in the health status of elderly individuals and identify any abnormalities." This prompt prompts the model to begin analyzing the sensor data and assist in detecting anomalies.

[0283] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0284] Step 1:

[0285] The terminal collects data within the home through an acoustic sensor, a temperature sensor, and a pressure sensor. The sound data obtained by the acoustic sensor is directly converted into a digital signal, and the loudness and frequency of the sound are measured. The temperature sensor accurately measures the temperature of the floor surface and is updated in real time. The pressure sensor senses the pressure change on the floor surface and prepares to transmit each data to the server. The input is the raw data from each sensor, and the output is the processed digital data.

[0286] Step 2:

[0287] The server receives the acoustic, temperature, and pressure data transmitted from the terminal. The server receives these data and performs preprocessing to reduce noise and standardize the data. The input is the digital data from the terminal, and the output is the clean data for analysis.

[0288] Step 3:

[0289] The server analyzes the received data using an artificial intelligence model based on TensorFlow. The acoustic, temperature, and pressure data are input into the AI model to calculate the probability for detecting outliers. Here, a prompt sentence based on the generated AI model is used to set the analysis conditions for identifying anomalies. The input is the clean data set, and the output is the result of the anomaly determination and the speculation regarding changes in the activity pattern.

[0290] Step 4:

[0291] When an anomaly is determined by the AI model, the server pushes a notification of that information to an external terminal. The notification content is detailed information including the type of anomaly, the location of occurrence, and the time of occurrence. The push notification is sent via a communication means, and the input is the information of the anomaly determination, and the output is the notification message to the user's external terminal.

[0292] Step 5:

[0293] Users receive notifications on their smartphones or PCs and check the details of the anomaly. Based on the notification details, users can quickly check the situation within their home. The input is a push notification message, and the output is the initiation of user verification action.

[0294] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0295] This invention relates to a system for detecting abnormalities and recognizing user emotions in the home of an elderly person living alone. This system is comprised of an acoustic signal processing means, a temperature detection means, a pressure detection means, an artificial intelligence model means for analyzing this data, and an emotion engine for recognizing user emotions.

[0296] Terminal embodiment

[0297] The device is equipped with acoustic, temperature, and pressure sensors to constantly monitor environmental factors within the home. Each sensor collects data in real time, which is processed immediately within the device. If abnormal data deviating from the normal range is detected during this processing, the information is sent to the server. In addition, the device collects emotional data from the user's voice and facial expressions and analyzes it using an emotion engine. This emotion engine identifies the user's current emotional state and helps customize notification content.

[0298] Server Embodiment

[0299] The server comprehensively analyzes the data transmitted from the terminal and performs anomaly detection. When an anomaly is detected, it considers the emotional data from the emotion engine and generates an appropriate notification for the user. For example, if a sense of urgency is recognized, the urgency of the notification is increased to encourage the user to take a quick action.

[0300] User Embodiment

[0301] The user can receive notifications sent from the server through their external terminal, such as a smartphone. The notifications are customized according to the user's current emotional state and can prompt more appropriate responses. For example, when the user is feeling anxious, a message that gives a greater sense of security is sent.

[0302] Specific Example

[0303] Consider the case where an elderly person suddenly loses consciousness in their own room at night. At this time, the sensors of the terminal detect abnormal volume, temperature changes, and a decrease in pressure. These data obtained from the sensors are immediately sent by the terminal to the server. The server's AI model recognizes this as a serious abnormality. At the same time, if the emotion engine detects an emotion indicating urgency from the elderly person's expression and voice, the server quickly sends an emergency notification to the family's smartphone to that effect. The family can promptly confirm the safety of the elderly person upon receiving this notification and take the necessary actions. Thereby, the present invention provides an environment in which the elderly can live more securely.

[0304] The following describes the processing flow.

[0305] Step 1:

[0306] The terminal collects real-time data within the home using an acoustic sensor, a temperature sensor, and a pressure sensor. The acoustic sensor monitors the surrounding sound, the temperature sensor observes the heat changes in the floor, and the pressure sensor detects the weight applied to the floor. These data are quickly aggregated inside the terminal.

[0307] Step 2:

[0308] The device immediately evaluates the aggregated data by comparing it to a set threshold. If an anomaly is detected, the information is flagged and marked as data to be sent to the server. At this point, the device captures voice and facial expression changes, and collects emotional data in parallel.

[0309] Step 3:

[0310] The device sends emotional data collected along with anomaly data to the server. The transmitted information includes the type of anomaly, the exact time it occurred, and the acquired emotional indicators.

[0311] Step 4:

[0312] When the server receives data from a terminal, it uses an AI model to analyze the severity of the anomaly. At the same time, it evaluates the user's emotional state, as provided by the emotion engine, and adjusts the urgency level accordingly.

[0313] Step 5:

[0314] The server generates an anomaly notification based on the analysis results. A message tailored to the user's emotional state is created. For example, if the user is feeling urgent, the message will include a prompt action.

[0315] Step 6:

[0316] Users receive notifications sent from the server on their smartphones or PCs. These notifications include details of detected anomalies and recommended actions, allowing users to take appropriate action based on this information. Personalized messages based on detected emotions enable users to quickly take the most appropriate response to the situation.

[0317] (Example 2)

[0318] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0319] There is a need to provide a system that enables a swift and appropriate response in the event of an emergency in households where elderly people live alone. In particular, a challenge is to determine if an abnormality has occurred while taking into account the emotional state of the elderly person and to notify them of appropriate countermeasures.

[0320] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0321] In this invention, the server includes means for processing acoustic information, means for detecting temperature changes, and means for detecting pressure fluctuations. This makes it possible to immediately determine abnormalities in the home environment of an elderly person living alone and, based on the results, transmit appropriate notifications tailored to the emotional state of the elderly person to an external information processing device.

[0322] "Means for processing acoustic information" refers to technologies for detecting sound, analyzing its characteristics, and detecting anomalies.

[0323] A "means for detecting temperature changes" refers to a device that measures the ambient temperature in real time and observes its fluctuations.

[0324] "Means for detecting pressure fluctuations" refers to sensor technology that continuously monitors the pressure acting on an object and captures changes in it.

[0325] An "artificial intelligence tool" is a program that analyzes collected data and recognizes specific patterns or anomalies.

[0326] "Emotional analysis methods" refer to technologies that analyze an individual's voice and facial expression data to determine their emotional state.

[0327] "Communication infrastructure" refers to the infrastructure used to send and receive information with other devices and systems.

[0328] An "external information processing device" is an electronic device capable of processing data outside of a system, such as a mobile phone or computer.

[0329] This system is designed for anomaly detection and emotion recognition in the homes of elderly people. The terminal is equipped with sensors to detect sound, temperature, and pressure. These sensors collect data in real time and process it immediately within the terminal. When the terminal identifies data deemed abnormal, it sends the information to the server. The server analyzes this data using a generative AI model to determine if it is an anomaly. Furthermore, it uses speech recognition and facial expression analysis technologies to analyze the user's emotions and adjust notifications according to their emotional state.

[0330] When generating notifications, the server considers the user's emotional state to determine appropriate content and urgency. For example, if the server determines that the user is in a state of stress, it will deliver a more urgent notification. This allows family members to quickly check on the elderly person's situation. In this way, the entire system works efficiently together to support the safety and well-being of the elderly.

[0331] A concrete example is a situation where an elderly person loses consciousness at night. In this case, the device's sensors acquire data such as abnormal volume, sudden temperature changes, and pressure drops. When this data is recognized as a serious anomaly by the server, the emotion engine detects emotions indicating the elderly person's urgency, and an emergency notification is promptly sent to the family's smartphone.

[0332] An example of a prompt message would be: "When an elderly person loses consciousness at night, what data should the device's sensors detect? Also, what notification should the server generate based on that data?" This prompt message clearly indicates the system's operation and the content of the notification.

[0333] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0334] Step 1:

[0335] The device collects environmental data in real time using acoustic, temperature, and pressure sensors. Specifically, it records sound with a microphone, measures room temperature with a temperature sensor, and measures floor load with a pressure sensor. The input is raw data obtained from the sensors, which is recorded as time-series data. The output is the sequenced data described above, which is supplied to the next processing step.

[0336] Step 2:

[0337] The terminal immediately analyzes the collected environmental data and detects anomalies by comparing it to known standards. Data processing includes calculating the average and peak volume, temperature fluctuation range, and pressure change rate. The input is the time-series data acquired in step 1, and the output is flag data indicating an anomaly. When an anomaly is detected, this flag is sent to the server.

[0338] Step 3:

[0339] The device collects user voice and facial expression data and performs emotion analysis. Specifically, it uses a camera and microphone to record the user's facial expressions and voice, and analyzes them using an emotion engine. It analyzes tone and intonation from the voice, and subtle changes from the facial expressions. The input is raw video and audio data, and the output is data indicating the user's emotional state.

[0340] Step 4:

[0341] The server receives anomaly and emotion data transmitted from the terminal and uses a generative AI model to perform anomaly detection and emotion analysis. Here, a series of data is used to determine the severity of the anomaly and the necessary response. The input is the output data from steps 2 and 3, and the output is a suggestion of appropriate countermeasures according to the type of anomaly and the emotional state.

[0342] Step 5:

[0343] The server generates a notification based on the analysis results and sends it to the user. The notification content will be text or voice message depending on the situation. For example, if an anomaly is detected and the user expresses anxiety, a message such as "The room temperature has dropped rapidly. Are you feeling alright?" will be generated. The input is the analysis results from step 4, and the output is notification information for family members or caregivers.

[0344] (Application Example 2)

[0345] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0346] In modern society, the number of elderly people living alone is increasing, making their safety a major challenge. In particular, a system that can respond immediately to sudden abnormalities or emergencies is required. However, elderly people living alone may have difficulty recognizing abnormalities themselves and notifying others. Furthermore, if the elderly person's emotional state is unstable, there is a risk of delays in appropriate responses, thus necessitating the development of more effective support systems.

[0347] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0348] In this invention, the server includes acoustic signal processing means, temperature detection means, and pressure detection means. This enables rapid detection of environmental abnormalities in the living space of elderly people, analysis of these abnormalities to determine their nature, and recognition of the elderly person's emotional state using an emotion engine. Furthermore, if an abnormality or change in emotion is detected, an automatic notification can be sent to an external device to prompt immediate action.

[0349] "Acoustic signal processing means" refers to means for receiving acoustic data and detecting abnormal sounds from that data.

[0350] A "temperature detection means" is a means for measuring the temperature in the environment and collecting that data.

[0351] A "pressure detection means" is a means for measuring changes in the pressure applied to the environment and collecting that data.

[0352] An "information processing model means" is a means for analyzing collected acoustic, temperature, and pressure data to determine whether or not there is an anomaly.

[0353] An "emotion engine" is a means of recognizing a user's emotional state by analyzing their voice and facial expression data.

[0354] "Information transmission means" refers to means for notifying an external device of the detected abnormal information.

[0355] An "external device" is a device capable of receiving data, such as a portable device or computer, used to receive abnormality notifications.

[0356] The system for realizing this invention is comprised of a combination of acoustic signal processing means, temperature detection means, pressure detection means, information processing model means, emotion engine, and information transmission means. The system uses multiple sensors within the home to constantly monitor environmental data such as acoustics, temperature, and pressure. Each sensor is managed by a microcontroller, such as a Raspberry Pi, which collects data in real time and transmits it to a server for analysis.

[0357] The server receives data using a web framework such as Flask and analyzes it using an information processing model based on Scikit-learn. If an anomaly is detected, the user's emotional state is also evaluated by an emotion engine utilizing Google Cloud's Speech-to-Text and Face Recognition. Based on the results, push notifications are sent to mobile devices via the Twilio API to quickly inform family members and caregivers of the urgency of the elderly person's situation.

[0358] For example, if there is a sound such as an elderly person falling, the acoustic signal processing means detects it, and the server determines it to be an anomaly. At this point, if the emotion engine detects the elderly person's anxious voice, the information transmission means sends a more urgent message to the family member's smartphone. This procedure enables prompt and appropriate support for the elderly.

[0359] An example of a prompt statement is, "Please propose a system for detecting abnormalities or emotional changes in the environment where elderly people live alone."

[0360] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0361] Step 1:

[0362] The device collects environmental data in real time from acoustic, temperature, and pressure sensors. Each sensor receives acoustic signals, temperature data, and pressure data as input, and outputs this data in a digital format. This allows for the recording of changes in the surrounding environment.

[0363] Step 2:

[0364] The device sends the collected data to the Raspberry Pi, where it is temporarily stored. At this stage, the data is still raw and requires analysis. The input is raw data from the device, which is stored on the Raspberry Pi as received data.

[0365] Step 3:

[0366] The server receives data from the Raspberry Pi using Flask. The Flask server prepares to begin analysis using the data as input and outputs data in a format that can be processed. At this stage, data formatting is performed as a preprocessing step.

[0367] Step 4:

[0368] The server uses Scikit-learn to analyze data using an information processing model. The input consists of pre-processed acoustic, temperature, and pressure data, and the model determines whether anomalies exist. The output is the result of the determination regarding the presence or absence of anomalies. If an anomaly is detected, the server proceeds to the next step.

[0369] Step 5:

[0370] The server uses an emotion engine powered by Google Cloud's Speech-to-Text and Face Recognition to analyze user emotions. Input is audio or image data, and after analysis, it provides output that identifies the user's emotional state. This allows for the determination of the user's emotions.

[0371] Step 6:

[0372] If an anomaly or emergency is detected, the server uses the Twilio API to send a notification to an external device. The input consists of the anomaly detection result and the emotional state, and a customized message is generated and output as the notification. This allows remote users to understand the situation immediately.

[0373] Step 7:

[0374] Users receive notifications sent from the server on their mobile devices. The input is the notification message from the server, and the output is the user confirming the information and taking immediate action. This creates a system that allows for the rapid confirmation of the safety of elderly people.

[0375] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0376] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0377] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0378] [Third Embodiment]

[0379] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0380] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0381] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0382] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0383] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0384] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0385] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0386] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0387] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0388] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0389] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0390] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0391] This invention relates to an anomaly detection system installed in the homes of elderly people living alone. This system is composed of acoustic signal processing means, temperature detection means, pressure detection means, artificial intelligence modeling means, and communication means.

[0392] Terminal embodiment

[0393] The device is equipped with multiple sensors to continuously monitor ambient noise, floor temperature, and pressure within the home. Specifically, acoustic sensors collect sounds within the home and process their intensity and frequency as digital signals. Temperature sensors accurately measure floor temperature and collect this data. Pressure sensors have the ability to detect changes in pressure on the floor. Data from these sensors is aggregated in real time into a data processing unit within the device. The device immediately analyzes this data and, if it identifies any anomalies exceeding specified values, sends the information to a server.

[0394] Server Embodiment

[0395] The server constantly receives data transmitted from the terminal and uses an AI model to analyze for anomalies. Specifically, the server compares acoustic, temperature, and pressure data to detect abnormal deviations from past patterns. If an anomaly is detected, the server prepares a notification as the next step and sends a push notification to the user's external terminal via communication means.

[0396] User Embodiment

[0397] Users receive real-time anomaly notifications on their smartphones or PCs. These notifications include details such as the location and time of the anomaly, and the type of anomaly detected (e.g., unusual sounds, elevated body temperature). This allows users to quickly verify and respond to the issue.

[0398] Specific example

[0399] Consider a scenario where an elderly person falls in their room one night. At this time, a loud noise is detected by an acoustic sensor, the temperature of the part of the body that came into contact with the floor rises sharply, and a pressure sensor also records a significant change. The terminal immediately transmits this data to a server. The server uses an AI model to analyze this data and determines that it is an anomaly indicating a fall. As a result, a notification is sent to the user's smartphone stating, "Fall detected. Please check immediately." Upon receiving this notification, the user can quickly check for any abnormalities in their home. In this way, the present invention enables elderly people to live in a safer environment.

[0400] The following describes the processing flow.

[0401] Step 1:

[0402] The device continuously collects sound, temperature, and pressure data from within the home using its respective sensors. Acoustic sensors convert sound into digital signals, temperature sensors measure floor surface temperature, and pressure sensors record the pressure on the floor. All of this data is aggregated in the device's data processing unit.

[0403] Step 2:

[0404] The terminal immediately analyzes the collected data and compares it to a pre-set normal range. If changes in volume, temperature, or pressure exceed a set threshold, the data is flagged as abnormal and prepared to be sent to the server as abnormal detection data.

[0405] Step 3:

[0406] The terminal sends data with an abnormality flag set to the server. At this time, it generates packet data containing detailed information such as the type, time, and location of the abnormality, and sends this data to the server via the communication means.

[0407] Step 4:

[0408] The server passes the abnormal data received from the terminal to an AI model for processing and analysis. The AI ​​model compares the data with previously accumulated data to determine whether the anomaly is temporary or indicates a serious problem.

[0409] Step 5:

[0410] Based on the AI ​​analysis results, if the server determines that an anomaly is serious, it will use communication means to generate a push notification to the user's external device. The notification will include the details of the anomaly and the recommended next action.

[0411] Step 6:

[0412] Users receive notifications from the server via their smartphones or PCs. Upon receiving a notification, users review the content and, if necessary, directly check on the situation in their home or arrange for necessary assistance.

[0413] (Example 1)

[0414] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0415] In modern society, the number of elderly people living alone is increasing, but effective anomaly detection systems to ensure their safety remain insufficient. In particular, the challenge lies in quickly detecting abnormal situations such as falls or sudden changes in health and prompting appropriate responses. In response to this, the present invention aims to provide a system that enables real-time anomaly detection and rapid external notification.

[0416] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0417] In this invention, the server includes means for processing acoustic information and detecting anomalies, means for measuring the temperature of the floor surface, and means for measuring changes in pressure on the floor surface. This makes it possible to quickly and accurately detect abnormal situations and immediately notify users in remote locations.

[0418] "Acoustic information" refers to digital data about the intensity and frequency of sounds collected from the environment.

[0419] "Temperature information" refers to measurement data that indicates the temperature state of an object.

[0420] "Pressure information" refers to data showing measurement results regarding the forces applied to floors and other surfaces.

[0421] A "generative AI model" is an artificial intelligence technique that learns patterns from large amounts of data and makes judgments and predictions for specific tasks.

[0422] "Communication methods" refer to technologies used to transmit data and information obtained from sensors and servers to external terminals.

[0423] "Means for collecting data from sensors in real time and performing initial analysis" refers to technologies that have the function of analyzing data acquired from various sensors immediately and quickly detecting signs of anomalies.

[0424] This invention relates to an anomaly detection system for ensuring the safety of the elderly. This system continuously collects acoustic, temperature, and pressure information and analyzes it using a generative AI model to quickly detect abnormal situations and enable notification to external terminals.

[0425] Terminal embodiment

[0426] The terminal is installed in the home and collects acoustic, temperature, and pressure information from the environment in real time through multiple sensors. These environmental sensors include acoustic sensors, temperature sensors, and pressure sensors. For example, the acoustic sensor detects noise levels and unusual sounds in the home and converts this into digital data. The temperature sensor measures changes in floor temperature at regular intervals and provides this as temperature data. The pressure sensor detects changes in pressure on the floor and collects this as pressure data. The terminal aggregates the data from these sensors into a data processing unit and analyzes early signs of anomalies.

[0427] Server Embodiment

[0428] The server receives information transmitted from the terminal and analyzes this data in detail using a generating AI model. Acoustic, temperature, and pressure data are compared to past normal patterns to check for abnormal fluctuations. For example, if an unusual sound occurs outside of normal activity hours, the data is identified as an anomaly. If the server detects an anomaly, a push notification is sent to the external user's device via communication.

[0429] User Embodiment

[0430] Users can receive anomaly notifications on their smartphones or PCs. The notifications include detailed information such as the location, time, and type of anomaly detected, allowing users to quickly check the situation and take necessary action.

[0431] Specific example

[0432] If an elderly person falls at home one night, an acoustic sensor detects a loud noise, a temperature sensor records a rapid rise in floor temperature, and a pressure sensor detects an abnormal pressure change. This data is transmitted from the terminal to a server, where an AI model recognizes it as a "fall." As a result, the user's smartphone receives a notification stating, "Fall detected. Please check immediately," allowing the user to quickly check the scene. In this way, the present invention can contribute to the safety of the elderly.

[0433] An example of a prompt message is: "Please explain in detail how the anomaly detection system for the elderly analyzes sound, temperature, and pressure data in real time to detect anomalies."

[0434] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0435] Step 1:

[0436] The device collects data in real time through acoustic, temperature, and pressure sensors placed throughout the home. The inputs from these sensors are raw data regarding environmental acoustics, floor temperature, and pressure changes, respectively. The device processes this raw data and organizes it as digital signals. For example, acoustic data is converted into sound intensity and frequency, and temperature data is the measurement itself from the sensor. Pressure data is output as the amount of pressure change per unit time.

[0437] Step 2:

[0438] The terminal integrates digital signals from each sensor and performs initial analysis in the data processing unit. The input data is compared to past normal values ​​to determine if there are any abnormalities. This process analyzes, for example, whether the volume exceeds a threshold, whether there is a sudden temperature rise, or whether there is a pressure change that exceeds the prediction. The output is represented as a flag indicating normal or abnormal.

[0439] Step 3:

[0440] If an anomaly flag is set, the terminal sends the data to the server. At this time, the terminal encrypts the data and transmits it through a secure communication channel. The transmitted data includes acoustic data, temperature data, and pressure data related to the anomaly, and a secure data stream is generated as the communication output.

[0441] Step 4:

[0442] The server receives anomalous data transmitted from the terminal and performs a detailed analysis using a generating AI model. The input to the server is an anomalous data stream from the terminal. The server feeds this data into an analysis algorithm and further examines the anomaly by comparing it with past data. The output is information that describes the anomaly and its occurrence in detail.

[0443] Step 5:

[0444] When the server detects an anomaly, it sends a notification to an external user terminal using a communication method. The server's input is the analysis results from a generative AI model. Based on these results, the server creates a notification message and sends it to the user as a push notification. The output is a notification message that can be received on the user's terminal.

[0445] Step 6:

[0446] Users receive push notifications on their smartphones or PCs to check the details of an anomaly. User input is the push notification data sent from the server. Output is detailed information displayed on the screen, including the location, time, and type of the anomaly. Based on this information, users can quickly take corrective action.

[0447] (Application Example 1)

[0448] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0449] In environments where elderly people live alone, there is a need to quickly detect unexpected physical abnormalities or changes in daily routines and accurately notify caregivers who are located remotely. However, conventional systems have challenges in detecting abnormalities accurately and identifying changes in activity patterns, making it difficult to respond quickly. Therefore, a system capable of more accurate and real-time monitoring and notification is needed.

[0450] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0451] In this invention, the server includes artificial intelligence model means for analyzing acoustic signals, temperature data, and pressure data; lifestyle pattern identification means for learning and detecting changes in activity patterns within the home; and communication means for notifying abnormalities and changes in activity patterns. This enables real-time detection of sudden accidents or changes in health conditions within the home, allowing caregivers to respond quickly.

[0452] An "acoustic signal processing device" is a device that collects ambient sounds and analyzes their magnitude and frequency as digital signals.

[0453] A "temperature detection means" is a device that measures the temperature of the floor surface and obtains accurate temperature data.

[0454] A "pressure detection device" is a device that measures changes in the pressure applied to the floor surface and collects that data.

[0455] An "artificial intelligence modeling device" is a device equipped with an algorithm for analyzing acoustic signals, temperature data, and pressure data to determine whether or not there is an anomaly.

[0456] A "communication device" is a device used to send notifications to external terminals when an anomaly is detected.

[0457] A "lifestyle pattern identification device" is a device that learns activity patterns within a household and detects changes in those patterns.

[0458] The server receives acoustic signals, temperature data, and pressure data collected through an anomaly detection system. The system's terminals are installed within the home and continuously monitor this data using multiple sensors. The terminals are equipped with the ability to acquire sounds within the home using acoustic sensors and analyze the characteristics of their digital signals. They also acquire floor temperature using temperature sensors and detect temperature changes in real time. Furthermore, pressure sensors detect changes in pressure on the floor, enabling the system to sense situations such as falls.

[0459] Data from the device is aggregated and analyzed on a server based on cloud infrastructure. The server processes this data using an artificial intelligence model based on TensorFlow to determine whether or not anomalies are present. In particular, the lifestyle pattern recognition function learns patterns of daily activities and identifies sudden changes. When an anomaly or activity change is detected, a push notification is sent via communication to the user's external device, such as a smartphone. This communication function allows the user to respond immediately to anomalies, even from a remote location.

[0460] Users can receive notifications on their smartphones or computers, allowing them to stay informed about the situation in their homes. For example, if an elderly person falls at home, the system will simultaneously detect a loud noise, a change in floor temperature, and an abnormality in the pressure sensor, sending the user a message saying, "Fall detected. Please check immediately." This allows users to respond quickly, further enhancing the safety of the elderly.

[0461] An example of a prompt for a generated AI model is, "Analyze the following household sensor data to detect changes in the health status of elderly individuals and identify any abnormalities." This prompt prompts the model to begin analyzing the sensor data and assist in detecting anomalies.

[0462] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0463] Step 1:

[0464] The device collects data from within the home through acoustic, temperature, and pressure sensors. Sound data acquired by the acoustic sensor is directly converted into a digital signal, and the loudness and frequency of the sound are measured. The temperature sensor accurately measures the floor temperature, and this data is updated in real time. The pressure sensor detects changes in pressure on the floor and prepares the data to be sent to the server. The input is raw data from each sensor, and the output is processed digital data.

[0465] Step 2:

[0466] The server receives acoustic, temperature, and pressure data transmitted from the terminal. The server receives this data and performs preprocessing to reduce noise and standardize the data. The input is digital data from the terminal, and the output is clean data for analysis.

[0467] Step 3:

[0468] The server analyzes the received data using an artificial intelligence model based on TensorFlow. Acoustic, temperature, and pressure data are input to the AI ​​model, and the probability of detecting anomalies is calculated. Here, prompts based on the generated AI model are used to set the analysis conditions for identifying anomalies. The input is a clean dataset, and the output is the result of the anomaly detection and inferences about changes in activity patterns.

[0469] Step 4:

[0470] When the server detects an anomaly using the AI ​​model, it pushes that information to an external terminal. The notification contains detailed information, including the type of anomaly, its location, and the time it occurred. The push notification is sent via a communication method, with the input being the anomaly detection information and the output being a notification message to the user's external terminal.

[0471] Step 5:

[0472] Users receive notifications on their smartphones or PCs and check the details of the anomaly. Based on the notification details, users can quickly check the situation within their home. The input is a push notification message, and the output is the initiation of user verification action.

[0473] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0474] This invention relates to a system for detecting abnormalities and recognizing user emotions in the home of an elderly person living alone. This system is comprised of an acoustic signal processing means, a temperature detection means, a pressure detection means, an artificial intelligence model means for analyzing this data, and an emotion engine for recognizing user emotions.

[0475] Terminal embodiment

[0476] The device is equipped with acoustic, temperature, and pressure sensors to constantly monitor environmental factors within the home. Each sensor collects data in real time, which is processed immediately within the device. If abnormal data deviating from the normal range is detected during this processing, the information is sent to the server. In addition, the device collects emotional data from the user's voice and facial expressions and analyzes it using an emotion engine. This emotion engine identifies the user's current emotional state and helps customize notification content.

[0477] Server Embodiment

[0478] The server comprehensively analyzes the data transmitted from the terminal and performs anomaly detection. When an anomaly is detected, it considers the emotional data from the emotion engine and generates an appropriate notification for the user. For example, if a sense of urgency is recognized, the urgency of the notification is increased to encourage the user to take a quick action.

[0479] User Embodiment

[0480] Users can receive notifications sent from the server through their external devices, such as smartphones. These notifications are customized to the user's current emotional state, allowing for more appropriate responses. For example, when a user is feeling anxious, a message designed to provide reassurance is sent.

[0481] Specific example

[0482] Consider a scenario where an elderly person suddenly loses consciousness in their room at night. In this case, the terminal's sensors detect abnormal noise levels, temperature changes, and pressure drops. This data obtained from the sensors is immediately sent from the terminal to the server. The server's AI model recognizes this as a serious anomaly. Simultaneously, if the emotion engine detects an emotion indicating urgency from the elderly person's facial expressions and voice, the server promptly sends an emergency notification to the family's smartphone. Upon receiving this notification, the family can quickly check on the elderly person's well-being and take necessary action. Thus, this invention provides an environment in which elderly people can live with greater peace of mind.

[0483] The following describes the processing flow.

[0484] Step 1:

[0485] The device collects real-time data within the home using acoustic, temperature, and pressure sensors. The acoustic sensor monitors ambient sounds, the temperature sensor observes changes in floor temperature, and the pressure sensor detects the weight on the floor. This data is quickly aggregated within the device.

[0486] Step 2:

[0487] The device immediately evaluates the aggregated data by comparing it to a set threshold. If an anomaly is detected, the information is flagged and marked as data to be sent to the server. At this point, the device captures voice and facial expression changes, and collects emotional data in parallel.

[0488] Step 3:

[0489] The device sends emotional data collected along with anomaly data to the server. The transmitted information includes the type of anomaly, the exact time it occurred, and the acquired emotional indicators.

[0490] Step 4:

[0491] When the server receives data from a terminal, it uses an AI model to analyze the severity of the anomaly. At the same time, it evaluates the user's emotional state, as provided by the emotion engine, and adjusts the urgency level accordingly.

[0492] Step 5:

[0493] The server generates an anomaly notification based on the analysis results. A message tailored to the user's emotional state is created. For example, if the user is feeling urgent, the message will include a prompt action.

[0494] Step 6:

[0495] Users receive notifications sent from the server on their smartphones or PCs. These notifications include details of detected anomalies and recommended actions, allowing users to take appropriate action based on this information. Personalized messages based on detected emotions enable users to quickly take the most appropriate response to the situation.

[0496] (Example 2)

[0497] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0498] There is a need to provide a system that enables a swift and appropriate response in the event of an emergency in households where elderly people live alone. In particular, a challenge is to determine if an abnormality has occurred while taking into account the emotional state of the elderly person and to notify them of appropriate countermeasures.

[0499] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0500] In this invention, the server includes means for processing acoustic information, means for detecting temperature changes, and means for detecting pressure fluctuations. This makes it possible to immediately determine abnormalities in the home environment of an elderly person living alone and, based on the results, transmit appropriate notifications tailored to the emotional state of the elderly person to an external information processing device.

[0501] "Means for processing acoustic information" refers to technologies for detecting sound, analyzing its characteristics, and detecting anomalies.

[0502] A "means for detecting temperature changes" refers to a device that measures the ambient temperature in real time and observes its fluctuations.

[0503] "Means for detecting pressure fluctuations" refers to sensor technology that continuously monitors the pressure acting on an object and captures changes in it.

[0504] An "artificial intelligence tool" is a program that analyzes collected data and recognizes specific patterns or anomalies.

[0505] "Emotional analysis methods" refer to technologies that analyze an individual's voice and facial expression data to determine their emotional state.

[0506] "Communication infrastructure" refers to the infrastructure used to send and receive information with other devices and systems.

[0507] An "external information processing device" is an electronic device capable of processing data outside of a system, such as a mobile phone or computer.

[0508] This system is designed for anomaly detection and emotion recognition in the homes of elderly people. The terminal is equipped with sensors to detect sound, temperature, and pressure. These sensors collect data in real time and process it immediately within the terminal. When the terminal identifies data deemed abnormal, it sends the information to the server. The server analyzes this data using a generative AI model to determine if it is an anomaly. Furthermore, it uses speech recognition and facial expression analysis technologies to analyze the user's emotions and adjust notifications according to their emotional state.

[0509] When generating notifications, the server considers the user's emotional state to determine appropriate content and urgency. For example, if the server determines that the user is in a state of stress, it will deliver a more urgent notification. This allows family members to quickly check on the elderly person's situation. In this way, the entire system works efficiently together to support the safety and well-being of the elderly.

[0510] A concrete example is a situation where an elderly person loses consciousness at night. In this case, the device's sensors acquire data such as abnormal volume, sudden temperature changes, and pressure drops. When this data is recognized as a serious anomaly by the server, the emotion engine detects emotions indicating the elderly person's urgency, and an emergency notification is promptly sent to the family's smartphone.

[0511] An example of a prompt message would be: "When an elderly person loses consciousness at night, what data should the device's sensors detect? Also, what notification should the server generate based on that data?" This prompt message clearly indicates the system's operation and the content of the notification.

[0512] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0513] Step 1:

[0514] The device collects environmental data in real time using acoustic, temperature, and pressure sensors. Specifically, it records sound with a microphone, measures room temperature with a temperature sensor, and measures floor load with a pressure sensor. The input is raw data obtained from the sensors, which is recorded as time-series data. The output is the sequenced data described above, which is supplied to the next processing step.

[0515] Step 2:

[0516] The terminal immediately analyzes the collected environmental data and detects anomalies by comparing it to known standards. Data processing includes calculating the average and peak volume, temperature fluctuation range, and pressure change rate. The input is the time-series data acquired in step 1, and the output is flag data indicating an anomaly. When an anomaly is detected, this flag is sent to the server.

[0517] Step 3:

[0518] The device collects user voice and facial expression data and performs emotion analysis. Specifically, it uses a camera and microphone to record the user's facial expressions and voice, and analyzes them using an emotion engine. It analyzes tone and intonation from the voice, and subtle changes from the facial expressions. The input is raw video and audio data, and the output is data indicating the user's emotional state.

[0519] Step 4:

[0520] The server receives anomaly and emotion data transmitted from the terminal and uses a generative AI model to perform anomaly detection and emotion analysis. Here, a series of data is used to determine the severity of the anomaly and the necessary response. The input is the output data from steps 2 and 3, and the output is a suggestion of appropriate countermeasures according to the type of anomaly and the emotional state.

[0521] Step 5:

[0522] The server generates a notification based on the analysis results and sends it to the user. The notification content will be text or voice message depending on the situation. For example, if an anomaly is detected and the user expresses anxiety, a message such as "The room temperature has dropped rapidly. Are you feeling alright?" will be generated. The input is the analysis results from step 4, and the output is notification information for family members or caregivers.

[0523] (Application Example 2)

[0524] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] In modern society, the number of elderly people living alone is increasing, making their safety a major challenge. In particular, a system that can respond immediately to sudden abnormalities or emergencies is required. However, elderly people living alone may have difficulty recognizing abnormalities themselves and notifying others. Furthermore, if the elderly person's emotional state is unstable, there is a risk of delays in appropriate responses, thus necessitating the development of more effective support systems.

[0526] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0527] In this invention, the server includes acoustic signal processing means, temperature detection means, and pressure detection means. This enables rapid detection of environmental abnormalities in the living space of elderly people, analysis of these abnormalities to determine their nature, and recognition of the elderly person's emotional state using an emotion engine. Furthermore, if an abnormality or change in emotion is detected, an automatic notification can be sent to an external device to prompt immediate action.

[0528] "Acoustic signal processing means" refers to means for receiving acoustic data and detecting abnormal sounds from that data.

[0529] A "temperature detection means" is a means for measuring the temperature in the environment and collecting that data.

[0530] A "pressure detection means" is a means for measuring changes in the pressure applied to the environment and collecting that data.

[0531] An "information processing model means" is a means for analyzing collected acoustic, temperature, and pressure data to determine whether or not there is an anomaly.

[0532] An "emotion engine" is a means of recognizing a user's emotional state by analyzing their voice and facial expression data.

[0533] "Information transmission means" refers to means for notifying an external device of the detected abnormal information.

[0534] An "external device" is a device capable of receiving data, such as a portable device or computer, used to receive abnormality notifications.

[0535] The system for realizing this invention is comprised of a combination of acoustic signal processing means, temperature detection means, pressure detection means, information processing model means, emotion engine, and information transmission means. The system uses multiple sensors within the home to constantly monitor environmental data such as acoustics, temperature, and pressure. Each sensor is managed by a microcontroller, such as a Raspberry Pi, which collects data in real time and transmits it to a server for analysis.

[0536] The server receives data using a web framework such as Flask and analyzes it using an information processing model based on Scikit-learn. If an anomaly is detected, the user's emotional state is also evaluated by an emotion engine utilizing Google Cloud's Speech-to-Text and Face Recognition. Based on the results, push notifications are sent to mobile devices via the Twilio API to quickly inform family members and caregivers of the urgency of the elderly person's situation.

[0537] For example, if there is a sound such as an elderly person falling, the acoustic signal processing means detects it, and the server determines it to be an anomaly. At this point, if the emotion engine detects the elderly person's anxious voice, the information transmission means sends a more urgent message to the family member's smartphone. This procedure enables prompt and appropriate support for the elderly.

[0538] An example of a prompt statement is, "Please propose a system for detecting abnormalities or emotional changes in the environment where elderly people live alone."

[0539] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0540] Step 1:

[0541] The device collects environmental data in real time from acoustic, temperature, and pressure sensors. Each sensor receives acoustic signals, temperature data, and pressure data as input, and outputs this data in a digital format. This allows for the recording of changes in the surrounding environment.

[0542] Step 2:

[0543] The device sends the collected data to the Raspberry Pi, where it is temporarily stored. At this stage, the data is still raw and requires analysis. The input is raw data from the device, which is stored on the Raspberry Pi as received data.

[0544] Step 3:

[0545] The server receives data from the Raspberry Pi using Flask. The Flask server prepares to begin analysis using the data as input and outputs data in a format that can be processed. At this stage, data formatting is performed as a preprocessing step.

[0546] Step 4:

[0547] The server uses Scikit-learn to analyze data using an information processing model. The input consists of pre-processed acoustic, temperature, and pressure data, and the model determines whether anomalies exist. The output is the result of the determination regarding the presence or absence of anomalies. If an anomaly is detected, the server proceeds to the next step.

[0548] Step 5:

[0549] The server uses an emotion engine powered by Google Cloud's Speech-to-Text and Face Recognition to analyze user emotions. Input is audio or image data, and after analysis, it provides output that identifies the user's emotional state. This allows for the determination of the user's emotions.

[0550] Step 6:

[0551] If an anomaly or emergency is detected, the server uses the Twilio API to send a notification to an external device. The input consists of the anomaly detection result and the emotional state, and a customized message is generated and output as the notification. This allows remote users to understand the situation immediately.

[0552] Step 7:

[0553] Users receive notifications sent from the server on their mobile devices. The input is the notification message from the server, and the output is the user confirming the information and taking immediate action. This creates a system that allows for the rapid confirmation of the safety of elderly people.

[0554] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0555] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0556] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0557] [Fourth Embodiment]

[0558] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0559] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0560] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0561] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0562] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0563] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0564] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0565] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0566] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0567] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0568] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0569] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0570] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0571] This invention relates to an anomaly detection system installed in the homes of elderly people living alone. This system is composed of acoustic signal processing means, temperature detection means, pressure detection means, artificial intelligence modeling means, and communication means.

[0572] Terminal embodiment

[0573] The device is equipped with multiple sensors to continuously monitor ambient noise, floor temperature, and pressure within the home. Specifically, acoustic sensors collect sounds within the home and process their intensity and frequency as digital signals. Temperature sensors accurately measure floor temperature and collect this data. Pressure sensors have the ability to detect changes in pressure on the floor. Data from these sensors is aggregated in real time into a data processing unit within the device. The device immediately analyzes this data and, if it identifies any anomalies exceeding specified values, sends the information to a server.

[0574] Server Embodiment

[0575] The server constantly receives data transmitted from the terminal and uses an AI model to analyze for anomalies. Specifically, the server compares acoustic, temperature, and pressure data to detect abnormal deviations from past patterns. If an anomaly is detected, the server prepares a notification as the next step and sends a push notification to the user's external terminal via communication means.

[0576] User Embodiment

[0577] Users receive real-time anomaly notifications on their smartphones or PCs. These notifications include details such as the location and time of the anomaly, and the type of anomaly detected (e.g., unusual sounds, elevated body temperature). This allows users to quickly verify and respond to the issue.

[0578] Specific example

[0579] Consider a scenario where an elderly person falls in their room one night. At this time, a loud noise is detected by an acoustic sensor, the temperature of the part of the body that came into contact with the floor rises sharply, and a pressure sensor also records a significant change. The terminal immediately transmits this data to a server. The server uses an AI model to analyze this data and determines that it is an anomaly indicating a fall. As a result, a notification is sent to the user's smartphone stating, "Fall detected. Please check immediately." Upon receiving this notification, the user can quickly check for any abnormalities in their home. In this way, the present invention enables elderly people to live in a safer environment.

[0580] The following describes the processing flow.

[0581] Step 1:

[0582] The device continuously collects sound, temperature, and pressure data from within the home using its respective sensors. Acoustic sensors convert sound into digital signals, temperature sensors measure floor surface temperature, and pressure sensors record the pressure on the floor. All of this data is aggregated in the device's data processing unit.

[0583] Step 2:

[0584] The terminal immediately analyzes the collected data and compares it to a pre-set normal range. If changes in volume, temperature, or pressure exceed a set threshold, the data is flagged as abnormal and prepared to be sent to the server as abnormal detection data.

[0585] Step 3:

[0586] The terminal sends data with an abnormality flag set to the server. At this time, it generates packet data containing detailed information such as the type, time, and location of the abnormality, and sends this data to the server via the communication means.

[0587] Step 4:

[0588] The server passes the abnormal data received from the terminal to an AI model for processing and analysis. The AI ​​model compares the data with previously accumulated data to determine whether the anomaly is temporary or indicates a serious problem.

[0589] Step 5:

[0590] Based on the AI ​​analysis results, if the server determines that an anomaly is serious, it will use communication means to generate a push notification to the user's external device. The notification will include the details of the anomaly and the recommended next action.

[0591] Step 6:

[0592] Users receive notifications from the server via their smartphones or PCs. Upon receiving a notification, users review the content and, if necessary, directly check on the situation in their home or arrange for necessary assistance.

[0593] (Example 1)

[0594] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0595] In modern society, the number of elderly people living alone is increasing, but effective anomaly detection systems to ensure their safety remain insufficient. In particular, the challenge lies in quickly detecting abnormal situations such as falls or sudden changes in health and prompting appropriate responses. In response to this, the present invention aims to provide a system that enables real-time anomaly detection and rapid external notification.

[0596] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0597] In this invention, the server includes means for processing acoustic information and detecting anomalies, means for measuring the temperature of the floor surface, and means for measuring changes in pressure on the floor surface. This makes it possible to quickly and accurately detect abnormal situations and immediately notify users in remote locations.

[0598] "Acoustic information" refers to digital data about the intensity and frequency of sounds collected from the environment.

[0599] "Temperature information" refers to measurement data that indicates the temperature state of an object.

[0600] "Pressure information" refers to data showing measurement results regarding the forces applied to floors and other surfaces.

[0601] A "generative AI model" is an artificial intelligence technique that learns patterns from large amounts of data and makes judgments and predictions for specific tasks.

[0602] "Communication methods" refer to technologies used to transmit data and information obtained from sensors and servers to external terminals.

[0603] "Means for collecting data from sensors in real time and performing initial analysis" refers to technologies that have the function of analyzing data acquired from various sensors immediately and quickly detecting signs of anomalies.

[0604] This invention relates to an anomaly detection system for ensuring the safety of the elderly. This system continuously collects acoustic, temperature, and pressure information and analyzes it using a generative AI model to quickly detect abnormal situations and enable notification to external terminals.

[0605] Terminal embodiment

[0606] The terminal is installed in the home and collects acoustic, temperature, and pressure information from the environment in real time through multiple sensors. These environmental sensors include acoustic sensors, temperature sensors, and pressure sensors. For example, the acoustic sensor detects noise levels and unusual sounds in the home and converts this into digital data. The temperature sensor measures changes in floor temperature at regular intervals and provides this as temperature data. The pressure sensor detects changes in pressure on the floor and collects this as pressure data. The terminal aggregates the data from these sensors into a data processing unit and analyzes early signs of anomalies.

[0607] Server Embodiment

[0608] The server receives information transmitted from the terminal and analyzes this data in detail using a generating AI model. Acoustic, temperature, and pressure data are compared to past normal patterns to check for abnormal fluctuations. For example, if an unusual sound occurs outside of normal activity hours, the data is identified as an anomaly. If the server detects an anomaly, a push notification is sent to the external user's device via communication.

[0609] User Embodiment

[0610] Users can receive anomaly notifications on their smartphones or PCs. The notifications include detailed information such as the location, time, and type of anomaly detected, allowing users to quickly check the situation and take necessary action.

[0611] Specific example

[0612] If an elderly person falls at home one night, an acoustic sensor detects a loud noise, a temperature sensor records a rapid rise in floor temperature, and a pressure sensor detects an abnormal pressure change. This data is transmitted from the terminal to a server, where an AI model recognizes it as a "fall." As a result, the user's smartphone receives a notification stating, "Fall detected. Please check immediately," allowing the user to quickly check the scene. In this way, the present invention can contribute to the safety of the elderly.

[0613] An example of a prompt message is: "Please explain in detail how the anomaly detection system for the elderly analyzes sound, temperature, and pressure data in real time to detect anomalies."

[0614] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0615] Step 1:

[0616] The device collects data in real time through acoustic, temperature, and pressure sensors placed throughout the home. The inputs from these sensors are raw data regarding environmental acoustics, floor temperature, and pressure changes, respectively. The device processes this raw data and organizes it as digital signals. For example, acoustic data is converted into sound intensity and frequency, and temperature data is the measurement itself from the sensor. Pressure data is output as the amount of pressure change per unit time.

[0617] Step 2:

[0618] The terminal integrates digital signals from each sensor and performs initial analysis in the data processing unit. The input data is compared to past normal values ​​to determine if there are any abnormalities. This process analyzes, for example, whether the volume exceeds a threshold, whether there is a sudden temperature rise, or whether there is a pressure change that exceeds the prediction. The output is represented as a flag indicating normal or abnormal.

[0619] Step 3:

[0620] If an anomaly flag is set, the terminal sends the data to the server. At this time, the terminal encrypts the data and transmits it through a secure communication channel. The transmitted data includes acoustic data, temperature data, and pressure data related to the anomaly, and a secure data stream is generated as the communication output.

[0621] Step 4:

[0622] The server receives anomalous data transmitted from the terminal and performs a detailed analysis using a generating AI model. The input to the server is an anomalous data stream from the terminal. The server feeds this data into an analysis algorithm and further examines the anomaly by comparing it with past data. The output is information that describes the anomaly and its occurrence in detail.

[0623] Step 5:

[0624] When the server detects an anomaly, it sends a notification to an external user terminal using a communication method. The server's input is the analysis results from a generative AI model. Based on these results, the server creates a notification message and sends it to the user as a push notification. The output is a notification message that can be received on the user's terminal.

[0625] Step 6:

[0626] Users receive push notifications on their smartphones or PCs to check the details of an anomaly. User input is the push notification data sent from the server. Output is detailed information displayed on the screen, including the location, time, and type of the anomaly. Based on this information, users can quickly take corrective action.

[0627] (Application Example 1)

[0628] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0629] In environments where elderly people live alone, there is a need to quickly detect unexpected physical abnormalities or changes in daily routines and accurately notify caregivers who are located remotely. However, conventional systems have challenges in detecting abnormalities accurately and identifying changes in activity patterns, making it difficult to respond quickly. Therefore, a system capable of more accurate and real-time monitoring and notification is needed.

[0630] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0631] In this invention, the server includes artificial intelligence model means for analyzing acoustic signals, temperature data, and pressure data; lifestyle pattern identification means for learning and detecting changes in activity patterns within the home; and communication means for notifying abnormalities and changes in activity patterns. This enables real-time detection of sudden accidents or changes in health conditions within the home, allowing caregivers to respond quickly.

[0632] An "acoustic signal processing device" is a device that collects ambient sounds and analyzes their magnitude and frequency as digital signals.

[0633] A "temperature detection means" is a device that measures the temperature of the floor surface and obtains accurate temperature data.

[0634] A "pressure detection device" is a device that measures changes in the pressure applied to the floor surface and collects that data.

[0635] An "artificial intelligence modeling device" is a device equipped with an algorithm for analyzing acoustic signals, temperature data, and pressure data to determine whether or not there is an anomaly.

[0636] A "communication device" is a device used to send notifications to external terminals when an anomaly is detected.

[0637] A "lifestyle pattern identification device" is a device that learns activity patterns within a household and detects changes in those patterns.

[0638] The server receives acoustic signals, temperature data, and pressure data collected through an anomaly detection system. The system's terminals are installed within the home and continuously monitor this data using multiple sensors. The terminals are equipped with the ability to acquire sounds within the home using acoustic sensors and analyze the characteristics of their digital signals. They also acquire floor temperature using temperature sensors and detect temperature changes in real time. Furthermore, pressure sensors detect changes in pressure on the floor, enabling the system to sense situations such as falls.

[0639] Data from the device is aggregated and analyzed on a server based on cloud infrastructure. The server processes this data using an artificial intelligence model based on TensorFlow to determine whether or not anomalies are present. In particular, the lifestyle pattern recognition function learns patterns of daily activities and identifies sudden changes. When an anomaly or activity change is detected, a push notification is sent via communication to the user's external device, such as a smartphone. This communication function allows the user to respond immediately to anomalies, even from a remote location.

[0640] Users can receive notifications on their smartphones or computers, allowing them to stay informed about the situation in their homes. For example, if an elderly person falls at home, the system will simultaneously detect a loud noise, a change in floor temperature, and an abnormality in the pressure sensor, sending the user a message saying, "Fall detected. Please check immediately." This allows users to respond quickly, further enhancing the safety of the elderly.

[0641] An example of a prompt for a generated AI model is, "Analyze the following household sensor data to detect changes in the health status of elderly individuals and identify any abnormalities." This prompt prompts the model to begin analyzing the sensor data and assist in detecting anomalies.

[0642] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0643] Step 1:

[0644] The device collects data from within the home through acoustic, temperature, and pressure sensors. Sound data acquired by the acoustic sensor is directly converted into a digital signal, and the loudness and frequency of the sound are measured. The temperature sensor accurately measures the floor temperature, and this data is updated in real time. The pressure sensor detects changes in pressure on the floor and prepares the data to be sent to the server. The input is raw data from each sensor, and the output is processed digital data.

[0645] Step 2:

[0646] The server receives acoustic, temperature, and pressure data transmitted from the terminal. The server receives this data and performs preprocessing to reduce noise and standardize the data. The input is digital data from the terminal, and the output is clean data for analysis.

[0647] Step 3:

[0648] The server analyzes the received data using an artificial intelligence model based on TensorFlow. Acoustic, temperature, and pressure data are input to the AI ​​model, and the probability of detecting anomalies is calculated. Here, prompts based on the generated AI model are used to set the analysis conditions for identifying anomalies. The input is a clean dataset, and the output is the result of the anomaly detection and inferences about changes in activity patterns.

[0649] Step 4:

[0650] When the server detects an anomaly using the AI ​​model, it pushes that information to an external terminal. The notification contains detailed information, including the type of anomaly, its location, and the time it occurred. The push notification is sent via a communication method, with the input being the anomaly detection information and the output being a notification message to the user's external terminal.

[0651] Step 5:

[0652] Users receive notifications on their smartphones or PCs and check the details of the anomaly. Based on the notification details, users can quickly check the situation within their home. The input is a push notification message, and the output is the initiation of user verification action.

[0653] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0654] This invention relates to a system for detecting abnormalities and recognizing user emotions in the home of an elderly person living alone. This system is comprised of an acoustic signal processing means, a temperature detection means, a pressure detection means, an artificial intelligence model means for analyzing this data, and an emotion engine for recognizing user emotions.

[0655] Terminal embodiment

[0656] The device is equipped with acoustic, temperature, and pressure sensors to constantly monitor environmental factors within the home. Each sensor collects data in real time, which is processed immediately within the device. If abnormal data deviating from the normal range is detected during this processing, the information is sent to the server. In addition, the device collects emotional data from the user's voice and facial expressions and analyzes it using an emotion engine. This emotion engine identifies the user's current emotional state and helps customize notification content.

[0657] Server Embodiment

[0658] The server comprehensively analyzes the data transmitted from the terminal and performs anomaly detection. When an anomaly is detected, it considers the emotional data from the emotion engine and generates an appropriate notification for the user. For example, if a sense of urgency is recognized, the urgency of the notification is increased to encourage the user to take a quick action.

[0659] User Embodiment

[0660] Users can receive notifications sent from the server through their external devices, such as smartphones. These notifications are customized to the user's current emotional state, allowing for more appropriate responses. For example, when a user is feeling anxious, a message designed to provide reassurance is sent.

[0661] Specific example

[0662] Consider a scenario where an elderly person suddenly loses consciousness in their room at night. In this case, the terminal's sensors detect abnormal noise levels, temperature changes, and pressure drops. This data obtained from the sensors is immediately sent from the terminal to the server. The server's AI model recognizes this as a serious anomaly. Simultaneously, if the emotion engine detects an emotion indicating urgency from the elderly person's facial expressions and voice, the server promptly sends an emergency notification to the family's smartphone. Upon receiving this notification, the family can quickly check on the elderly person's well-being and take necessary action. Thus, this invention provides an environment in which elderly people can live with greater peace of mind.

[0663] The following describes the processing flow.

[0664] Step 1:

[0665] The device collects real-time data within the home using acoustic, temperature, and pressure sensors. The acoustic sensor monitors ambient sounds, the temperature sensor observes changes in floor temperature, and the pressure sensor detects the weight on the floor. This data is quickly aggregated within the device.

[0666] Step 2:

[0667] The device immediately evaluates the aggregated data by comparing it to a set threshold. If an anomaly is detected, the information is flagged and marked as data to be sent to the server. At this point, the device captures voice and facial expression changes, and collects emotional data in parallel.

[0668] Step 3:

[0669] The device sends emotional data collected along with anomaly data to the server. The transmitted information includes the type of anomaly, the exact time it occurred, and the acquired emotional indicators.

[0670] Step 4:

[0671] When the server receives data from a terminal, it uses an AI model to analyze the severity of the anomaly. At the same time, it evaluates the user's emotional state, as provided by the emotion engine, and adjusts the urgency level accordingly.

[0672] Step 5:

[0673] The server generates an anomaly notification based on the analysis results. A message tailored to the user's emotional state is created. For example, if the user is feeling urgent, the message will include a prompt action.

[0674] Step 6:

[0675] Users receive notifications sent from the server on their smartphones or PCs. These notifications include details of detected anomalies and recommended actions, allowing users to take appropriate action based on this information. Personalized messages based on detected emotions enable users to quickly take the most appropriate response to the situation.

[0676] (Example 2)

[0677] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] There is a need to provide a system that enables a swift and appropriate response in the event of an emergency in households where elderly people live alone. In particular, a challenge is to determine if an abnormality has occurred while taking into account the emotional state of the elderly person and to notify them of appropriate countermeasures.

[0679] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0680] In this invention, the server includes means for processing acoustic information, means for detecting temperature changes, and means for detecting pressure fluctuations. This makes it possible to immediately determine abnormalities in the home environment of an elderly person living alone and, based on the results, transmit appropriate notifications tailored to the emotional state of the elderly person to an external information processing device.

[0681] "Means for processing acoustic information" refers to technologies for detecting sound, analyzing its characteristics, and detecting anomalies.

[0682] A "means for detecting temperature changes" refers to a device that measures the ambient temperature in real time and observes its fluctuations.

[0683] "Means for detecting pressure fluctuations" refers to sensor technology that continuously monitors the pressure acting on an object and captures changes in it.

[0684] An "artificial intelligence tool" is a program that analyzes collected data and recognizes specific patterns or anomalies.

[0685] "Emotional analysis methods" refer to technologies that analyze an individual's voice and facial expression data to determine their emotional state.

[0686] "Communication infrastructure" refers to the infrastructure used to send and receive information with other devices and systems.

[0687] An "external information processing device" is an electronic device capable of processing data outside of a system, such as a mobile phone or computer.

[0688] This system is designed for anomaly detection and emotion recognition in the homes of elderly people. The terminal is equipped with sensors to detect sound, temperature, and pressure. These sensors collect data in real time and process it immediately within the terminal. When the terminal identifies data deemed abnormal, it sends the information to the server. The server analyzes this data using a generative AI model to determine if it is an anomaly. Furthermore, it uses speech recognition and facial expression analysis technologies to analyze the user's emotions and adjust notifications according to their emotional state.

[0689] When generating notifications, the server considers the user's emotional state to determine appropriate content and urgency. For example, if the server determines that the user is in a state of stress, it will deliver a more urgent notification. This allows family members to quickly check on the elderly person's situation. In this way, the entire system works efficiently together to support the safety and well-being of the elderly.

[0690] A concrete example is a situation where an elderly person loses consciousness at night. In this case, the device's sensors acquire data such as abnormal volume, sudden temperature changes, and pressure drops. When this data is recognized as a serious anomaly by the server, the emotion engine detects emotions indicating the elderly person's urgency, and an emergency notification is promptly sent to the family's smartphone.

[0691] An example of a prompt message would be: "When an elderly person loses consciousness at night, what data should the device's sensors detect? Also, what notification should the server generate based on that data?" This prompt message clearly indicates the system's operation and the content of the notification.

[0692] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0693] Step 1:

[0694] The device collects environmental data in real time using acoustic, temperature, and pressure sensors. Specifically, it records sound with a microphone, measures room temperature with a temperature sensor, and measures floor load with a pressure sensor. The input is raw data obtained from the sensors, which is recorded as time-series data. The output is the sequenced data described above, which is supplied to the next processing step.

[0695] Step 2:

[0696] The terminal immediately analyzes the collected environmental data and detects anomalies by comparing it to known standards. Data processing includes calculating the average and peak volume, temperature fluctuation range, and pressure change rate. The input is the time-series data acquired in step 1, and the output is flag data indicating an anomaly. When an anomaly is detected, this flag is sent to the server.

[0697] Step 3:

[0698] The device collects user voice and facial expression data and performs emotion analysis. Specifically, it uses a camera and microphone to record the user's facial expressions and voice, and analyzes them using an emotion engine. It analyzes tone and intonation from the voice, and subtle changes from the facial expressions. The input is raw video and audio data, and the output is data indicating the user's emotional state.

[0699] Step 4:

[0700] The server receives anomaly and emotion data transmitted from the terminal and uses a generative AI model to perform anomaly detection and emotion analysis. Here, a series of data is used to determine the severity of the anomaly and the necessary response. The input is the output data from steps 2 and 3, and the output is a suggestion of appropriate countermeasures according to the type of anomaly and the emotional state.

[0701] Step 5:

[0702] The server generates a notification based on the analysis results and sends it to the user. The notification content will be text or voice message depending on the situation. For example, if an anomaly is detected and the user expresses anxiety, a message such as "The room temperature has dropped rapidly. Are you feeling alright?" will be generated. The input is the analysis results from step 4, and the output is notification information for family members or caregivers.

[0703] (Application Example 2)

[0704] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0705] In modern society, the number of elderly people living alone is increasing, making their safety a major challenge. In particular, a system that can respond immediately to sudden abnormalities or emergencies is required. However, elderly people living alone may have difficulty recognizing abnormalities themselves and notifying others. Furthermore, if the elderly person's emotional state is unstable, there is a risk of delays in appropriate responses, thus necessitating the development of more effective support systems.

[0706] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0707] In this invention, the server includes acoustic signal processing means, temperature detection means, and pressure detection means. This enables rapid detection of environmental abnormalities in the living space of elderly people, analysis of these abnormalities to determine their nature, and recognition of the elderly person's emotional state using an emotion engine. Furthermore, if an abnormality or change in emotion is detected, an automatic notification can be sent to an external device to prompt immediate action.

[0708] "Acoustic signal processing means" refers to means for receiving acoustic data and detecting abnormal sounds from that data.

[0709] A "temperature detection means" is a means for measuring the temperature in the environment and collecting that data.

[0710] A "pressure detection means" is a means for measuring changes in the pressure applied to the environment and collecting that data.

[0711] An "information processing model means" is a means for analyzing collected acoustic, temperature, and pressure data to determine whether or not there is an anomaly.

[0712] An "emotion engine" is a means of recognizing a user's emotional state by analyzing their voice and facial expression data.

[0713] "Information transmission means" refers to means for notifying an external device of the detected abnormal information.

[0714] An "external device" is a device capable of receiving data, such as a portable device or computer, used to receive abnormality notifications.

[0715] The system for realizing this invention is comprised of a combination of acoustic signal processing means, temperature detection means, pressure detection means, information processing model means, emotion engine, and information transmission means. The system uses multiple sensors within the home to constantly monitor environmental data such as acoustics, temperature, and pressure. Each sensor is managed by a microcontroller, such as a Raspberry Pi, which collects data in real time and transmits it to a server for analysis.

[0716] The server receives data using a web framework such as Flask and analyzes it using an information processing model based on Scikit-learn. If an anomaly is detected, the user's emotional state is also evaluated by an emotion engine utilizing Google Cloud's Speech-to-Text and Face Recognition. Based on the results, push notifications are sent to mobile devices via the Twilio API to quickly inform family members and caregivers of the urgency of the elderly person's situation.

[0717] For example, if there is a sound such as an elderly person falling, the acoustic signal processing means detects it, and the server determines it to be an anomaly. At this point, if the emotion engine detects the elderly person's anxious voice, the information transmission means sends a more urgent message to the family member's smartphone. This procedure enables prompt and appropriate support for the elderly.

[0718] An example of a prompt statement is, "Please propose a system for detecting abnormalities or emotional changes in the environment where elderly people live alone."

[0719] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0720] Step 1:

[0721] The device collects environmental data in real time from acoustic, temperature, and pressure sensors. Each sensor receives acoustic signals, temperature data, and pressure data as input, and outputs this data in a digital format. This allows for the recording of changes in the surrounding environment.

[0722] Step 2:

[0723] The device sends the collected data to the Raspberry Pi, where it is temporarily stored. At this stage, the data is still raw and requires analysis. The input is raw data from the device, which is stored on the Raspberry Pi as received data.

[0724] Step 3:

[0725] The server receives data from the Raspberry Pi using Flask. The Flask server prepares to begin analysis using the data as input and outputs data in a format that can be processed. At this stage, data formatting is performed as a preprocessing step.

[0726] Step 4:

[0727] The server uses Scikit-learn to analyze data using an information processing model. The input consists of pre-processed acoustic, temperature, and pressure data, and the model determines whether anomalies exist. The output is the result of the determination regarding the presence or absence of anomalies. If an anomaly is detected, the server proceeds to the next step.

[0728] Step 5:

[0729] The server uses an emotion engine powered by Google Cloud's Speech-to-Text and Face Recognition to analyze user emotions. Input is audio or image data, and after analysis, it provides output that identifies the user's emotional state. This allows for the determination of the user's emotions.

[0730] Step 6:

[0731] If an anomaly or emergency is detected, the server uses the Twilio API to send a notification to an external device. The input consists of the anomaly detection result and the emotional state, and a customized message is generated and output as the notification. This allows remote users to understand the situation immediately.

[0732] Step 7:

[0733] Users receive notifications sent from the server on their mobile devices. The input is the notification message from the server, and the output is the user confirming the information and taking immediate action. This creates a system that allows for the rapid confirmation of the safety of elderly people.

[0734] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0735] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0736] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0737] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0738] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0739] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0740] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0741] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0742] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0743] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0744] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0745] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0746] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0747] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0748] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0749] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0750] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0751] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0752] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0753] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0754] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0755] The following is further disclosed regarding the embodiments described above.

[0756] (Claim 1)

[0757] Acoustic signal processing means for detecting abnormal sounds,

[0758] A temperature sensing means for measuring the temperature distribution of the floor surface,

[0759] A pressure sensing means for measuring pressure changes on the floor surface,

[0760] An artificial intelligence model means for analyzing data collected from the aforementioned acoustic signal processing means, temperature detection means, and pressure detection means to determine anomalies,

[0761] A communication method for sending notifications to external terminals when an anomaly is detected,

[0762] A system that includes this.

[0763] (Claim 2)

[0764] The system according to claim 1, characterized in that the artificial intelligence model means has an algorithm for determining an anomaly using acoustic signals, temperature data, and pressure data.

[0765] (Claim 3)

[0766] The system according to claim 1, characterized in that the communication means sends a push notification to a user's mobile terminal in a remote location in the event of an abnormality.

[0767] "Example 1"

[0768] (Claim 1)

[0769] A means for processing acoustic information and detecting anomalies,

[0770] A means for measuring the temperature of the floor surface,

[0771] A means for measuring changes in pressure on the floor surface,

[0772] A means for analyzing collected data using a generative AI model and determining anomalies,

[0773] A means for sending a notification to an external user's terminal when an anomaly is detected,

[0774] A means of collecting data from sensors in real time and performing initial analysis,

[0775] A system that includes this.

[0776] (Claim 2)

[0777] The system according to claim 1, characterized in that the generating AI model includes an algorithm for determining anomalies using acoustic information, temperature information, and pressure information.

[0778] (Claim 3)

[0779] The system according to claim 1, characterized in that the communication means sends a push notification to a user's mobile device in a remote location when an anomaly is detected.

[0780] "Application Example 1"

[0781] (Claim 1)

[0782] Acoustic signal processing means for detecting abnormal sounds,

[0783] A temperature sensing means for measuring the temperature distribution of the floor surface,

[0784] A pressure sensing means for measuring pressure changes on the floor surface,

[0785] An artificial intelligence model means for analyzing data collected from the aforementioned acoustic signal processing means, temperature detection means, and pressure detection means to determine anomalies,

[0786] A communication method for sending notifications to external terminals when an anomaly is detected,

[0787] A means for identifying lifestyle patterns to learn activity patterns within the home, detect changes therein, and notify,

[0788] A system that includes this.

[0789] (Claim 2)

[0790] The system according to claim 1, characterized in that the artificial intelligence model means has an algorithm that determines anomalies using acoustic signals, temperature data, and pressure data, and identifies changes in specific activity patterns.

[0791] (Claim 3)

[0792] The system according to claim 1, characterized in that the communication means sends a push notification to a user's mobile terminal in a remote location when it detects an abnormality or a change in a specific activity pattern.

[0793] "Example 2 of combining an emotion engine"

[0794] (Claim 1)

[0795] Means for processing acoustic information,

[0796] Means for detecting temperature changes,

[0797] Means for detecting pressure fluctuations,

[0798] An artificial intelligence means for analyzing the aforementioned acoustic information, temperature changes, and pressure fluctuations to determine anomalies,

[0799] A means of sentiment analysis for identifying the user's emotional state,

[0800] A communication means for generating a notification based on an anomaly detection and transmitting it to an external information processing device,

[0801] A system that includes this.

[0802] (Claim 2)

[0803] The system according to claim 1, characterized in that the artificial intelligence means uses information on sound, temperature, and pressure to determine anomalies, and further adjusts the notification content by also considering data obtained by the emotion analysis means.

[0804] (Claim 3)

[0805] The system according to claim 1, characterized in that the communication means transmits a specific emergency message to a remote information processing device in the event of an abnormality.

[0806] "Application example 2 when combining with an emotional engine"

[0807] (Claim 1)

[0808] Acoustic signal processing means for detecting abnormal sounds,

[0809] A temperature sensing means for measuring the ambient temperature,

[0810] A pressure sensing means for measuring pressure changes,

[0811] An information processing model means for analyzing data collected from the aforementioned acoustic signal processing means, temperature detection means, and pressure detection means to determine anomalies,

[0812] An emotion engine for recognizing and analyzing user emotions,

[0813] An information transmission means for sending a notification to an external device when an abnormality is detected,

[0814] A system that includes this.

[0815] (Claim 2)

[0816] The system according to claim 1, characterized in that the information processing model means has an algorithm for determining an anomaly using acoustic signals, temperature data, pressure data, and user emotion data.

[0817] (Claim 3)

[0818] The system according to claim 1, characterized in that the information transmission means transmits a customized notification to a user's mobile device in a remote location in the event of an abnormality. [Explanation of symbols]

[0819] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Acoustic signal processing means for detecting abnormal sounds, A temperature sensing means for measuring the temperature distribution of the floor surface, A pressure sensing means for measuring pressure changes on the floor surface, An artificial intelligence model means for analyzing data collected from the aforementioned acoustic signal processing means, temperature detection means, and pressure detection means to determine anomalies, A communication method for sending notifications to external terminals when an anomaly is detected, A means for identifying lifestyle patterns to learn activity patterns within the home, detect changes therein, and notify, A system that includes this.

2. The system according to claim 1, characterized in that the artificial intelligence model means has an algorithm that determines anomalies using acoustic signals, temperature data, and pressure data, and identifies changes in specific activity patterns.

3. The system according to claim 1, characterized in that the communication means sends a push notification to a user's mobile terminal in a remote location when it detects an abnormality or a change in a specific activity pattern.

Images