System

Abstract

A system is provided.SOLUTION: The system includes a video acquisition means, a means for preprocessing video data, a means for learning the preprocessed video data and modeling a life pattern, a means for analyzing the video data in real time and detecting an abnormality, a means for transmitting an abnormality detection result to a notification destination, and a means for collecting feedback and improving a learning model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is important to reduce the risks that occur when elderly people live alone. Specifically, situations can arise where they are unable to call for help due to falls or sudden illness. However, existing "monitoring" services lack video analysis and real-time anomaly detection capabilities, making it difficult to respond quickly. There is a need for a "monitoring" system that can improve these situations, provide safer and faster responses, and provide a safer and more efficient solution. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The video acquisition means acquires video data from a webcam, and the preprocessing means converts the data into a format that is easy to analyze. The preprocessed video data is then modeled into lifestyle patterns by the learning means. Using this model, the real-time analysis means analyzes the video data in real time, and if an abnormality is detected, the notification means sends an alert to the specified notification destination. In addition, feedback is collected from users through the feedback collection means, and the learning model is improved, thereby improving the accuracy of the system.

[0007] Furthermore, this invention achieves more accurate anomaly detection by receiving data from IoT devices in the home network and processing it in sync with video analysis. Furthermore, when detecting anomalies, it reduces false positives by combining the analysis results of consecutive frames with data from additional sensors, providing a highly reliable "monitoring" service.

[0008] "Video acquisition means" refers to a function that collects video data in real time using a webcam or other video capture device.

[0009] The "preprocessing means" is a process for converting the acquired video data into a format that is easy to analyze, and includes steps such as data decoding, normalization, and contour extraction.

[0010] The "learning method" is a technology that uses preprocessed video data to train a machine learning model and model the subject's lifestyle patterns.

[0011] "Real-time analysis means" refers to an analysis engine that detects abnormalities in real time based on acquired and pre-processed video data.

[0012] "Notification means" refers to a function that sends an alert to a designated notification recipient when an abnormality is detected, and includes methods such as push notification, email, and SMS.

[0013] The "feedback collection means" is a function that receives feedback from users regarding the results of anomaly detection and uses this feedback to improve the accuracy of the system.

[0014] "IoT devices in a home network" refers to various sensor devices that are connected to a network within the home and provide environmental and operational information.

[0015] A "learning model" is a model generated by a machine learning algorithm that learns lifestyle patterns, and serves as the basis for anomaly detection.

[0016] "Anomaly detection" is a function that detects deviations in real-time analyzed data from learned lifestyle patterns.

[0017] The "designated notification destination" is a contact point such as a family member or medical institution that is set to receive a notification when an abnormality is detected. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] MODE FOR CARRYING OUT THE INVENTION

[0040] The system of the present invention integrates the functions of video acquisition, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. Specifically, this system is as follows.

[0041] 1. Video acquisition

[0042] Device (Web camera)

[0043] The webcam installed as a terminal constantly monitors the elderly's living space and captures video data in real time, which is then sent to a server via a network.

[0044] 2. Video preprocessing

[0045] server

[0046] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, noise removal, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[0047] 3. Learning about daily life patterns

[0048] server

[0049] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[0050] 4. Real-time analysis and anomaly detection

[0051] server

[0052] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[0053] 5. Abnormality notification

[0054] server

[0055] If an abnormality is detected, the server generates a notification message and sends it to designated recipients, such as family members or medical institutions. The notification includes details such as the type of abnormality, the time of occurrence, and location information. Notifications can be sent via push notification, email, SMS, etc.

[0056] 6. Gather feedback and improve the model

[0057] User

[0058] The user who receives the notification sends feedback about the anomaly back to the server, including the actual situation, whether there was a false positive, and whether the response was appropriate.

[0059] server

[0060] The server uses the collected feedback to retrain the machine learning model and make improvements to reduce false positives and improve detection accuracy, making the system more reliable over time.

[0061] Specific examples

[0062] Example 1: Fall detection

[0063] server

[0064] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion is detected and recognized as an abnormality. The server immediately sends a notification to the family member saying, "A fall has been detected in the living room."

[0065] Example 2: Detecting abnormal activity times

[0066] server

[0067] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, the server sends a notification to the family saying, "No activity was observed by 8:00 AM." The notification also includes a message requesting feedback to confirm the actual situation.

[0068] As described above, this invention is a system that integrates the functions of image acquisition, preprocessing, learning, analysis, notification, and feedback collection to monitor the lives of the elderly in real time and enable rapid response. This system greatly improves the safety of the elderly and allows their families and medical institutions to monitor them with peace of mind.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] Device (Web camera)

[0072] The webcam captures video in real time and generates video data that is compressed and sent over the home network to a server.

[0073] Step 2:

[0074] server

[0075] The server receives the video data sent from the webcam, performs an error check to see if there are any packet losses, and requests packet retransmission if necessary.

[0076] Step 3:

[0077] server

[0078] The received video data is decoded and converted into a format that can be easily processed by the analysis engine. This preprocessing includes data normalization, noise removal, and contour extraction.

[0079] Step 4:

[0080] server

[0081] The preprocessed data is stored in a database, which includes not only the video frames but also time information and activity metadata.

[0082] Step 5:

[0083] server

[0084] The machine learning model is trained using data stored in the database. In the initial stage, the model is built based on data from a certain period of time and learns the lifestyle patterns of elderly people.

[0085] Step 6:

[0086] server

[0087] The system analyzes video data received in real time and compares it with a learning model to detect the subject's position, movement, and gestures such as falls.

[0088] Step 7:

[0089] server

[0090] It combines the analysis results of consecutive frames with data from other IoT devices in the home network to detect anomalies, for example, when it receives fall data from a smart sensor, it compares it with the corresponding video analysis results.

[0091] Step 8:

[0092] server

[0093] When an anomaly is detected, a notification message is generated and sent to the specified notification recipients, containing details such as the type of anomaly, the time of occurrence, and location information.

[0094] Step 9:

[0095] User

[0096] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[0097] Step 10:

[0098] User

[0099] Feedback regarding the anomaly notification is sent back to the server, including the actual situation and whether or not there was a false positive.

[0100] Step 11:

[0101] server

[0102] The machine learning model is retrained based on the collected feedback, which improves the model's accuracy, reducing false positives and improving anomaly detection.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] In elderly care services, rapid detection and response to abnormal behavior is important. However, conventional systems have issues with monitoring accuracy and real-time performance, resulting in frequent false positives. Furthermore, the process of feedback collection and model improvement is not well coordinated, resulting in a lack of system reliability. The present invention aims to solve these problems and improve the safety of elderly people.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for constantly capturing video for monitoring the elderly person's living space, means for decoding the received video data and dividing it into frames, means for normalizing the frames and performing contour extraction and noise removal, means for learning the preprocessed video data using a machine learning algorithm to model lifestyle patterns, means for analyzing the video data in real time and comparing it with an existing learning model to detect abnormal behavior, means for generating a notification message including the type, time of occurrence, and location information of the detected abnormal behavior and sending it to a specified notification destination, and means for collecting feedback from users who receive the notification and re-training and improving the learning model. This enables rapid and accurate detection and notification of abnormal behavior, and makes it possible to improve the accuracy and reliability of the system based on the feedback.

[0108] "Video acquisition means" refers to devices or systems that monitor the elderly person's living space and capture video images at all times.

[0109] "Decoding" refers to the process of converting received video data into a frame-by-frame format that can be analyzed.

[0110] The term "means for dividing into frames" refers to a process for dividing video data into individual frames and subjecting each frame to analysis.

[0111] "Normalization" refers to the process of standardizing video data frames to a specific size and format to improve the accuracy of analysis.

[0112] "Contour extraction" refers to the process of extracting the main shape information from a frame of video data and emphasizing the necessary information.

[0113] "Noise reduction" refers to the process of removing unnecessary information from video data to improve the quality of the data.

[0114] "Machine learning algorithm" refers to a mathematical model and its execution program that learns patterns from data and makes predictions and classifications.

[0115] "Modeling lifestyle patterns" refers to learning the daily behavior of elderly people and saving those behavior patterns as a mathematical model.

[0116] "Means for analyzing in real time" refers to a method for instantly analyzing currently received video data and instantly detecting abnormal behavior.

[0117] "Abnormal behavior" refers to behavior that deviates from the elderly person's daily behavior patterns, including, for example, prolonged immobility or sudden falls.

[0118] "Notification message" refers to a message that includes information about detected abnormal behavior (such as type, time of occurrence, location information, etc.) and is sent to a designated recipient.

[0119] "Feedback" refers to information provided by users who receive notifications about the actual situation, whether there were any false positives, and the appropriateness of the response.

[0120] "Retraining" refers to the process of updating an existing machine learning model based on new data and feedback to improve its accuracy and reliability.

[0121] MODE FOR CARRYING OUT THE INVENTION

[0122] The system of the present invention integrates the functions of video capture, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. To implement this system, the following specific hardware and software are used.

[0123] Hardware

[0124] Terminal (Web camera): Installed to monitor the living space of the elderly, it captures video 24 hours a day and transmits it to a server via the network.

[0125] software

[0126] Decoding and pre-processing: After receiving the video data from the webcam, the server decodes it using the FFmpeg library and splits the data into frames. Then, it normalizes the frames, extracts contours, and removes noise using the OpenCV library.

[0127] Machine learning: Using the preprocessed data, machine learning libraries such as TensorFlow and PyTorch are used on the server to learn the lifestyle patterns, thereby modeling the daily behavior of elderly people.

[0128] Real-time analysis and anomaly detection: The server uses analysis engines such as YOLOv5 and OpenPose to analyze video data in real time and compare it with existing learning models to detect abnormal behavior.

[0129] Notification: If an anomaly is detected, the server uses Twilio or Firebase Cloud Messaging (FCM) to generate a notification message containing the type of abnormal behavior, the time of occurrence, and location information, and sends it to the specified notification recipient.

[0130] Feedback collection and model improvement: After receiving the notification, users provide feedback through the application. The server receives this feedback, stores it in a database for analysis, and improves the model by retraining it.

[0131] Specific examples

[0132] Specific operation of fall detection

[0133] If an elderly person falls in the living room, a webcam captures the moment and sends the video data to a server. The server preprocesses the video data and analyzes it in real time. If the analysis detects a fall, it is recorded as an abnormal behavior and the server sends a notification message to the family saying, "A fall has been detected in the living room."

[0134] Specific operation of abnormal activity time detection

[0135] If an elderly person normally wakes up at 7:00 a.m. but does not wake up by 8:00 a.m., the server will compare the current data with the lifestyle patterns learned from past data to detect anomalies. If no activity is observed for a long period of time, a notification will be sent to the family stating, "No activity was observed by 8:00 a.m." The notification will also include a message requesting feedback to confirm the actual situation.

[0136] Prompt Sentence Examples

[0137] "In an elderly care system, please explain in detail the steps by which a webcam detects an elderly person's fall, and the server preprocesses and analyzes the video data and notifies the family."

[0138] This invention provides a system that can monitor the lives of elderly people in real time, quickly detect abnormal behavior, and notify them. This system is highly reliable because it continuously improves its model based on user feedback, thereby improving the safety of elderly people.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] Video Acquisition

[0142] The device (webcam) starts up and monitors the elderly person's living space.

[0143] How it works: The webcam captures video streams 24 / 7, obtaining video data in real time.

[0144] Input: Real-time video footage of an elderly person's living space.

[0145] Output: Captured video data (video stream).

[0146] Step 2:

[0147] Video data preprocessing

[0148] The server receives the video data sent from the web camera.

[0149] What happens: The server uses the FFmpeg library to decode the video stream and grab each frame.

[0150] Input: Video data (video stream) sent from the webcam.

[0151] Output: Decoded video frames.

[0152] Step 3:

[0153] Frame normalization and denoising

[0154] The server normalizes each frame and performs contour extraction and noise removal.

[0155] Specific operation: Using the OpenCV library, the frame size is standardized to 1280x720 pixels, converted to grayscale, and edge detection and noise removal are performed using the Canny method.

[0156] Input: Decoded video frames.

[0157] Output: Normalized and denoised frames.

[0158] Step 4:

[0159] Learning life patterns

[0160] The server inputs the pre-processed video frames into a machine learning algorithm.

[0161] Specific operation: Using TensorFlow and PyTorch, the system learns and models the daily behavioral patterns of elderly people based on data from a certain period of time.

[0162] Input: Preprocessed video frames.

[0163] Output: Learned behavioral pattern model.

[0164] Step 5:

[0165] Real-time analysis and anomaly detection

[0166] The server analyzes the video data sent from the webcam in real time.

[0167] Specific operation: Utilizing YOLOv5 and OpenPose, we compare with the learning model and detect behavior that deviates from everyday life patterns (abnormal behavior).

[0168] Input: Real-time video data, learning model.

[0169] Output: A log of the anomalous behavior detected.

[0170] Step 6:

[0171] Abnormal notification

[0172] The server generates a notification message if an abnormality is detected.

[0173] Specific operation: Using Twilio or Firebase Cloud Messaging (FCM), a notification containing the type of anomaly, the time of occurrence, and location information is sent to the specified destination.

[0174] Input: The log in which the anomalous behavior was detected.

[0175] Output: Notification message sent to family and healthcare providers.

[0176] Step 7:

[0177] Gathering feedback and improving the model

[0178] The user receives a notification and sends feedback back to the server.

[0179] Specific operation: The user reports the actual situation regarding the anomaly and whether there were any false positives through the application.

[0180] Input: User feedback.

[0181] Output: Feedback is collected and stored in a database.

[0182] Step 8:

[0183] Improving the model through retraining

[0184] The server retrains and improves the machine learning model based on the feedback collected.

[0185] How it works: Using TensorFlow, we incorporate feedback data to retrain existing models, improving detection accuracy and reliability.

[0186] Input: User feedback data, existing learning model.

[0187] Output: An improved learning model.

[0188] This allows the system to monitor elderly people in real time, quickly and accurately detect and notify abnormal behavior, and constantly improve the model based on user feedback.

[0189] (Application example 1)

[0190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0191] While conventional elderly monitoring systems are effective, especially in fixed living environments, they are difficult to monitor while moving, and passenger safety monitoring has not been fully realized, especially in self-driving vehicles. Furthermore, while there is a need for a function that can respond quickly and appropriately when a passenger experiences an abnormality, current systems often respond slowly. Therefore, there is an urgent need to develop a system that can monitor passenger safety in self-driving vehicles and provide immediate notification in the event of an abnormality.

[0192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0193] In this invention, the server includes a video acquisition means, a means for preprocessing the video data, a means for learning the preprocessed video data and modeling a lifestyle pattern, a means for analyzing the video data in real time and detecting anomalies, a means for sending the anomaly detection results to a notification destination, a means for collecting feedback and improving the learning model, a means for monitoring the status of passengers in the autonomous vehicle, and a means for detecting and notifying passenger anomalies in real time. This makes it possible to monitor the safety of passengers in the autonomous vehicle and to quickly notify passengers in the event of an anomaly.

[0194] "Video acquisition means" means a means for capturing video data of passengers in real time using a camera within the autonomous vehicle.

[0195] The "means for preprocessing video data" refers to a means for performing processes such as adjusting resolution and removing noise in order to convert the acquired video data into an analyzable format.

[0196] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for using preprocessed video data to learn passengers' normal behavioral patterns through a machine learning algorithm and modeling them.

[0197] "Means for analyzing video data in real time and detecting abnormalities" refers to a means for detecting abnormal behavior by comparing video data acquired in real time with past learning models.

[0198] The "means for transmitting an abnormality detection result to a notification destination" is a means for transmitting a notification message to a pre-specified contact point when an abnormality is detected.

[0199] The "means for collecting feedback and improving the learning model" refers to a means for collecting feedback from the recipients of notifications and retraining and improving the learning model based on that data.

[0200] "Means for monitoring the status of passengers inside an autonomous vehicle" refers to means for constantly monitoring the behavior and status of passengers using cameras and sensors installed inside the vehicle.

[0201] "Means for detecting and notifying passenger abnormalities in real time" refers to a means for detecting abnormal passenger behavior (e.g., sudden falls or prolonged periods of inactivity) in real time and immediately notifying the relevant parties.

[0202] The system for realizing this invention monitors the status of passengers in an autonomous vehicle in real time and promptly notifies them if an abnormality is detected. This system is specifically implemented by combining the following multiple means.

[0203] 1. Video acquisition method

[0204] The server captures passenger video data using cameras installed inside the autonomous vehicle, and the real-time video data is transmitted to the server.

[0205] 2. Video data preprocessing methods

[0206] The server receives the video data sent from the camera and performs pre-processing, which includes decoding the video data, adjusting the resolution, removing noise, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[0207] 3. A means of learning about life patterns

[0208] The server uses the preprocessed video data to model passengers' usual behavior patterns using machine learning algorithms, which are then saved as a learning model using machine learning frameworks such as TensorFlow and PyTorch.

[0209] 4. Real-time analysis and anomaly detection methods

[0210] The server analyzes the video data transmitted in real time and compares it with the learning model. If the analysis engine detects an abnormality (e.g., a sudden fall or prolonged inactivity), it records it. This anomaly detection uses a pre-trained generative AI model.

[0211] 5. Abnormality notification means

[0212] If an abnormality is detected, the server generates a notification message and sends it to the designated notification recipient (for example, family or a medical institution). This notification includes the type of abnormality, the time of occurrence, location information, etc. Notification methods include push notification, email, and SMS.

[0213] 6. Feedback Collection Methods

[0214] The user who receives the notification sends feedback about the situation to the server, including the actual situation and whether there were any false positives.

[0215] 7. Ways to improve the learning model

[0216] The server uses the collected feedback to retrain the machine learning model, reducing false positives and improving detection accuracy, again using machine learning frameworks such as TensorFlow or PyTorch.

[0217] Example 1: Fall detection

[0218] The server detects when a passenger in an autonomous vehicle suddenly falls. The camera captures the moment of the fall, and the preprocessed video data is sent to the server. Analysis detects the falling motion and recognizes it as an abnormality. The server then sends an abnormality notification to the specified notification destination.

[0219] Example 2: Detecting long periods of inactivity

[0220] For example, if a passenger who normally does not stay seated for more than 10 minutes remains motionless for more than 30 minutes, the server will detect the abnormality by comparing it with past data and send a notification such as, "The period of inactivity has exceeded normal."

[0221] Example prompt sentence:

[0222] "We will build a system that will detect abnormal behavior from camera footage and send a notification to a specified email address when an abnormality is detected. Specifically, we will detect when a passenger in an autonomous vehicle loses consciousness and continues to sit there, and send an email notification on the spot."

[0223] In this way, it is possible to monitor the safety of passengers in autonomous vehicles and respond quickly.

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] The server acquires video data in real time from the terminal (camera inside the autonomous vehicle). This data is a raw video stream that is sent from the terminal to the server. Specifically, the camera's video data is periodically captured and sent to the server via the network. The input data is the camera video, and the output is the video data sent to the server.

[0227] Step 2:

[0228] The server preprocesses the received video data, specifically decoding, adjusting resolution, and removing noise. The preprocessed video data is then converted into a format suitable for analysis. The input data is raw video data, and the output is preprocessed video data.

[0229] Step 3:

[0230] The server uses preprocessed video data to learn and model passengers' usual behavioral patterns using machine learning algorithms. Specifically, it collects data over a certain period of time and uses TensorFlow and PyTorch to learn lifestyle patterns. The input data is the preprocessed video data, and the output is a learning model.

[0231] Step 4:

[0232] The server analyzes the video data transmitted in real time and compares it with the learning model to detect anomalies. Specifically, the video data is input into a generative AI model to detect abnormal behavior. The input data are the video data transmitted in real time and the learning model, and the output is the anomaly detection result.

[0233] Step 5:

[0234] If an anomaly is detected, the server generates a notification message and sends it to a designated recipient (for example, family or a medical institution). Specifically, the notification includes details such as the type of anomaly, the time of occurrence, and location information, and is sent via push notification, email, or SMS. The input data is the anomaly detection result, and the output is the notification message.

[0235] Step 6:

[0236] The user who receives the notification sends back feedback. Specifically, they click a link or button included in the notification to report the actual situation, whether there were any false positives, etc. to the server. The input data is the user feedback, and the output is the feedback sent to the server.

[0237] Step 7:

[0238] The server uses the collected feedback to retrain the machine learning model and improve its detection accuracy. Specifically, it takes the feedback as a dataset and retrains the model using TensorFlow or PyTorch. The input data is the feedback and the existing model, and the output is a new, improved trained model.

[0239] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0240] MODE FOR CARRYING OUT THE INVENTION

[0241] This invention combines functions for an elderly care service, including video capture, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection, with an emotion engine that recognizes the user's emotions. This provides a system that takes emotion data into account in anomaly detection and realizes highly accurate and appropriate responses.

[0242] 1. Video acquisition

[0243] Device (Web camera)

[0244] The webcams used as terminals constantly monitor the elderly's living spaces, capturing video data in real time, which is then compressed and sent to a server via the home network.

[0245] 2. Video preprocessing

[0246] server

[0247] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, and noise reduction. At this stage, the video data is converted into a format suitable for the analysis engine.

[0248] 3. Learning about daily life patterns

[0249] server

[0250] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[0251] 4. Real-time analysis and anomaly detection

[0252] server

[0253] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[0254] 5. Use of Emotion Engine

[0255] server

[0256] The server extracts the user's facial expressions from the video data and uses an emotion engine to analyze their emotional state in real time, resulting in emotional states such as joy, sadness, anger, surprise, and fear.

[0257] 6. Consideration of abnormality notification and emotional data

[0258] server

[0259] If an abnormality is detected, the server generates and sends a notification message taking into account the analysis results of the emotion engine. For example, if the emotion data indicates "fear" or "sadness" when an abnormality is detected, the urgency of the notification will be increased. The content of the notification will also be adjusted based on the user's emotional state, prompting the user to take a more appropriate action.

[0260] 7. Gather feedback and improve the model

[0261] User

[0262] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[0263] server

[0264] The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the model, reducing false positives and improving anomaly detection.

[0265] Specific examples

[0266] Example 1: Fall detection and emotional response

[0267] server

[0268] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion and facial expressions of "fear" or "pain" are detected. The server immediately sends a notification to the family saying, "A fall has been detected in the living room, and a facial expression of fear has been confirmed."

[0269] Example 2: Detection of abnormal activity times and emotion assessment

[0270] server

[0271] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, and if "sadness" or "no facial expression" is continuously detected during that time, the server will send a notification to the family stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification will also include a message requesting feedback to confirm the actual situation.

[0272] This system improves the accuracy of anomaly detection by combining an emotion engine, further enhancing the safety of elderly people. It also encourages families and medical institutions to take quicker and more appropriate action.

[0273] The processing flow will be explained below.

[0274] Step 1:

[0275] Device (Web camera)

[0276] The webcam monitors the elderly's living space in real time and captures video data, which is compressed and transmitted to a server over the home network.

[0277] Step 2:

[0278] server

[0279] The server receives the video data sent from the webcam, performs error checking on the received data, and requests retransmission if necessary.

[0280] Step 3:

[0281] server

[0282] The server decodes the received video data and converts it into a format that can be easily processed by the analysis engine. Pre-processing includes data normalization, noise removal, and contour extraction.

[0283] Step 4:

[0284] server

[0285] The server stores the pre-processed video data in a database, which includes not only the video frames but also time information and activity metadata.

[0286] Step 5:

[0287] server

[0288] The server uses the data stored in the database to run machine learning algorithms and model the elderly's daily life patterns.In the initial stage, data on the elderly's daily life patterns is collected over a certain period of time.

[0289] Step 6:

[0290] server

[0291] The system analyzes video data received in real time and compares it with learned lifestyle patterns to detect the subject's position, movements, and gestures such as falls.

[0292] Step 7:

[0293] server

[0294] In parallel with the real-time analysis, the emotion engine extracts facial expressions from the video data and analyzes emotions. Emotion recognition involves facial expression analysis using machine learning models.

[0295] Step 8:

[0296] server

[0297] The behavioral and emotional data obtained as a result of the analysis are integrated to perform anomaly detection, taking into account the analysis results of consecutive frames and data from other IoT devices in the home network.

[0298] Step 9:

[0299] server

[0300] If an abnormality is detected, a notification message is generated. This message includes the type of abnormality, the time of occurrence, location information, and emotional state data. If the emotional state is "fear" or "sadness," the message is made more urgent.

[0301] Step 10:

[0302] server

[0303] The generated notification message is sent to the designated recipients, such as family members or medical institutions, via push notification, email, SMS, etc.

[0304] Step 11:

[0305] User

[0306] Upon receiving the notification, family members or medical institutions will assess the situation and take prompt action if necessary, such as contacting the affected person to confirm the situation and arranging for rescue if necessary.

[0307] Step 12:

[0308] User

[0309] Feedback on the anomaly detection is sent to the server, including the actual situation, the appropriateness of the anomaly, and whether there were any false positives.

[0310] Step 13:

[0311] server

[0312] Based on the collected feedback, the machine learning model and sentiment engine are retrained, which improves the accuracy of the model, reducing false positives and improving anomaly detection accuracy.

[0313] Example 2

[0314] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] In elderly care systems, it is important to monitor the daily lives of the elderly and detect abnormalities, but conventional systems have difficulty in accurately detecting abnormalities and have a problem of frequent false positives.In addition to detecting abnormalities, it is also necessary to take into account the emotional state of the elderly and provide appropriate notifications.However, conventional systems do not perform emotional analysis, so they have the problem of being unable to respond quickly to highly urgent situations.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0317] In this invention, the server includes a video acquisition means, a video data preprocessing means, a learning means for learning the preprocessed video data and modeling a lifestyle pattern, a facial expression analysis means for evaluating an emotional state, a real-time video data analysis means for detecting anomalies, a means for generating and sending a notification message based on the anomaly detection result and the emotion analysis result, and a means for collecting feedback and improving the learning model. This enables highly accurate anomaly detection that takes into account the lifestyle patterns and emotions of the elderly person and appropriate adjustment of the urgency of notifications.

[0318] "Video acquisition means" refers to a means of capturing real-time video data in the elderly person's living space using a device such as a webcam.

[0319] "Means for preprocessing video data" refers to the means for decoding, normalizing, extracting contours, removing noise, etc. from compressed and transmitted video data, and converting it into a format suitable for analysis.

[0320] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for modeling the daily life patterns of elderly people using a machine learning algorithm, using preprocessed video data collected over a certain period of time.

[0321] "Means for analyzing video data in real time and detecting abnormalities" refers to means for comparing video data acquired in real time with past lifestyle pattern models to detect abnormal behavior that deviates from daily life patterns.

[0322] The "means for analyzing facial expressions and assessing emotional states" is a means for analyzing facial expressions extracted from video data and determining emotional states such as joy, sadness, anger, etc.

[0323] "Means for generating and sending notification messages based on the results of anomaly detection and emotion analysis" refers to means for generating notification messages with appropriate urgency and content based on the results of anomaly detection and emotion analysis, and sending them to family members or medical institutions.

[0324] "Means for collecting feedback and improving the learning model" refers to collecting feedback from families and medical institutions, and using that feedback to retrain the machine learning model and emotion engine, thereby improving the accuracy of the system.

[0325] MODE FOR CARRYING OUT THE INVENTION

[0326] This invention relates to an elderly care service system that combines functions for image acquisition, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection with an emotion engine that recognizes the user's emotions. Each element of the system and its configuration method are described in detail below.

[0327] 1. Video acquisition

[0328] Device (Web camera)

[0329] The webcam, which serves as the terminal, monitors the elderly's living space 24 hours a day and captures video data in real time. The captured video data is compressed for communication efficiency. For example, the data is compressed in MPEG-4 format and sent to a server via the home network.

[0330] 2. Video preprocessing

[0331] server

[0332] The server receives the video data sent from the webcam and performs the following pre-processing:

[0333] Decoding: Reconstructing the received data into the original video data.

[0334] Normalization: Standardize frames to make them easier to analyze.

[0335] Contour extraction: Applying algorithms to clarify the contours of objects and people.

[0336] Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[0337] 3. Learning about daily life patterns

[0338] server

[0339] Preprocessed video data is collected over a period of time, and a machine learning algorithm is used to model the elderly person's daily life patterns. This allows normal behavioral patterns to be learned and used for anomaly detection. For example, the system learns that an elderly person wakes up at 7:00 every morning.

[0340] 4. Real-time analysis and anomaly detection

[0341] server

[0342] The system analyzes video data sent in real time and compares it with the learning model. Examples of anomaly detection include a lack of movement at normal wake-up time or a sudden fall.

[0343] 5. Use of Emotion Engine

[0344] server

[0345] The system extracts the user's facial expressions from the video data and uses an emotion engine to evaluate their emotional state in real time. Analysis identifies emotional states such as joy, sadness, anger, surprise, and fear.

[0346] 6. Consideration of abnormality notification and emotional data

[0347] server

[0348] If an abnormality is detected, a notification message is generated based on the analysis results of the emotion engine and sent to family members or medical institutions. For example, if emotions such as "fear" or "sadness" are confirmed when an abnormality is detected, a notification with a higher level of urgency will be sent.

[0349] 7. Gather feedback and improve the model

[0350] User

[0351] Upon receiving the notification, family members or medical institutions will check the situation and take necessary measures, such as calling the elderly person to check on their condition.

[0352] server

[0353] Feedback from family members and medical institutions is collected and used to retrain the lifestyle pattern model and emotion engine, improving the system's accuracy and reducing false positives.

[0354] Specific examples

[0355] Example 1: Fall detection and emotional response

[0356] server

[0357] If an elderly person falls in the living room, the webcam captures the video and sends it to the server. The server preprocesses the video and analyzes it in real time. As a result of this analysis, a fall and "fear" are detected at the same time. In this case, a notification is sent to the family saying, "A fall has been detected in the living room and a fearful expression has been confirmed."

[0358] Example 2: Detection of abnormal activity times and emotion assessment

[0359] server

[0360] This scenario assumes that an elderly person normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares this with learned lifestyle patterns based on past data and detects abnormalities. If the person remains motionless for a long period of time and detects "sadness" or "expressionlessness," it sends a notification to the family member stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification also includes a message requesting feedback to check the situation.

[0361] Example prompt

[0362] Examples of prompts used in generative AI models are as follows:

[0363] "Please explain the elderly monitoring system. This system monitors the daily life of the elderly, learns their life patterns, performs real-time analysis, detects anomalies, uses an emotion engine, notifies users of abnormalities, and collects feedback. Please explain the specific processing steps and the specific actions that are performed within them."

[0364] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0365] Processing steps of the elderly monitoring system

[0366] Step 1: Capture footage

[0367] Device (Web camera)

[0368] Input: A webcam captures real-time video data from the elderly person's living space.

[0369] How it works: The webcam monitors the elderly's living space 24 hours a day, providing detailed footage with a 360-degree field of view.

[0370] Output: Captured real-time video data is generated.

[0371] Step 2: Data compression and transmission

[0372] Device (Web camera)

[0373] Input: Captured video data.

[0374] Specific operation: The captured video data is compressed in a format such as MPEG-4, which reduces the data size and reduces the network load.

[0375] Output: The compressed video data is sent to a server via the home network.

[0376] Step 3: Data reception and decoding

[0377] server

[0378] Input: Compressed video data transmitted over a home network.

[0379] Specific operation: The server receives the video data and decodes the compressed data to restore it to its original video format.

[0380] Output: The original decoded video data is generated.

[0381] Step 4: Preprocessing the video data

[0382] server

[0383] Input: The decoded original video data.

[0384] Specific behavior:

[0385] 1. Normalization: Standardize each frame to a standard format.

[0386] 2. Contour extraction: Clarifying the contours of objects and people.

[0387] 3. Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[0388] Output: Pre-processed video data is generated.

[0389] Step 5: Learning Life Patterns

[0390] server

[0391] Input: Preprocessed video data.

[0392] How it works: The server collects pre-processed video data and uses machine learning algorithms to model the elderly person's daily life patterns, such as specific behaviors (such as waking up and eating) at the same time every day.

[0393] Output: Model data that learns the lifestyle patterns of elderly people is generated.

[0394] Step 6: Analyze real-time data and detect anomalies

[0395] server

[0396] Input: Video data and learning model sent in real time.

[0397] Specific operation: The server analyzes real-time video data and compares it with a pre-trained lifestyle pattern model. If abnormal behavior (such as prolonged inactivity or a sudden fall) is detected, the information is recorded as an abnormality.

[0398] Output: The anomaly detection results are generated.

[0399] Step 7: Facial expression analysis and emotion assessment

[0400] server

[0401] Input: Real-time video data.

[0402] Specific operation: Facial expressions are extracted from video data and the emotional state is evaluated in real time using an emotion engine. Through analysis, emotional states such as joy, sadness, anger, surprise, and fear are identified.

[0403] Output: The result of the sentiment analysis is generated.

[0404] Step 8: Generate and send a notification message

[0405] server

[0406] Input: Anomaly detection results and sentiment analysis results.

[0407] Specific operation: The server generates a notification message based on the results of anomaly detection and emotion analysis. For example, it might say, "A fall was detected in the living room and a frightened expression was observed."

[0408] Output: The generated notification message is sent to the family or healthcare provider.

[0409] Step 9: Assess the situation and respond

[0410] User

[0411] Input: Notification message from the server.

[0412] Specific actions: Family members or medical institutions will receive a notification, check the situation, and take action such as contacting the elderly person and arranging rescue if necessary.

[0413] Output: The situation is checked and action is taken.

[0414] Step 10: Gather feedback and improve the model

[0415] server

[0416] Input: Feedback from family or healthcare provider.

[0417] How it works: The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the system and reducing false positives.

[0418] Output: An improved learning model and emotion engine are generated.

[0419] (Application example 2)

[0420] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] The main function of conventional elderly monitoring systems was to detect abnormalities by analyzing video data, but the accuracy of the abnormality detection was not always high, and appropriate responses were sometimes delayed.In addition, because the abnormality detection and notification did not take into account the emotions of the elderly, it was difficult to accurately determine the urgency of the situation.This posed the issue of not being able to completely ensure the safety and security of the elderly.

[0422] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a video acquisition means, a means for preprocessing video data, a means for learning from the preprocessed video data and modeling lifestyle patterns, a means for analyzing the video data in real time and detecting anomalies, a means for recognizing the user's emotions and considering them together with the anomaly detection results, a means for generating and transmitting a notification message based on the anomaly detection results and emotion data, and a means for collecting feedback and improving the learning model. This makes it possible to ensure the safety of elderly people in real time and quickly take appropriate measures according to the level of urgency.

[0423] The "video acquisition means" is a device that captures the subject's behavior and facial expressions in real time and acquires them as video data.

[0424] The "means for preprocessing video data" refers to a processing device or software that performs decoding, noise removal, frame normalization, etc. to convert the acquired video data into a format that is easy to analyze.

[0425] The "means for modeling lifestyle patterns" is a system that uses preprocessed video data to learn the subject's daily behavioral patterns using a machine learning algorithm and saves them as a model.

[0426] The "means for detecting anomalies" is an algorithm that compares video data acquired in real time with a learning model to detect behavior that deviates from normal life patterns.

[0427] The "means for recognizing user emotions" is an engine that analyzes the subject's facial expressions from the acquired video data and evaluates their emotional state, such as joy, sadness, or fear, in real time.

[0428] The "means for generating and transmitting a notification message based on the abnormality detection result and emotional data" is a system that creates a notification message that takes into account the detected abnormality and the user's emotional data, and transmits it to the home or medical personnel.

[0429] "Means for collecting feedback and improving the learning model" refers to a mechanism for collecting feedback from households and medical institutions regarding the results of anomaly detection and notification, and using that data to improve the accuracy of the machine learning model.

[0430] The present invention provides specific means and methods for effectively realizing elderly care. The system uses smart glasses as a terminal to monitor the user's behavior and emotions in real time, and sends appropriate notifications to family members and medical personnel when abnormalities occur.

[0431] Hardware Configuration

[0432] The smart glasses are equipped with a built-in camera that constantly captures the user's actions. The video data undergoes initial pre-processing by the smart glasses' on-device processor.

[0433] Software Configuration

[0434] The server receives the data sent from the terminal and performs the following processing.

[0435] 1. Video pre-processing:

[0436] The server decodes the video data sent from the smart glasses, performs noise reduction and frame normalization, and converts it into a format suitable for analysis.

[0437] 2. Learning Life Patterns:

[0438] Using the preprocessed data, the server learns the elderly person's daily life patterns using machine learning algorithms (e.g., TensorFlow / Keras). In the initial stage, data is collected over several weeks to create a model of their normal behavior.

[0439] 3. Real-time analysis and anomaly detection:

[0440] The server compares the video data acquired in real time with the trained model to detect anomalies, and if an anomaly is found, the information is processed immediately.

[0441] 4. Emotion analysis:

[0442] Facial expressions are extracted from the video data, and an emotion analysis engine (e.g., TensorFlow model) is used to analyze the user's emotional state in real time. The resulting emotion data is included in the anomaly detection results.

[0443] 5. Abnormal notification:

[0444] If an anomaly is detected, the server generates a notification message taking into account the user's emotional data. For example, if the user shows a fearful expression, the notification will be sent with a high level of urgency. The notification will be sent via email or a dedicated application.

[0445] 6. Gather feedback and improve the model:

[0446] Family members and medical institutions receive notifications, check the situation, respond, and then provide feedback. The server uses this feedback to retrain the learning model and improve detection accuracy.

[0447] Specific examples

[0448] Suppose an elderly person is wearing smart glasses and going about their daily life at home. Suddenly, they fall, and the smart glasses capture the video. The server immediately analyzes the data and detects the abnormality. At the same time, the emotion analysis engine recognizes that the user has an expression of "fear" on their face. This causes the server to send a high-level emergency notification to their family, urging them to take prompt action. This system transmits highly accurate information to their family and medical institutions in real time, ensuring the safety of the elderly.

[0449] Example prompts for generative AI models

[0450] 1. Capture footage using a webcam or smart glasses.

[0451] 2. Preprocess the video data to remove noise and unnecessary information.

[0452] 3. Use a machine learning model to detect anomalies by comparing learned patterns of daily life with the video data.

[0453] 4. Analyze facial expression data in real time to determine emotional states.

[0454] 5. If an abnormality is detected, adjust the urgency of the notification message to be sent to family members, taking into account the emotional data.

[0455] 6. Collect feedback from families and healthcare providers to retrain and improve your machine learning models.

[0456] The detailed configuration and processing means of the present invention make it possible to provide a highly accurate monitoring system for ensuring the safety of elderly people.

[0457] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0458] Step 1:

[0459] The device (smart glasses) uses a built-in camera to capture the elderly person's behavior and facial expressions in real time. Real-time video data is acquired as input, and this video data is temporarily stored in the device as output.

[0460] Step 2:

[0461] The device preprocesses the captured video data, specifically decoding, denoising, and frame normalizing the video data. The input is the captured video data, and the output is the preprocessed video data.

[0462] Step 3:

[0463] The pre-processed video data is sent to the server through the home network. The input is the pre-processed video data, and the output is obtained by sending this data to the server.

[0464] Step 4:

[0465] The server receives the preprocessed video data and uses a machine learning algorithm to learn the elderly's daily life patterns. The input is the preprocessed video data, and the output is the learned lifestyle pattern model.

[0466] Step 5:

[0467] The server analyzes newly received video data in real time and compares it with the trained model to detect anomalies. The input is the real-time video data and the trained lifestyle pattern model, and the output is the anomaly detection results.

[0468] Step 6:

[0469] The server extracts the user's facial expressions from the video data and analyzes their emotional state in real time using an emotion analysis engine. The input is the video data, and the output is the analysis result of the emotional state.

[0470] Step 7:

[0471] When an anomaly is detected, the server takes into account the emotion analysis results and generates an anomaly notification message with a set urgency level. The input is the anomaly detection result and the emotion analysis result, and the output is the notification message.

[0472] Step 8:

[0473] The server sends the generated notification message to the terminal of the home or medical personnel. The input is the notification message, and the output is the terminal of the home or medical personnel that receives the notification.

[0474] Step 9:

[0475] Users (family members or medical professionals) receive notifications, check and respond, and provide feedback to the server. The input is the notification message, and the output is feedback information.

[0476] Step 10:

[0477] The server uses the received feedback to retrain the learning model and improve the accuracy of anomaly detection. The input is the feedback information, and the output is an improved learning model.

[0478] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0479] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0480] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0481] [Second embodiment]

[0482] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0483] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0484] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0485] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0486] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0487] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0488] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0489] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0490] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0491] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0492] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0493] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0494] MODE FOR CARRYING OUT THE INVENTION

[0495] The system of the present invention integrates the functions of video acquisition, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. Specifically, this system is as follows.

[0496] 1. Video acquisition

[0497] Device (Web camera)

[0498] The webcam installed as a terminal constantly monitors the elderly's living space and captures video data in real time, which is then sent to a server via a network.

[0499] 2. Video preprocessing

[0500] server

[0501] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, noise removal, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[0502] 3. Learning about daily life patterns

[0503] server

[0504] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[0505] 4. Real-time analysis and anomaly detection

[0506] server

[0507] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[0508] 5. Abnormality notification

[0509] server

[0510] If an abnormality is detected, the server generates a notification message and sends it to designated recipients, such as family members or medical institutions. The notification includes details such as the type of abnormality, the time of occurrence, and location information. Notifications can be sent via push notification, email, SMS, etc.

[0511] 6. Gather feedback and improve the model

[0512] User

[0513] The user who receives the notification sends feedback about the anomaly back to the server, including the actual situation, whether there was a false positive, and whether the response was appropriate.

[0514] server

[0515] The server uses the collected feedback to retrain the machine learning model and make improvements to reduce false positives and improve detection accuracy, making the system more reliable over time.

[0516] Specific examples

[0517] Example 1: Fall detection

[0518] server

[0519] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion is detected and recognized as an abnormality. The server immediately sends a notification to the family member saying, "A fall has been detected in the living room."

[0520] Example 2: Detecting abnormal activity times

[0521] server

[0522] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, the server sends a notification to the family saying, "No activity was observed by 8:00 AM." The notification also includes a message requesting feedback to confirm the actual situation.

[0523] As described above, this invention is a system that integrates the functions of image acquisition, preprocessing, learning, analysis, notification, and feedback collection to monitor the lives of the elderly in real time and enable rapid response. This system greatly improves the safety of the elderly and allows their families and medical institutions to monitor them with peace of mind.

[0524] The processing flow will be explained below.

[0525] Step 1:

[0526] Device (Web camera)

[0527] The webcam captures video in real time and generates video data that is compressed and sent over the home network to a server.

[0528] Step 2:

[0529] server

[0530] The server receives the video data sent from the webcam, performs an error check to see if there are any packet losses, and requests packet retransmission if necessary.

[0531] Step 3:

[0532] server

[0533] The received video data is decoded and converted into a format that can be easily processed by the analysis engine. This preprocessing includes data normalization, noise removal, and contour extraction.

[0534] Step 4:

[0535] server

[0536] The preprocessed data is stored in a database, which includes not only the video frames but also time information and activity metadata.

[0537] Step 5:

[0538] server

[0539] The machine learning model is trained using data stored in the database. In the initial stage, the model is built based on data from a certain period of time and learns the lifestyle patterns of elderly people.

[0540] Step 6:

[0541] server

[0542] The system analyzes video data received in real time and compares it with a learning model to detect the subject's position, movement, and gestures such as falls.

[0543] Step 7:

[0544] server

[0545] It combines the analysis results of consecutive frames with data from other IoT devices in the home network to detect anomalies, for example, when it receives fall data from a smart sensor, it compares it with the corresponding video analysis results.

[0546] Step 8:

[0547] server

[0548] When an anomaly is detected, a notification message is generated and sent to the specified notification recipients, containing details such as the type of anomaly, the time of occurrence, and location information.

[0549] Step 9:

[0550] User

[0551] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[0552] Step 10:

[0553] User

[0554] Feedback regarding the anomaly notification is sent back to the server, including the actual situation and whether or not there was a false positive.

[0555] Step 11:

[0556] server

[0557] The machine learning model is retrained based on the collected feedback, which improves the model's accuracy, reducing false positives and improving anomaly detection.

[0558] Example 1

[0559] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0560] In elderly care services, rapid detection and response to abnormal behavior is important. However, conventional systems have issues with monitoring accuracy and real-time performance, resulting in frequent false positives. Furthermore, the process of feedback collection and model improvement is not well coordinated, resulting in a lack of system reliability. The present invention aims to solve these problems and improve the safety of elderly people.

[0561] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0562] In this invention, the server includes means for constantly capturing video for monitoring the elderly person's living space, means for decoding the received video data and dividing it into frames, means for normalizing the frames and performing contour extraction and noise removal, means for learning the preprocessed video data using a machine learning algorithm to model lifestyle patterns, means for analyzing the video data in real time and comparing it with an existing learning model to detect abnormal behavior, means for generating a notification message including the type, time of occurrence, and location information of the detected abnormal behavior and sending it to a specified notification destination, and means for collecting feedback from users who receive the notification and re-training and improving the learning model. This enables rapid and accurate detection and notification of abnormal behavior, and makes it possible to improve the accuracy and reliability of the system based on the feedback.

[0563] "Video acquisition means" refers to devices or systems that monitor the elderly person's living space and capture video images at all times.

[0564] "Decoding" refers to the process of converting received video data into a frame-by-frame format that can be analyzed.

[0565] The term "means for dividing into frames" refers to a process for dividing video data into individual frames and subjecting each frame to analysis.

[0566] "Normalization" refers to the process of standardizing video data frames to a specific size and format to improve the accuracy of analysis.

[0567] "Contour extraction" refers to the process of extracting the main shape information from a frame of video data and emphasizing the necessary information.

[0568] "Noise reduction" refers to the process of removing unnecessary information from video data to improve the quality of the data.

[0569] "Machine learning algorithm" refers to a mathematical model and its execution program that learns patterns from data and makes predictions and classifications.

[0570] "Modeling lifestyle patterns" refers to learning the daily behavior of elderly people and saving those behavior patterns as a mathematical model.

[0571] "Means for analyzing in real time" refers to a method for instantly analyzing currently received video data and instantly detecting abnormal behavior.

[0572] "Abnormal behavior" refers to behavior that deviates from the elderly person's daily behavior patterns, including, for example, prolonged immobility or sudden falls.

[0573] "Notification message" refers to a message that includes information about detected abnormal behavior (such as type, time of occurrence, location information, etc.) and is sent to a designated recipient.

[0574] "Feedback" refers to information provided by users who receive notifications about the actual situation, whether there were any false positives, and the appropriateness of the response.

[0575] "Retraining" refers to the process of updating an existing machine learning model based on new data and feedback to improve its accuracy and reliability.

[0576] MODE FOR CARRYING OUT THE INVENTION

[0577] The system of the present invention integrates the functions of video capture, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. To implement this system, the following specific hardware and software are used.

[0578] Hardware

[0579] Terminal (Web camera): Installed to monitor the living space of the elderly, it captures video 24 hours a day and transmits it to a server via the network.

[0580] software

[0581] Decoding and pre-processing: After receiving the video data from the webcam, the server decodes it using the FFmpeg library and splits the data into frames. Then, it normalizes the frames, extracts contours, and removes noise using the OpenCV library.

[0582] Machine learning: Using the preprocessed data, machine learning libraries such as TensorFlow and PyTorch are used on the server to learn the lifestyle patterns, thereby modeling the daily behavior of elderly people.

[0583] Real-time analysis and anomaly detection: The server uses analysis engines such as YOLOv5 and OpenPose to analyze video data in real time and compare it with existing learning models to detect abnormal behavior.

[0584] Notification: If an anomaly is detected, the server uses Twilio or Firebase Cloud Messaging (FCM) to generate a notification message containing the type of abnormal behavior, the time of occurrence, and location information, and sends it to the specified notification recipient.

[0585] Feedback collection and model improvement: After receiving the notification, users provide feedback through the application. The server receives this feedback, stores it in a database for analysis, and improves the model by retraining it.

[0586] Specific examples

[0587] Specific operation of fall detection

[0588] If an elderly person falls in the living room, a webcam captures the moment and sends the video data to a server. The server preprocesses the video data and analyzes it in real time. If the analysis detects a fall, it is recorded as an abnormal behavior and the server sends a notification message to the family saying, "A fall has been detected in the living room."

[0589] Specific operation of abnormal activity time detection

[0590] If an elderly person normally wakes up at 7:00 a.m. but does not wake up by 8:00 a.m., the server will compare the current data with the lifestyle patterns learned from past data to detect anomalies. If no activity is observed for a long period of time, a notification will be sent to the family stating, "No activity was observed by 8:00 a.m." The notification will also include a message requesting feedback to confirm the actual situation.

[0591] Prompt Sentence Examples

[0592] "In an elderly care system, please explain in detail the steps by which a webcam detects an elderly person's fall, and the server preprocesses and analyzes the video data and notifies the family."

[0593] This invention provides a system that can monitor the lives of elderly people in real time, quickly detect abnormal behavior, and notify them. This system is highly reliable because it continuously improves its model based on user feedback, thereby improving the safety of elderly people.

[0594] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0595] Step 1:

[0596] Video Acquisition

[0597] The device (webcam) starts up and monitors the elderly person's living space.

[0598] How it works: The webcam captures video streams 24 / 7, obtaining video data in real time.

[0599] Input: Real-time video footage of an elderly person's living space.

[0600] Output: Captured video data (video stream).

[0601] Step 2:

[0602] Video data preprocessing

[0603] The server receives the video data sent from the web camera.

[0604] What happens: The server uses the FFmpeg library to decode the video stream and grab each frame.

[0605] Input: Video data (video stream) sent from the webcam.

[0606] Output: Decoded video frames.

[0607] Step 3:

[0608] Frame normalization and denoising

[0609] The server normalizes each frame and performs contour extraction and noise removal.

[0610] Specific operation: Using the OpenCV library, the frame size is standardized to 1280x720 pixels, converted to grayscale, and edge detection and noise removal are performed using the Canny method.

[0611] Input: Decoded video frames.

[0612] Output: Normalized and denoised frames.

[0613] Step 4:

[0614] Learning life patterns

[0615] The server inputs the pre-processed video frames into a machine learning algorithm.

[0616] Specific operation: Using TensorFlow and PyTorch, the system learns and models the daily behavioral patterns of elderly people based on data from a certain period of time.

[0617] Input: Preprocessed video frames.

[0618] Output: Learned behavioral pattern model.

[0619] Step 5:

[0620] Real-time analysis and anomaly detection

[0621] The server analyzes the video data sent from the webcam in real time.

[0622] Specific operation: Utilizing YOLOv5 and OpenPose, we compare with the learning model and detect behavior that deviates from everyday life patterns (abnormal behavior).

[0623] Input: Real-time video data, learning model.

[0624] Output: A log of the anomalous behavior detected.

[0625] Step 6:

[0626] Abnormal notification

[0627] The server generates a notification message if an abnormality is detected.

[0628] Specific operation: Using Twilio or Firebase Cloud Messaging (FCM), a notification containing the type of anomaly, the time of occurrence, and location information is sent to the specified destination.

[0629] Input: The log in which the anomalous behavior was detected.

[0630] Output: Notification message sent to family and healthcare providers.

[0631] Step 7:

[0632] Gathering feedback and improving the model

[0633] The user receives a notification and sends feedback back to the server.

[0634] Specific operation: The user reports the actual situation regarding the anomaly and whether there were any false positives through the application.

[0635] Input: User feedback.

[0636] Output: Feedback is collected and stored in a database.

[0637] Step 8:

[0638] Improving the model through retraining

[0639] The server retrains and improves the machine learning model based on the feedback collected.

[0640] How it works: Using TensorFlow, we incorporate feedback data to retrain existing models, improving detection accuracy and reliability.

[0641] Input: User feedback data, existing learning model.

[0642] Output: An improved learning model.

[0643] This allows the system to monitor elderly people in real time, quickly and accurately detect and notify abnormal behavior, and constantly improve the model based on user feedback.

[0644] (Application example 1)

[0645] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] While conventional elderly monitoring systems are effective, especially in fixed living environments, they are difficult to monitor while moving, and passenger safety monitoring has not been fully realized, especially in self-driving vehicles. Furthermore, while there is a need for a function that can respond quickly and appropriately when a passenger experiences an abnormality, current systems often respond slowly. Therefore, there is an urgent need to develop a system that can monitor passenger safety in self-driving vehicles and provide immediate notification in the event of an abnormality.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0648] In this invention, the server includes a video acquisition means, a means for preprocessing the video data, a means for learning the preprocessed video data and modeling a lifestyle pattern, a means for analyzing the video data in real time and detecting anomalies, a means for sending the anomaly detection results to a notification destination, a means for collecting feedback and improving the learning model, a means for monitoring the status of passengers in the autonomous vehicle, and a means for detecting and notifying passenger anomalies in real time. This makes it possible to monitor the safety of passengers in the autonomous vehicle and to quickly notify passengers in the event of an anomaly.

[0649] "Video acquisition means" means a means for capturing video data of passengers in real time using a camera within the autonomous vehicle.

[0650] The "means for preprocessing video data" refers to a means for performing processes such as adjusting resolution and removing noise in order to convert the acquired video data into an analyzable format.

[0651] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for using preprocessed video data to learn passengers' normal behavioral patterns through a machine learning algorithm and modeling them.

[0652] "Means for analyzing video data in real time and detecting abnormalities" refers to a means for detecting abnormal behavior by comparing video data acquired in real time with past learning models.

[0653] The "means for transmitting an abnormality detection result to a notification destination" is a means for transmitting a notification message to a pre-specified contact point when an abnormality is detected.

[0654] The "means for collecting feedback and improving the learning model" refers to a means for collecting feedback from the recipients of notifications and retraining and improving the learning model based on that data.

[0655] "Means for monitoring the status of passengers inside an autonomous vehicle" refers to means for constantly monitoring the behavior and status of passengers using cameras and sensors installed inside the vehicle.

[0656] "Means for detecting and notifying passenger abnormalities in real time" refers to a means for detecting abnormal passenger behavior (e.g., sudden falls or prolonged periods of inactivity) in real time and immediately notifying the relevant parties.

[0657] The system for realizing this invention monitors the status of passengers in an autonomous vehicle in real time and promptly notifies them if an abnormality is detected. This system is specifically implemented by combining the following multiple means.

[0658] 1. Video acquisition method

[0659] The server captures passenger video data using cameras installed inside the autonomous vehicle, and the real-time video data is transmitted to the server.

[0660] 2. Video data preprocessing methods

[0661] The server receives the video data sent from the camera and performs pre-processing, which includes decoding the video data, adjusting the resolution, removing noise, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[0662] 3. A means of learning about life patterns

[0663] The server uses the preprocessed video data to model passengers' usual behavior patterns using machine learning algorithms, which are then saved as a learning model using machine learning frameworks such as TensorFlow and PyTorch.

[0664] 4. Real-time analysis and anomaly detection methods

[0665] The server analyzes the video data transmitted in real time and compares it with the learning model. If the analysis engine detects an abnormality (e.g., a sudden fall or prolonged inactivity), it records it. This anomaly detection uses a pre-trained generative AI model.

[0666] 5. Abnormality notification means

[0667] If an abnormality is detected, the server generates a notification message and sends it to the designated notification recipient (for example, family or a medical institution). This notification includes the type of abnormality, the time of occurrence, location information, etc. Notification methods include push notification, email, and SMS.

[0668] 6. Feedback Collection Methods

[0669] The user who receives the notification sends feedback about the situation to the server, including the actual situation and whether there were any false positives.

[0670] 7. Ways to improve the learning model

[0671] The server uses the collected feedback to retrain the machine learning model, reducing false positives and improving detection accuracy, again using machine learning frameworks such as TensorFlow or PyTorch.

[0672] Example 1: Fall detection

[0673] The server detects when a passenger in an autonomous vehicle suddenly falls. The camera captures the moment of the fall, and the preprocessed video data is sent to the server. Analysis detects the falling motion and recognizes it as an abnormality. The server then sends an abnormality notification to the specified notification destination.

[0674] Example 2: Detecting long periods of inactivity

[0675] For example, if a passenger who normally does not stay seated for more than 10 minutes remains motionless for more than 30 minutes, the server will detect the abnormality by comparing it with past data and send a notification such as, "The period of inactivity has exceeded normal."

[0676] Example prompt sentence:

[0677] "We will build a system that will detect abnormal behavior from camera footage and send a notification to a specified email address when an abnormality is detected. Specifically, we will detect when a passenger in an autonomous vehicle loses consciousness and continues to sit there, and send an email notification on the spot."

[0678] In this way, it is possible to monitor the safety of passengers in autonomous vehicles and respond quickly.

[0679] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0680] Step 1:

[0681] The server acquires video data in real time from the terminal (camera inside the autonomous vehicle). This data is a raw video stream that is sent from the terminal to the server. Specifically, the camera's video data is periodically captured and sent to the server via the network. The input data is the camera video, and the output is the video data sent to the server.

[0682] Step 2:

[0683] The server preprocesses the received video data, specifically decoding, adjusting resolution, and removing noise. The preprocessed video data is then converted into a format suitable for analysis. The input data is raw video data, and the output is preprocessed video data.

[0684] Step 3:

[0685] The server uses preprocessed video data to learn and model passengers' usual behavioral patterns using machine learning algorithms. Specifically, it collects data over a certain period of time and uses TensorFlow and PyTorch to learn lifestyle patterns. The input data is the preprocessed video data, and the output is a learning model.

[0686] Step 4:

[0687] The server analyzes the video data transmitted in real time and compares it with the learning model to detect anomalies. Specifically, the video data is input into a generative AI model to detect abnormal behavior. The input data are the video data transmitted in real time and the learning model, and the output is the anomaly detection result.

[0688] Step 5:

[0689] If an anomaly is detected, the server generates a notification message and sends it to a designated recipient (for example, family or a medical institution). Specifically, the notification includes details such as the type of anomaly, the time of occurrence, and location information, and is sent via push notification, email, or SMS. The input data is the anomaly detection result, and the output is the notification message.

[0690] Step 6:

[0691] The user who receives the notification sends back feedback. Specifically, they click a link or button included in the notification to report the actual situation, whether there were any false positives, etc. to the server. The input data is the user feedback, and the output is the feedback sent to the server.

[0692] Step 7:

[0693] The server uses the collected feedback to retrain the machine learning model and improve its detection accuracy. Specifically, it takes the feedback as a dataset and retrains the model using TensorFlow or PyTorch. The input data is the feedback and the existing model, and the output is a new, improved trained model.

[0694] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0695] MODE FOR CARRYING OUT THE INVENTION

[0696] This invention combines functions for an elderly care service, including video capture, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection, with an emotion engine that recognizes the user's emotions. This provides a system that takes emotion data into account in anomaly detection and realizes highly accurate and appropriate responses.

[0697] 1. Video acquisition

[0698] Device (Web camera)

[0699] The webcams used as terminals constantly monitor the elderly's living spaces, capturing video data in real time, which is then compressed and sent to a server via the home network.

[0700] 2. Video preprocessing

[0701] server

[0702] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, and noise reduction. At this stage, the video data is converted into a format suitable for the analysis engine.

[0703] 3. Learning about daily life patterns

[0704] server

[0705] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[0706] 4. Real-time analysis and anomaly detection

[0707] server

[0708] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[0709] 5. Use of Emotion Engine

[0710] server

[0711] The server extracts the user's facial expressions from the video data and uses an emotion engine to analyze their emotional state in real time, resulting in emotional states such as joy, sadness, anger, surprise, and fear.

[0712] 6. Consideration of abnormality notification and emotional data

[0713] server

[0714] If an abnormality is detected, the server generates and sends a notification message taking into account the analysis results of the emotion engine. For example, if the emotion data indicates "fear" or "sadness" when an abnormality is detected, the urgency of the notification will be increased. The content of the notification will also be adjusted based on the user's emotional state, prompting the user to take a more appropriate action.

[0715] 7. Gather feedback and improve the model

[0716] User

[0717] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[0718] server

[0719] The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the model, reducing false positives and improving anomaly detection.

[0720] Specific examples

[0721] Example 1: Fall detection and emotional response

[0722] server

[0723] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion and facial expressions of "fear" or "pain" are detected. The server immediately sends a notification to the family saying, "A fall has been detected in the living room, and a facial expression of fear has been confirmed."

[0724] Example 2: Detection of abnormal activity times and emotion assessment

[0725] server

[0726] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, and if "sadness" or "no facial expression" is continuously detected during that time, the server will send a notification to the family stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification will also include a message requesting feedback to confirm the actual situation.

[0727] This system improves the accuracy of anomaly detection by combining an emotion engine, further enhancing the safety of elderly people. It also encourages families and medical institutions to take quicker and more appropriate action.

[0728] The processing flow will be explained below.

[0729] Step 1:

[0730] Device (Web camera)

[0731] The webcam monitors the elderly's living space in real time and captures video data, which is compressed and transmitted to a server over the home network.

[0732] Step 2:

[0733] server

[0734] The server receives the video data sent from the webcam, performs error checking on the received data, and requests retransmission if necessary.

[0735] Step 3:

[0736] server

[0737] The server decodes the received video data and converts it into a format that can be easily processed by the analysis engine. Pre-processing includes data normalization, noise removal, and contour extraction.

[0738] Step 4:

[0739] server

[0740] The server stores the pre-processed video data in a database, which includes not only the video frames but also time information and activity metadata.

[0741] Step 5:

[0742] server

[0743] The server uses the data stored in the database to run machine learning algorithms and model the elderly's daily life patterns.In the initial stage, data on the elderly's daily life patterns is collected over a certain period of time.

[0744] Step 6:

[0745] server

[0746] The system analyzes video data received in real time and compares it with learned lifestyle patterns to detect the subject's position, movements, and gestures such as falls.

[0747] Step 7:

[0748] server

[0749] In parallel with the real-time analysis, the emotion engine extracts facial expressions from the video data and analyzes emotions. Emotion recognition involves facial expression analysis using machine learning models.

[0750] Step 8:

[0751] server

[0752] The behavioral and emotional data obtained as a result of the analysis are integrated to perform anomaly detection, taking into account the analysis results of consecutive frames and data from other IoT devices in the home network.

[0753] Step 9:

[0754] server

[0755] If an abnormality is detected, a notification message is generated. This message includes the type of abnormality, the time of occurrence, location information, and emotional state data. If the emotional state is "fear" or "sadness," the message is made more urgent.

[0756] Step 10:

[0757] server

[0758] The generated notification message is sent to the designated recipients, such as family members or medical institutions, via push notification, email, SMS, etc.

[0759] Step 11:

[0760] User

[0761] Upon receiving the notification, family members or medical institutions will assess the situation and take prompt action if necessary, such as contacting the affected person to confirm the situation and arranging for rescue if necessary.

[0762] Step 12:

[0763] User

[0764] Feedback on the anomaly detection is sent to the server, including the actual situation, the appropriateness of the anomaly, and whether there were any false positives.

[0765] Step 13:

[0766] server

[0767] Based on the collected feedback, the machine learning model and sentiment engine are retrained, which improves the accuracy of the model, reducing false positives and improving anomaly detection accuracy.

[0768] Example 2

[0769] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0770] In elderly care systems, it is important to monitor the daily lives of the elderly and detect abnormalities, but conventional systems have difficulty in accurately detecting abnormalities and have a problem of frequent false positives.In addition to detecting abnormalities, it is also necessary to take into account the emotional state of the elderly and provide appropriate notifications.However, conventional systems do not perform emotional analysis, so they have the problem of being unable to respond quickly to highly urgent situations.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0772] In this invention, the server includes a video acquisition means, a video data preprocessing means, a learning means for learning the preprocessed video data and modeling a lifestyle pattern, a facial expression analysis means for evaluating an emotional state, a real-time video data analysis means for detecting anomalies, a means for generating and sending a notification message based on the anomaly detection result and the emotion analysis result, and a means for collecting feedback and improving the learning model. This enables highly accurate anomaly detection that takes into account the lifestyle patterns and emotions of the elderly person and appropriate adjustment of the urgency of notifications.

[0773] "Video acquisition means" refers to a means of capturing real-time video data in the elderly person's living space using a device such as a webcam.

[0774] "Means for preprocessing video data" refers to the means for decoding, normalizing, extracting contours, removing noise, etc. from compressed and transmitted video data, and converting it into a format suitable for analysis.

[0775] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for modeling the daily life patterns of elderly people using a machine learning algorithm, using preprocessed video data collected over a certain period of time.

[0776] "Means for analyzing video data in real time and detecting abnormalities" refers to means for comparing video data acquired in real time with past lifestyle pattern models to detect abnormal behavior that deviates from daily life patterns.

[0777] The "means for analyzing facial expressions and assessing emotional states" is a means for analyzing facial expressions extracted from video data and determining emotional states such as joy, sadness, anger, etc.

[0778] "Means for generating and sending notification messages based on the results of anomaly detection and emotion analysis" refers to means for generating notification messages with appropriate urgency and content based on the results of anomaly detection and emotion analysis, and sending them to family members or medical institutions.

[0779] "Means for collecting feedback and improving the learning model" refers to collecting feedback from families and medical institutions, and using that feedback to retrain the machine learning model and emotion engine, thereby improving the accuracy of the system.

[0780] MODE FOR CARRYING OUT THE INVENTION

[0781] This invention relates to an elderly care service system that combines functions for image acquisition, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection with an emotion engine that recognizes the user's emotions. Each element of the system and its configuration method are described in detail below.

[0782] 1. Video acquisition

[0783] Device (Web camera)

[0784] The webcam, which serves as the terminal, monitors the elderly's living space 24 hours a day and captures video data in real time. The captured video data is compressed for communication efficiency. For example, the data is compressed in MPEG-4 format and sent to a server via the home network.

[0785] 2. Video preprocessing

[0786] server

[0787] The server receives the video data sent from the webcam and performs the following pre-processing:

[0788] Decoding: Reconstructing the received data into the original video data.

[0789] Normalization: Standardize frames to make them easier to analyze.

[0790] Contour extraction: Applying algorithms to clarify the contours of objects and people.

[0791] Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[0792] 3. Learning about daily life patterns

[0793] server

[0794] Preprocessed video data is collected over a period of time, and a machine learning algorithm is used to model the elderly person's daily life patterns. This allows normal behavioral patterns to be learned and used for anomaly detection. For example, the system learns that an elderly person wakes up at 7:00 every morning.

[0795] 4. Real-time analysis and anomaly detection

[0796] server

[0797] The system analyzes video data sent in real time and compares it with the learning model. Examples of anomaly detection include a lack of movement at normal wake-up time or a sudden fall.

[0798] 5. Use of Emotion Engine

[0799] server

[0800] The system extracts the user's facial expressions from the video data and uses an emotion engine to evaluate their emotional state in real time. Analysis identifies emotional states such as joy, sadness, anger, surprise, and fear.

[0801] 6. Consideration of abnormality notification and emotional data

[0802] server

[0803] If an abnormality is detected, a notification message is generated based on the analysis results of the emotion engine and sent to family members or medical institutions. For example, if emotions such as "fear" or "sadness" are confirmed when an abnormality is detected, a notification with a higher level of urgency will be sent.

[0804] 7. Gather feedback and improve the model

[0805] User

[0806] Upon receiving the notification, family members or medical institutions will check the situation and take necessary measures, such as calling the elderly person to check on their condition.

[0807] server

[0808] Feedback from family members and medical institutions is collected and used to retrain the lifestyle pattern model and emotion engine, improving the system's accuracy and reducing false positives.

[0809] Specific examples

[0810] Example 1: Fall detection and emotional response

[0811] server

[0812] If an elderly person falls in the living room, the webcam captures the video and sends it to the server. The server preprocesses the video and analyzes it in real time. As a result of this analysis, a fall and "fear" are detected at the same time. In this case, a notification is sent to the family saying, "A fall has been detected in the living room and a fearful expression has been confirmed."

[0813] Example 2: Detection of abnormal activity times and emotion assessment

[0814] server

[0815] This scenario assumes that an elderly person normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares this with learned lifestyle patterns based on past data and detects abnormalities. If the person remains motionless for a long period of time and detects "sadness" or "expressionlessness," it sends a notification to the family member stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification also includes a message requesting feedback to check the situation.

[0816] Example prompt

[0817] Examples of prompts used in generative AI models are as follows:

[0818] "Please explain the elderly monitoring system. This system monitors the daily life of the elderly, learns their life patterns, performs real-time analysis, detects anomalies, uses an emotion engine, notifies users of abnormalities, and collects feedback. Please explain the specific processing steps and the specific actions that are performed within them."

[0819] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0820] Processing steps of the elderly monitoring system

[0821] Step 1: Capture footage

[0822] Device (Web camera)

[0823] Input: A webcam captures real-time video data from the elderly person's living space.

[0824] How it works: The webcam monitors the elderly's living space 24 hours a day, providing detailed footage with a 360-degree field of view.

[0825] Output: Captured real-time video data is generated.

[0826] Step 2: Data compression and transmission

[0827] Device (Web camera)

[0828] Input: Captured video data.

[0829] Specific operation: The captured video data is compressed in a format such as MPEG-4, which reduces the data size and reduces the network load.

[0830] Output: The compressed video data is sent to a server via the home network.

[0831] Step 3: Data reception and decoding

[0832] server

[0833] Input: Compressed video data transmitted over a home network.

[0834] Specific operation: The server receives the video data and decodes the compressed data to restore it to its original video format.

[0835] Output: The original decoded video data is generated.

[0836] Step 4: Preprocessing the video data

[0837] server

[0838] Input: The decoded original video data.

[0839] Specific behavior:

[0840] 1. Normalization: Standardize each frame to a standard format.

[0841] 2. Contour extraction: Clarifying the contours of objects and people.

[0842] 3. Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[0843] Output: Pre-processed video data is generated.

[0844] Step 5: Learning Life Patterns

[0845] server

[0846] Input: Preprocessed video data.

[0847] How it works: The server collects pre-processed video data and uses machine learning algorithms to model the elderly person's daily life patterns, such as specific behaviors (such as waking up and eating) at the same time every day.

[0848] Output: Model data that learns the lifestyle patterns of elderly people is generated.

[0849] Step 6: Analyze real-time data and detect anomalies

[0850] server

[0851] Input: Video data and learning model sent in real time.

[0852] Specific operation: The server analyzes real-time video data and compares it with a pre-trained lifestyle pattern model. If abnormal behavior (such as prolonged inactivity or a sudden fall) is detected, the information is recorded as an abnormality.

[0853] Output: The anomaly detection results are generated.

[0854] Step 7: Facial expression analysis and emotion assessment

[0855] server

[0856] Input: Real-time video data.

[0857] Specific operation: Facial expressions are extracted from video data and the emotional state is evaluated in real time using an emotion engine. Through analysis, emotional states such as joy, sadness, anger, surprise, and fear are identified.

[0858] Output: The result of the sentiment analysis is generated.

[0859] Step 8: Generate and send a notification message

[0860] server

[0861] Input: Anomaly detection results and sentiment analysis results.

[0862] Specific operation: The server generates a notification message based on the results of anomaly detection and emotion analysis. For example, it might say, "A fall was detected in the living room and a frightened expression was observed."

[0863] Output: The generated notification message is sent to the family or healthcare provider.

[0864] Step 9: Assess the situation and respond

[0865] User

[0866] Input: Notification message from the server.

[0867] Specific actions: Family members or medical institutions will receive a notification, check the situation, and take action such as contacting the elderly person and arranging rescue if necessary.

[0868] Output: The situation is checked and action is taken.

[0869] Step 10: Gather feedback and improve the model

[0870] server

[0871] Input: Feedback from family or healthcare provider.

[0872] How it works: The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the system and reducing false positives.

[0873] Output: An improved learning model and emotion engine are generated.

[0874] (Application example 2)

[0875] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0876] The main function of conventional elderly monitoring systems was to detect abnormalities by analyzing video data, but the accuracy of the abnormality detection was not always high, and appropriate responses were sometimes delayed.In addition, because the abnormality detection and notification did not take into account the emotions of the elderly, it was difficult to accurately determine the urgency of the situation.This posed the issue of not being able to completely ensure the safety and security of the elderly.

[0877] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a video acquisition means, a means for preprocessing video data, a means for learning from the preprocessed video data and modeling lifestyle patterns, a means for analyzing the video data in real time and detecting anomalies, a means for recognizing the user's emotions and considering them together with the anomaly detection results, a means for generating and transmitting a notification message based on the anomaly detection results and emotion data, and a means for collecting feedback and improving the learning model. This makes it possible to ensure the safety of elderly people in real time and quickly take appropriate measures according to the level of urgency.

[0878] The "video acquisition means" is a device that captures the subject's behavior and facial expressions in real time and acquires them as video data.

[0879] The "means for preprocessing video data" refers to a processing device or software that performs decoding, noise removal, frame normalization, etc. to convert the acquired video data into a format that is easy to analyze.

[0880] The "means for modeling lifestyle patterns" is a system that uses preprocessed video data to learn the subject's daily behavioral patterns using a machine learning algorithm and saves them as a model.

[0881] The "means for detecting anomalies" is an algorithm that compares video data acquired in real time with a learning model to detect behavior that deviates from normal life patterns.

[0882] The "means for recognizing user emotions" is an engine that analyzes the subject's facial expressions from the acquired video data and evaluates their emotional state, such as joy, sadness, or fear, in real time.

[0883] The "means for generating and transmitting a notification message based on the abnormality detection result and emotional data" is a system that creates a notification message that takes into account the detected abnormality and the user's emotional data, and transmits it to the home or medical personnel.

[0884] "Means for collecting feedback and improving the learning model" refers to a mechanism for collecting feedback from households and medical institutions regarding the results of anomaly detection and notification, and using that data to improve the accuracy of the machine learning model.

[0885] The present invention provides specific means and methods for effectively realizing elderly care. The system uses smart glasses as a terminal to monitor the user's behavior and emotions in real time, and sends appropriate notifications to family members and medical personnel when abnormalities occur.

[0886] Hardware Configuration

[0887] The smart glasses are equipped with a built-in camera that constantly captures the user's actions. The video data undergoes initial pre-processing by the smart glasses' on-device processor.

[0888] Software Configuration

[0889] The server receives the data sent from the terminal and performs the following processing.

[0890] 1. Video pre-processing:

[0891] The server decodes the video data sent from the smart glasses, performs noise reduction and frame normalization, and converts it into a format suitable for analysis.

[0892] 2. Learning Life Patterns:

[0893] Using the preprocessed data, the server learns the elderly person's daily life patterns using machine learning algorithms (e.g., TensorFlow / Keras). In the initial stage, data is collected over several weeks to create a model of their normal behavior.

[0894] 3. Real-time analysis and anomaly detection:

[0895] The server compares the video data acquired in real time with the trained model to detect anomalies, and if an anomaly is found, the information is processed immediately.

[0896] 4. Emotion analysis:

[0897] Facial expressions are extracted from the video data, and an emotion analysis engine (e.g., TensorFlow model) is used to analyze the user's emotional state in real time. The resulting emotion data is included in the anomaly detection results.

[0898] 5. Abnormal notification:

[0899] If an anomaly is detected, the server generates a notification message taking into account the user's emotional data. For example, if the user shows a fearful expression, the notification will be sent with a high level of urgency. The notification will be sent via email or a dedicated application.

[0900] 6. Gather feedback and improve the model:

[0901] Family members and medical institutions receive notifications, check the situation, respond, and then provide feedback. The server uses this feedback to retrain the learning model and improve detection accuracy.

[0902] Specific examples

[0903] Suppose an elderly person is wearing smart glasses and going about their daily life at home. Suddenly, they fall, and the smart glasses capture the video. The server immediately analyzes the data and detects the abnormality. At the same time, the emotion analysis engine recognizes that the user has an expression of "fear" on their face. This causes the server to send a high-level emergency notification to their family, urging them to take prompt action. This system transmits highly accurate information to their family and medical institutions in real time, ensuring the safety of the elderly.

[0904] Example prompts for generative AI models

[0905] 1. Capture footage using a webcam or smart glasses.

[0906] 2. Preprocess the video data to remove noise and unnecessary information.

[0907] 3. Use a machine learning model to detect anomalies by comparing learned patterns of daily life with the video data.

[0908] 4. Analyze facial expression data in real time to determine emotional states.

[0909] 5. If an abnormality is detected, adjust the urgency of the notification message to be sent to family members, taking into account the emotional data.

[0910] 6. Collect feedback from families and healthcare providers to retrain and improve your machine learning models.

[0911] The detailed configuration and processing means of the present invention make it possible to provide a highly accurate monitoring system for ensuring the safety of elderly people.

[0912] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0913] Step 1:

[0914] The device (smart glasses) uses a built-in camera to capture the elderly person's behavior and facial expressions in real time. Real-time video data is acquired as input, and this video data is temporarily stored in the device as output.

[0915] Step 2:

[0916] The device preprocesses the captured video data, specifically decoding, denoising, and frame normalizing the video data. The input is the captured video data, and the output is the preprocessed video data.

[0917] Step 3:

[0918] The pre-processed video data is sent to the server through the home network. The input is the pre-processed video data, and the output is obtained by sending this data to the server.

[0919] Step 4:

[0920] The server receives the preprocessed video data and uses a machine learning algorithm to learn the elderly's daily life patterns. The input is the preprocessed video data, and the output is the learned lifestyle pattern model.

[0921] Step 5:

[0922] The server analyzes newly received video data in real time and compares it with the trained model to detect anomalies. The input is the real-time video data and the trained lifestyle pattern model, and the output is the anomaly detection results.

[0923] Step 6:

[0924] The server extracts the user's facial expressions from the video data and analyzes their emotional state in real time using an emotion analysis engine. The input is the video data, and the output is the analysis result of the emotional state.

[0925] Step 7:

[0926] When an anomaly is detected, the server takes into account the emotion analysis results and generates an anomaly notification message with a set urgency level. The input is the anomaly detection result and the emotion analysis result, and the output is the notification message.

[0927] Step 8:

[0928] The server sends the generated notification message to the terminal of the home or medical personnel. The input is the notification message, and the output is the terminal of the home or medical personnel that receives the notification.

[0929] Step 9:

[0930] Users (family members or medical professionals) receive notifications, check and respond, and provide feedback to the server. The input is the notification message, and the output is feedback information.

[0931] Step 10:

[0932] The server uses the received feedback to retrain the learning model and improve the accuracy of anomaly detection. The input is the feedback information, and the output is an improved learning model.

[0933] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0934] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0935] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0936] [Third embodiment]

[0937] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0938] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0939] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0940] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0941] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0942] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0943] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0944] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0945] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0946] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0947] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0948] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0949] MODE FOR CARRYING OUT THE INVENTION

[0950] The system of the present invention integrates the functions of video acquisition, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. Specifically, this system is as follows.

[0951] 1. Video acquisition

[0952] Device (Web camera)

[0953] The webcam installed as a terminal constantly monitors the elderly's living space and captures video data in real time, which is then sent to a server via a network.

[0954] 2. Video preprocessing

[0955] server

[0956] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, noise removal, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[0957] 3. Learning about daily life patterns

[0958] server

[0959] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[0960] 4. Real-time analysis and anomaly detection

[0961] server

[0962] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[0963] 5. Abnormality notification

[0964] server

[0965] If an abnormality is detected, the server generates a notification message and sends it to designated recipients, such as family members or medical institutions. The notification includes details such as the type of abnormality, the time of occurrence, and location information. Notifications can be sent via push notification, email, SMS, etc.

[0966] 6. Gather feedback and improve the model

[0967] User

[0968] The user who receives the notification sends feedback about the anomaly back to the server, including the actual situation, whether there was a false positive, and whether the response was appropriate.

[0969] server

[0970] The server uses the collected feedback to retrain the machine learning model and make improvements to reduce false positives and improve detection accuracy, making the system more reliable over time.

[0971] Specific examples

[0972] Example 1: Fall detection

[0973] server

[0974] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion is detected and recognized as an abnormality. The server immediately sends a notification to the family member saying, "A fall has been detected in the living room."

[0975] Example 2: Detecting abnormal activity times

[0976] server

[0977] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, the server sends a notification to the family saying, "No activity was observed by 8:00 AM." The notification also includes a message requesting feedback to confirm the actual situation.

[0978] As described above, this invention is a system that integrates the functions of image acquisition, preprocessing, learning, analysis, notification, and feedback collection to monitor the lives of the elderly in real time and enable rapid response. This system greatly improves the safety of the elderly and allows their families and medical institutions to monitor them with peace of mind.

[0979] The processing flow will be explained below.

[0980] Step 1:

[0981] Device (Web camera)

[0982] The webcam captures video in real time and generates video data that is compressed and sent over the home network to a server.

[0983] Step 2:

[0984] server

[0985] The server receives the video data sent from the webcam, performs an error check to see if there are any packet losses, and requests packet retransmission if necessary.

[0986] Step 3:

[0987] server

[0988] The received video data is decoded and converted into a format that can be easily processed by the analysis engine. This preprocessing includes data normalization, noise removal, and contour extraction.

[0989] Step 4:

[0990] server

[0991] The preprocessed data is stored in a database, which includes not only the video frames but also time information and activity metadata.

[0992] Step 5:

[0993] server

[0994] The machine learning model is trained using data stored in the database. In the initial stage, the model is built based on data from a certain period of time and learns the lifestyle patterns of elderly people.

[0995] Step 6:

[0996] server

[0997] The system analyzes video data received in real time and compares it with a learning model to detect the subject's position, movement, and gestures such as falls.

[0998] Step 7:

[0999] server

[1000] It combines the analysis results of consecutive frames with data from other IoT devices in the home network to detect anomalies, for example, when it receives fall data from a smart sensor, it compares it with the corresponding video analysis results.

[1001] Step 8:

[1002] server

[1003] When an anomaly is detected, a notification message is generated and sent to the specified notification recipients, containing details such as the type of anomaly, the time of occurrence, and location information.

[1004] Step 9:

[1005] User

[1006] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[1007] Step 10:

[1008] User

[1009] Feedback regarding the anomaly notification is sent back to the server, including the actual situation and whether or not there was a false positive.

[1010] Step 11:

[1011] server

[1012] The machine learning model is retrained based on the collected feedback, which improves the model's accuracy, reducing false positives and improving anomaly detection.

[1013] Example 1

[1014] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] In elderly care services, rapid detection and response to abnormal behavior is important. However, conventional systems have issues with monitoring accuracy and real-time performance, resulting in frequent false positives. Furthermore, the process of feedback collection and model improvement is not well coordinated, resulting in a lack of system reliability. The present invention aims to solve these problems and improve the safety of elderly people.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1017] In this invention, the server includes means for constantly capturing video for monitoring the elderly person's living space, means for decoding the received video data and dividing it into frames, means for normalizing the frames and performing contour extraction and noise removal, means for learning the preprocessed video data using a machine learning algorithm to model lifestyle patterns, means for analyzing the video data in real time and comparing it with an existing learning model to detect abnormal behavior, means for generating a notification message including the type, time of occurrence, and location information of the detected abnormal behavior and sending it to a specified notification destination, and means for collecting feedback from users who receive the notification and re-training and improving the learning model. This enables rapid and accurate detection and notification of abnormal behavior, and makes it possible to improve the accuracy and reliability of the system based on the feedback.

[1018] "Video acquisition means" refers to devices or systems that monitor the elderly person's living space and capture video images at all times.

[1019] "Decoding" refers to the process of converting received video data into a frame-by-frame format that can be analyzed.

[1020] The term "means for dividing into frames" refers to a process for dividing video data into individual frames and subjecting each frame to analysis.

[1021] "Normalization" refers to the process of standardizing video data frames to a specific size and format to improve the accuracy of analysis.

[1022] "Contour extraction" refers to the process of extracting the main shape information from a frame of video data and emphasizing the necessary information.

[1023] "Noise reduction" refers to the process of removing unnecessary information from video data to improve the quality of the data.

[1024] "Machine learning algorithm" refers to a mathematical model and its execution program that learns patterns from data and makes predictions and classifications.

[1025] "Modeling lifestyle patterns" refers to learning the daily behavior of elderly people and saving those behavior patterns as a mathematical model.

[1026] "Means for analyzing in real time" refers to a method for instantly analyzing currently received video data and instantly detecting abnormal behavior.

[1027] "Abnormal behavior" refers to behavior that deviates from the elderly person's daily behavior patterns, including, for example, prolonged immobility or sudden falls.

[1028] "Notification message" refers to a message that includes information about detected abnormal behavior (such as type, time of occurrence, location information, etc.) and is sent to a designated recipient.

[1029] "Feedback" refers to information provided by users who receive notifications about the actual situation, whether there were any false positives, and the appropriateness of the response.

[1030] "Retraining" refers to the process of updating an existing machine learning model based on new data and feedback to improve its accuracy and reliability.

[1031] MODE FOR CARRYING OUT THE INVENTION

[1032] The system of the present invention integrates the functions of video capture, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. To implement this system, the following specific hardware and software are used.

[1033] Hardware

[1034] Terminal (Web camera): Installed to monitor the living space of the elderly, it captures video 24 hours a day and transmits it to a server via the network.

[1035] software

[1036] Decoding and pre-processing: After receiving the video data from the webcam, the server decodes it using the FFmpeg library and splits the data into frames. Then, it normalizes the frames, extracts contours, and removes noise using the OpenCV library.

[1037] Machine learning: Using the preprocessed data, machine learning libraries such as TensorFlow and PyTorch are used on the server to learn the lifestyle patterns, thereby modeling the daily behavior of elderly people.

[1038] Real-time analysis and anomaly detection: The server uses analysis engines such as YOLOv5 and OpenPose to analyze video data in real time and compare it with existing learning models to detect abnormal behavior.

[1039] Notification: If an anomaly is detected, the server uses Twilio or Firebase Cloud Messaging (FCM) to generate a notification message containing the type of abnormal behavior, the time of occurrence, and location information, and sends it to the specified notification recipient.

[1040] Feedback collection and model improvement: After receiving the notification, users provide feedback through the application. The server receives this feedback, stores it in a database for analysis, and improves the model by retraining it.

[1041] Specific examples

[1042] Specific operation of fall detection

[1043] If an elderly person falls in the living room, a webcam captures the moment and sends the video data to a server. The server preprocesses the video data and analyzes it in real time. If the analysis detects a fall, it is recorded as an abnormal behavior and the server sends a notification message to the family saying, "A fall has been detected in the living room."

[1044] Specific operation of abnormal activity time detection

[1045] If an elderly person normally wakes up at 7:00 a.m. but does not wake up by 8:00 a.m., the server will compare the current data with the lifestyle patterns learned from past data to detect anomalies. If no activity is observed for a long period of time, a notification will be sent to the family stating, "No activity was observed by 8:00 a.m." The notification will also include a message requesting feedback to confirm the actual situation.

[1046] Prompt Sentence Examples

[1047] "In an elderly care system, please explain in detail the steps by which a webcam detects an elderly person's fall, and the server preprocesses and analyzes the video data and notifies the family."

[1048] This invention provides a system that can monitor the lives of elderly people in real time, quickly detect abnormal behavior, and notify them. This system is highly reliable because it continuously improves its model based on user feedback, thereby improving the safety of elderly people.

[1049] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1050] Step 1:

[1051] Video Acquisition

[1052] The device (webcam) starts up and monitors the elderly person's living space.

[1053] How it works: The webcam captures video streams 24 / 7, obtaining video data in real time.

[1054] Input: Real-time video footage of an elderly person's living space.

[1055] Output: Captured video data (video stream).

[1056] Step 2:

[1057] Video data preprocessing

[1058] The server receives the video data sent from the web camera.

[1059] What happens: The server uses the FFmpeg library to decode the video stream and grab each frame.

[1060] Input: Video data (video stream) sent from the webcam.

[1061] Output: Decoded video frames.

[1062] Step 3:

[1063] Frame normalization and denoising

[1064] The server normalizes each frame and performs contour extraction and noise removal.

[1065] Specific operation: Using the OpenCV library, the frame size is standardized to 1280x720 pixels, converted to grayscale, and edge detection and noise removal are performed using the Canny method.

[1066] Input: Decoded video frames.

[1067] Output: Normalized and denoised frames.

[1068] Step 4:

[1069] Learning life patterns

[1070] The server inputs the pre-processed video frames into a machine learning algorithm.

[1071] Specific operation: Using TensorFlow and PyTorch, the system learns and models the daily behavioral patterns of elderly people based on data from a certain period of time.

[1072] Input: Preprocessed video frames.

[1073] Output: Learned behavioral pattern model.

[1074] Step 5:

[1075] Real-time analysis and anomaly detection

[1076] The server analyzes the video data sent from the webcam in real time.

[1077] Specific operation: Utilizing YOLOv5 and OpenPose, we compare with the learning model and detect behavior that deviates from everyday life patterns (abnormal behavior).

[1078] Input: Real-time video data, learning model.

[1079] Output: A log of the anomalous behavior detected.

[1080] Step 6:

[1081] Abnormal notification

[1082] The server generates a notification message if an abnormality is detected.

[1083] Specific operation: Using Twilio or Firebase Cloud Messaging (FCM), a notification containing the type of anomaly, the time of occurrence, and location information is sent to the specified destination.

[1084] Input: The log in which the anomalous behavior was detected.

[1085] Output: Notification message sent to family and healthcare providers.

[1086] Step 7:

[1087] Gathering feedback and improving the model

[1088] The user receives a notification and sends feedback back to the server.

[1089] Specific operation: The user reports the actual situation regarding the anomaly and whether there were any false positives through the application.

[1090] Input: User feedback.

[1091] Output: Feedback is collected and stored in a database.

[1092] Step 8:

[1093] Improving the model through retraining

[1094] The server retrains and improves the machine learning model based on the feedback collected.

[1095] How it works: Using TensorFlow, we incorporate feedback data to retrain existing models, improving detection accuracy and reliability.

[1096] Input: User feedback data, existing learning model.

[1097] Output: An improved learning model.

[1098] This allows the system to monitor elderly people in real time, quickly and accurately detect and notify abnormal behavior, and constantly improve the model based on user feedback.

[1099] (Application example 1)

[1100] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1101] While conventional elderly monitoring systems are effective, especially in fixed living environments, they are difficult to monitor while moving, and passenger safety monitoring has not been fully realized, especially in self-driving vehicles. Furthermore, while there is a need for a function that can respond quickly and appropriately when a passenger experiences an abnormality, current systems often respond slowly. Therefore, there is an urgent need to develop a system that can monitor passenger safety in self-driving vehicles and provide immediate notification in the event of an abnormality.

[1102] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1103] In this invention, the server includes a video acquisition means, a means for preprocessing the video data, a means for learning the preprocessed video data and modeling a lifestyle pattern, a means for analyzing the video data in real time and detecting anomalies, a means for sending the anomaly detection results to a notification destination, a means for collecting feedback and improving the learning model, a means for monitoring the status of passengers in the autonomous vehicle, and a means for detecting and notifying passenger anomalies in real time. This makes it possible to monitor the safety of passengers in the autonomous vehicle and to quickly notify passengers in the event of an anomaly.

[1104] "Video acquisition means" means a means for capturing video data of passengers in real time using a camera within the autonomous vehicle.

[1105] The "means for preprocessing video data" refers to a means for performing processes such as adjusting resolution and removing noise in order to convert the acquired video data into an analyzable format.

[1106] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for using preprocessed video data to learn passengers' normal behavioral patterns through a machine learning algorithm and modeling them.

[1107] "Means for analyzing video data in real time and detecting abnormalities" refers to a means for detecting abnormal behavior by comparing video data acquired in real time with past learning models.

[1108] The "means for transmitting an abnormality detection result to a notification destination" is a means for transmitting a notification message to a pre-specified contact point when an abnormality is detected.

[1109] The "means for collecting feedback and improving the learning model" refers to a means for collecting feedback from the recipients of notifications and retraining and improving the learning model based on that data.

[1110] "Means for monitoring the status of passengers inside an autonomous vehicle" refers to means for constantly monitoring the behavior and status of passengers using cameras and sensors installed inside the vehicle.

[1111] "Means for detecting and notifying passenger abnormalities in real time" refers to a means for detecting abnormal passenger behavior (e.g., sudden falls or prolonged periods of inactivity) in real time and immediately notifying the relevant parties.

[1112] The system for realizing this invention monitors the status of passengers in an autonomous vehicle in real time and promptly notifies them if an abnormality is detected. This system is specifically implemented by combining the following multiple means.

[1113] 1. Video acquisition method

[1114] The server captures passenger video data using cameras installed inside the autonomous vehicle, and the real-time video data is transmitted to the server.

[1115] 2. Video data preprocessing methods

[1116] The server receives the video data sent from the camera and performs pre-processing, which includes decoding the video data, adjusting the resolution, removing noise, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[1117] 3. A means of learning about life patterns

[1118] The server uses the preprocessed video data to model passengers' usual behavior patterns using machine learning algorithms, which are then saved as a learning model using machine learning frameworks such as TensorFlow and PyTorch.

[1119] 4. Real-time analysis and anomaly detection methods

[1120] The server analyzes the video data transmitted in real time and compares it with the learning model. If the analysis engine detects an abnormality (e.g., a sudden fall or prolonged inactivity), it records it. This anomaly detection uses a pre-trained generative AI model.

[1121] 5. Abnormality notification means

[1122] If an abnormality is detected, the server generates a notification message and sends it to the designated notification recipient (for example, family or a medical institution). This notification includes the type of abnormality, the time of occurrence, location information, etc. Notification methods include push notification, email, and SMS.

[1123] 6. Feedback Collection Methods

[1124] The user who receives the notification sends feedback about the situation to the server, including the actual situation and whether there were any false positives.

[1125] 7. Ways to improve the learning model

[1126] The server uses the collected feedback to retrain the machine learning model, reducing false positives and improving detection accuracy, again using machine learning frameworks such as TensorFlow or PyTorch.

[1127] Example 1: Fall detection

[1128] The server detects when a passenger in an autonomous vehicle suddenly falls. The camera captures the moment of the fall, and the preprocessed video data is sent to the server. Analysis detects the falling motion and recognizes it as an abnormality. The server then sends an abnormality notification to the specified notification destination.

[1129] Example 2: Detecting long periods of inactivity

[1130] For example, if a passenger who normally does not stay seated for more than 10 minutes remains motionless for more than 30 minutes, the server will detect the abnormality by comparing it with past data and send a notification such as, "The period of inactivity has exceeded normal."

[1131] Example prompt sentence:

[1132] "We will build a system that will detect abnormal behavior from camera footage and send a notification to a specified email address when an abnormality is detected. Specifically, we will detect when a passenger in an autonomous vehicle loses consciousness and continues to sit there, and send an email notification on the spot."

[1133] In this way, it is possible to monitor the safety of passengers in autonomous vehicles and respond quickly.

[1134] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1135] Step 1:

[1136] The server acquires video data in real time from the terminal (camera inside the autonomous vehicle). This data is a raw video stream that is sent from the terminal to the server. Specifically, the camera's video data is periodically captured and sent to the server via the network. The input data is the camera video, and the output is the video data sent to the server.

[1137] Step 2:

[1138] The server preprocesses the received video data, specifically decoding, adjusting resolution, and removing noise. The preprocessed video data is then converted into a format suitable for analysis. The input data is raw video data, and the output is preprocessed video data.

[1139] Step 3:

[1140] The server uses preprocessed video data to learn and model passengers' usual behavioral patterns using machine learning algorithms. Specifically, it collects data over a certain period of time and uses TensorFlow and PyTorch to learn lifestyle patterns. The input data is the preprocessed video data, and the output is a learning model.

[1141] Step 4:

[1142] The server analyzes the video data transmitted in real time and compares it with the learning model to detect anomalies. Specifically, the video data is input into a generative AI model to detect abnormal behavior. The input data are the video data transmitted in real time and the learning model, and the output is the anomaly detection result.

[1143] Step 5:

[1144] If an anomaly is detected, the server generates a notification message and sends it to a designated recipient (for example, family or a medical institution). Specifically, the notification includes details such as the type of anomaly, the time of occurrence, and location information, and is sent via push notification, email, or SMS. The input data is the anomaly detection result, and the output is the notification message.

[1145] Step 6:

[1146] The user who receives the notification sends back feedback. Specifically, they click a link or button included in the notification to report the actual situation, whether there were any false positives, etc. to the server. The input data is the user feedback, and the output is the feedback sent to the server.

[1147] Step 7:

[1148] The server uses the collected feedback to retrain the machine learning model and improve its detection accuracy. Specifically, it takes the feedback as a dataset and retrains the model using TensorFlow or PyTorch. The input data is the feedback and the existing model, and the output is a new, improved trained model.

[1149] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1150] MODE FOR CARRYING OUT THE INVENTION

[1151] This invention combines functions for an elderly care service, including video capture, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection, with an emotion engine that recognizes the user's emotions. This provides a system that takes emotion data into account in anomaly detection and realizes highly accurate and appropriate responses.

[1152] 1. Video acquisition

[1153] Device (Web camera)

[1154] The webcams used as terminals constantly monitor the elderly's living spaces, capturing video data in real time, which is then compressed and sent to a server via the home network.

[1155] 2. Video preprocessing

[1156] server

[1157] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, and noise reduction. At this stage, the video data is converted into a format suitable for the analysis engine.

[1158] 3. Learning about daily life patterns

[1159] server

[1160] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[1161] 4. Real-time analysis and anomaly detection

[1162] server

[1163] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[1164] 5. Use of Emotion Engine

[1165] server

[1166] The server extracts the user's facial expressions from the video data and uses an emotion engine to analyze their emotional state in real time, resulting in emotional states such as joy, sadness, anger, surprise, and fear.

[1167] 6. Consideration of abnormality notification and emotional data

[1168] server

[1169] If an abnormality is detected, the server generates and sends a notification message taking into account the analysis results of the emotion engine. For example, if the emotion data indicates "fear" or "sadness" when an abnormality is detected, the urgency of the notification will be increased. The content of the notification will also be adjusted based on the user's emotional state, prompting the user to take a more appropriate action.

[1170] 7. Gather feedback and improve the model

[1171] User

[1172] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[1173] server

[1174] The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the model, reducing false positives and improving anomaly detection.

[1175] Specific examples

[1176] Example 1: Fall detection and emotional response

[1177] server

[1178] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion and facial expressions of "fear" or "pain" are detected. The server immediately sends a notification to the family saying, "A fall has been detected in the living room, and a facial expression of fear has been confirmed."

[1179] Example 2: Detection of abnormal activity times and emotion assessment

[1180] server

[1181] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, and if "sadness" or "no facial expression" is continuously detected during that time, the server will send a notification to the family stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification will also include a message requesting feedback to confirm the actual situation.

[1182] This system improves the accuracy of anomaly detection by combining an emotion engine, further enhancing the safety of elderly people. It also encourages families and medical institutions to take quicker and more appropriate action.

[1183] The processing flow will be explained below.

[1184] Step 1:

[1185] Device (Web camera)

[1186] The webcam monitors the elderly's living space in real time and captures video data, which is compressed and transmitted to a server over the home network.

[1187] Step 2:

[1188] server

[1189] The server receives the video data sent from the webcam, performs error checking on the received data, and requests retransmission if necessary.

[1190] Step 3:

[1191] server

[1192] The server decodes the received video data and converts it into a format that can be easily processed by the analysis engine. Pre-processing includes data normalization, noise removal, and contour extraction.

[1193] Step 4:

[1194] server

[1195] The server stores the pre-processed video data in a database, which includes not only the video frames but also time information and activity metadata.

[1196] Step 5:

[1197] server

[1198] The server uses the data stored in the database to run machine learning algorithms and model the elderly's daily life patterns.In the initial stage, data on the elderly's daily life patterns is collected over a certain period of time.

[1199] Step 6:

[1200] server

[1201] The system analyzes video data received in real time and compares it with learned lifestyle patterns to detect the subject's position, movements, and gestures such as falls.

[1202] Step 7:

[1203] server

[1204] In parallel with the real-time analysis, the emotion engine extracts facial expressions from the video data and analyzes emotions. Emotion recognition involves facial expression analysis using machine learning models.

[1205] Step 8:

[1206] server

[1207] The behavioral and emotional data obtained as a result of the analysis are integrated to perform anomaly detection, taking into account the analysis results of consecutive frames and data from other IoT devices in the home network.

[1208] Step 9:

[1209] server

[1210] If an abnormality is detected, a notification message is generated. This message includes the type of abnormality, the time of occurrence, location information, and emotional state data. If the emotional state is "fear" or "sadness," the message is made more urgent.

[1211] Step 10:

[1212] server

[1213] The generated notification message is sent to the designated recipients, such as family members or medical institutions, via push notification, email, SMS, etc.

[1214] Step 11:

[1215] User

[1216] Upon receiving the notification, family members or medical institutions will assess the situation and take prompt action if necessary, such as contacting the affected person to confirm the situation and arranging for rescue if necessary.

[1217] Step 12:

[1218] User

[1219] Feedback on the anomaly detection is sent to the server, including the actual situation, the appropriateness of the anomaly, and whether there were any false positives.

[1220] Step 13:

[1221] server

[1222] Based on the collected feedback, the machine learning model and sentiment engine are retrained, which improves the accuracy of the model, reducing false positives and improving anomaly detection accuracy.

[1223] Example 2

[1224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] In elderly care systems, it is important to monitor the daily lives of the elderly and detect abnormalities, but conventional systems have difficulty in accurately detecting abnormalities and have a problem of frequent false positives.In addition to detecting abnormalities, it is also necessary to take into account the emotional state of the elderly and provide appropriate notifications.However, conventional systems do not perform emotional analysis, so they have the problem of being unable to respond quickly to highly urgent situations.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1227] In this invention, the server includes a video acquisition means, a video data preprocessing means, a learning means for learning the preprocessed video data and modeling a lifestyle pattern, a facial expression analysis means for evaluating an emotional state, a real-time video data analysis means for detecting anomalies, a means for generating and sending a notification message based on the anomaly detection result and the emotion analysis result, and a means for collecting feedback and improving the learning model. This enables highly accurate anomaly detection that takes into account the lifestyle patterns and emotions of the elderly person and appropriate adjustment of the urgency of notifications.

[1228] "Video acquisition means" refers to a means of capturing real-time video data in the elderly person's living space using a device such as a webcam.

[1229] "Means for preprocessing video data" refers to the means for decoding, normalizing, extracting contours, removing noise, etc. from compressed and transmitted video data, and converting it into a format suitable for analysis.

[1230] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for modeling the daily life patterns of elderly people using a machine learning algorithm, using preprocessed video data collected over a certain period of time.

[1231] "Means for analyzing video data in real time and detecting abnormalities" refers to means for comparing video data acquired in real time with past lifestyle pattern models to detect abnormal behavior that deviates from daily life patterns.

[1232] The "means for analyzing facial expressions and assessing emotional states" is a means for analyzing facial expressions extracted from video data and determining emotional states such as joy, sadness, anger, etc.

[1233] "Means for generating and sending notification messages based on the results of anomaly detection and emotion analysis" refers to means for generating notification messages with appropriate urgency and content based on the results of anomaly detection and emotion analysis, and sending them to family members or medical institutions.

[1234] "Means for collecting feedback and improving the learning model" refers to collecting feedback from families and medical institutions, and using that feedback to retrain the machine learning model and emotion engine, thereby improving the accuracy of the system.

[1235] MODE FOR CARRYING OUT THE INVENTION

[1236] This invention relates to an elderly care service system that combines functions for image acquisition, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection with an emotion engine that recognizes the user's emotions. Each element of the system and its configuration method are described in detail below.

[1237] 1. Video acquisition

[1238] Device (Web camera)

[1239] The webcam, which serves as the terminal, monitors the elderly's living space 24 hours a day and captures video data in real time. The captured video data is compressed for communication efficiency. For example, the data is compressed in MPEG-4 format and sent to a server via the home network.

[1240] 2. Video preprocessing

[1241] server

[1242] The server receives the video data sent from the webcam and performs the following pre-processing:

[1243] Decoding: Reconstructing the received data into the original video data.

[1244] Normalization: Standardize frames to make them easier to analyze.

[1245] Contour extraction: Applying algorithms to clarify the contours of objects and people.

[1246] Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[1247] 3. Learning about daily life patterns

[1248] server

[1249] Preprocessed video data is collected over a period of time, and a machine learning algorithm is used to model the elderly person's daily life patterns. This allows normal behavioral patterns to be learned and used for anomaly detection. For example, the system learns that an elderly person wakes up at 7:00 every morning.

[1250] 4. Real-time analysis and anomaly detection

[1251] server

[1252] The system analyzes video data sent in real time and compares it with the learning model. Examples of anomaly detection include a lack of movement at normal wake-up time or a sudden fall.

[1253] 5. Use of Emotion Engine

[1254] server

[1255] The system extracts the user's facial expressions from the video data and uses an emotion engine to evaluate their emotional state in real time. Analysis identifies emotional states such as joy, sadness, anger, surprise, and fear.

[1256] 6. Consideration of abnormality notification and emotional data

[1257] server

[1258] If an abnormality is detected, a notification message is generated based on the analysis results of the emotion engine and sent to family members or medical institutions. For example, if emotions such as "fear" or "sadness" are confirmed when an abnormality is detected, a notification with a higher level of urgency will be sent.

[1259] 7. Gather feedback and improve the model

[1260] User

[1261] Upon receiving the notification, family members or medical institutions will check the situation and take necessary measures, such as calling the elderly person to check on their condition.

[1262] server

[1263] Feedback from family members and medical institutions is collected and used to retrain the lifestyle pattern model and emotion engine, improving the system's accuracy and reducing false positives.

[1264] Specific examples

[1265] Example 1: Fall detection and emotional response

[1266] server

[1267] If an elderly person falls in the living room, the webcam captures the video and sends it to the server. The server preprocesses the video and analyzes it in real time. As a result of this analysis, a fall and "fear" are detected at the same time. In this case, a notification is sent to the family saying, "A fall has been detected in the living room and a fearful expression has been confirmed."

[1268] Example 2: Detection of abnormal activity times and emotion assessment

[1269] server

[1270] This scenario assumes that an elderly person normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares this with learned lifestyle patterns based on past data and detects abnormalities. If the person remains motionless for a long period of time and detects "sadness" or "expressionlessness," it sends a notification to the family member stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification also includes a message requesting feedback to check the situation.

[1271] Example prompt

[1272] Examples of prompts used in generative AI models are as follows:

[1273] "Please explain the elderly monitoring system. This system monitors the daily life of the elderly, learns their life patterns, performs real-time analysis, detects anomalies, uses an emotion engine, notifies users of abnormalities, and collects feedback. Please explain the specific processing steps and the specific actions that are performed within them."

[1274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1275] Processing steps of the elderly monitoring system

[1276] Step 1: Capture footage

[1277] Device (Web camera)

[1278] Input: A webcam captures real-time video data from the elderly person's living space.

[1279] How it works: The webcam monitors the elderly's living space 24 hours a day, providing detailed footage with a 360-degree field of view.

[1280] Output: Captured real-time video data is generated.

[1281] Step 2: Data compression and transmission

[1282] Device (Web camera)

[1283] Input: Captured video data.

[1284] Specific operation: The captured video data is compressed in a format such as MPEG-4, which reduces the data size and reduces the network load.

[1285] Output: The compressed video data is sent to a server via the home network.

[1286] Step 3: Data reception and decoding

[1287] server

[1288] Input: Compressed video data transmitted over a home network.

[1289] Specific operation: The server receives the video data and decodes the compressed data to restore it to its original video format.

[1290] Output: The original decoded video data is generated.

[1291] Step 4: Preprocessing the video data

[1292] server

[1293] Input: The decoded original video data.

[1294] Specific behavior:

[1295] 1. Normalization: Standardize each frame to a standard format.

[1296] 2. Contour extraction: Clarifying the contours of objects and people.

[1297] 3. Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[1298] Output: Pre-processed video data is generated.

[1299] Step 5: Learning Life Patterns

[1300] server

[1301] Input: Preprocessed video data.

[1302] How it works: The server collects pre-processed video data and uses machine learning algorithms to model the elderly person's daily life patterns, such as specific behaviors (such as waking up and eating) at the same time every day.

[1303] Output: Model data that learns the lifestyle patterns of elderly people is generated.

[1304] Step 6: Analyze real-time data and detect anomalies

[1305] server

[1306] Input: Video data and learning model sent in real time.

[1307] Specific operation: The server analyzes real-time video data and compares it with a pre-trained lifestyle pattern model. If abnormal behavior (such as prolonged inactivity or a sudden fall) is detected, the information is recorded as an abnormality.

[1308] Output: The anomaly detection results are generated.

[1309] Step 7: Facial expression analysis and emotion assessment

[1310] server

[1311] Input: Real-time video data.

[1312] Specific operation: Facial expressions are extracted from video data and the emotional state is evaluated in real time using an emotion engine. Through analysis, emotional states such as joy, sadness, anger, surprise, and fear are identified.

[1313] Output: The result of the sentiment analysis is generated.

[1314] Step 8: Generate and send a notification message

[1315] server

[1316] Input: Anomaly detection results and sentiment analysis results.

[1317] Specific operation: The server generates a notification message based on the results of anomaly detection and emotion analysis. For example, it might say, "A fall was detected in the living room and a frightened expression was observed."

[1318] Output: The generated notification message is sent to the family or healthcare provider.

[1319] Step 9: Assess the situation and respond

[1320] User

[1321] Input: Notification message from the server.

[1322] Specific actions: Family members or medical institutions will receive a notification, check the situation, and take action such as contacting the elderly person and arranging rescue if necessary.

[1323] Output: The situation is checked and action is taken.

[1324] Step 10: Gather feedback and improve the model

[1325] server

[1326] Input: Feedback from family or healthcare provider.

[1327] How it works: The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the system and reducing false positives.

[1328] Output: An improved learning model and emotion engine are generated.

[1329] (Application example 2)

[1330] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1331] The main function of conventional elderly monitoring systems was to detect abnormalities by analyzing video data, but the accuracy of the abnormality detection was not always high, and appropriate responses were sometimes delayed.In addition, because the abnormality detection and notification did not take into account the emotions of the elderly, it was difficult to accurately determine the urgency of the situation.This posed the issue of not being able to completely ensure the safety and security of the elderly.

[1332] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a video acquisition means, a means for preprocessing video data, a means for learning from the preprocessed video data and modeling lifestyle patterns, a means for analyzing the video data in real time and detecting anomalies, a means for recognizing the user's emotions and considering them together with the anomaly detection results, a means for generating and transmitting a notification message based on the anomaly detection results and emotion data, and a means for collecting feedback and improving the learning model. This makes it possible to ensure the safety of elderly people in real time and quickly take appropriate measures according to the level of urgency.

[1333] The "video acquisition means" is a device that captures the subject's behavior and facial expressions in real time and acquires them as video data.

[1334] The "means for preprocessing video data" refers to a processing device or software that performs decoding, noise removal, frame normalization, etc. to convert the acquired video data into a format that is easy to analyze.

[1335] The "means for modeling lifestyle patterns" is a system that uses preprocessed video data to learn the subject's daily behavioral patterns using a machine learning algorithm and saves them as a model.

[1336] The "means for detecting anomalies" is an algorithm that compares video data acquired in real time with a learning model to detect behavior that deviates from normal life patterns.

[1337] The "means for recognizing user emotions" is an engine that analyzes the subject's facial expressions from the acquired video data and evaluates their emotional state, such as joy, sadness, or fear, in real time.

[1338] The "means for generating and transmitting a notification message based on the abnormality detection result and emotional data" is a system that creates a notification message that takes into account the detected abnormality and the user's emotional data, and transmits it to the home or medical personnel.

[1339] "Means for collecting feedback and improving the learning model" refers to a mechanism for collecting feedback from households and medical institutions regarding the results of anomaly detection and notification, and using that data to improve the accuracy of the machine learning model.

[1340] The present invention provides specific means and methods for effectively realizing elderly care. The system uses smart glasses as a terminal to monitor the user's behavior and emotions in real time, and sends appropriate notifications to family members and medical personnel when abnormalities occur.

[1341] Hardware Configuration

[1342] The smart glasses are equipped with a built-in camera that constantly captures the user's actions. The video data undergoes initial pre-processing by the smart glasses' on-device processor.

[1343] Software Configuration

[1344] The server receives the data sent from the terminal and performs the following processing.

[1345] 1. Video pre-processing:

[1346] The server decodes the video data sent from the smart glasses, performs noise reduction and frame normalization, and converts it into a format suitable for analysis.

[1347] 2. Learning Life Patterns:

[1348] Using the preprocessed data, the server learns the elderly person's daily life patterns using machine learning algorithms (e.g., TensorFlow / Keras). In the initial stage, data is collected over several weeks to create a model of their normal behavior.

[1349] 3. Real-time analysis and anomaly detection:

[1350] The server compares the video data acquired in real time with the trained model to detect anomalies, and if an anomaly is found, the information is processed immediately.

[1351] 4. Emotion analysis:

[1352] Facial expressions are extracted from the video data, and an emotion analysis engine (e.g., TensorFlow model) is used to analyze the user's emotional state in real time. The resulting emotion data is included in the anomaly detection results.

[1353] 5. Abnormal notification:

[1354] If an anomaly is detected, the server generates a notification message taking into account the user's emotional data. For example, if the user shows a fearful expression, the notification will be sent with a high level of urgency. The notification will be sent via email or a dedicated application.

[1355] 6. Gather feedback and improve the model:

[1356] Family members and medical institutions receive notifications, check the situation, respond, and then provide feedback. The server uses this feedback to retrain the learning model and improve detection accuracy.

[1357] Specific examples

[1358] Suppose an elderly person is wearing smart glasses and going about their daily life at home. Suddenly, they fall, and the smart glasses capture the video. The server immediately analyzes the data and detects the abnormality. At the same time, the emotion analysis engine recognizes that the user has an expression of "fear" on their face. This causes the server to send a high-level emergency notification to their family, urging them to take prompt action. This system transmits highly accurate information to their family and medical institutions in real time, ensuring the safety of the elderly.

[1359] Example prompts for generative AI models

[1360] 1. Capture footage using a webcam or smart glasses.

[1361] 2. Preprocess the video data to remove noise and unnecessary information.

[1362] 3. Use a machine learning model to detect anomalies by comparing learned patterns of daily life with the video data.

[1363] 4. Analyze facial expression data in real time to determine emotional states.

[1364] 5. If an abnormality is detected, adjust the urgency of the notification message to be sent to family members, taking into account the emotional data.

[1365] 6. Collect feedback from families and healthcare providers to retrain and improve your machine learning models.

[1366] The detailed configuration and processing means of the present invention make it possible to provide a highly accurate monitoring system for ensuring the safety of elderly people.

[1367] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1368] Step 1:

[1369] The device (smart glasses) uses a built-in camera to capture the elderly person's behavior and facial expressions in real time. Real-time video data is acquired as input, and this video data is temporarily stored in the device as output.

[1370] Step 2:

[1371] The device preprocesses the captured video data, specifically decoding, denoising, and frame normalizing the video data. The input is the captured video data, and the output is the preprocessed video data.

[1372] Step 3:

[1373] The pre-processed video data is sent to the server through the home network. The input is the pre-processed video data, and the output is obtained by sending this data to the server.

[1374] Step 4:

[1375] The server receives the preprocessed video data and uses a machine learning algorithm to learn the elderly's daily life patterns. The input is the preprocessed video data, and the output is the learned lifestyle pattern model.

[1376] Step 5:

[1377] The server analyzes newly received video data in real time and compares it with the trained model to detect anomalies. The input is the real-time video data and the trained lifestyle pattern model, and the output is the anomaly detection results.

[1378] Step 6:

[1379] The server extracts the user's facial expressions from the video data and analyzes their emotional state in real time using an emotion analysis engine. The input is the video data, and the output is the analysis result of the emotional state.

[1380] Step 7:

[1381] When an anomaly is detected, the server takes into account the emotion analysis results and generates an anomaly notification message with a set urgency level. The input is the anomaly detection result and the emotion analysis result, and the output is the notification message.

[1382] Step 8:

[1383] The server sends the generated notification message to the terminal of the home or medical personnel. The input is the notification message, and the output is the terminal of the home or medical personnel that receives the notification.

[1384] Step 9:

[1385] Users (family members or medical professionals) receive notifications, check and respond, and provide feedback to the server. The input is the notification message, and the output is feedback information.

[1386] Step 10:

[1387] The server uses the received feedback to retrain the learning model and improve the accuracy of anomaly detection. The input is the feedback information, and the output is an improved learning model.

[1388] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1389] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1390] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1391] [Fourth embodiment]

[1392] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1393] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1394] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1395] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1396] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1397] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1398] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1399] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1400] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1401] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1402] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1403] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1404] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1405] MODE FOR CARRYING OUT THE INVENTION

[1406] The system of the present invention integrates the functions of video acquisition, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. Specifically, this system is as follows.

[1407] 1. Video acquisition

[1408] Device (Web camera)

[1409] The webcam installed as a terminal constantly monitors the elderly's living space and captures video data in real time, which is then sent to a server via a network.

[1410] 2. Video preprocessing

[1411] server

[1412] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, noise removal, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[1413] 3. Learning about daily life patterns

[1414] server

[1415] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[1416] 4. Real-time analysis and anomaly detection

[1417] server

[1418] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[1419] 5. Abnormality notification

[1420] server

[1421] If an abnormality is detected, the server generates a notification message and sends it to designated recipients, such as family members or medical institutions. The notification includes details such as the type of abnormality, the time of occurrence, and location information. Notifications can be sent via push notification, email, SMS, etc.

[1422] 6. Gather feedback and improve the model

[1423] User

[1424] The user who receives the notification sends feedback about the anomaly back to the server, including the actual situation, whether there was a false positive, and whether the response was appropriate.

[1425] server

[1426] The server uses the collected feedback to retrain the machine learning model and make improvements to reduce false positives and improve detection accuracy, making the system more reliable over time.

[1427] Specific examples

[1428] Example 1: Fall detection

[1429] server

[1430] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion is detected and recognized as an abnormality. The server immediately sends a notification to the family member saying, "A fall has been detected in the living room."

[1431] Example 2: Detecting abnormal activity times

[1432] server

[1433] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, the server sends a notification to the family saying, "No activity was observed by 8:00 AM." The notification also includes a message requesting feedback to confirm the actual situation.

[1434] As described above, this invention is a system that integrates the functions of image acquisition, preprocessing, learning, analysis, notification, and feedback collection to monitor the lives of the elderly in real time and enable rapid response. This system greatly improves the safety of the elderly and allows their families and medical institutions to monitor them with peace of mind.

[1435] The processing flow will be explained below.

[1436] Step 1:

[1437] Device (Web camera)

[1438] The webcam captures video in real time and generates video data that is compressed and sent over the home network to a server.

[1439] Step 2:

[1440] server

[1441] The server receives the video data sent from the webcam, performs an error check to see if there are any packet losses, and requests packet retransmission if necessary.

[1442] Step 3:

[1443] server

[1444] The received video data is decoded and converted into a format that can be easily processed by the analysis engine. This preprocessing includes data normalization, noise removal, and contour extraction.

[1445] Step 4:

[1446] server

[1447] The preprocessed data is stored in a database, which includes not only the video frames but also time information and activity metadata.

[1448] Step 5:

[1449] server

[1450] The machine learning model is trained using data stored in the database. In the initial stage, the model is built based on data from a certain period of time and learns the lifestyle patterns of elderly people.

[1451] Step 6:

[1452] server

[1453] The system analyzes video data received in real time and compares it with a learning model to detect the subject's position, movement, and gestures such as falls.

[1454] Step 7:

[1455] server

[1456] It combines the analysis results of consecutive frames with data from other IoT devices in the home network to detect anomalies, for example, when it receives fall data from a smart sensor, it compares it with the corresponding video analysis results.

[1457] Step 8:

[1458] server

[1459] When an anomaly is detected, a notification message is generated and sent to the specified notification recipients, containing details such as the type of anomaly, the time of occurrence, and location information.

[1460] Step 9:

[1461] User

[1462] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[1463] Step 10:

[1464] User

[1465] Feedback regarding the anomaly notification is sent back to the server, including the actual situation and whether or not there was a false positive.

[1466] Step 11:

[1467] server

[1468] The machine learning model is retrained based on the collected feedback, which improves the model's accuracy, reducing false positives and improving anomaly detection.

[1469] Example 1

[1470] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1471] In elderly care services, rapid detection and response to abnormal behavior is important. However, conventional systems have issues with monitoring accuracy and real-time performance, resulting in frequent false positives. Furthermore, the process of feedback collection and model improvement is not well coordinated, resulting in a lack of system reliability. The present invention aims to solve these problems and improve the safety of elderly people.

[1472] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1473] In this invention, the server includes means for constantly capturing video for monitoring the elderly person's living space, means for decoding the received video data and dividing it into frames, means for normalizing the frames and performing contour extraction and noise removal, means for learning the preprocessed video data using a machine learning algorithm to model lifestyle patterns, means for analyzing the video data in real time and comparing it with an existing learning model to detect abnormal behavior, means for generating a notification message including the type, time of occurrence, and location information of the detected abnormal behavior and sending it to a specified notification destination, and means for collecting feedback from users who receive the notification and re-training and improving the learning model. This enables rapid and accurate detection and notification of abnormal behavior, and makes it possible to improve the accuracy and reliability of the system based on the feedback.

[1474] "Video acquisition means" refers to devices or systems that monitor the elderly person's living space and capture video images at all times.

[1475] "Decoding" refers to the process of converting received video data into a frame-by-frame format that can be analyzed.

[1476] The term "means for dividing into frames" refers to a process for dividing video data into individual frames and subjecting each frame to analysis.

[1477] "Normalization" refers to the process of standardizing video data frames to a specific size and format to improve the accuracy of analysis.

[1478] "Contour extraction" refers to the process of extracting the main shape information from a frame of video data and emphasizing the necessary information.

[1479] "Noise reduction" refers to the process of removing unnecessary information from video data to improve the quality of the data.

[1480] "Machine learning algorithm" refers to a mathematical model and its execution program that learns patterns from data and makes predictions and classifications.

[1481] "Modeling lifestyle patterns" refers to learning the daily behavior of elderly people and saving those behavior patterns as a mathematical model.

[1482] "Means for analyzing in real time" refers to a method for instantly analyzing currently received video data and instantly detecting abnormal behavior.

[1483] "Abnormal behavior" refers to behavior that deviates from the elderly person's daily behavior patterns, including, for example, prolonged immobility or sudden falls.

[1484] "Notification message" refers to a message that includes information about detected abnormal behavior (such as type, time of occurrence, location information, etc.) and is sent to a designated recipient.

[1485] "Feedback" refers to information provided by users who receive notifications about the actual situation, whether there were any false positives, and the appropriateness of the response.

[1486] "Retraining" refers to the process of updating an existing machine learning model based on new data and feedback to improve its accuracy and reliability.

[1487] MODE FOR CARRYING OUT THE INVENTION

[1488] The system of the present invention integrates the functions of video capture, pre-processing, learning, real-time analysis, anomaly detection, notification, and feedback collection in an elderly care service. To implement this system, the following specific hardware and software are used.

[1489] Hardware

[1490] Terminal (Web camera): Installed to monitor the living space of the elderly, it captures video 24 hours a day and transmits it to a server via the network.

[1491] software

[1492] Decoding and pre-processing: After receiving the video data from the webcam, the server decodes it using the FFmpeg library and splits the data into frames. Then, it normalizes the frames, extracts contours, and removes noise using the OpenCV library.

[1493] Machine learning: Using the preprocessed data, machine learning libraries such as TensorFlow and PyTorch are used on the server to learn the lifestyle patterns, thereby modeling the daily behavior of elderly people.

[1494] Real-time analysis and anomaly detection: The server uses analysis engines such as YOLOv5 and OpenPose to analyze video data in real time and compare it with existing learning models to detect abnormal behavior.

[1495] Notification: If an anomaly is detected, the server uses Twilio or Firebase Cloud Messaging (FCM) to generate a notification message containing the type of abnormal behavior, the time of occurrence, and location information, and sends it to the specified notification recipient.

[1496] Feedback collection and model improvement: After receiving the notification, users provide feedback through the application. The server receives this feedback, stores it in a database for analysis, and improves the model by retraining it.

[1497] Specific examples

[1498] Specific operation of fall detection

[1499] If an elderly person falls in the living room, a webcam captures the moment and sends the video data to a server. The server preprocesses the video data and analyzes it in real time. If the analysis detects a fall, it is recorded as an abnormal behavior and the server sends a notification message to the family saying, "A fall has been detected in the living room."

[1500] Specific operation of abnormal activity time detection

[1501] If an elderly person normally wakes up at 7:00 a.m. but does not wake up by 8:00 a.m., the server will compare the current data with the lifestyle patterns learned from past data to detect anomalies. If no activity is observed for a long period of time, a notification will be sent to the family stating, "No activity was observed by 8:00 a.m." The notification will also include a message requesting feedback to confirm the actual situation.

[1502] Prompt Sentence Examples

[1503] "In an elderly care system, please explain in detail the steps by which a webcam detects an elderly person's fall, and the server preprocesses and analyzes the video data and notifies the family."

[1504] This invention provides a system that can monitor the lives of elderly people in real time, quickly detect abnormal behavior, and notify them. This system is highly reliable because it continuously improves its model based on user feedback, thereby improving the safety of elderly people.

[1505] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1506] Step 1:

[1507] Video Acquisition

[1508] The device (webcam) starts up and monitors the elderly person's living space.

[1509] How it works: The webcam captures video streams 24 / 7, obtaining video data in real time.

[1510] Input: Real-time video footage of an elderly person's living space.

[1511] Output: Captured video data (video stream).

[1512] Step 2:

[1513] Video data preprocessing

[1514] The server receives the video data sent from the web camera.

[1515] What happens: The server uses the FFmpeg library to decode the video stream and grab each frame.

[1516] Input: Video data (video stream) sent from the webcam.

[1517] Output: Decoded video frames.

[1518] Step 3:

[1519] Frame normalization and denoising

[1520] The server normalizes each frame and performs contour extraction and noise removal.

[1521] Specific operation: Using the OpenCV library, the frame size is standardized to 1280x720 pixels, converted to grayscale, and edge detection and noise removal are performed using the Canny method.

[1522] Input: Decoded video frames.

[1523] Output: Normalized and denoised frames.

[1524] Step 4:

[1525] Learning life patterns

[1526] The server inputs the pre-processed video frames into a machine learning algorithm.

[1527] Specific operation: Using TensorFlow and PyTorch, the system learns and models the daily behavioral patterns of elderly people based on data from a certain period of time.

[1528] Input: Preprocessed video frames.

[1529] Output: Learned behavioral pattern model.

[1530] Step 5:

[1531] Real-time analysis and anomaly detection

[1532] The server analyzes the video data sent from the webcam in real time.

[1533] Specific operation: Utilizing YOLOv5 and OpenPose, we compare with the learning model and detect behavior that deviates from everyday life patterns (abnormal behavior).

[1534] Input: Real-time video data, learning model.

[1535] Output: A log of the anomalous behavior detected.

[1536] Step 6:

[1537] Abnormal notification

[1538] The server generates a notification message if an abnormality is detected.

[1539] Specific operation: Using Twilio or Firebase Cloud Messaging (FCM), a notification containing the type of anomaly, the time of occurrence, and location information is sent to the specified destination.

[1540] Input: The log in which the anomalous behavior was detected.

[1541] Output: Notification message sent to family and healthcare providers.

[1542] Step 7:

[1543] Gathering feedback and improving the model

[1544] The user receives a notification and sends feedback back to the server.

[1545] Specific operation: The user reports the actual situation regarding the anomaly and whether there were any false positives through the application.

[1546] Input: User feedback.

[1547] Output: Feedback is collected and stored in a database.

[1548] Step 8:

[1549] Improving the model through retraining

[1550] The server retrains and improves the machine learning model based on the feedback collected.

[1551] How it works: Using TensorFlow, we incorporate feedback data to retrain existing models, improving detection accuracy and reliability.

[1552] Input: User feedback data, existing learning model.

[1553] Output: An improved learning model.

[1554] This allows the system to monitor elderly people in real time, quickly and accurately detect and notify abnormal behavior, and constantly improve the model based on user feedback.

[1555] (Application example 1)

[1556] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1557] While conventional elderly monitoring systems are effective, especially in fixed living environments, they are difficult to monitor while moving, and passenger safety monitoring has not been fully realized, especially in self-driving vehicles. Furthermore, while there is a need for a function that can respond quickly and appropriately when a passenger experiences an abnormality, current systems often respond slowly. Therefore, there is an urgent need to develop a system that can monitor passenger safety in self-driving vehicles and provide immediate notification in the event of an abnormality.

[1558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1559] In this invention, the server includes a video acquisition means, a means for preprocessing the video data, a means for learning the preprocessed video data and modeling a lifestyle pattern, a means for analyzing the video data in real time and detecting anomalies, a means for sending the anomaly detection results to a notification destination, a means for collecting feedback and improving the learning model, a means for monitoring the status of passengers in the autonomous vehicle, and a means for detecting and notifying passenger anomalies in real time. This makes it possible to monitor the safety of passengers in the autonomous vehicle and to quickly notify passengers in the event of an anomaly.

[1560] "Video acquisition means" means a means for capturing video data of passengers in real time using a camera within the autonomous vehicle.

[1561] The "means for preprocessing video data" refers to a means for performing processes such as adjusting resolution and removing noise in order to convert the acquired video data into an analyzable format.

[1562] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for using preprocessed video data to learn passengers' normal behavioral patterns through a machine learning algorithm and modeling them.

[1563] "Means for analyzing video data in real time and detecting abnormalities" refers to a means for detecting abnormal behavior by comparing video data acquired in real time with past learning models.

[1564] The "means for transmitting an abnormality detection result to a notification destination" is a means for transmitting a notification message to a pre-specified contact point when an abnormality is detected.

[1565] The "means for collecting feedback and improving the learning model" refers to a means for collecting feedback from the recipients of notifications and retraining and improving the learning model based on that data.

[1566] "Means for monitoring the status of passengers inside an autonomous vehicle" refers to means for constantly monitoring the behavior and status of passengers using cameras and sensors installed inside the vehicle.

[1567] "Means for detecting and notifying passenger abnormalities in real time" refers to a means for detecting abnormal passenger behavior (e.g., sudden falls or prolonged periods of inactivity) in real time and immediately notifying the relevant parties.

[1568] The system for realizing this invention monitors the status of passengers in an autonomous vehicle in real time and promptly notifies them if an abnormality is detected. This system is specifically implemented by combining the following multiple means.

[1569] 1. Video acquisition method

[1570] The server captures passenger video data using cameras installed inside the autonomous vehicle, and the real-time video data is transmitted to the server.

[1571] 2. Video data preprocessing methods

[1572] The server receives the video data sent from the camera and performs pre-processing, which includes decoding the video data, adjusting the resolution, removing noise, etc. At this stage, the video data is converted into a format suitable for the analysis engine.

[1573] 3. A means of learning about life patterns

[1574] The server uses the preprocessed video data to model passengers' usual behavior patterns using machine learning algorithms, which are then saved as a learning model using machine learning frameworks such as TensorFlow and PyTorch.

[1575] 4. Real-time analysis and anomaly detection methods

[1576] The server analyzes the video data transmitted in real time and compares it with the learning model. If the analysis engine detects an abnormality (e.g., a sudden fall or prolonged inactivity), it records it. This anomaly detection uses a pre-trained generative AI model.

[1577] 5. Abnormality notification means

[1578] If an abnormality is detected, the server generates a notification message and sends it to the designated notification recipient (for example, family or a medical institution). This notification includes the type of abnormality, the time of occurrence, location information, etc. Notification methods include push notification, email, and SMS.

[1579] 6. Feedback Collection Methods

[1580] The user who receives the notification sends feedback about the situation to the server, including the actual situation and whether there were any false positives.

[1581] 7. Ways to improve the learning model

[1582] The server uses the collected feedback to retrain the machine learning model, reducing false positives and improving detection accuracy, again using machine learning frameworks such as TensorFlow or PyTorch.

[1583] Example 1: Fall detection

[1584] The server detects when a passenger in an autonomous vehicle suddenly falls. The camera captures the moment of the fall, and the preprocessed video data is sent to the server. Analysis detects the falling motion and recognizes it as an abnormality. The server then sends an abnormality notification to the specified notification destination.

[1585] Example 2: Detecting long periods of inactivity

[1586] For example, if a passenger who normally does not stay seated for more than 10 minutes remains motionless for more than 30 minutes, the server will detect the abnormality by comparing it with past data and send a notification such as, "The period of inactivity has exceeded normal."

[1587] Example prompt sentence:

[1588] "We will build a system that will detect abnormal behavior from camera footage and send a notification to a specified email address when an abnormality is detected. Specifically, we will detect when a passenger in an autonomous vehicle loses consciousness and continues to sit there, and send an email notification on the spot."

[1589] In this way, it is possible to monitor the safety of passengers in autonomous vehicles and respond quickly.

[1590] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1591] Step 1:

[1592] The server acquires video data in real time from the terminal (camera inside the autonomous vehicle). This data is a raw video stream that is sent from the terminal to the server. Specifically, the camera's video data is periodically captured and sent to the server via the network. The input data is the camera video, and the output is the video data sent to the server.

[1593] Step 2:

[1594] The server preprocesses the received video data, specifically decoding, adjusting resolution, and removing noise. The preprocessed video data is then converted into a format suitable for analysis. The input data is raw video data, and the output is preprocessed video data.

[1595] Step 3:

[1596] The server uses preprocessed video data to learn and model passengers' usual behavioral patterns using machine learning algorithms. Specifically, it collects data over a certain period of time and uses TensorFlow and PyTorch to learn lifestyle patterns. The input data is the preprocessed video data, and the output is a learning model.

[1597] Step 4:

[1598] The server analyzes the video data transmitted in real time and compares it with the learning model to detect anomalies. Specifically, the video data is input into a generative AI model to detect abnormal behavior. The input data are the video data transmitted in real time and the learning model, and the output is the anomaly detection result.

[1599] Step 5:

[1600] If an anomaly is detected, the server generates a notification message and sends it to a designated recipient (for example, family or a medical institution). Specifically, the notification includes details such as the type of anomaly, the time of occurrence, and location information, and is sent via push notification, email, or SMS. The input data is the anomaly detection result, and the output is the notification message.

[1601] Step 6:

[1602] The user who receives the notification sends back feedback. Specifically, they click a link or button included in the notification to report the actual situation, whether there were any false positives, etc. to the server. The input data is the user feedback, and the output is the feedback sent to the server.

[1603] Step 7:

[1604] The server uses the collected feedback to retrain the machine learning model and improve its detection accuracy. Specifically, it takes the feedback as a dataset and retrains the model using TensorFlow or PyTorch. The input data is the feedback and the existing model, and the output is a new, improved trained model.

[1605] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1606] MODE FOR CARRYING OUT THE INVENTION

[1607] This invention combines functions for an elderly care service, including video capture, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection, with an emotion engine that recognizes the user's emotions. This provides a system that takes emotion data into account in anomaly detection and realizes highly accurate and appropriate responses.

[1608] 1. Video acquisition

[1609] Device (Web camera)

[1610] The webcams used as terminals constantly monitor the elderly's living spaces, capturing video data in real time, which is then compressed and sent to a server via the home network.

[1611] 2. Video preprocessing

[1612] server

[1613] The server receives the video data sent from the webcam and performs preprocessing, which includes data decoding, frame normalization, contour extraction, and noise reduction. At this stage, the video data is converted into a format suitable for the analysis engine.

[1614] 3. Learning about daily life patterns

[1615] server

[1616] The server uses the preprocessed video data to train the machine learning algorithm. In the initial stage, data is collected for a certain period of time to model the elderly's daily life patterns. This allows the elderly's normal behavior patterns to be saved as a learning model.

[1617] 4. Real-time analysis and anomaly detection

[1618] server

[1619] The server analyzes the video data sent from the webcam in real time. The analysis engine compares it with the learning model and detects behavior that deviates from daily life patterns. For example, if a person is inactive for a long period of time or suddenly falls, the information is recorded as an abnormality.

[1620] 5. Use of Emotion Engine

[1621] server

[1622] The server extracts the user's facial expressions from the video data and uses an emotion engine to analyze their emotional state in real time, resulting in emotional states such as joy, sadness, anger, surprise, and fear.

[1623] 6. Consideration of abnormality notification and emotional data

[1624] server

[1625] If an abnormality is detected, the server generates and sends a notification message taking into account the analysis results of the emotion engine. For example, if the emotion data indicates "fear" or "sadness" when an abnormality is detected, the urgency of the notification will be increased. The content of the notification will also be adjusted based on the user's emotional state, prompting the user to take a more appropriate action.

[1626] 7. Gather feedback and improve the model

[1627] User

[1628] Family or medical personnel are notified and can assess or respond to the situation, for example, contacting the individual for identification and arranging for immediate assistance if necessary.

[1629] server

[1630] The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the model, reducing false positives and improving anomaly detection.

[1631] Specific examples

[1632] Example 1: Fall detection and emotional response

[1633] server

[1634] Consider the case where an elderly person falls in the living room. A webcam captures the moment of the fall, and the video data is sent to a server. The server preprocesses the data and analyzes it in real time. As a result of the analysis, the falling motion and facial expressions of "fear" or "pain" are detected. The server immediately sends a notification to the family saying, "A fall has been detected in the living room, and a facial expression of fear has been confirmed."

[1635] Example 2: Detection of abnormal activity times and emotion assessment

[1636] server

[1637] Consider an elderly person who normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares the current data with the lifestyle patterns learned from past data to detect abnormalities. If no activity is observed for a long period of time, and if "sadness" or "no facial expression" is continuously detected during that time, the server will send a notification to the family stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification will also include a message requesting feedback to confirm the actual situation.

[1638] This system improves the accuracy of anomaly detection by combining an emotion engine, further enhancing the safety of elderly people. It also encourages families and medical institutions to take quicker and more appropriate action.

[1639] The processing flow will be explained below.

[1640] Step 1:

[1641] Device (Web camera)

[1642] The webcam monitors the elderly's living space in real time and captures video data, which is compressed and transmitted to a server over the home network.

[1643] Step 2:

[1644] server

[1645] The server receives the video data sent from the webcam, performs error checking on the received data, and requests retransmission if necessary.

[1646] Step 3:

[1647] server

[1648] The server decodes the received video data and converts it into a format that can be easily processed by the analysis engine. Pre-processing includes data normalization, noise removal, and contour extraction.

[1649] Step 4:

[1650] server

[1651] The server stores the pre-processed video data in a database, which includes not only the video frames but also time information and activity metadata.

[1652] Step 5:

[1653] server

[1654] The server uses the data stored in the database to run machine learning algorithms and model the elderly's daily life patterns.In the initial stage, data on the elderly's daily life patterns is collected over a certain period of time.

[1655] Step 6:

[1656] server

[1657] The system analyzes video data received in real time and compares it with learned lifestyle patterns to detect the subject's position, movements, and gestures such as falls.

[1658] Step 7:

[1659] server

[1660] In parallel with the real-time analysis, the emotion engine extracts facial expressions from the video data and analyzes emotions. Emotion recognition involves facial expression analysis using machine learning models.

[1661] Step 8:

[1662] server

[1663] The behavioral and emotional data obtained as a result of the analysis are integrated to perform anomaly detection, taking into account the analysis results of consecutive frames and data from other IoT devices in the home network.

[1664] Step 9:

[1665] server

[1666] If an abnormality is detected, a notification message is generated. This message includes the type of abnormality, the time of occurrence, location information, and emotional state data. If the emotional state is "fear" or "sadness," the message is made more urgent.

[1667] Step 10:

[1668] server

[1669] The generated notification message is sent to the designated recipients, such as family members or medical institutions, via push notification, email, SMS, etc.

[1670] Step 11:

[1671] User

[1672] Upon receiving the notification, family members or medical institutions will assess the situation and take prompt action if necessary, such as contacting the affected person to confirm the situation and arranging for rescue if necessary.

[1673] Step 12:

[1674] User

[1675] Feedback on the anomaly detection is sent to the server, including the actual situation, the appropriateness of the anomaly, and whether there were any false positives.

[1676] Step 13:

[1677] server

[1678] Based on the collected feedback, the machine learning model and sentiment engine are retrained, which improves the accuracy of the model, reducing false positives and improving anomaly detection accuracy.

[1679] Example 2

[1680] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1681] In elderly care systems, it is important to monitor the daily lives of the elderly and detect abnormalities, but conventional systems have difficulty in accurately detecting abnormalities and have a problem of frequent false positives.In addition to detecting abnormalities, it is also necessary to take into account the emotional state of the elderly and provide appropriate notifications.However, conventional systems do not perform emotional analysis, so they have the problem of being unable to respond quickly to highly urgent situations.

[1682] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1683] In this invention, the server includes a video acquisition means, a video data preprocessing means, a learning means for learning the preprocessed video data and modeling a lifestyle pattern, a facial expression analysis means for evaluating an emotional state, a real-time video data analysis means for detecting anomalies, a means for generating and sending a notification message based on the anomaly detection result and the emotion analysis result, and a means for collecting feedback and improving the learning model. This enables highly accurate anomaly detection that takes into account the lifestyle patterns and emotions of the elderly person and appropriate adjustment of the urgency of notifications.

[1684] "Video acquisition means" refers to a means of capturing real-time video data in the elderly person's living space using a device such as a webcam.

[1685] "Means for preprocessing video data" refers to the means for decoding, normalizing, extracting contours, removing noise, etc. from compressed and transmitted video data, and converting it into a format suitable for analysis.

[1686] The "means for learning from preprocessed video data and modeling lifestyle patterns" refers to a means for modeling the daily life patterns of elderly people using a machine learning algorithm, using preprocessed video data collected over a certain period of time.

[1687] "Means for analyzing video data in real time and detecting abnormalities" refers to means for comparing video data acquired in real time with past lifestyle pattern models to detect abnormal behavior that deviates from daily life patterns.

[1688] The "means for analyzing facial expressions and assessing emotional states" is a means for analyzing facial expressions extracted from video data and determining emotional states such as joy, sadness, anger, etc.

[1689] "Means for generating and sending notification messages based on the results of anomaly detection and emotion analysis" refers to means for generating notification messages with appropriate urgency and content based on the results of anomaly detection and emotion analysis, and sending them to family members or medical institutions.

[1690] "Means for collecting feedback and improving the learning model" refers to collecting feedback from families and medical institutions, and using that feedback to retrain the machine learning model and emotion engine, thereby improving the accuracy of the system.

[1691] MODE FOR CARRYING OUT THE INVENTION

[1692] This invention relates to an elderly care service system that combines functions for image acquisition, preprocessing, learning of lifestyle patterns, real-time analysis, anomaly detection, notification, and feedback collection with an emotion engine that recognizes the user's emotions. Each element of the system and its configuration method are described in detail below.

[1693] 1. Video acquisition

[1694] Device (Web camera)

[1695] The webcam, which serves as the terminal, monitors the elderly's living space 24 hours a day and captures video data in real time. The captured video data is compressed for communication efficiency. For example, the data is compressed in MPEG-4 format and sent to a server via the home network.

[1696] 2. Video preprocessing

[1697] server

[1698] The server receives the video data sent from the webcam and performs the following pre-processing:

[1699] Decoding: Reconstructing the received data into the original video data.

[1700] Normalization: Standardize frames to make them easier to analyze.

[1701] Contour extraction: Applying algorithms to clarify the contours of objects and people.

[1702] Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[1703] 3. Learning about daily life patterns

[1704] server

[1705] Preprocessed video data is collected over a period of time, and a machine learning algorithm is used to model the elderly person's daily life patterns. This allows normal behavioral patterns to be learned and used for anomaly detection. For example, the system learns that an elderly person wakes up at 7:00 every morning.

[1706] 4. Real-time analysis and anomaly detection

[1707] server

[1708] The system analyzes video data sent in real time and compares it with the learning model. Examples of anomaly detection include a lack of movement at normal wake-up time or a sudden fall.

[1709] 5. Use of Emotion Engine

[1710] server

[1711] The system extracts the user's facial expressions from the video data and uses an emotion engine to evaluate their emotional state in real time. Analysis identifies emotional states such as joy, sadness, anger, surprise, and fear.

[1712] 6. Consideration of abnormality notification and emotional data

[1713] server

[1714] If an abnormality is detected, a notification message is generated based on the analysis results of the emotion engine and sent to family members or medical institutions. For example, if emotions such as "fear" or "sadness" are confirmed when an abnormality is detected, a notification with a higher level of urgency will be sent.

[1715] 7. Gather feedback and improve the model

[1716] User

[1717] Upon receiving the notification, family members or medical institutions will check the situation and take necessary measures, such as calling the elderly person to check on their condition.

[1718] server

[1719] Feedback from family members and medical institutions is collected and used to retrain the lifestyle pattern model and emotion engine, improving the system's accuracy and reducing false positives.

[1720] Specific examples

[1721] Example 1: Fall detection and emotional response

[1722] server

[1723] If an elderly person falls in the living room, the webcam captures the video and sends it to the server. The server preprocesses the video and analyzes it in real time. As a result of this analysis, a fall and "fear" are detected at the same time. In this case, a notification is sent to the family saying, "A fall has been detected in the living room and a fearful expression has been confirmed."

[1724] Example 2: Detection of abnormal activity times and emotion assessment

[1725] server

[1726] This scenario assumes that an elderly person normally wakes up at 7:00 AM, but does not wake up by 8:00 AM. The server compares this with learned lifestyle patterns based on past data and detects abnormalities. If the person remains motionless for a long period of time and detects "sadness" or "expressionlessness," it sends a notification to the family member stating, "No activity was observed by 8:00 AM, and a sad expression was confirmed." This notification also includes a message requesting feedback to check the situation.

[1727] Example prompt

[1728] Examples of prompts used in generative AI models are as follows:

[1729] "Please explain the elderly monitoring system. This system monitors the daily life of the elderly, learns their life patterns, performs real-time analysis, detects anomalies, uses an emotion engine, notifies users of abnormalities, and collects feedback. Please explain the specific processing steps and the specific actions that are performed within them."

[1730] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1731] Processing steps of the elderly monitoring system

[1732] Step 1: Capture footage

[1733] Device (Web camera)

[1734] Input: A webcam captures real-time video data from the elderly person's living space.

[1735] How it works: The webcam monitors the elderly's living space 24 hours a day, providing detailed footage with a 360-degree field of view.

[1736] Output: Captured real-time video data is generated.

[1737] Step 2: Data compression and transmission

[1738] Device (Web camera)

[1739] Input: Captured video data.

[1740] Specific operation: The captured video data is compressed in a format such as MPEG-4, which reduces the data size and reduces the network load.

[1741] Output: The compressed video data is sent to a server via the home network.

[1742] Step 3: Data reception and decoding

[1743] server

[1744] Input: Compressed video data transmitted over a home network.

[1745] Specific operation: The server receives the video data and decodes the compressed data to restore it to its original video format.

[1746] Output: The original decoded video data is generated.

[1747] Step 4: Preprocessing the video data

[1748] server

[1749] Input: The decoded original video data.

[1750] Specific behavior:

[1751] 1. Normalization: Standardize each frame to a standard format.

[1752] 2. Contour extraction: Clarifying the contours of objects and people.

[1753] 3. Noise reduction: Remove noise from the image using a Gaussian filter or similar.

[1754] Output: Pre-processed video data is generated.

[1755] Step 5: Learning Life Patterns

[1756] server

[1757] Input: Preprocessed video data.

[1758] How it works: The server collects pre-processed video data and uses machine learning algorithms to model the elderly person's daily life patterns, such as specific behaviors (such as waking up and eating) at the same time every day.

[1759] Output: Model data that learns the lifestyle patterns of elderly people is generated.

[1760] Step 6: Analyze real-time data and detect anomalies

[1761] server

[1762] Input: Video data and learning model sent in real time.

[1763] Specific operation: The server analyzes real-time video data and compares it with a pre-trained lifestyle pattern model. If abnormal behavior (such as prolonged inactivity or a sudden fall) is detected, the information is recorded as an abnormality.

[1764] Output: The anomaly detection results are generated.

[1765] Step 7: Facial expression analysis and emotion assessment

[1766] server

[1767] Input: Real-time video data.

[1768] Specific operation: Facial expressions are extracted from video data and the emotional state is evaluated in real time using an emotion engine. Through analysis, emotional states such as joy, sadness, anger, surprise, and fear are identified.

[1769] Output: The result of the sentiment analysis is generated.

[1770] Step 8: Generate and send a notification message

[1771] server

[1772] Input: Anomaly detection results and sentiment analysis results.

[1773] Specific operation: The server generates a notification message based on the results of anomaly detection and emotion analysis. For example, it might say, "A fall was detected in the living room and a frightened expression was observed."

[1774] Output: The generated notification message is sent to the family or healthcare provider.

[1775] Step 9: Assess the situation and respond

[1776] User

[1777] Input: Notification message from the server.

[1778] Specific actions: Family members or medical institutions will receive a notification, check the situation, and take action such as contacting the elderly person and arranging rescue if necessary.

[1779] Output: The situation is checked and action is taken.

[1780] Step 10: Gather feedback and improve the model

[1781] server

[1782] Input: Feedback from family or healthcare provider.

[1783] How it works: The server retrains the machine learning model and sentiment engine based on the collected feedback, improving the accuracy of the system and reducing false positives.

[1784] Output: An improved learning model and emotion engine are generated.

[1785] (Application example 2)

[1786] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1787] The main function of conventional elderly monitoring systems was to detect abnormalities by analyzing video data, but the accuracy of the abnormality detection was not always high, and appropriate responses were sometimes delayed.In addition, because the abnormality detection and notification did not take into account the emotions of the elderly, it was difficult to accurately determine the urgency of the situation.This posed the issue of not being able to completely ensure the safety and security of the elderly.

[1788] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a video acquisition means, a means for preprocessing video data, a means for learning from the preprocessed video data and modeling lifestyle patterns, a means for analyzing the video data in real time and detecting anomalies, a means for recognizing the user's emotions and considering them together with the anomaly detection results, a means for generating and transmitting a notification message based on the anomaly detection results and emotion data, and a means for collecting feedback and improving the learning model. This makes it possible to ensure the safety of elderly people in real time and quickly take appropriate measures according to the level of urgency.

[1789] The "video acquisition means" is a device that captures the subject's behavior and facial expressions in real time and acquires them as video data.

[1790] The "means for preprocessing video data" refers to a processing device or software that performs decoding, noise removal, frame normalization, etc. to convert the acquired video data into a format that is easy to analyze.

[1791] The "means for modeling lifestyle patterns" is a system that uses preprocessed video data to learn the subject's daily behavioral patterns using a machine learning algorithm and saves them as a model.

[1792] The "means for detecting anomalies" is an algorithm that compares video data acquired in real time with a learning model to detect behavior that deviates from normal life patterns.

[1793] The "means for recognizing user emotions" is an engine that analyzes the subject's facial expressions from the acquired video data and evaluates their emotional state, such as joy, sadness, or fear, in real time.

[1794] The "means for generating and transmitting a notification message based on the abnormality detection result and emotional data" is a system that creates a notification message that takes into account the detected abnormality and the user's emotional data, and transmits it to the home or medical personnel.

[1795] "Means for collecting feedback and improving the learning model" refers to a mechanism for collecting feedback from households and medical institutions regarding the results of anomaly detection and notification, and using that data to improve the accuracy of the machine learning model.

[1796] The present invention provides specific means and methods for effectively realizing elderly care. The system uses smart glasses as a terminal to monitor the user's behavior and emotions in real time, and sends appropriate notifications to family members and medical personnel when abnormalities occur.

[1797] Hardware Configuration

[1798] The smart glasses are equipped with a built-in camera that constantly captures the user's actions. The video data undergoes initial pre-processing by the smart glasses' on-device processor.

[1799] Software Configuration

[1800] The server receives the data sent from the terminal and performs the following processing.

[1801] 1. Video pre-processing:

[1802] The server decodes the video data sent from the smart glasses, performs noise reduction and frame normalization, and converts it into a format suitable for analysis.

[1803] 2. Learning Life Patterns:

[1804] Using the preprocessed data, the server learns the elderly person's daily life patterns using machine learning algorithms (e.g., TensorFlow / Keras). In the initial stage, data is collected over several weeks to create a model of their normal behavior.

[1805] 3. Real-time analysis and anomaly detection:

[1806] The server compares the video data acquired in real time with the trained model to detect anomalies, and if an anomaly is found, the information is processed immediately.

[1807] 4. Emotion analysis:

[1808] Facial expressions are extracted from the video data, and an emotion analysis engine (e.g., TensorFlow model) is used to analyze the user's emotional state in real time. The resulting emotion data is included in the anomaly detection results.

[1809] 5. Abnormal notification:

[1810] If an anomaly is detected, the server generates a notification message taking into account the user's emotional data. For example, if the user shows a fearful expression, the notification will be sent with a high level of urgency. The notification will be sent via email or a dedicated application.

[1811] 6. Gather feedback and improve the model:

[1812] Family members and medical institutions receive notifications, check the situation, respond, and then provide feedback. The server uses this feedback to retrain the learning model and improve detection accuracy.

[1813] Specific examples

[1814] Suppose an elderly person is wearing smart glasses and going about their daily life at home. Suddenly, they fall, and the smart glasses capture the video. The server immediately analyzes the data and detects the abnormality. At the same time, the emotion analysis engine recognizes that the user has an expression of "fear" on their face. This causes the server to send a high-level emergency notification to their family, urging them to take prompt action. This system transmits highly accurate information to their family and medical institutions in real time, ensuring the safety of the elderly.

[1815] Example prompts for generative AI models

[1816] 1. Capture footage using a webcam or smart glasses.

[1817] 2. Preprocess the video data to remove noise and unnecessary information.

[1818] 3. Use a machine learning model to detect anomalies by comparing learned patterns of daily life with the video data.

[1819] 4. Analyze facial expression data in real time to determine emotional states.

[1820] 5. If an abnormality is detected, adjust the urgency of the notification message to be sent to family members, taking into account the emotional data.

[1821] 6. Collect feedback from families and healthcare providers to retrain and improve your machine learning models.

[1822] The detailed configuration and processing means of the present invention make it possible to provide a highly accurate monitoring system for ensuring the safety of elderly people.

[1823] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1824] Step 1:

[1825] The device (smart glasses) uses a built-in camera to capture the elderly person's behavior and facial expressions in real time. Real-time video data is acquired as input, and this video data is temporarily stored in the device as output.

[1826] Step 2:

[1827] The device preprocesses the captured video data, specifically decoding, denoising, and frame normalizing the video data. The input is the captured video data, and the output is the preprocessed video data.

[1828] Step 3:

[1829] The pre-processed video data is sent to the server through the home network. The input is the pre-processed video data, and the output is obtained by sending this data to the server.

[1830] Step 4:

[1831] The server receives the preprocessed video data and uses a machine learning algorithm to learn the elderly's daily life patterns. The input is the preprocessed video data, and the output is the learned lifestyle pattern model.

[1832] Step 5:

[1833] The server analyzes newly received video data in real time and compares it with the trained model to detect anomalies. The input is the real-time video data and the trained lifestyle pattern model, and the output is the anomaly detection results.

[1834] Step 6:

[1835] The server extracts the user's facial expressions from the video data and analyzes their emotional state in real time using an emotion analysis engine. The input is the video data, and the output is the analysis result of the emotional state.

[1836] Step 7:

[1837] When an anomaly is detected, the server takes into account the emotion analysis results and generates an anomaly notification message with a set urgency level. The input is the anomaly detection result and the emotion analysis result, and the output is the notification message.

[1838] Step 8:

[1839] The server sends the generated notification message to the terminal of the home or medical personnel. The input is the notification message, and the output is the terminal of the home or medical personnel that receives the notification.

[1840] Step 9:

[1841] Users (family members or medical professionals) receive notifications, check and respond, and provide feedback to the server. The input is the notification message, and the output is feedback information.

[1842] Step 10:

[1843] The server uses the received feedback to retrain the learning model and improve the accuracy of anomaly detection. The input is the feedback information, and the output is an improved learning model.

[1844] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1845] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1846] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1847] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1848] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1849] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1850] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1851] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1852] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1853] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1854] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1855] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1856] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1857] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1858] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1859] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1860] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1861] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1862] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1863] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1864] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1865] The following is further disclosed regarding the above embodiment.

[1866] (Claim 1)

[1867] An image acquisition means;

[1868] means for preprocessing the video data;

[1869] A means for learning from the preprocessed video data and modeling life patterns;

[1870] A means of analyzing video data in real time and detecting abnormalities,

[1871] means for transmitting the anomaly detection result to a notification destination;

[1872] A system that includes a means to collect feedback and improve the learning model.

[1873] (Claim 2)

[1874] 10. The system of claim 1, further comprising means for receiving and processing data from IoT devices in a home network in synchronization with the video analytics.

[1875] (Claim 3)

[1876] 10. The system of claim 1, further comprising means for determining whether an anomaly is detected by combining analysis results of successive frames and data from additional sensors.

[1877] "Example 1"

[1878] (Claim 1)

[1879] A means of constantly capturing video to monitor the elderly person's living space;

[1880] means for decoding received video data and dividing it into frames;

[1881] means for normalizing frames, extracting contours and removing noise;

[1882] A means for learning the preprocessed video data using a machine learning algorithm to model lifestyle patterns;

[1883] A means to analyze video data in real time and compare it with existing learning models to detect abnormal behavior;

[1884] means for generating a notification message including the type of abnormal behavior detected, the time of occurrence, and location information, and transmitting the message to a designated notification destination;

[1885] The system includes a means for collecting feedback from users who receive the notifications and retraining and improving the learning model.

[1886] (Claim 2)

[1887] 10. The system of claim 1, further comprising means for receiving and processing data from IoT devices in a home network in synchronization with the video analytics.

[1888] (Claim 3)

[1889] 10. The system of claim 1, further comprising means for determining whether an anomaly is detected by combining analysis results of successive frames and data from additional sensors.

[1890] "Application Example 1"

[1891] (Claim 1)

[1892] An image acquisition means;

[1893] means for preprocessing the video data;

[1894] A means for learning from the preprocessed video data and modeling life patterns;

[1895] A means of analyzing video data in real time and detecting abnormalities,

[1896] means for transmitting the anomaly detection result to a notification destination;

[1897] a means of gathering feedback and improving the learning model;

[1898] a means for monitoring the status of a passenger in the autonomous vehicle;

[1899] A system that includes a means for detecting and notifying passenger anomalies in real time.

[1900] (Claim 2)

[1901] 10. The system of claim 1, further comprising means for receiving and processing data from IoT devices in a home network in synchronization with the video analytics.

[1902] (Claim 3)

[1903] 10. The system of claim 1, further comprising means for determining whether an anomaly is detected by combining analysis results of successive frames and data from additional sensors.

[1904] "Example 2: Combining Emotion Engines"

[1905] (Claim 1)

[1906] An image acquisition means;

[1907] means for preprocessing the video data;

[1908] A means for learning from the preprocessed video data and modeling life patterns;

[1909] A means of analyzing video data in real time and detecting abnormalities,

[1910] means for analyzing facial expressions and assessing emotional state;

[1911] means for generating and transmitting a notification message based on the anomaly detection result and the emotion analysis result;

[1912] A system that includes a means to collect feedback and improve the learning model.

[1913] (Claim 2)

[1914] 10. The system of claim 1, further comprising means for receiving and processing data from IoT devices in a home network in synchronization with the video analytics.

[1915] (Claim 3)

[1916] 10. The system of claim 1, further comprising means for determining whether an anomaly is detected by combining analysis results of successive frames and data from additional sensors.

[1917] "Application example 2 when combining emotion engines"

[1918] (Claim 1)

[1919] An image acquisition means;

[1920] means for preprocessing the video data;

[1921] A means for learning from the preprocessed video data and modeling life patterns;

[1922] A means of analyzing video data in real time and detecting abnormalities,

[1923] A means of recognizing user emotions and taking them into account along with the anomaly detection results;

[1924] means for generating and transmitting a notification message based on the anomaly detection result and emotion data;

[1925] A system that includes a means to collect feedback and improve the learning model.

[1926] (Claim 2)

[1927] 10. The system of claim 1, further comprising means for receiving and processing data from devices in a home network in synchronization with the video analytics.

[1928] (Claim 3)

[1929] 10. The system of claim 1, further comprising means for determining whether an anomaly is detected by combining analysis results of successive frames and data from additional sensors. [Explanation of symbols]

[1930] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. An image acquisition means; means for preprocessing the video data; A means for learning from the preprocessed video data and modeling life patterns; A means of analyzing video data in real time and detecting abnormalities, means for transmitting the anomaly detection result to a notification destination; A system that includes a means to collect feedback and improve the learning model.

2. 10. The system of claim 1, further comprising means for receiving and processing data from IoT devices in a home network in synchronization with the video analytics.

3. The system of claim 1 , further comprising means for determining whether an anomaly is detected by combining analysis results of successive frames and data from additional sensors.

Images