System

Abstract

A system is provided.SOLUTION: A system including means for acquiring video data from a camera device installed in a home to monitor a symptom, means for uploading the acquired video data to a server, means for analyzing the video data uploaded to the server and extracting a feature amount of the symptom, means for using an artificial intelligence model for generating an explanatory text of the symptom on the basis of the extracted feature amount, means for storing the generated explanatory text and an analysis result in a database, means for notifying a user terminal of the stored data, and means for triggering an emergency response on the basis of a degree of urgency.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the spread of telework and remote care in modern times, the importance of monitoring and watching over medical conditions at home is increasing. It is particularly difficult for busy individuals working in companies to monitor sick or elderly people on a daily basis, and it is difficult to respond immediately when traveling or on business trips. Furthermore, it is often difficult to properly explain specific symptoms when seeing a doctor, increasing the risk of delayed diagnosis. To address these issues, there is a need for a system that can provide effective support. [Means for solving the problem]

[0005] The present invention provides a means for analyzing video data acquired from a camera device installed in the home and extracting symptom features. It also incorporates a means for generating a specific symptom description using a generative AI model based on the extracted features. The generated description and analysis results are stored in a database and notified to the user's device, enabling daily monitoring. The system also provides a means for evaluating the severity and urgency of symptoms and, if necessary, issuing an audio warning or dispatching an ambulance, enabling a prompt and appropriate emergency response. This system can also improve the accuracy of diagnoses by presenting the analysis results to the user during a doctor's consultation.

[0006] "Symptom monitoring" refers to the continuous observation of the condition of sick people, elderly people, etc. using camera devices installed in the home.

[0007] A "camera device" refers to a device used to acquire video data, and is typically installed in various places in the home.

[0008] "Video data" refers to information about video captured by a camera device that has been recorded in digital format and is used to analyze the characteristics of a medical condition.

[0009] "Server" refers to the computer system that receives and stores video data, performs analysis, and manages the data.

[0010] "Upload" refers to the process of sending data (in this case, video data) from a device to a server.

[0011] "Analysis" refers to the operations and calculations performed to process the information contained in the video data and extract specific symptoms or characteristics.

[0012] "Symptom features" refers to specific numerical values ​​or characteristics (such as movement patterns or changes in facial expressions) that indicate the condition of the disease, extracted through analysis.

[0013] "Explanatory text" refers to a sentence that specifically describes the symptoms, generated by the generative AI model based on the extracted symptom features.

[0014] "Generative AI model" refers to an artificial intelligence model used to analyze symptom features from video data and generate explanatory text based on that.

[0015] A "database" refers to a system for systematically storing and managing information such as analysis results and generated explanatory text.

[0016] A "user terminal" is a device used by a user, such as a smartphone or computer, that is used to receive notifications and display analysis results.

[0017] "Notification" refers to the act of transmitting information to a user's device, such as analysis results or urgency assessment results.

[0018] "Urgency" refers to an index that evaluates how urgent a symptom is based on the extracted feature quantities of the symptom.

[0019] "Emergency response" means taking immediate action in response to highly urgent symptoms, and specifically includes issuing audio alerts and arranging for an ambulance.

[0020] "Voice warning" refers to the act of issuing a warning sound or voice message through a speaker installed on a user terminal or in the home in the event of a high level of emergency.

[0021] "Dispatching an ambulance" refers to the process in which the system automatically calls an ambulance when a highly urgent symptom occurs. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention is a system for monitoring the condition of sick people and the elderly and providing timely medical care. This system is realized using a camera device, a server, and a user terminal installed in the home.

[0044] System configuration

[0045] The system consists of the following main components:

[0046] Camera device: Installed in a specific location in the home (living room, bedroom, etc.) and captures video data continuously.

[0047] Terminal: Manages the video data acquired from the camera and uploads it to the server. The terminal is also a device (usually the user's smartphone or tablet) that receives notifications from the server.

[0048] Server: Receives video data, performs analysis, stores the results in a database, and takes emergency action based on the analysis results.

[0049] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[0050] Program processing

[0051] 1. Recording and uploading the video

[0052] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[0053] 2. Video analysis and symptom feature extraction

[0054] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for specifically describing the symptoms.

[0055] 3. Symptom Description Generation

[0056] Based on the extracted features, the generative AI model uses natural language processing techniques to generate a specific symptom description. This description is written in a format that is easy for users and doctors to understand. The server stores this generated description in a database for future reference.

[0057] 4. Urgency Assessment and Notification

[0058] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[0059] Specific examples

[0060] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify the family.

[0061] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User: A camera installed in the home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[0065] Step 2:

[0066] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[0067] Step 3:

[0068] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[0069] Step 4:

[0070] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[0071] Step 5:

[0072] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[0073] Step 6:

[0074] Generative AI model: Based on the extracted features, a specific description of the symptoms is generated using natural language processing technology. The generated description is structured in a format that is easy for users to understand.

[0075] Step 7:

[0076] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[0077] Step 8:

[0078] Server: After confirming that the data has been saved, it sends a notification to the user device. The notification includes a summary of the analysis results and a generated explanation.

[0079] Step 9:

[0080] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[0081] Step 10:

[0082] Server: Evaluates the urgency of symptoms based on the saved analysis data. If the urgency is determined to be high, an emergency response flag is set.

[0083] Step 11:

[0084] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[0085] Step 12:

[0086] Terminal: The Y terminal issues a voice alert announcing, "An ambulance has been called and will arrive shortly." At the same time, a notification is sent to the user's family, who are informed, "Symptoms have been observed and emergency response is required."

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In modern society, the number of elderly people and patients with medical conditions is increasing, making home health monitoring systems increasingly important. However, conventional systems lack the ability to detect sudden health abnormalities occurring within the home in real time and to promptly take appropriate measures. In particular, there is a need for a system that can accurately grasp the specific characteristics of symptoms and accurately determine the level of urgency. There is also a lack of means for smoothly sharing information with medical providers. In light of this background, the present invention aims to provide a system that accurately monitors health abnormalities occurring within the home and promptly takes appropriate measures.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for acquiring video data from a video recording device installed in the home to monitor symptoms, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting symptom features, means for using an artificial intelligence system to generate a symptom description based on the extracted features, means for storing the generated description and analysis results in a data storage system, means for notifying a user terminal of the stored data, means for triggering an emergency response based on the urgency level, and means for coordinating and executing each stage of processing. This enables real-time detection of health abnormalities occurring in the home and enables prompt and appropriate response. Furthermore, smooth information sharing with medical providers can support accurate diagnosis and treatment.

[0092] A "symptom" is a change in bodily function or condition that occurs due to an abnormality or disease in the human body.

[0093] "Monitoring" is the act of carefully observing and recording events or situations in progress.

[0094] "Home" refers to the space inside a building where a family lives and conducts their daily lives.

[0095] A "video recording device" is a device such as a camera or surveillance camera that captures and stores video data.

[0096] "Video data" refers to the digital data of the captured video.

[0097] A "central processing unit" is a server or computer system that receives, processes, and analyzes data.

[0098] "Analysis" is the act of examining data or information in detail and understanding its content and structure.

[0099] "Features" are important parameters or indicators extracted from data to represent symptoms or events.

[0100] An "artificial intelligence system" is a computer system that uses certain algorithms to process data, learn, and make inferences.

[0101] A "description" is a text written in natural language about a specific phenomenon or symptom.

[0102] A "data storage system" is a system for safely storing data for a long period of time.

[0103] "User terminal" means a device that is directly operated by a user to receive and display information.

[0104] "Notification" is the act of conveying specific information to a recipient.

[0105] "Emergency response" refers to immediate response measures taken in response to a sudden event.

[0106] "Collaboration" refers to the act of multiple systems or processes working together.

[0107] This invention relates to a system for real-time monitoring of health abnormalities occurring in the home and for promptly taking appropriate measures. This system is realized using a video recording device, a central processing unit (server), a user terminal, and an artificial intelligence system.

[0108] Hardware and software used

[0109] 1. Video Recording Device

[0110] A user installs a video recording device in a specific location in the home (e.g., the living room or bedroom). The video recording device is a home surveillance camera (e.g., a home IP camera). This device continuously captures video data and automatically records the video if an abnormality is detected.

[0111] 2. User terminal

[0112] The terminal is responsible for managing the video data acquired from the video recording device and uploading it to the central processing unit. The terminal can be a smartphone or tablet. After receiving the video data, the user terminal sends it to the central processing unit and also receives notifications from the server.

[0113] 3. Central Processing Unit (Server)

[0114] The server receives and analyzes the video data uploaded from the device. Specifically, it uses an AI analysis platform (for example, a cloud service such as AWS or Microsoft Azure). The server analyzes the video data, extracts symptom features, and passes them to the AI ​​model. The server also stores the analysis results and generated explanations in a database and takes action based on the urgency assessment.

[0115] 4. Artificial Intelligence Systems

[0116] The generative AI model extracts symptom features from video data and generates symptom descriptions based on them. The AI ​​model used uses deep learning frameworks such as TensorFlow and PyTorch. In particular, the generative AI model uses natural language processing technology (such as GPT-3) to generate specific symptom descriptions.

[0117] Specific operation of the system

[0118] A user keeps a camera device installed in their home turned on at all times. The camera device continues to capture images of the surroundings in real time, and if an abnormality is detected, the image is automatically recorded.

[0119] The device periodically receives video data from the camera and uploads it to a central processing unit (server). The server adds the received video data to an analysis queue and passes it sequentially to the generative AI model. The generative AI model analyzes the video data frame by frame and extracts features such as abnormal movements, changes in facial expression, and changes in skin color.

[0120] Based on the extracted features, the generative AI model uses natural language processing to generate a specific description of the symptoms, such as "an elderly person suddenly collapsed, their face was pale, and their breathing was becoming labored."

[0121] The server stores the generated explanation and analysis results in a database and sends emergency notifications to the user's device as needed. If the emergency is severe, the server will issue a voice warning or automatically arrange for emergency medical transport.

[0122] Examples of concrete examples and prompts

[0123] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to a server via the device. The server receives the video data and begins analysis using a generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, arrange for emergency medical transport, and notify the user's family.

[0124] An example of a prompt sentence is, "Please generate an analysis result of a scene in which an elderly person collapses." The model will extract the features of the specified scene and generate a detailed description.

[0125] This system allows users to respond quickly and appropriately to emergencies that occur at home and provides accurate information to medical providers.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1: Capture and transmit video data

[0128] A user turns on a video recording device installed in their home. The camera device continuously captures video data and automatically records the video if an abnormality occurs. The recorded video is sent to a terminal in real time. The input is the video data captured by the camera device, and the output is the recorded video data sent to the terminal. Specifically, the camera device detects a sudden fall by an elderly person and records the scene.

[0129] Step 2: Receiving and uploading video data

[0130] The terminal receives video data sent from the camera device and uploads it to the server at regular intervals. The input is the recorded video data received from the camera device, and the output is the video data uploaded to the server. Specifically, the terminal receives video data from the camera device and immediately uploads it to the cloud server.

[0131] Step 3: Video data analysis and feature extraction

[0132] The server receives the video data uploaded from the device and adds it to the analysis queue. The server then passes the video data in the analysis queue to the generative AI model, which then begins analysis. The generative AI model analyzes the video data frame by frame and extracts symptom features such as abnormal movements, changes in facial expression, and changes in skin color. The input is the video data uploaded to the server, and the output is the extracted symptom features. Specifically, this is the AI ​​model's behavior in identifying elderly people's falling movements and changes in facial color.

[0133] Step 4: Generate symptom descriptions

[0134] The server inputs the extracted features into a generative AI model to generate a description of the symptoms. The generative AI model uses natural language processing technology to generate a description and returns it to the server. The input is the extracted features of the symptoms, and the output is the generated description of the symptoms. Specifically, the operation generates a description such as "After the elderly person collapsed, their face turned pale and their breathing became rough."

[0135] Step 5: Save the description and analysis results

[0136] The server saves the generated symptom descriptions and analysis results in a database. The saved data is archived for future reference. The input is the generated symptom descriptions and analysis results, and the output is the records stored in the database. Specifically, the operation is to store the generated descriptions and analysis results in an SQL database.

[0137] Step 6: Urgency assessment and notification

[0138] The server evaluates the generated analysis results and symptom descriptions to determine the level of urgency. If the level of urgency is high, the server sends an audio warning instruction to the terminal and arranges for emergency medical transport if necessary. The input is the generated analysis results and symptom descriptions, and the output is an alert instruction and response arrangements based on the level of urgency. Specifically, the server determines that the level of urgency is high and sends an audio warning notification to the user's smartphone.

[0139] Step 7: Implementing notifications and following up

[0140] The device receives the voice warning instruction sent from the server and notifies the user. At the same time, if the emergency level is high, the device notifies the user's family as well. The input is the warning instruction from the server, and the output is a voice notification and an alert to the user. Specifically, the smartphone sends a voice message to the elderly person's family saying, "An emergency has occurred. Please respond immediately."

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] There is a need to safely monitor the elderly and sick at home, quickly detect any sudden changes in their condition, and take appropriate action. However, existing systems have difficulty simultaneously monitoring their condition and security, making it difficult to respond quickly and accurately. Therefore, a highly reliable system that can comprehensively monitor health and safety at home is needed.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes means for acquiring video data from a video device installed in the home to monitor symptoms and abnormalities, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting feature quantities of symptoms and abnormalities, means for using a generative artificial intelligence model to generate an explanatory text based on the extracted feature quantities, means for saving the generated explanatory text and analysis results in a recording medium, means for notifying a user terminal of the saved data, and means for triggering medical and security responses based on the urgency level. This enables comprehensive monitoring of health and safety within the home, and enables appropriate medical and security responses to be taken immediately when an abnormality is detected.

[0146] "Symptoms and abnormalities" refers to changes in the health status of a person or abnormal behavior, such as collapsing or changes in facial color, that are not normal.

[0147] "Video equipment" refers to cameras and recording devices installed in the home that continuously capture video data.

[0148] "Video data" refers to video information captured within the home by a video device.

[0149] A "central processing unit" is a device, such as a server or computer system, that analyzes acquired video data and performs the necessary calculations.

[0150] "Features" are specific attributes or patterns extracted from video data to explain symptoms or abnormalities.

[0151] A "generative artificial intelligence model" is a model that uses artificial intelligence technology to extract features of symptoms and abnormalities from video data and generate explanatory text in natural language based on this.

[0152] "Recording medium" refers to a database or storage device for saving the analysis results and generated explanations.

[0153] A "user terminal" is a device used to notify the analysis results and generated explanatory text, and typically refers to a smartphone or tablet.

[0154] "Urgency" refers to the result of evaluating the seriousness of detected symptoms or abnormalities, and serves as a criterion for determining whether immediate action is required.

[0155] "Medical response" refers to medical treatment or intervention based on the urgency of the situation, such as dispatching an ambulance.

[0156] "Security response" refers to crime prevention measures taken when an abnormality is detected within the home, including issuing emergency notifications and warnings.

[0157] System Overview

[0158] The system for implementing this invention is composed of a video device, a central processing unit, a user terminal, and software for linking these devices installed in the home. The system is designed to simultaneously monitor the user's health and the safety of the home, and to respond immediately if an abnormality is detected.

[0159] Hardware and software used

[0160] Hardware:

[0161] Video device (camera): Continuously captures video from within the home.

[0162] Central processing unit (server): Receives captured video data and performs analysis.

[0163] User device (smartphone / tablet / smart glasses): A device that notifies the analysis results.

[0164] software:

[0165] OpenCV: A library for camera operation and image processing.

[0166] TensorFlow: A library for analyzing video data using generative artificial intelligence models.

[0167] smtplib: A library for sending analysis results and notifications by email.

[0168] Program processing

[0169] The server periodically acquires video data from cameras installed in the home and analyzes the data using a central processing unit. During the analysis, a generative artificial intelligence model is used to extract abnormal features (e.g., falling, changes in facial color). The extracted features are converted into explanatory text using natural language processing technology, and the analysis results are saved on a recording medium.

[0170] Program execution procedure

[0171] 1. Acquisition of video data:

[0172] The video device continuously monitors a specific area in the home (e.g., living room, bedroom) and acquires video data.

[0173] 2. Upload data:

[0174] The captured video data is uploaded to the central processing unit via the user terminal.

[0175] 3. Data Analysis:

[0176] The central processing unit analyzes the uploaded video data using AI libraries such as TensorFlow.

[0177] Based on the analysis, feature quantities of symptoms and abnormalities are extracted.

[0178] 4. Description generation:

[0179] Based on the extracted features, a generative artificial intelligence model generates a natural language explanation.

[0180] 5. Data Retention and Notification:

[0181] The generated explanation and analysis results are stored on a recording medium, and a notification is sent to the user terminal according to the level of urgency.

[0182] Specific examples

[0183] For example, if an elderly person collapses at night, the camera detects the movement and sends the video data to a central processing unit. The central processing unit extracts features such as the collapse movement and changes in facial color, and generates a description such as "The elderly person suddenly collapsed, their face turned pale, and their breathing became heavy." This result is saved in the central processing unit, and if the emergency is deemed high, medical and security measures are immediately implemented. A notification is immediately sent to the user's device, and an emergency vehicle may be dispatched, and an alert may also be sent to the family.

[0184] Prompt Sentence Examples

[0185] "We are building a new system to monitor family members. We would like the system to use a camera to detect abnormalities such as elderly or sick members collapsing or becoming pale, and then take appropriate action. What kind of prompt would be appropriate?"

[0186] In this way, the system can take appropriate measures quickly and effectively support health and safety monitoring within the home.

[0187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0188] Step 1:

[0189] A video device (camera) in the home captures video data. The camera continuously monitors a specific area, such as the living room or bedroom, and collects video data at a constant frame rate. The input is real-time video from the camera, and the output is the captured video frames.

[0190] Step 2:

[0191] The terminal uploads the video data acquired from the camera to a central processing unit (server). The terminal buffers video frames at regular intervals and transmits them to the central processing unit via the network. The input is the video data from the video device, and the output is the video data uploaded to the server.

[0192] Step 3:

[0193] The server receives the uploaded video data and begins analysis. The server divides the video data into frames and inputs them into a generative AI model. The input is the uploaded video data, and the output is image data for each frame to perform analysis.

[0194] Step 4:

[0195] The server uses a generative AI model to analyze the video data and extract features of symptoms and abnormalities. Typically, an AI library such as TensorFlow is used, and the model detects abnormal behavior and changes from the video data. The input is frame image data for analysis, and the output is feature data (e.g., the behavior of falling, changes in facial color).

[0196] Step 5:

[0197] The server uses a generative AI model to generate a description of the symptoms or abnormalities based on the extracted features. The generated description is created using natural language processing (NLP) technology. The input is feature data, and the output is a description in a format that is easy for humans to understand.

[0198] Step 6:

[0199] The server stores the generated explanation and analysis results on a recording medium. The server stores the generated explanation and related analysis results in a database. The input is the generated explanation and analysis results, and the output is the data stored in the database.

[0200] Step 7:

[0201] The server evaluates the urgency and sends a notification to the user device if it is determined to be highly urgent. The urgency evaluation is performed using an AI model or rule-based system. The input is the generated explanation and analysis results, and the output is notification data sent to the user device.

[0202] Step 8:

[0203] The server sends a notification to the user's smart device (e.g., smartphone, tablet, smart glasses). The user device responds based on the received notification, such as issuing an audio warning. The input is the result of the urgency assessment and notification data, and the output is a real-time notification and audio warning.

[0204] Step 9:

[0205] The user checks the notification and takes additional action as necessary (e.g., contacting a medical institution or reporting to a security agency). The input is the notification sent to the user terminal, and the output is the user's response action.

[0206] Step 10:

[0207] The server automatically triggers appropriate actions (e.g. dispatching emergency vehicles, notifying security). In the case of a high-urgency situation, the system automatically executes triggers to ensure a prompt response. The input is the result of the urgency assessment, and the output is the automatic trigger action.

[0208] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0209] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. This system analyzes changes in the user's medical condition and emotional state in real time, enabling appropriate medical response.

[0210] System configuration

[0211] The system consists of the following main components:

[0212] Camera device: Installed in a specific location in the home, it constantly captures video data.

[0213] Terminal: Manages video data and uploads it to the server. The terminal is also a device that receives notifications from the server.

[0214] Server: Receives and analyzes video data, stores the results in a database, and takes emergency action based on the analysis results.

[0215] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[0216] Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis.

[0217] Program processing

[0218] Recording and uploading video

[0219] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[0220] Video analysis and symptom feature extraction

[0221] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for describing specific symptoms.

[0222] Further analysis of the emotion engine

[0223] Furthermore, the emotion engine analyzes the user's emotional state from the video and audio data. The emotion engine uses, for example, facial expression analysis and audio analysis to identify the user's emotions (e.g., surprise, fear, anger, etc.). This emotional state information is integrated with the symptom analysis results to provide comprehensive information for judgment.

[0224] Symptom description generation

[0225] Based on the extracted features and the analysis results of the emotion engine, the generative AI model uses natural language processing technology to generate a specific description of the symptoms. This description is written in a format that is easy for users and doctors to understand. The server stores the generated description in a database.

[0226] Urgency assessment and notification

[0227] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. The results of the emotion engine are also taken into account here. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[0228] Specific examples

[0229] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "The elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[0230] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[0231] The processing flow will be explained below.

[0232] Step 1:

[0233] User: A camera installed in the user's home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[0234] Step 2:

[0235] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[0236] Step 3:

[0237] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[0238] Step 4:

[0239] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[0240] Step 5:

[0241] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[0242] Step 6:

[0243] Server: The extracted features are formatted for storage in a database, making them possible to integrate with existing symptom data.

[0244] Step 7:

[0245] Emotion engine: Analyzes the user's emotional state from video and audio data. For example, facial expression analysis and audio analysis are used to identify emotions such as surprise, fear, and anger. Emotional state information is integrated with symptom analysis results.

[0246] Step 8:

[0247] Generative AI model: Based on the extracted features and the analysis results of the emotion engine, a specific description of the symptoms is generated using natural language processing technology. This description is constructed in a format that is easy for users to understand.

[0248] Step 9:

[0249] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[0250] Step 10:

[0251] Server: After confirming that the data has been saved, it sends a notification to the user device, which includes a summary of the analysis results and a generated explanation.

[0252] Step 11:

[0253] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[0254] Step 12:

[0255] Server: Evaluates the urgency of symptoms based on the saved analysis data. Taking into account the results of the emotion engine, if the urgency is evaluated as high, sets an emergency response flag.

[0256] Step 13:

[0257] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[0258] Step 14:

[0259] Device: The device issues a voice alert, announcing that an ambulance has been called and will arrive shortly. At the same time, a notification is sent to the user's family, informing them that symptoms have been observed and that emergency response is required.

[0260] This detailed processing step allows for real-time monitoring of the user's medical condition and emotional state, enabling prompt and accurate responses, and the stored data can then be used by doctors for more accurate and effective treatment.

[0261] Example 2

[0262] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0263] When the condition of an elderly person or a sick person suddenly changes at home, it is difficult to quickly and accurately monitor and judge the change and take appropriate measures promptly. In particular, for elderly people living alone, there is a lack of means to grasp changes in the condition in real time and take emergency measures, so a system that solves this problem is needed.

[0264] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server via a terminal, means for analyzing the video data uploaded to the server and extracting symptom features, means for using a generative AI model to generate a symptom description based on the extracted features, means for analyzing an emotional state from video data and audio data, means for storing the generated description and analysis results in a database, means for notifying a user terminal of the stored data, and means for triggering an emergency response based on the urgency. This makes it possible to monitor changes in the condition of elderly people and sick people in real time and take prompt and appropriate medical action.

[0265] A "symptom" is a sign or phenomenon that indicates an abnormal physical or mental condition or malfunction.

[0266] A "camera device" is a hardware device for acquiring video data.

[0267] "Video data" refers to digital data containing visual information captured by a camera device.

[0268] A "terminal" is a relay device for uploading video data to a server.

[0269] "Server" means a computer system on a network that receives, analyzes, and stores video data.

[0270] "Features" refer to important attributes or elements that make up symptoms and are extracted from the subject of analysis.

[0271] A "generative AI model" is an artificial intelligence that generates symptom descriptions based on extracted features.

[0272] "Emotional state" refers to the mental state and emotions of the user that are analyzed from video data and audio data.

[0273] "Database" refers to an information storage system for storing generated descriptions and analysis results.

[0274] A "user terminal" is a device that receives information stored in a database and notifies the user.

[0275] "Urgency" is an indicator that shows the seriousness of the symptoms and the need for response.

[0276] "Emergency response" refers to the actions and means to provide prompt medical treatment according to the degree of urgency.

[0277] The present invention is a system for monitoring the medical and emotional state of elderly and sick people and for responding quickly in emergencies. The system consists of the following main components:

[0278] 1. Camera device: A device that is installed in a specific location in the home and captures video data at all times. In this invention, the general term camera device is used.

[0279] 2. Device: A device that manages video data and uploads it to the server. The device also receives notifications from the server. Examples of devices that fall under this category include, but are not limited to, the Google Pixel and the Apple iPad.

[0280] 3. Server: This is a system that receives and analyzes video data and stores the results in a database. As a specific example, we use an AWS EC2 instance.

[0281] 4. Generative AI model: This is an artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these. A TensorFlow-based model is used as the AI ​​model.

[0282] 5. Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis. An example is IBM Watson's emotion analysis engine.

[0283] 6. Database: This is an information storage system for saving the generated explanations and analysis results. Database systems such as MySQL or MongoDB are used.

[0284] 7. User terminal: A device that receives information stored in the database and notifies the user. Devices such as Raspberry Pi fall into this category.

[0285] Operation overview

[0286] A user constantly monitors video data using a camera device installed in the home. For example, when the camera device detects an elderly person falling, it records the video.

[0287] The device automatically uploads the recorded video to the server. For example, if the device is a Google Pixel smartphone, the video data is stored from the device in an AWS S3 bucket and then sent to an AWS EC2 instance for analysis.

[0288] When the server receives the uploaded video data, it begins analyzing it using an AI model. For example, a TensorFlow-based AI model analyzes the video data frame by frame and extracts features such as abnormal behavior and changes in facial color.

[0289] The emotion engine further analyzes the emotional state based on the extracted features. For example, IBM Watson's emotion analysis engine can be used to identify emotions such as "pain" or "fear" from the user's facial expressions and voice.

[0290] Based on these features and the analysis results of the emotion engine, the generative AI model generates a specific description of the symptoms, such as, "An elderly person suddenly collapsed, their face was pale, and their breathing was heavy. They appeared to be in pain."

[0291] The server stores the generated descriptions in a database, for example a MySQL database.

[0292] The user device receives the stored data and notifies the user or their family. For example, the Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio.

[0293] Prompt Sentence Examples

[0294] "If the user is elderly and suddenly collapses in the middle of the night, please explain in detail the process of detecting abnormal movements and changes in facial expressions from video data recorded by a camera device, identifying the emotion of pain using an emotion analysis engine, and dispatching an ambulance based on the results."

[0295] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0296] Step 1:

[0297] Users constantly monitor their own situation using cameras installed in their homes. If the camera detects an abnormality, it starts recording. For example, if the user suddenly collapses, the video of the incident is automatically recorded. This video data becomes the input.

[0298] Step 2:

[0299] The device automatically uploads the recorded video to the server. During this process, the video data is temporarily stored on the device and then sent to the server via the network. For example, devices such as the Google Pixel and Apple iPad are used. The device inputs the video data (mp4 format) and uploads it to the server.

[0300] Step 3:

[0301] The server receives the uploaded video data. A dedicated script running on an AWS EC2 instance adds the received video data to the analysis queue. The server receives the video data as input and processes it by passing it to the AI ​​model.

[0302] Step 4:

[0303] The server passes the received video data to a TensorFlow-based AI model and begins analysis. The AI ​​model analyzes each video frame and extracts features such as abnormal behavior and changes in facial color. In this step, the video data is input and features (abnormal behavior, changes in facial color, etc.) are obtained as output. Specifically, the AI ​​model analyzes each frame and detects abnormal patterns.

[0304] Step 5:

[0305] The server passes the features to IBM Watson's emotion analysis engine for additional analysis. The emotion analysis engine analyzes the user's emotional state (e.g., pain, fear) from the video and audio data. In this step, the features are the input and the emotional state is obtained as the output. Specifically, the emotion analysis engine identifies changes in facial expressions and voice to analyze the user's emotions.

[0306] Step 6:

[0307] The server uses a generative AI model to generate a symptom description based on the results obtained from the AI ​​model and the emotion analysis engine. For example, a description such as "An elderly person suddenly collapsed, his face was pale, his breathing was rough, and he was in pain" is generated. In this step, the features and emotional state are input, and a symptom description is obtained as the output. Specifically, the generative AI model constructs the description using natural language processing technology.

[0308] Step 7:

[0309] The server stores the generated explanations and analysis results in a database. For example, this information is stored in MySQL or MongoDB. In this step, the symptom explanations and analysis results are input, and the output is stored in the database.

[0310] Step 8:

[0311] The user device receives the stored data and notifies the user and their family. For example, a Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio. Specifically, notifications from the database are input, and based on that, voice alerts and dispatch of an ambulance are output.

[0312] In this way, each processing step works closely together to realize a system that monitors the user's condition and responds to emergencies quickly and appropriately.

[0313] (Application example 2)

[0314] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] Conventional home monitoring systems have problems with insufficient monitoring of the condition of elderly and sick people, making it difficult to respond quickly and appropriately when an emergency occurs. Furthermore, because the level of urgency is determined solely by symptom analysis without taking into account the user's emotional state, appropriate responses may be delayed. Furthermore, there is a lack of a way to present the analysis results in an easy-to-understand format, making it difficult to provide accurate information to family members and doctors.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0317] In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server, means for analyzing the video data uploaded to the server and extracting features of the symptoms, means for using an artificial intelligence model to generate a description of the symptoms based on the extracted features, means for saving the generated description and analysis results in a database, means for notifying a user terminal of the saved data, means for triggering an emergency response based on the level of urgency, means for analyzing an emotional state from the acquired video data and audio data using an emotion engine, and means for integrating the analysis results of the emotional state with the analysis results of the symptoms. This enables the user's medical condition and emotional state to be analyzed in real time, enabling prompt and appropriate response in emergencies.

[0318] A "camera device" is a device that is installed in the home and is used to acquire video data.

[0319] A "server" is a central processing unit that receives acquired video data, performs analysis, and stores and notifies the results.

[0320] "Video data" is a collection of continuous image information captured by a camera device.

[0321] "Upload" refers to the process of sending video data acquired from a camera device to a server.

[0322] "Analysis" is the process of extracting specific information from the acquired video data and organizing the detailed information using machine learning models and artificial intelligence technology.

[0323] A "feature" is a specific pattern or indicator of data extracted through analysis, and is an element that represents symptoms or emotional states.

[0324] The "explanation" is a detailed document describing the patient's condition and emotional state in natural language, generated based on the extracted features.

[0325] An "artificial intelligence model" is a set of algorithms and programs used to analyze acquired data, extract features, and generate explanatory text.

[0326] A "database" is a system for storing and managing analysis results and generated explanatory text.

[0327] A "user terminal" is a terminal device for receiving and displaying notifications of stored data.

[0328] The "emotion engine" is a software engine that analyzes the user's emotional state from video and audio data and outputs the results.

[0329] "Urgency" refers to the level of necessity for emergency response, determined based on the analysis results.

[0330] "Emergency response" is a process of arranging for an ambulance and notifying family members based on the level of urgency.

[0331] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. A specific embodiment based on an application called "Safeguard Watch" will be described.

[0332] The server acquires video data from a camera device installed in the home. This camera device continuously monitors the user's daily life in real time and acquires important video data. It also has an audio input device, so it can acquire the user's voice data.

[0333] Video and audio data are uploaded to a server via a device installed in the home. The server receives this data and prepares it for storage. The server then analyzes the uploaded video and audio data. A generative AI model and an emotion engine are used for the analysis. The generative AI model extracts symptom features and emotional states from the video and audio data. For example, abnormal movements, changes in facial color, changes in facial expression, and trembling voice are analyzed as features. The emotion engine also analyzes the user's emotional state (e.g., surprise, fear, anger, etc.).

[0334] Based on the extracted features and emotional state, the generative AI model generates a specific description of the symptoms. The generated description is presented in a format that is easy for users and doctors to understand. This description and the analysis results are stored in a database on the server. The saved data is then sent to the user's device. The user's device can receive the notification and issue a warning to the user. It can also trigger an emergency response based on the level of urgency. If the level of urgency is high, the server immediately issues an audio warning and automatically dispatches an ambulance. The system also notifies the user's family members about the symptoms that have occurred.

[0335] This system allows users to respond quickly and appropriately to emergency situations. It also enables doctors to receive accurate information. It is expected to effectively support daily disease management and monitoring. Specific examples are shown below.

[0336] If an elderly person suddenly collapses at home in the middle of the night, the camera automatically records the footage and uploads it to a server via a device. The server receives the video data and begins analysis using a generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[0337] Prompt Sentence Examples

[0338] "Please analyze the user's falling motion and changes in facial expressions from multiple angles to identify changes in the patient's condition in real time."

[0339] "Analyze the behavioral patterns of elderly people when their complexion suddenly changes and suggest appropriate emergency responses."

[0340] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0341] Step 1:

[0342] The terminal acquires video data in real time from a camera device installed in the home, and this video data is immediately transmitted to the terminal along with audio data collected by an audio input device installed in the home.

[0343] input:

[0344] Real-time video and audio data from home camera devices and audio input devices.

[0345] output:

[0346] Capture of acquired video and audio data.

[0347] Step 2:

[0348] The device uploads the captured video and audio data to the server, where it undergoes format conversion and compression before being sent in a format that the server can use for analysis.

[0349] input:

[0350] Video data and audio data acquired from a camera device and an audio input device.

[0351] output:

[0352] Video and audio data that has been converted and compressed into an analyzable format.

[0353] Step 3:

[0354] The server receives the uploaded video and audio data and passes it to the generative AI model and emotion engine. The generative AI model extracts symptom features from the video data, and the emotion engine analyzes the emotional state from the audio data.

[0355] input:

[0356] Video and audio data uploaded to the server.

[0357] output:

[0358] Emotional state data analyzed from symptom features of video data and audio data.

[0359] Specific behavior:

[0360] It uses a generative AI model to detect abnormalities in behavior and facial expressions in each frame of video data, and an emotion engine to identify emotional states from audio data.

[0361] Step 4:

[0362] The server generates a specific description of the symptoms based on the analysis results obtained from the generative AI model and the emotion engine. The generative AI model documents the description using natural language processing (NLP) techniques.

[0363] input:

[0364] Symptom features from the generative AI model and emotional state data from the emotion engine.

[0365] output:

[0366] A detailed description of the symptoms.

[0367] Specific behavior:

[0368] Using NLP technology, a description such as "An elderly person suddenly collapsed, his face was pale, and his breathing was rough" was generated based on the analysis results.

[0369] Step 5:

[0370] The server stores the generated descriptions and analysis results in a database for future reference and emergency response decisions.

[0371] input:

[0372] Generated explanations and analysis results.

[0373] output:

[0374] Descriptions and analysis results stored in a database.

[0375] Specific behavior:

[0376] The data is saved to the database and managed with a timestamp.

[0377] Step 6:

[0378] The server notifies the user device of the stored data, which then receives the notification, issues a warning to the user, and triggers an emergency response based on the urgency level.

[0379] input:

[0380] Descriptions and analysis results stored in a database.

[0381] output:

[0382] Notifications to user devices and emergency response triggers.

[0383] Specific behavior:

[0384] A push notification is sent to the user's device. If the emergency is severe, an audio warning is issued and an ambulance is automatically dispatched. The user's family is also notified of the situation.

[0385] The above steps enable real-time monitoring of the medical conditions and emotional states of elderly and sick people at home, and enable prompt and appropriate response to emergencies.

[0386] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0387] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0388] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0389] [Second embodiment]

[0390] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0391] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0392] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0393] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0394] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0395] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0396] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0397] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0398] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0399] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0400] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0401] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0402] The present invention is a system for monitoring the condition of sick people and the elderly and providing timely medical care. This system is realized using a camera device, a server, and a user terminal installed in the home.

[0403] System configuration

[0404] The system consists of the following main components:

[0405] Camera device: Installed in a specific location in the home (living room, bedroom, etc.) and captures video data continuously.

[0406] Terminal: Manages the video data acquired from the camera and uploads it to the server. The terminal is also a device (usually the user's smartphone or tablet) that receives notifications from the server.

[0407] Server: Receives video data, performs analysis, stores the results in a database, and takes emergency action based on the analysis results.

[0408] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[0409] Program processing

[0410] 1. Recording and uploading the video

[0411] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[0412] 2. Video analysis and symptom feature extraction

[0413] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for specifically describing the symptoms.

[0414] 3. Symptom Description Generation

[0415] Based on the extracted features, the generative AI model uses natural language processing techniques to generate a specific symptom description. This description is written in a format that is easy for users and doctors to understand. The server stores this generated description in a database for future reference.

[0416] 4. Urgency Assessment and Notification

[0417] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[0418] Specific examples

[0419] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify the family.

[0420] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[0421] The processing flow will be explained below.

[0422] Step 1:

[0423] User: A camera installed in the home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[0424] Step 2:

[0425] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[0426] Step 3:

[0427] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[0428] Step 4:

[0429] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[0430] Step 5:

[0431] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[0432] Step 6:

[0433] Generative AI model: Based on the extracted features, a specific description of the symptoms is generated using natural language processing technology. The generated description is structured in a format that is easy for users to understand.

[0434] Step 7:

[0435] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[0436] Step 8:

[0437] Server: After confirming that the data has been saved, it sends a notification to the user device. The notification includes a summary of the analysis results and a generated explanation.

[0438] Step 9:

[0439] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[0440] Step 10:

[0441] Server: Evaluates the urgency of symptoms based on the saved analysis data. If the urgency is determined to be high, an emergency response flag is set.

[0442] Step 11:

[0443] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[0444] Step 12:

[0445] Terminal: The Y terminal issues a voice alert announcing, "An ambulance has been called and will arrive shortly." At the same time, a notification is sent to the user's family, who are informed, "Symptoms have been observed and emergency response is required."

[0446] Example 1

[0447] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0448] In modern society, the number of elderly people and patients with medical conditions is increasing, making home health monitoring systems increasingly important. However, conventional systems lack the ability to detect sudden health abnormalities occurring within the home in real time and to promptly take appropriate measures. In particular, there is a need for a system that can accurately grasp the specific characteristics of symptoms and accurately determine the level of urgency. There is also a lack of means for smoothly sharing information with medical providers. In light of this background, the present invention aims to provide a system that accurately monitors health abnormalities occurring within the home and promptly takes appropriate measures.

[0449] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0450] In this invention, the server includes means for acquiring video data from a video recording device installed in the home to monitor symptoms, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting symptom features, means for using an artificial intelligence system to generate a symptom description based on the extracted features, means for storing the generated description and analysis results in a data storage system, means for notifying a user terminal of the stored data, means for triggering an emergency response based on the urgency level, and means for coordinating and executing each stage of processing. This enables real-time detection of health abnormalities occurring in the home and enables prompt and appropriate response. Furthermore, smooth information sharing with medical providers can support accurate diagnosis and treatment.

[0451] A "symptom" is a change in bodily function or condition that occurs due to an abnormality or disease in the human body.

[0452] "Monitoring" is the act of carefully observing and recording events or situations in progress.

[0453] "Home" refers to the space inside a building where a family lives and conducts their daily lives.

[0454] A "video recording device" is a device such as a camera or surveillance camera that captures and stores video data.

[0455] "Video data" refers to the digital data of the captured video.

[0456] A "central processing unit" is a server or computer system that receives, processes, and analyzes data.

[0457] "Analysis" is the act of examining data or information in detail and understanding its content and structure.

[0458] "Features" are important parameters or indicators extracted from data to represent symptoms or events.

[0459] An "artificial intelligence system" is a computer system that uses certain algorithms to process data, learn, and make inferences.

[0460] A "description" is a text written in natural language about a specific phenomenon or symptom.

[0461] A "data storage system" is a system for safely storing data for a long period of time.

[0462] "User terminal" means a device that is directly operated by a user to receive and display information.

[0463] "Notification" is the act of conveying specific information to a recipient.

[0464] "Emergency response" refers to immediate response measures taken in response to a sudden event.

[0465] "Collaboration" refers to the act of multiple systems or processes working together.

[0466] This invention relates to a system for real-time monitoring of health abnormalities occurring in the home and for promptly taking appropriate measures. This system is realized using a video recording device, a central processing unit (server), a user terminal, and an artificial intelligence system.

[0467] Hardware and software used

[0468] 1. Video Recording Device

[0469] A user installs a video recording device in a specific location in the home (e.g., the living room or bedroom). The video recording device is a home surveillance camera (e.g., a home IP camera). This device continuously captures video data and automatically records the video if an abnormality is detected.

[0470] 2. User terminal

[0471] The terminal is responsible for managing the video data acquired from the video recording device and uploading it to the central processing unit. The terminal can be a smartphone or tablet. After receiving the video data, the user terminal sends it to the central processing unit and also receives notifications from the server.

[0472] 3. Central Processing Unit (Server)

[0473] The server receives and analyzes the video data uploaded from the device. Specifically, it uses an AI analysis platform (for example, a cloud service such as AWS or Microsoft Azure). The server analyzes the video data, extracts symptom features, and passes them to the AI ​​model. The server also stores the analysis results and generated explanations in a database and takes action based on the urgency assessment.

[0474] 4. Artificial Intelligence Systems

[0475] The generative AI model extracts symptom features from video data and generates symptom descriptions based on them. The AI ​​model used uses deep learning frameworks such as TensorFlow and PyTorch. In particular, the generative AI model uses natural language processing technology (such as GPT-3) to generate specific symptom descriptions.

[0476] Specific operation of the system

[0477] A user keeps a camera device installed in their home turned on at all times. The camera device continues to capture images of the surroundings in real time, and if an abnormality is detected, the image is automatically recorded.

[0478] The device periodically receives video data from the camera and uploads it to a central processing unit (server). The server adds the received video data to an analysis queue and passes it sequentially to the generative AI model. The generative AI model analyzes the video data frame by frame and extracts features such as abnormal movements, changes in facial expression, and changes in skin color.

[0479] Based on the extracted features, the generative AI model uses natural language processing to generate a specific description of the symptoms, such as "an elderly person suddenly collapsed, their face was pale, and their breathing was becoming labored."

[0480] The server stores the generated explanation and analysis results in a database and sends emergency notifications to the user's device as needed. If the emergency is severe, the server will issue a voice warning or automatically arrange for emergency medical transport.

[0481] Examples of concrete examples and prompts

[0482] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to a server via the device. The server receives the video data and begins analysis using a generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, arrange for emergency medical transport, and notify the user's family.

[0483] An example of a prompt sentence is, "Please generate an analysis result of a scene in which an elderly person collapses." The model will extract the features of the specified scene and generate a detailed description.

[0484] This system allows users to respond quickly and appropriately to emergencies that occur at home and provides accurate information to medical providers.

[0485] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0486] Step 1: Capture and transmit video data

[0487] A user turns on a video recording device installed in their home. The camera device continuously captures video data and automatically records the video if an abnormality occurs. The recorded video is sent to a terminal in real time. The input is the video data captured by the camera device, and the output is the recorded video data sent to the terminal. Specifically, the camera device detects a sudden fall by an elderly person and records the scene.

[0488] Step 2: Receiving and uploading video data

[0489] The terminal receives video data sent from the camera device and uploads it to the server at regular intervals. The input is the recorded video data received from the camera device, and the output is the video data uploaded to the server. Specifically, the terminal receives video data from the camera device and immediately uploads it to the cloud server.

[0490] Step 3: Video data analysis and feature extraction

[0491] The server receives the video data uploaded from the device and adds it to the analysis queue. The server then passes the video data in the analysis queue to the generative AI model, which then begins analysis. The generative AI model analyzes the video data frame by frame and extracts symptom features such as abnormal movements, changes in facial expression, and changes in skin color. The input is the video data uploaded to the server, and the output is the extracted symptom features. Specifically, this is the AI ​​model's behavior in identifying elderly people's falling movements and changes in facial color.

[0492] Step 4: Generate symptom descriptions

[0493] The server inputs the extracted features into a generative AI model to generate a description of the symptoms. The generative AI model uses natural language processing technology to generate a description and returns it to the server. The input is the extracted features of the symptoms, and the output is the generated description of the symptoms. Specifically, the operation generates a description such as "After the elderly person collapsed, their face turned pale and their breathing became rough."

[0494] Step 5: Save the description and analysis results

[0495] The server saves the generated symptom descriptions and analysis results in a database. The saved data is archived for future reference. The input is the generated symptom descriptions and analysis results, and the output is the records stored in the database. Specifically, the operation is to store the generated descriptions and analysis results in an SQL database.

[0496] Step 6: Urgency assessment and notification

[0497] The server evaluates the generated analysis results and symptom descriptions to determine the level of urgency. If the level of urgency is high, the server sends an audio warning instruction to the terminal and arranges for emergency medical transport if necessary. The input is the generated analysis results and symptom descriptions, and the output is an alert instruction and response arrangements based on the level of urgency. Specifically, the server determines that the level of urgency is high and sends an audio warning notification to the user's smartphone.

[0498] Step 7: Implementing notifications and following up

[0499] The device receives the voice warning instruction sent from the server and notifies the user. At the same time, if the emergency level is high, the device notifies the user's family as well. The input is the warning instruction from the server, and the output is a voice notification and an alert to the user. Specifically, the smartphone sends a voice message to the elderly person's family saying, "An emergency has occurred. Please respond immediately."

[0500] (Application example 1)

[0501] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0502] There is a need to safely monitor the elderly and sick at home, quickly detect any sudden changes in their condition, and take appropriate action. However, existing systems have difficulty simultaneously monitoring their condition and security, making it difficult to respond quickly and accurately. Therefore, a highly reliable system that can comprehensively monitor health and safety at home is needed.

[0503] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0504] In this invention, the server includes means for acquiring video data from a video device installed in the home to monitor symptoms and abnormalities, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting feature quantities of symptoms and abnormalities, means for using a generative artificial intelligence model to generate an explanatory text based on the extracted feature quantities, means for saving the generated explanatory text and analysis results in a recording medium, means for notifying a user terminal of the saved data, and means for triggering medical and security responses based on the urgency level. This enables comprehensive monitoring of health and safety within the home, and enables appropriate medical and security responses to be taken immediately when an abnormality is detected.

[0505] "Symptoms and abnormalities" refers to changes in the health status of a person or abnormal behavior, such as collapsing or changes in facial color, that are not normal.

[0506] "Video equipment" refers to cameras and recording devices installed in the home that continuously capture video data.

[0507] "Video data" refers to video information captured within the home by a video device.

[0508] A "central processing unit" is a device, such as a server or computer system, that analyzes acquired video data and performs the necessary calculations.

[0509] "Features" are specific attributes or patterns extracted from video data to explain symptoms or abnormalities.

[0510] A "generative artificial intelligence model" is a model that uses artificial intelligence technology to extract features of symptoms and abnormalities from video data and generate explanatory text in natural language based on this.

[0511] "Recording medium" refers to a database or storage device for saving the analysis results and generated explanations.

[0512] A "user terminal" is a device used to notify the analysis results and generated explanatory text, and typically refers to a smartphone or tablet.

[0513] "Urgency" refers to the result of evaluating the seriousness of detected symptoms or abnormalities, and serves as a criterion for determining whether immediate action is required.

[0514] "Medical response" refers to medical treatment or intervention based on the urgency of the situation, such as dispatching an ambulance.

[0515] "Security response" refers to crime prevention measures taken when an abnormality is detected within the home, including issuing emergency notifications and warnings.

[0516] System Overview

[0517] The system for implementing this invention is composed of a video device, a central processing unit, a user terminal, and software for linking these devices installed in the home. The system is designed to simultaneously monitor the user's health and the safety of the home, and to respond immediately if an abnormality is detected.

[0518] Hardware and software used

[0519] Hardware:

[0520] Video device (camera): Continuously captures video from within the home.

[0521] Central processing unit (server): Receives captured video data and performs analysis.

[0522] User device (smartphone / tablet / smart glasses): A device that notifies the analysis results.

[0523] software:

[0524] OpenCV: A library for camera operation and image processing.

[0525] TensorFlow: A library for analyzing video data using generative artificial intelligence models.

[0526] smtplib: A library for sending analysis results and notifications by email.

[0527] Program processing

[0528] The server periodically acquires video data from cameras installed in the home and analyzes the data using a central processing unit. During the analysis, a generative artificial intelligence model is used to extract abnormal features (e.g., falling, changes in facial color). The extracted features are converted into explanatory text using natural language processing technology, and the analysis results are saved on a recording medium.

[0529] Program execution procedure

[0530] 1. Acquisition of video data:

[0531] The video device continuously monitors a specific area in the home (e.g., living room, bedroom) and acquires video data.

[0532] 2. Upload data:

[0533] The captured video data is uploaded to the central processing unit via the user terminal.

[0534] 3. Data Analysis:

[0535] The central processing unit analyzes the uploaded video data using AI libraries such as TensorFlow.

[0536] Based on the analysis, feature quantities of symptoms and abnormalities are extracted.

[0537] 4. Description generation:

[0538] Based on the extracted features, a generative artificial intelligence model generates a natural language explanation.

[0539] 5. Data Retention and Notification:

[0540] The generated explanation and analysis results are stored on a recording medium, and a notification is sent to the user terminal according to the level of urgency.

[0541] Specific examples

[0542] For example, if an elderly person collapses at night, the camera detects the movement and sends the video data to a central processing unit. The central processing unit extracts features such as the collapse movement and changes in facial color, and generates a description such as "The elderly person suddenly collapsed, their face turned pale, and their breathing became heavy." This result is saved in the central processing unit, and if the emergency is deemed high, medical and security measures are immediately implemented. A notification is immediately sent to the user's device, and an emergency vehicle may be dispatched, and an alert may also be sent to the family.

[0543] Prompt Sentence Examples

[0544] "We are building a new system to monitor family members. We would like the system to use a camera to detect abnormalities such as elderly or sick members collapsing or becoming pale, and then take appropriate action. What kind of prompt would be appropriate?"

[0545] In this way, the system can take appropriate measures quickly and effectively support health and safety monitoring within the home.

[0546] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0547] Step 1:

[0548] A video device (camera) in the home captures video data. The camera continuously monitors a specific area, such as the living room or bedroom, and collects video data at a constant frame rate. The input is real-time video from the camera, and the output is the captured video frames.

[0549] Step 2:

[0550] The terminal uploads the video data acquired from the camera to a central processing unit (server). The terminal buffers video frames at regular intervals and transmits them to the central processing unit via the network. The input is the video data from the video device, and the output is the video data uploaded to the server.

[0551] Step 3:

[0552] The server receives the uploaded video data and begins analysis. The server divides the video data into frames and inputs them into a generative AI model. The input is the uploaded video data, and the output is image data for each frame to perform analysis.

[0553] Step 4:

[0554] The server uses a generative AI model to analyze the video data and extract features of symptoms and abnormalities. Typically, an AI library such as TensorFlow is used, and the model detects abnormal behavior and changes from the video data. The input is frame image data for analysis, and the output is feature data (e.g., the behavior of falling, changes in facial color).

[0555] Step 5:

[0556] The server uses a generative AI model to generate a description of the symptoms or abnormalities based on the extracted features. The generated description is created using natural language processing (NLP) technology. The input is feature data, and the output is a description in a format that is easy for humans to understand.

[0557] Step 6:

[0558] The server stores the generated explanation and analysis results on a recording medium. The server stores the generated explanation and related analysis results in a database. The input is the generated explanation and analysis results, and the output is the data stored in the database.

[0559] Step 7:

[0560] The server evaluates the urgency and sends a notification to the user device if it is determined to be highly urgent. The urgency evaluation is performed using an AI model or rule-based system. The input is the generated explanation and analysis results, and the output is notification data sent to the user device.

[0561] Step 8:

[0562] The server sends a notification to the user's smart device (e.g., smartphone, tablet, smart glasses). The user device responds based on the received notification, such as issuing an audio warning. The input is the result of the urgency assessment and notification data, and the output is a real-time notification and audio warning.

[0563] Step 9:

[0564] The user checks the notification and takes additional action as necessary (e.g., contacting a medical institution or reporting to a security agency). The input is the notification sent to the user terminal, and the output is the user's response action.

[0565] Step 10:

[0566] The server automatically triggers appropriate actions (e.g. dispatching emergency vehicles, notifying security). In the case of a high-urgency situation, the system automatically executes triggers to ensure a prompt response. The input is the result of the urgency assessment, and the output is the automatic trigger action.

[0567] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0568] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. This system analyzes changes in the user's medical condition and emotional state in real time, enabling appropriate medical response.

[0569] System configuration

[0570] The system consists of the following main components:

[0571] Camera device: Installed in a specific location in the home, it constantly captures video data.

[0572] Terminal: Manages video data and uploads it to the server. The terminal is also a device that receives notifications from the server.

[0573] Server: Receives and analyzes video data, stores the results in a database, and takes emergency action based on the analysis results.

[0574] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[0575] Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis.

[0576] Program processing

[0577] Recording and uploading video

[0578] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[0579] Video analysis and symptom feature extraction

[0580] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for describing specific symptoms.

[0581] Further analysis of the emotion engine

[0582] Furthermore, the emotion engine analyzes the user's emotional state from the video and audio data. The emotion engine uses, for example, facial expression analysis and audio analysis to identify the user's emotions (e.g., surprise, fear, anger, etc.). This emotional state information is integrated with the symptom analysis results to provide comprehensive information for judgment.

[0583] Symptom description generation

[0584] Based on the extracted features and the analysis results of the emotion engine, the generative AI model uses natural language processing technology to generate a specific description of the symptoms. This description is written in a format that is easy for users and doctors to understand. The server stores the generated description in a database.

[0585] Urgency assessment and notification

[0586] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. The results of the emotion engine are also taken into account here. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[0587] Specific examples

[0588] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "The elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[0589] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[0590] The processing flow will be explained below.

[0591] Step 1:

[0592] User: A camera installed in the user's home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[0593] Step 2:

[0594] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[0595] Step 3:

[0596] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[0597] Step 4:

[0598] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[0599] Step 5:

[0600] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[0601] Step 6:

[0602] Server: The extracted features are formatted for storage in a database, making them possible to integrate with existing symptom data.

[0603] Step 7:

[0604] Emotion engine: Analyzes the user's emotional state from video and audio data. For example, facial expression analysis and audio analysis are used to identify emotions such as surprise, fear, and anger. Emotional state information is integrated with symptom analysis results.

[0605] Step 8:

[0606] Generative AI model: Based on the extracted features and the analysis results of the emotion engine, a specific description of the symptoms is generated using natural language processing technology. This description is constructed in a format that is easy for users to understand.

[0607] Step 9:

[0608] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[0609] Step 10:

[0610] Server: After confirming that the data has been saved, it sends a notification to the user device, which includes a summary of the analysis results and a generated explanation.

[0611] Step 11:

[0612] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[0613] Step 12:

[0614] Server: Evaluates the urgency of symptoms based on the saved analysis data. Taking into account the results of the emotion engine, if the urgency is evaluated as high, sets an emergency response flag.

[0615] Step 13:

[0616] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[0617] Step 14:

[0618] Device: The device issues a voice alert, announcing that an ambulance has been called and will arrive shortly. At the same time, a notification is sent to the user's family, informing them that symptoms have been observed and that emergency response is required.

[0619] This detailed processing step allows for real-time monitoring of the user's medical condition and emotional state, enabling prompt and accurate responses, and the stored data can then be used by doctors for more accurate and effective treatment.

[0620] Example 2

[0621] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0622] When the condition of an elderly person or a sick person suddenly changes at home, it is difficult to quickly and accurately monitor and judge the change and take appropriate measures promptly. In particular, for elderly people living alone, there is a lack of means to grasp changes in the condition in real time and take emergency measures, so a system that solves this problem is needed.

[0623] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server via a terminal, means for analyzing the video data uploaded to the server and extracting symptom features, means for using a generative AI model to generate a symptom description based on the extracted features, means for analyzing an emotional state from video data and audio data, means for storing the generated description and analysis results in a database, means for notifying a user terminal of the stored data, and means for triggering an emergency response based on the urgency. This makes it possible to monitor changes in the condition of elderly people and sick people in real time and take prompt and appropriate medical action.

[0624] A "symptom" is a sign or phenomenon that indicates an abnormal physical or mental condition or malfunction.

[0625] A "camera device" is a hardware device for acquiring video data.

[0626] "Video data" refers to digital data containing visual information captured by a camera device.

[0627] A "terminal" is a relay device for uploading video data to a server.

[0628] "Server" means a computer system on a network that receives, analyzes, and stores video data.

[0629] "Features" refer to important attributes or elements that make up symptoms and are extracted from the subject of analysis.

[0630] A "generative AI model" is an artificial intelligence that generates symptom descriptions based on extracted features.

[0631] "Emotional state" refers to the mental state and emotions of the user that are analyzed from video data and audio data.

[0632] "Database" refers to an information storage system for storing generated descriptions and analysis results.

[0633] A "user terminal" is a device that receives information stored in a database and notifies the user.

[0634] "Urgency" is an indicator that shows the seriousness of the symptoms and the need for response.

[0635] "Emergency response" refers to the actions and means to provide prompt medical treatment according to the degree of urgency.

[0636] The present invention is a system for monitoring the medical and emotional state of elderly and sick people and for responding quickly in emergencies. The system consists of the following main components:

[0637] 1. Camera device: A device that is installed in a specific location in the home and captures video data at all times. In this invention, the general term camera device is used.

[0638] 2. Device: A device that manages video data and uploads it to the server. The device also receives notifications from the server. Examples of devices that fall under this category include, but are not limited to, the Google Pixel and the Apple iPad.

[0639] 3. Server: This is a system that receives and analyzes video data and stores the results in a database. As a specific example, we use an AWS EC2 instance.

[0640] 4. Generative AI model: This is an artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these. A TensorFlow-based model is used as the AI ​​model.

[0641] 5. Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis. An example is IBM Watson's emotion analysis engine.

[0642] 6. Database: This is an information storage system for saving the generated explanations and analysis results. Database systems such as MySQL or MongoDB are used.

[0643] 7. User terminal: A device that receives information stored in the database and notifies the user. Devices such as Raspberry Pi fall into this category.

[0644] Operation overview

[0645] A user constantly monitors video data using a camera device installed in the home. For example, when the camera device detects an elderly person falling, it records the video.

[0646] The device automatically uploads the recorded video to the server. For example, if the device is a Google Pixel smartphone, the video data is stored from the device in an AWS S3 bucket and then sent to an AWS EC2 instance for analysis.

[0647] When the server receives the uploaded video data, it begins analyzing it using an AI model. For example, a TensorFlow-based AI model analyzes the video data frame by frame and extracts features such as abnormal behavior and changes in facial color.

[0648] The emotion engine further analyzes the emotional state based on the extracted features. For example, IBM Watson's emotion analysis engine can be used to identify emotions such as "pain" or "fear" from the user's facial expressions and voice.

[0649] Based on these features and the analysis results of the emotion engine, the generative AI model generates a specific description of the symptoms, such as, "An elderly person suddenly collapsed, their face was pale, and their breathing was heavy. They appeared to be in pain."

[0650] The server stores the generated descriptions in a database, for example a MySQL database.

[0651] The user device receives the stored data and notifies the user or their family. For example, the Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio.

[0652] Prompt Sentence Examples

[0653] "If the user is elderly and suddenly collapses in the middle of the night, please explain in detail the process of detecting abnormal movements and changes in facial expressions from video data recorded by a camera device, identifying the emotion of pain using an emotion analysis engine, and dispatching an ambulance based on the results."

[0654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0655] Step 1:

[0656] Users constantly monitor their own situation using cameras installed in their homes. If the camera detects an abnormality, it starts recording. For example, if the user suddenly collapses, the video of the incident is automatically recorded. This video data becomes the input.

[0657] Step 2:

[0658] The device automatically uploads the recorded video to the server. During this process, the video data is temporarily stored on the device and then sent to the server via the network. For example, devices such as the Google Pixel and Apple iPad are used. The device inputs the video data (mp4 format) and uploads it to the server.

[0659] Step 3:

[0660] The server receives the uploaded video data. A dedicated script running on an AWS EC2 instance adds the received video data to the analysis queue. The server receives the video data as input and processes it by passing it to the AI ​​model.

[0661] Step 4:

[0662] The server passes the received video data to a TensorFlow-based AI model and begins analysis. The AI ​​model analyzes each video frame and extracts features such as abnormal behavior and changes in facial color. In this step, the video data is input and features (abnormal behavior, changes in facial color, etc.) are obtained as output. Specifically, the AI ​​model analyzes each frame and detects abnormal patterns.

[0663] Step 5:

[0664] The server passes the features to IBM Watson's emotion analysis engine for additional analysis. The emotion analysis engine analyzes the user's emotional state (e.g., pain, fear) from the video and audio data. In this step, the features are the input and the emotional state is obtained as the output. Specifically, the emotion analysis engine identifies changes in facial expressions and voice to analyze the user's emotions.

[0665] Step 6:

[0666] The server uses a generative AI model to generate a symptom description based on the results obtained from the AI ​​model and the emotion analysis engine. For example, a description such as "An elderly person suddenly collapsed, his face was pale, his breathing was rough, and he was in pain" is generated. In this step, the features and emotional state are input, and a symptom description is obtained as the output. Specifically, the generative AI model constructs the description using natural language processing technology.

[0667] Step 7:

[0668] The server stores the generated explanations and analysis results in a database. For example, this information is stored in MySQL or MongoDB. In this step, the symptom explanations and analysis results are input, and the output is stored in the database.

[0669] Step 8:

[0670] The user device receives the stored data and notifies the user and their family. For example, a Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio. Specifically, notifications from the database are input, and based on that, voice alerts and dispatch of an ambulance are output.

[0671] In this way, each processing step works closely together to realize a system that monitors the user's condition and responds to emergencies quickly and appropriately.

[0672] (Application example 2)

[0673] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0674] Conventional home monitoring systems have problems with insufficient monitoring of the condition of elderly and sick people, making it difficult to respond quickly and appropriately when an emergency occurs. Furthermore, because the level of urgency is determined solely by symptom analysis without taking into account the user's emotional state, appropriate responses may be delayed. Furthermore, there is a lack of a way to present the analysis results in an easy-to-understand format, making it difficult to provide accurate information to family members and doctors.

[0675] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0676] In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server, means for analyzing the video data uploaded to the server and extracting features of the symptoms, means for using an artificial intelligence model to generate a description of the symptoms based on the extracted features, means for saving the generated description and analysis results in a database, means for notifying a user terminal of the saved data, means for triggering an emergency response based on the level of urgency, means for analyzing an emotional state from the acquired video data and audio data using an emotion engine, and means for integrating the analysis results of the emotional state with the analysis results of the symptoms. This enables the user's medical condition and emotional state to be analyzed in real time, enabling prompt and appropriate response in emergencies.

[0677] A "camera device" is a device that is installed in the home and is used to acquire video data.

[0678] A "server" is a central processing unit that receives acquired video data, performs analysis, and stores and notifies the results.

[0679] "Video data" is a collection of continuous image information captured by a camera device.

[0680] "Upload" refers to the process of sending video data acquired from a camera device to a server.

[0681] "Analysis" is the process of extracting specific information from the acquired video data and organizing the detailed information using machine learning models and artificial intelligence technology.

[0682] A "feature" is a specific pattern or indicator of data extracted through analysis, and is an element that represents symptoms or emotional states.

[0683] The "explanation" is a detailed document describing the patient's condition and emotional state in natural language, generated based on the extracted features.

[0684] An "artificial intelligence model" is a set of algorithms and programs used to analyze acquired data, extract features, and generate explanatory text.

[0685] A "database" is a system for storing and managing analysis results and generated explanatory text.

[0686] A "user terminal" is a terminal device for receiving and displaying notifications of stored data.

[0687] The "emotion engine" is a software engine that analyzes the user's emotional state from video and audio data and outputs the results.

[0688] "Urgency" refers to the level of necessity for emergency response, determined based on the analysis results.

[0689] "Emergency response" is a process of arranging for an ambulance and notifying family members based on the level of urgency.

[0690] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. A specific embodiment based on an application called "Safeguard Watch" will be described.

[0691] The server acquires video data from a camera device installed in the home. This camera device continuously monitors the user's daily life in real time and acquires important video data. It also has an audio input device, so it can acquire the user's voice data.

[0692] Video and audio data are uploaded to a server via a device installed in the home. The server receives this data and prepares it for storage. The server then analyzes the uploaded video and audio data. A generative AI model and an emotion engine are used for the analysis. The generative AI model extracts symptom features and emotional states from the video and audio data. For example, abnormal movements, changes in facial color, changes in facial expression, and trembling voice are analyzed as features. The emotion engine also analyzes the user's emotional state (e.g., surprise, fear, anger, etc.).

[0693] Based on the extracted features and emotional state, the generative AI model generates a specific description of the symptoms. The generated description is presented in a format that is easy for users and doctors to understand. This description and the analysis results are stored in a database on the server. The saved data is then sent to the user's device. The user's device can receive the notification and issue a warning to the user. It can also trigger an emergency response based on the level of urgency. If the level of urgency is high, the server immediately issues an audio warning and automatically dispatches an ambulance. The system also notifies the user's family members about the symptoms that have occurred.

[0694] This system allows users to respond quickly and appropriately to emergency situations. It also enables doctors to receive accurate information. It is expected to effectively support daily disease management and monitoring. Specific examples are shown below.

[0695] If an elderly person suddenly collapses at home in the middle of the night, the camera automatically records the footage and uploads it to a server via a device. The server receives the video data and begins analysis using a generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[0696] Prompt Sentence Examples

[0697] "Please analyze the user's falling motion and changes in facial expressions from multiple angles to identify changes in the patient's condition in real time."

[0698] "Analyze the behavioral patterns of elderly people when their complexion suddenly changes and suggest appropriate emergency responses."

[0699] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0700] Step 1:

[0701] The terminal acquires video data in real time from a camera device installed in the home, and this video data is immediately transmitted to the terminal along with audio data collected by an audio input device installed in the home.

[0702] input:

[0703] Real-time video and audio data from home camera devices and audio input devices.

[0704] output:

[0705] Capture of acquired video and audio data.

[0706] Step 2:

[0707] The device uploads the captured video and audio data to the server, where it undergoes format conversion and compression before being sent in a format that the server can use for analysis.

[0708] input:

[0709] Video data and audio data acquired from a camera device and an audio input device.

[0710] output:

[0711] Video and audio data that has been converted and compressed into an analyzable format.

[0712] Step 3:

[0713] The server receives the uploaded video and audio data and passes it to the generative AI model and emotion engine. The generative AI model extracts symptom features from the video data, and the emotion engine analyzes the emotional state from the audio data.

[0714] input:

[0715] Video and audio data uploaded to the server.

[0716] output:

[0717] Emotional state data analyzed from symptom features of video data and audio data.

[0718] Specific behavior:

[0719] It uses a generative AI model to detect abnormalities in behavior and facial expressions in each frame of video data, and an emotion engine to identify emotional states from audio data.

[0720] Step 4:

[0721] The server generates a specific description of the symptoms based on the analysis results obtained from the generative AI model and the emotion engine. The generative AI model documents the description using natural language processing (NLP) techniques.

[0722] input:

[0723] Symptom features from the generative AI model and emotional state data from the emotion engine.

[0724] output:

[0725] A detailed description of the symptoms.

[0726] Specific behavior:

[0727] Using NLP technology, a description such as "An elderly person suddenly collapsed, his face was pale, and his breathing was rough" was generated based on the analysis results.

[0728] Step 5:

[0729] The server stores the generated descriptions and analysis results in a database for future reference and emergency response decisions.

[0730] input:

[0731] Generated explanations and analysis results.

[0732] output:

[0733] Descriptions and analysis results stored in a database.

[0734] Specific behavior:

[0735] The data is saved to the database and managed with a timestamp.

[0736] Step 6:

[0737] The server notifies the user device of the stored data, which then receives the notification, issues a warning to the user, and triggers an emergency response based on the urgency level.

[0738] input:

[0739] Descriptions and analysis results stored in a database.

[0740] output:

[0741] Notifications to user devices and emergency response triggers.

[0742] Specific behavior:

[0743] A push notification is sent to the user's device. If the emergency is severe, an audio warning is issued and an ambulance is automatically dispatched. The user's family is also notified of the situation.

[0744] The above steps enable real-time monitoring of the medical conditions and emotional states of elderly and sick people at home, and enable prompt and appropriate response to emergencies.

[0745] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0746] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0747] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0748] [Third embodiment]

[0749] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0750] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0751] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0752] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0753] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0754] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0755] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0756] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0757] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0758] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0759] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0760] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0761] The present invention is a system for monitoring the condition of sick people and the elderly and providing timely medical care. This system is realized using a camera device, a server, and a user terminal installed in the home.

[0762] System configuration

[0763] The system consists of the following main components:

[0764] Camera device: Installed in a specific location in the home (living room, bedroom, etc.) and captures video data continuously.

[0765] Terminal: Manages the video data acquired from the camera and uploads it to the server. The terminal is also a device (usually the user's smartphone or tablet) that receives notifications from the server.

[0766] Server: Receives video data, performs analysis, stores the results in a database, and takes emergency action based on the analysis results.

[0767] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[0768] Program processing

[0769] 1. Recording and uploading the video

[0770] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[0771] 2. Video analysis and symptom feature extraction

[0772] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for specifically describing the symptoms.

[0773] 3. Symptom Description Generation

[0774] Based on the extracted features, the generative AI model uses natural language processing techniques to generate a specific symptom description. This description is written in a format that is easy for users and doctors to understand. The server stores this generated description in a database for future reference.

[0775] 4. Urgency Assessment and Notification

[0776] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[0777] Specific examples

[0778] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify the family.

[0779] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[0780] The processing flow will be explained below.

[0781] Step 1:

[0782] User: A camera installed in the home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[0783] Step 2:

[0784] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[0785] Step 3:

[0786] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[0787] Step 4:

[0788] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[0789] Step 5:

[0790] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[0791] Step 6:

[0792] Generative AI model: Based on the extracted features, a specific description of the symptoms is generated using natural language processing technology. The generated description is structured in a format that is easy for users to understand.

[0793] Step 7:

[0794] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[0795] Step 8:

[0796] Server: After confirming that the data has been saved, it sends a notification to the user device. The notification includes a summary of the analysis results and a generated explanation.

[0797] Step 9:

[0798] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[0799] Step 10:

[0800] Server: Evaluates the urgency of symptoms based on the saved analysis data. If the urgency is determined to be high, an emergency response flag is set.

[0801] Step 11:

[0802] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[0803] Step 12:

[0804] Terminal: The Y terminal issues a voice alert announcing, "An ambulance has been called and will arrive shortly." At the same time, a notification is sent to the user's family, who are informed, "Symptoms have been observed and emergency response is required."

[0805] Example 1

[0806] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0807] In modern society, the number of elderly people and patients with medical conditions is increasing, making home health monitoring systems increasingly important. However, conventional systems lack the ability to detect sudden health abnormalities occurring within the home in real time and to promptly take appropriate measures. In particular, there is a need for a system that can accurately grasp the specific characteristics of symptoms and accurately determine the level of urgency. There is also a lack of means for smoothly sharing information with medical providers. In light of this background, the present invention aims to provide a system that accurately monitors health abnormalities occurring within the home and promptly takes appropriate measures.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0809] In this invention, the server includes means for acquiring video data from a video recording device installed in the home to monitor symptoms, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting symptom features, means for using an artificial intelligence system to generate a symptom description based on the extracted features, means for storing the generated description and analysis results in a data storage system, means for notifying a user terminal of the stored data, means for triggering an emergency response based on the urgency level, and means for coordinating and executing each stage of processing. This enables real-time detection of health abnormalities occurring in the home and enables prompt and appropriate response. Furthermore, smooth information sharing with medical providers can support accurate diagnosis and treatment.

[0810] A "symptom" is a change in bodily function or condition that occurs due to an abnormality or disease in the human body.

[0811] "Monitoring" is the act of carefully observing and recording events or situations in progress.

[0812] "Home" refers to the space inside a building where a family lives and conducts their daily lives.

[0813] A "video recording device" is a device such as a camera or surveillance camera that captures and stores video data.

[0814] "Video data" refers to the digital data of the captured video.

[0815] A "central processing unit" is a server or computer system that receives, processes, and analyzes data.

[0816] "Analysis" is the act of examining data or information in detail and understanding its content and structure.

[0817] "Features" are important parameters or indicators extracted from data to represent symptoms or events.

[0818] An "artificial intelligence system" is a computer system that uses certain algorithms to process data, learn, and make inferences.

[0819] A "description" is a text written in natural language about a specific phenomenon or symptom.

[0820] A "data storage system" is a system for safely storing data for a long period of time.

[0821] "User terminal" means a device that is directly operated by a user to receive and display information.

[0822] "Notification" is the act of conveying specific information to a recipient.

[0823] "Emergency response" refers to immediate response measures taken in response to a sudden event.

[0824] "Collaboration" refers to the act of multiple systems or processes working together.

[0825] This invention relates to a system for real-time monitoring of health abnormalities occurring in the home and for promptly taking appropriate measures. This system is realized using a video recording device, a central processing unit (server), a user terminal, and an artificial intelligence system.

[0826] Hardware and software used

[0827] 1. Video Recording Device

[0828] A user installs a video recording device in a specific location in the home (e.g., the living room or bedroom). The video recording device is a home surveillance camera (e.g., a home IP camera). This device continuously captures video data and automatically records the video if an abnormality is detected.

[0829] 2. User terminal

[0830] The terminal is responsible for managing the video data acquired from the video recording device and uploading it to the central processing unit. The terminal can be a smartphone or tablet. After receiving the video data, the user terminal sends it to the central processing unit and also receives notifications from the server.

[0831] 3. Central Processing Unit (Server)

[0832] The server receives and analyzes the video data uploaded from the device. Specifically, it uses an AI analysis platform (for example, a cloud service such as AWS or Microsoft Azure). The server analyzes the video data, extracts symptom features, and passes them to the AI ​​model. The server also stores the analysis results and generated explanations in a database and takes action based on the urgency assessment.

[0833] 4. Artificial Intelligence Systems

[0834] The generative AI model extracts symptom features from video data and generates symptom descriptions based on them. The AI ​​model used uses deep learning frameworks such as TensorFlow and PyTorch. In particular, the generative AI model uses natural language processing technology (such as GPT-3) to generate specific symptom descriptions.

[0835] Specific operation of the system

[0836] A user keeps a camera device installed in their home turned on at all times. The camera device continues to capture images of the surroundings in real time, and if an abnormality is detected, the image is automatically recorded.

[0837] The device periodically receives video data from the camera and uploads it to a central processing unit (server). The server adds the received video data to an analysis queue and passes it sequentially to the generative AI model. The generative AI model analyzes the video data frame by frame and extracts features such as abnormal movements, changes in facial expression, and changes in skin color.

[0838] Based on the extracted features, the generative AI model uses natural language processing to generate a specific description of the symptoms, such as "an elderly person suddenly collapsed, their face was pale, and their breathing was becoming labored."

[0839] The server stores the generated explanation and analysis results in a database and sends emergency notifications to the user's device as needed. If the emergency is severe, the server will issue a voice warning or automatically arrange for emergency medical transport.

[0840] Examples of concrete examples and prompts

[0841] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to a server via the device. The server receives the video data and begins analysis using a generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, arrange for emergency medical transport, and notify the user's family.

[0842] An example of a prompt sentence is, "Please generate an analysis result of a scene in which an elderly person collapses." The model will extract the features of the specified scene and generate a detailed description.

[0843] This system allows users to respond quickly and appropriately to emergencies that occur at home and provides accurate information to medical providers.

[0844] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0845] Step 1: Capture and transmit video data

[0846] A user turns on a video recording device installed in their home. The camera device continuously captures video data and automatically records the video if an abnormality occurs. The recorded video is sent to a terminal in real time. The input is the video data captured by the camera device, and the output is the recorded video data sent to the terminal. Specifically, the camera device detects a sudden fall by an elderly person and records the scene.

[0847] Step 2: Receiving and uploading video data

[0848] The terminal receives video data sent from the camera device and uploads it to the server at regular intervals. The input is the recorded video data received from the camera device, and the output is the video data uploaded to the server. Specifically, the terminal receives video data from the camera device and immediately uploads it to the cloud server.

[0849] Step 3: Video data analysis and feature extraction

[0850] The server receives the video data uploaded from the device and adds it to the analysis queue. The server then passes the video data in the analysis queue to the generative AI model, which then begins analysis. The generative AI model analyzes the video data frame by frame and extracts symptom features such as abnormal movements, changes in facial expression, and changes in skin color. The input is the video data uploaded to the server, and the output is the extracted symptom features. Specifically, this is the AI ​​model's behavior in identifying elderly people's falling movements and changes in facial color.

[0851] Step 4: Generate symptom descriptions

[0852] The server inputs the extracted features into a generative AI model to generate a description of the symptoms. The generative AI model uses natural language processing technology to generate a description and returns it to the server. The input is the extracted features of the symptoms, and the output is the generated description of the symptoms. Specifically, the operation generates a description such as "After the elderly person collapsed, their face turned pale and their breathing became rough."

[0853] Step 5: Save the description and analysis results

[0854] The server saves the generated symptom descriptions and analysis results in a database. The saved data is archived for future reference. The input is the generated symptom descriptions and analysis results, and the output is the records stored in the database. Specifically, the operation is to store the generated descriptions and analysis results in an SQL database.

[0855] Step 6: Urgency assessment and notification

[0856] The server evaluates the generated analysis results and symptom descriptions to determine the level of urgency. If the level of urgency is high, the server sends an audio warning instruction to the terminal and arranges for emergency medical transport if necessary. The input is the generated analysis results and symptom descriptions, and the output is an alert instruction and response arrangements based on the level of urgency. Specifically, the server determines that the level of urgency is high and sends an audio warning notification to the user's smartphone.

[0857] Step 7: Implementing notifications and following up

[0858] The device receives the voice warning instruction sent from the server and notifies the user. At the same time, if the emergency level is high, the device notifies the user's family as well. The input is the warning instruction from the server, and the output is a voice notification and an alert to the user. Specifically, the smartphone sends a voice message to the elderly person's family saying, "An emergency has occurred. Please respond immediately."

[0859] (Application example 1)

[0860] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0861] There is a need to safely monitor the elderly and sick at home, quickly detect any sudden changes in their condition, and take appropriate action. However, existing systems have difficulty simultaneously monitoring their condition and security, making it difficult to respond quickly and accurately. Therefore, a highly reliable system that can comprehensively monitor health and safety at home is needed.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0863] In this invention, the server includes means for acquiring video data from a video device installed in the home to monitor symptoms and abnormalities, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting feature quantities of symptoms and abnormalities, means for using a generative artificial intelligence model to generate an explanatory text based on the extracted feature quantities, means for saving the generated explanatory text and analysis results in a recording medium, means for notifying a user terminal of the saved data, and means for triggering medical and security responses based on the urgency level. This enables comprehensive monitoring of health and safety within the home, and enables appropriate medical and security responses to be taken immediately when an abnormality is detected.

[0864] "Symptoms and abnormalities" refers to changes in the health status of a person or abnormal behavior, such as collapsing or changes in facial color, that are not normal.

[0865] "Video equipment" refers to cameras and recording devices installed in the home that continuously capture video data.

[0866] "Video data" refers to video information captured within the home by a video device.

[0867] A "central processing unit" is a device, such as a server or computer system, that analyzes acquired video data and performs the necessary calculations.

[0868] "Features" are specific attributes or patterns extracted from video data to explain symptoms or abnormalities.

[0869] A "generative artificial intelligence model" is a model that uses artificial intelligence technology to extract features of symptoms and abnormalities from video data and generate explanatory text in natural language based on this.

[0870] "Recording medium" refers to a database or storage device for saving the analysis results and generated explanations.

[0871] A "user terminal" is a device used to notify the analysis results and generated explanatory text, and typically refers to a smartphone or tablet.

[0872] "Urgency" refers to the result of evaluating the seriousness of detected symptoms or abnormalities, and serves as a criterion for determining whether immediate action is required.

[0873] "Medical response" refers to medical treatment or intervention based on the urgency of the situation, such as dispatching an ambulance.

[0874] "Security response" refers to crime prevention measures taken when an abnormality is detected within the home, including issuing emergency notifications and warnings.

[0875] System Overview

[0876] The system for implementing this invention is composed of a video device, a central processing unit, a user terminal, and software for linking these devices installed in the home. The system is designed to simultaneously monitor the user's health and the safety of the home, and to respond immediately if an abnormality is detected.

[0877] Hardware and software used

[0878] Hardware:

[0879] Video device (camera): Continuously captures video from within the home.

[0880] Central processing unit (server): Receives captured video data and performs analysis.

[0881] User device (smartphone / tablet / smart glasses): A device that notifies the analysis results.

[0882] software:

[0883] OpenCV: A library for camera operation and image processing.

[0884] TensorFlow: A library for analyzing video data using generative artificial intelligence models.

[0885] smtplib: A library for sending analysis results and notifications by email.

[0886] Program processing

[0887] The server periodically acquires video data from cameras installed in the home and analyzes the data using a central processing unit. During the analysis, a generative artificial intelligence model is used to extract abnormal features (e.g., falling, changes in facial color). The extracted features are converted into explanatory text using natural language processing technology, and the analysis results are saved on a recording medium.

[0888] Program execution procedure

[0889] 1. Acquisition of video data:

[0890] The video device continuously monitors a specific area in the home (e.g., living room, bedroom) and acquires video data.

[0891] 2. Upload data:

[0892] The captured video data is uploaded to the central processing unit via the user terminal.

[0893] 3. Data Analysis:

[0894] The central processing unit analyzes the uploaded video data using AI libraries such as TensorFlow.

[0895] Based on the analysis, feature quantities of symptoms and abnormalities are extracted.

[0896] 4. Description generation:

[0897] Based on the extracted features, a generative artificial intelligence model generates a natural language explanation.

[0898] 5. Data Retention and Notification:

[0899] The generated explanation and analysis results are stored on a recording medium, and a notification is sent to the user terminal according to the level of urgency.

[0900] Specific examples

[0901] For example, if an elderly person collapses at night, the camera detects the movement and sends the video data to a central processing unit. The central processing unit extracts features such as the collapse movement and changes in facial color, and generates a description such as "The elderly person suddenly collapsed, their face turned pale, and their breathing became heavy." This result is saved in the central processing unit, and if the emergency is deemed high, medical and security measures are immediately implemented. A notification is immediately sent to the user's device, and an emergency vehicle may be dispatched, and an alert may also be sent to the family.

[0902] Prompt Sentence Examples

[0903] "We are building a new system to monitor family members. We would like the system to use a camera to detect abnormalities such as elderly or sick members collapsing or becoming pale, and then take appropriate action. What kind of prompt would be appropriate?"

[0904] In this way, the system can take appropriate measures quickly and effectively support health and safety monitoring within the home.

[0905] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0906] Step 1:

[0907] A video device (camera) in the home captures video data. The camera continuously monitors a specific area, such as the living room or bedroom, and collects video data at a constant frame rate. The input is real-time video from the camera, and the output is the captured video frames.

[0908] Step 2:

[0909] The terminal uploads the video data acquired from the camera to a central processing unit (server). The terminal buffers video frames at regular intervals and transmits them to the central processing unit via the network. The input is the video data from the video device, and the output is the video data uploaded to the server.

[0910] Step 3:

[0911] The server receives the uploaded video data and begins analysis. The server divides the video data into frames and inputs them into a generative AI model. The input is the uploaded video data, and the output is image data for each frame to perform analysis.

[0912] Step 4:

[0913] The server uses a generative AI model to analyze the video data and extract features of symptoms and abnormalities. Typically, an AI library such as TensorFlow is used, and the model detects abnormal behavior and changes from the video data. The input is frame image data for analysis, and the output is feature data (e.g., the behavior of falling, changes in facial color).

[0914] Step 5:

[0915] The server uses a generative AI model to generate a description of the symptoms or abnormalities based on the extracted features. The generated description is created using natural language processing (NLP) technology. The input is feature data, and the output is a description in a format that is easy for humans to understand.

[0916] Step 6:

[0917] The server stores the generated explanation and analysis results on a recording medium. The server stores the generated explanation and related analysis results in a database. The input is the generated explanation and analysis results, and the output is the data stored in the database.

[0918] Step 7:

[0919] The server evaluates the urgency and sends a notification to the user device if it is determined to be highly urgent. The urgency evaluation is performed using an AI model or rule-based system. The input is the generated explanation and analysis results, and the output is notification data sent to the user device.

[0920] Step 8:

[0921] The server sends a notification to the user's smart device (e.g., smartphone, tablet, smart glasses). The user device responds based on the received notification, such as issuing an audio warning. The input is the result of the urgency assessment and notification data, and the output is a real-time notification and audio warning.

[0922] Step 9:

[0923] The user checks the notification and takes additional action as necessary (e.g., contacting a medical institution or reporting to a security agency). The input is the notification sent to the user terminal, and the output is the user's response action.

[0924] Step 10:

[0925] The server automatically triggers appropriate actions (e.g. dispatching emergency vehicles, notifying security). In the case of a high-urgency situation, the system automatically executes triggers to ensure a prompt response. The input is the result of the urgency assessment, and the output is the automatic trigger action.

[0926] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0927] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. This system analyzes changes in the user's medical condition and emotional state in real time, enabling appropriate medical response.

[0928] System configuration

[0929] The system consists of the following main components:

[0930] Camera device: Installed in a specific location in the home, it constantly captures video data.

[0931] Terminal: Manages video data and uploads it to the server. The terminal is also a device that receives notifications from the server.

[0932] Server: Receives and analyzes video data, stores the results in a database, and takes emergency action based on the analysis results.

[0933] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[0934] Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis.

[0935] Program processing

[0936] Recording and uploading video

[0937] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[0938] Video analysis and symptom feature extraction

[0939] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for describing specific symptoms.

[0940] Further analysis of the emotion engine

[0941] Furthermore, the emotion engine analyzes the user's emotional state from the video and audio data. The emotion engine uses, for example, facial expression analysis and audio analysis to identify the user's emotions (e.g., surprise, fear, anger, etc.). This emotional state information is integrated with the symptom analysis results to provide comprehensive information for judgment.

[0942] Symptom description generation

[0943] Based on the extracted features and the analysis results of the emotion engine, the generative AI model uses natural language processing technology to generate a specific description of the symptoms. This description is written in a format that is easy for users and doctors to understand. The server stores the generated description in a database.

[0944] Urgency assessment and notification

[0945] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. The results of the emotion engine are also taken into account here. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[0946] Specific examples

[0947] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "The elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[0948] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[0949] The processing flow will be explained below.

[0950] Step 1:

[0951] User: A camera installed in the user's home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[0952] Step 2:

[0953] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[0954] Step 3:

[0955] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[0956] Step 4:

[0957] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[0958] Step 5:

[0959] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[0960] Step 6:

[0961] Server: The extracted features are formatted for storage in a database, making them possible to integrate with existing symptom data.

[0962] Step 7:

[0963] Emotion engine: Analyzes the user's emotional state from video and audio data. For example, facial expression analysis and audio analysis are used to identify emotions such as surprise, fear, and anger. Emotional state information is integrated with symptom analysis results.

[0964] Step 8:

[0965] Generative AI model: Based on the extracted features and the analysis results of the emotion engine, a specific description of the symptoms is generated using natural language processing technology. This description is constructed in a format that is easy for users to understand.

[0966] Step 9:

[0967] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[0968] Step 10:

[0969] Server: After confirming that the data has been saved, it sends a notification to the user device, which includes a summary of the analysis results and a generated explanation.

[0970] Step 11:

[0971] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[0972] Step 12:

[0973] Server: Evaluates the urgency of symptoms based on the saved analysis data. Taking into account the results of the emotion engine, if the urgency is evaluated as high, sets an emergency response flag.

[0974] Step 13:

[0975] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[0976] Step 14:

[0977] Device: The device issues a voice alert, announcing that an ambulance has been called and will arrive shortly. At the same time, a notification is sent to the user's family, informing them that symptoms have been observed and that emergency response is required.

[0978] This detailed processing step allows for real-time monitoring of the user's medical condition and emotional state, enabling prompt and accurate responses, and the stored data can then be used by doctors for more accurate and effective treatment.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] When the condition of an elderly person or a sick person suddenly changes at home, it is difficult to quickly and accurately monitor and judge the change and take appropriate measures promptly. In particular, for elderly people living alone, there is a lack of means to grasp changes in the condition in real time and take emergency measures, so a system that solves this problem is needed.

[0982] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server via a terminal, means for analyzing the video data uploaded to the server and extracting symptom features, means for using a generative AI model to generate a symptom description based on the extracted features, means for analyzing an emotional state from video data and audio data, means for storing the generated description and analysis results in a database, means for notifying a user terminal of the stored data, and means for triggering an emergency response based on the urgency. This makes it possible to monitor changes in the condition of elderly people and sick people in real time and take prompt and appropriate medical action.

[0983] A "symptom" is a sign or phenomenon that indicates an abnormal physical or mental condition or malfunction.

[0984] A "camera device" is a hardware device for acquiring video data.

[0985] "Video data" refers to digital data containing visual information captured by a camera device.

[0986] A "terminal" is a relay device for uploading video data to a server.

[0987] "Server" means a computer system on a network that receives, analyzes, and stores video data.

[0988] "Features" refer to important attributes or elements that make up symptoms and are extracted from the subject of analysis.

[0989] A "generative AI model" is an artificial intelligence that generates symptom descriptions based on extracted features.

[0990] "Emotional state" refers to the mental state and emotions of the user that are analyzed from video data and audio data.

[0991] "Database" refers to an information storage system for storing generated descriptions and analysis results.

[0992] A "user terminal" is a device that receives information stored in a database and notifies the user.

[0993] "Urgency" is an indicator that shows the seriousness of the symptoms and the need for response.

[0994] "Emergency response" refers to the actions and means to provide prompt medical treatment according to the degree of urgency.

[0995] The present invention is a system for monitoring the medical and emotional state of elderly and sick people and for responding quickly in emergencies. The system consists of the following main components:

[0996] 1. Camera device: A device that is installed in a specific location in the home and captures video data at all times. In this invention, the general term camera device is used.

[0997] 2. Device: A device that manages video data and uploads it to the server. The device also receives notifications from the server. Examples of devices that fall under this category include, but are not limited to, the Google Pixel and the Apple iPad.

[0998] 3. Server: This is a system that receives and analyzes video data and stores the results in a database. As a specific example, we use an AWS EC2 instance.

[0999] 4. Generative AI model: This is an artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these. A TensorFlow-based model is used as the AI ​​model.

[1000] 5. Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis. An example is IBM Watson's emotion analysis engine.

[1001] 6. Database: This is an information storage system for saving the generated explanations and analysis results. Database systems such as MySQL or MongoDB are used.

[1002] 7. User terminal: A device that receives information stored in the database and notifies the user. Devices such as Raspberry Pi fall into this category.

[1003] Operation overview

[1004] A user constantly monitors video data using a camera device installed in the home. For example, when the camera device detects an elderly person falling, it records the video.

[1005] The device automatically uploads the recorded video to the server. For example, if the device is a Google Pixel smartphone, the video data is stored from the device in an AWS S3 bucket and then sent to an AWS EC2 instance for analysis.

[1006] When the server receives the uploaded video data, it begins analyzing it using an AI model. For example, a TensorFlow-based AI model analyzes the video data frame by frame and extracts features such as abnormal behavior and changes in facial color.

[1007] The emotion engine further analyzes the emotional state based on the extracted features. For example, IBM Watson's emotion analysis engine can be used to identify emotions such as "pain" or "fear" from the user's facial expressions and voice.

[1008] Based on these features and the analysis results of the emotion engine, the generative AI model generates a specific description of the symptoms, such as, "An elderly person suddenly collapsed, their face was pale, and their breathing was heavy. They appeared to be in pain."

[1009] The server stores the generated descriptions in a database, for example a MySQL database.

[1010] The user device receives the stored data and notifies the user or their family. For example, the Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio.

[1011] Prompt Sentence Examples

[1012] "If the user is elderly and suddenly collapses in the middle of the night, please explain in detail the process of detecting abnormal movements and changes in facial expressions from video data recorded by a camera device, identifying the emotion of pain using an emotion analysis engine, and dispatching an ambulance based on the results."

[1013] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1014] Step 1:

[1015] Users constantly monitor their own situation using cameras installed in their homes. If the camera detects an abnormality, it starts recording. For example, if the user suddenly collapses, the video of the incident is automatically recorded. This video data becomes the input.

[1016] Step 2:

[1017] The device automatically uploads the recorded video to the server. During this process, the video data is temporarily stored on the device and then sent to the server via the network. For example, devices such as the Google Pixel and Apple iPad are used. The device inputs the video data (mp4 format) and uploads it to the server.

[1018] Step 3:

[1019] The server receives the uploaded video data. A dedicated script running on an AWS EC2 instance adds the received video data to the analysis queue. The server receives the video data as input and processes it by passing it to the AI ​​model.

[1020] Step 4:

[1021] The server passes the received video data to a TensorFlow-based AI model and begins analysis. The AI ​​model analyzes each video frame and extracts features such as abnormal behavior and changes in facial color. In this step, the video data is input and features (abnormal behavior, changes in facial color, etc.) are obtained as output. Specifically, the AI ​​model analyzes each frame and detects abnormal patterns.

[1022] Step 5:

[1023] The server passes the features to IBM Watson's emotion analysis engine for additional analysis. The emotion analysis engine analyzes the user's emotional state (e.g., pain, fear) from the video and audio data. In this step, the features are the input and the emotional state is obtained as the output. Specifically, the emotion analysis engine identifies changes in facial expressions and voice to analyze the user's emotions.

[1024] Step 6:

[1025] The server uses a generative AI model to generate a symptom description based on the results obtained from the AI ​​model and the emotion analysis engine. For example, a description such as "An elderly person suddenly collapsed, his face was pale, his breathing was rough, and he was in pain" is generated. In this step, the features and emotional state are input, and a symptom description is obtained as the output. Specifically, the generative AI model constructs the description using natural language processing technology.

[1026] Step 7:

[1027] The server stores the generated explanations and analysis results in a database. For example, this information is stored in MySQL or MongoDB. In this step, the symptom explanations and analysis results are input, and the output is stored in the database.

[1028] Step 8:

[1029] The user device receives the stored data and notifies the user and their family. For example, a Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio. Specifically, notifications from the database are input, and based on that, voice alerts and dispatch of an ambulance are output.

[1030] In this way, each processing step works closely together to realize a system that monitors the user's condition and responds to emergencies quickly and appropriately.

[1031] (Application example 2)

[1032] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Conventional home monitoring systems have problems with insufficient monitoring of the condition of elderly and sick people, making it difficult to respond quickly and appropriately when an emergency occurs. Furthermore, because the level of urgency is determined solely by symptom analysis without taking into account the user's emotional state, appropriate responses may be delayed. Furthermore, there is a lack of a way to present the analysis results in an easy-to-understand format, making it difficult to provide accurate information to family members and doctors.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1035] In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server, means for analyzing the video data uploaded to the server and extracting features of the symptoms, means for using an artificial intelligence model to generate a description of the symptoms based on the extracted features, means for saving the generated description and analysis results in a database, means for notifying a user terminal of the saved data, means for triggering an emergency response based on the level of urgency, means for analyzing an emotional state from the acquired video data and audio data using an emotion engine, and means for integrating the analysis results of the emotional state with the analysis results of the symptoms. This enables the user's medical condition and emotional state to be analyzed in real time, enabling prompt and appropriate response in emergencies.

[1036] A "camera device" is a device that is installed in the home and is used to acquire video data.

[1037] A "server" is a central processing unit that receives acquired video data, performs analysis, and stores and notifies the results.

[1038] "Video data" is a collection of continuous image information captured by a camera device.

[1039] "Upload" refers to the process of sending video data acquired from a camera device to a server.

[1040] "Analysis" is the process of extracting specific information from the acquired video data and organizing the detailed information using machine learning models and artificial intelligence technology.

[1041] A "feature" is a specific pattern or indicator of data extracted through analysis, and is an element that represents symptoms or emotional states.

[1042] The "explanation" is a detailed document describing the patient's condition and emotional state in natural language, generated based on the extracted features.

[1043] An "artificial intelligence model" is a set of algorithms and programs used to analyze acquired data, extract features, and generate explanatory text.

[1044] A "database" is a system for storing and managing analysis results and generated explanatory text.

[1045] A "user terminal" is a terminal device for receiving and displaying notifications of stored data.

[1046] The "emotion engine" is a software engine that analyzes the user's emotional state from video and audio data and outputs the results.

[1047] "Urgency" refers to the level of necessity for emergency response, determined based on the analysis results.

[1048] "Emergency response" is a process of arranging for an ambulance and notifying family members based on the level of urgency.

[1049] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. A specific embodiment based on an application called "Safeguard Watch" will be described.

[1050] The server acquires video data from a camera device installed in the home. This camera device continuously monitors the user's daily life in real time and acquires important video data. It also has an audio input device, so it can acquire the user's voice data.

[1051] Video and audio data are uploaded to a server via a device installed in the home. The server receives this data and prepares it for storage. The server then analyzes the uploaded video and audio data. A generative AI model and an emotion engine are used for the analysis. The generative AI model extracts symptom features and emotional states from the video and audio data. For example, abnormal movements, changes in facial color, changes in facial expression, and trembling voice are analyzed as features. The emotion engine also analyzes the user's emotional state (e.g., surprise, fear, anger, etc.).

[1052] Based on the extracted features and emotional state, the generative AI model generates a specific description of the symptoms. The generated description is presented in a format that is easy for users and doctors to understand. This description and the analysis results are stored in a database on the server. The saved data is then sent to the user's device. The user's device can receive the notification and issue a warning to the user. It can also trigger an emergency response based on the level of urgency. If the level of urgency is high, the server immediately issues an audio warning and automatically dispatches an ambulance. The system also notifies the user's family members about the symptoms that have occurred.

[1053] This system allows users to respond quickly and appropriately to emergency situations. It also enables doctors to receive accurate information. It is expected to effectively support daily disease management and monitoring. Specific examples are shown below.

[1054] If an elderly person suddenly collapses at home in the middle of the night, the camera automatically records the footage and uploads it to a server via a device. The server receives the video data and begins analysis using a generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[1055] Prompt Sentence Examples

[1056] "Please analyze the user's falling motion and changes in facial expressions from multiple angles to identify changes in the patient's condition in real time."

[1057] "Analyze the behavioral patterns of elderly people when their complexion suddenly changes and suggest appropriate emergency responses."

[1058] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1059] Step 1:

[1060] The terminal acquires video data in real time from a camera device installed in the home, and this video data is immediately transmitted to the terminal along with audio data collected by an audio input device installed in the home.

[1061] input:

[1062] Real-time video and audio data from home camera devices and audio input devices.

[1063] output:

[1064] Capture of acquired video and audio data.

[1065] Step 2:

[1066] The device uploads the captured video and audio data to the server, where it undergoes format conversion and compression before being sent in a format that the server can use for analysis.

[1067] input:

[1068] Video data and audio data acquired from a camera device and an audio input device.

[1069] output:

[1070] Video and audio data that has been converted and compressed into an analyzable format.

[1071] Step 3:

[1072] The server receives the uploaded video and audio data and passes it to the generative AI model and emotion engine. The generative AI model extracts symptom features from the video data, and the emotion engine analyzes the emotional state from the audio data.

[1073] input:

[1074] Video and audio data uploaded to the server.

[1075] output:

[1076] Emotional state data analyzed from symptom features of video data and audio data.

[1077] Specific behavior:

[1078] It uses a generative AI model to detect abnormalities in behavior and facial expressions in each frame of video data, and an emotion engine to identify emotional states from audio data.

[1079] Step 4:

[1080] The server generates a specific description of the symptoms based on the analysis results obtained from the generative AI model and the emotion engine. The generative AI model documents the description using natural language processing (NLP) techniques.

[1081] input:

[1082] Symptom features from the generative AI model and emotional state data from the emotion engine.

[1083] output:

[1084] A detailed description of the symptoms.

[1085] Specific behavior:

[1086] Using NLP technology, a description such as "An elderly person suddenly collapsed, his face was pale, and his breathing was rough" was generated based on the analysis results.

[1087] Step 5:

[1088] The server stores the generated descriptions and analysis results in a database for future reference and emergency response decisions.

[1089] input:

[1090] Generated explanations and analysis results.

[1091] output:

[1092] Descriptions and analysis results stored in a database.

[1093] Specific behavior:

[1094] The data is saved to the database and managed with a timestamp.

[1095] Step 6:

[1096] The server notifies the user device of the stored data, which then receives the notification, issues a warning to the user, and triggers an emergency response based on the urgency level.

[1097] input:

[1098] Descriptions and analysis results stored in a database.

[1099] output:

[1100] Notifications to user devices and emergency response triggers.

[1101] Specific behavior:

[1102] A push notification is sent to the user's device. If the emergency is severe, an audio warning is issued and an ambulance is automatically dispatched. The user's family is also notified of the situation.

[1103] The above steps enable real-time monitoring of the medical conditions and emotional states of elderly and sick people at home, and enable prompt and appropriate response to emergencies.

[1104] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1105] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1106] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1107] [Fourth embodiment]

[1108] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1109] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1110] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1111] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1112] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1113] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1114] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1115] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1116] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1117] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1118] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1119] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1120] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1121] The present invention is a system for monitoring the condition of sick people and the elderly and providing timely medical care. This system is realized using a camera device, a server, and a user terminal installed in the home.

[1122] System configuration

[1123] The system consists of the following main components:

[1124] Camera device: Installed in a specific location in the home (living room, bedroom, etc.) and captures video data continuously.

[1125] Terminal: Manages the video data acquired from the camera and uploads it to the server. The terminal is also a device (usually the user's smartphone or tablet) that receives notifications from the server.

[1126] Server: Receives video data, performs analysis, stores the results in a database, and takes emergency action based on the analysis results.

[1127] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[1128] Program processing

[1129] 1. Recording and uploading the video

[1130] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[1131] 2. Video analysis and symptom feature extraction

[1132] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for specifically describing the symptoms.

[1133] 3. Symptom Description Generation

[1134] Based on the extracted features, the generative AI model uses natural language processing techniques to generate a specific symptom description. This description is written in a format that is easy for users and doctors to understand. The server stores this generated description in a database for future reference.

[1135] 4. Urgency Assessment and Notification

[1136] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[1137] Specific examples

[1138] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify the family.

[1139] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[1140] The processing flow will be explained below.

[1141] Step 1:

[1142] User: A camera installed in the home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[1143] Step 2:

[1144] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[1145] Step 3:

[1146] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[1147] Step 4:

[1148] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[1149] Step 5:

[1150] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[1151] Step 6:

[1152] Generative AI model: Based on the extracted features, a specific description of the symptoms is generated using natural language processing technology. The generated description is structured in a format that is easy for users to understand.

[1153] Step 7:

[1154] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[1155] Step 8:

[1156] Server: After confirming that the data has been saved, it sends a notification to the user device. The notification includes a summary of the analysis results and a generated explanation.

[1157] Step 9:

[1158] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[1159] Step 10:

[1160] Server: Evaluates the urgency of symptoms based on the saved analysis data. If the urgency is determined to be high, an emergency response flag is set.

[1161] Step 11:

[1162] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[1163] Step 12:

[1164] Terminal: The Y terminal issues a voice alert announcing, "An ambulance has been called and will arrive shortly." At the same time, a notification is sent to the user's family, who are informed, "Symptoms have been observed and emergency response is required."

[1165] Example 1

[1166] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1167] In modern society, the number of elderly people and patients with medical conditions is increasing, making home health monitoring systems increasingly important. However, conventional systems lack the ability to detect sudden health abnormalities occurring within the home in real time and to promptly take appropriate measures. In particular, there is a need for a system that can accurately grasp the specific characteristics of symptoms and accurately determine the level of urgency. There is also a lack of means for smoothly sharing information with medical providers. In light of this background, the present invention aims to provide a system that accurately monitors health abnormalities occurring within the home and promptly takes appropriate measures.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1169] In this invention, the server includes means for acquiring video data from a video recording device installed in the home to monitor symptoms, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting symptom features, means for using an artificial intelligence system to generate a symptom description based on the extracted features, means for storing the generated description and analysis results in a data storage system, means for notifying a user terminal of the stored data, means for triggering an emergency response based on the urgency level, and means for coordinating and executing each stage of processing. This enables real-time detection of health abnormalities occurring in the home and enables prompt and appropriate response. Furthermore, smooth information sharing with medical providers can support accurate diagnosis and treatment.

[1170] A "symptom" is a change in bodily function or condition that occurs due to an abnormality or disease in the human body.

[1171] "Monitoring" is the act of carefully observing and recording events or situations in progress.

[1172] "Home" refers to the space inside a building where a family lives and conducts their daily lives.

[1173] A "video recording device" is a device such as a camera or surveillance camera that captures and stores video data.

[1174] "Video data" refers to the digital data of the captured video.

[1175] A "central processing unit" is a server or computer system that receives, processes, and analyzes data.

[1176] "Analysis" is the act of examining data or information in detail and understanding its content and structure.

[1177] "Features" are important parameters or indicators extracted from data to represent symptoms or events.

[1178] An "artificial intelligence system" is a computer system that uses certain algorithms to process data, learn, and make inferences.

[1179] A "description" is a text written in natural language about a specific phenomenon or symptom.

[1180] A "data storage system" is a system for safely storing data for a long period of time.

[1181] "User terminal" means a device that is directly operated by a user to receive and display information.

[1182] "Notification" is the act of conveying specific information to a recipient.

[1183] "Emergency response" refers to immediate response measures taken in response to a sudden event.

[1184] "Collaboration" refers to the act of multiple systems or processes working together.

[1185] This invention relates to a system for real-time monitoring of health abnormalities occurring in the home and for promptly taking appropriate measures. This system is realized using a video recording device, a central processing unit (server), a user terminal, and an artificial intelligence system.

[1186] Hardware and software used

[1187] 1. Video Recording Device

[1188] A user installs a video recording device in a specific location in the home (e.g., the living room or bedroom). The video recording device is a home surveillance camera (e.g., a home IP camera). This device continuously captures video data and automatically records the video if an abnormality is detected.

[1189] 2. User terminal

[1190] The terminal is responsible for managing the video data acquired from the video recording device and uploading it to the central processing unit. The terminal can be a smartphone or tablet. After receiving the video data, the user terminal sends it to the central processing unit and also receives notifications from the server.

[1191] 3. Central Processing Unit (Server)

[1192] The server receives and analyzes the video data uploaded from the device. Specifically, it uses an AI analysis platform (for example, a cloud service such as AWS or Microsoft Azure). The server analyzes the video data, extracts symptom features, and passes them to the AI ​​model. The server also stores the analysis results and generated explanations in a database and takes action based on the urgency assessment.

[1193] 4. Artificial Intelligence Systems

[1194] The generative AI model extracts symptom features from video data and generates symptom descriptions based on them. The AI ​​model used uses deep learning frameworks such as TensorFlow and PyTorch. In particular, the generative AI model uses natural language processing technology (such as GPT-3) to generate specific symptom descriptions.

[1195] Specific operation of the system

[1196] A user keeps a camera device installed in their home turned on at all times. The camera device continues to capture images of the surroundings in real time, and if an abnormality is detected, the image is automatically recorded.

[1197] The device periodically receives video data from the camera and uploads it to a central processing unit (server). The server adds the received video data to an analysis queue and passes it sequentially to the generative AI model. The generative AI model analyzes the video data frame by frame and extracts features such as abnormal movements, changes in facial expression, and changes in skin color.

[1198] Based on the extracted features, the generative AI model uses natural language processing to generate a specific description of the symptoms, such as "an elderly person suddenly collapsed, their face was pale, and their breathing was becoming labored."

[1199] The server stores the generated explanation and analysis results in a database and sends emergency notifications to the user's device as needed. If the emergency is severe, the server will issue a voice warning or automatically arrange for emergency medical transport.

[1200] Examples of concrete examples and prompts

[1201] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to a server via the device. The server receives the video data and begins analysis using a generative AI model. The AI ​​model extracts features such as the collapse motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The results of this analysis are stored in a database, and if the emergency is deemed high, the server will issue an audio warning, arrange for emergency medical transport, and notify the user's family.

[1202] An example of a prompt sentence is, "Please generate an analysis result of a scene in which an elderly person collapses." The model will extract the features of the specified scene and generate a detailed description.

[1203] This system allows users to respond quickly and appropriately to emergencies that occur at home and provides accurate information to medical providers.

[1204] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1205] Step 1: Capture and transmit video data

[1206] A user turns on a video recording device installed in their home. The camera device continuously captures video data and automatically records the video if an abnormality occurs. The recorded video is sent to a terminal in real time. The input is the video data captured by the camera device, and the output is the recorded video data sent to the terminal. Specifically, the camera device detects a sudden fall by an elderly person and records the scene.

[1207] Step 2: Receiving and uploading video data

[1208] The terminal receives video data sent from the camera device and uploads it to the server at regular intervals. The input is the recorded video data received from the camera device, and the output is the video data uploaded to the server. Specifically, the terminal receives video data from the camera device and immediately uploads it to the cloud server.

[1209] Step 3: Video data analysis and feature extraction

[1210] The server receives the video data uploaded from the device and adds it to the analysis queue. The server then passes the video data in the analysis queue to the generative AI model, which then begins analysis. The generative AI model analyzes the video data frame by frame and extracts symptom features such as abnormal movements, changes in facial expression, and changes in skin color. The input is the video data uploaded to the server, and the output is the extracted symptom features. Specifically, this is the AI ​​model's behavior in identifying elderly people's falling movements and changes in facial color.

[1211] Step 4: Generate symptom descriptions

[1212] The server inputs the extracted features into a generative AI model to generate a description of the symptoms. The generative AI model uses natural language processing technology to generate a description and returns it to the server. The input is the extracted features of the symptoms, and the output is the generated description of the symptoms. Specifically, the operation generates a description such as "After the elderly person collapsed, their face turned pale and their breathing became rough."

[1213] Step 5: Save the description and analysis results

[1214] The server saves the generated symptom descriptions and analysis results in a database. The saved data is archived for future reference. The input is the generated symptom descriptions and analysis results, and the output is the records stored in the database. Specifically, the operation is to store the generated descriptions and analysis results in an SQL database.

[1215] Step 6: Urgency assessment and notification

[1216] The server evaluates the generated analysis results and symptom descriptions to determine the level of urgency. If the level of urgency is high, the server sends an audio warning instruction to the terminal and arranges for emergency medical transport if necessary. The input is the generated analysis results and symptom descriptions, and the output is an alert instruction and response arrangements based on the level of urgency. Specifically, the server determines that the level of urgency is high and sends an audio warning notification to the user's smartphone.

[1217] Step 7: Implementing notifications and following up

[1218] The device receives the voice warning instruction sent from the server and notifies the user. At the same time, if the emergency level is high, the device notifies the user's family as well. The input is the warning instruction from the server, and the output is a voice notification and an alert to the user. Specifically, the smartphone sends a voice message to the elderly person's family saying, "An emergency has occurred. Please respond immediately."

[1219] (Application example 1)

[1220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1221] There is a need to safely monitor the elderly and sick at home, quickly detect any sudden changes in their condition, and take appropriate action. However, existing systems have difficulty simultaneously monitoring their condition and security, making it difficult to respond quickly and accurately. Therefore, a highly reliable system that can comprehensively monitor health and safety at home is needed.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1223] In this invention, the server includes means for acquiring video data from a video device installed in the home to monitor symptoms and abnormalities, means for uploading the acquired video data to a central processing unit, means for analyzing the video data uploaded to the central processing unit and extracting feature quantities of symptoms and abnormalities, means for using a generative artificial intelligence model to generate an explanatory text based on the extracted feature quantities, means for saving the generated explanatory text and analysis results in a recording medium, means for notifying a user terminal of the saved data, and means for triggering medical and security responses based on the urgency level. This enables comprehensive monitoring of health and safety within the home, and enables appropriate medical and security responses to be taken immediately when an abnormality is detected.

[1224] "Symptoms and abnormalities" refers to changes in the health status of a person or abnormal behavior, such as collapsing or changes in facial color, that are not normal.

[1225] "Video equipment" refers to cameras and recording devices installed in the home that continuously capture video data.

[1226] "Video data" refers to video information captured within the home by a video device.

[1227] A "central processing unit" is a device, such as a server or computer system, that analyzes acquired video data and performs the necessary calculations.

[1228] "Features" are specific attributes or patterns extracted from video data to explain symptoms or abnormalities.

[1229] A "generative artificial intelligence model" is a model that uses artificial intelligence technology to extract features of symptoms and abnormalities from video data and generate explanatory text in natural language based on this.

[1230] "Recording medium" refers to a database or storage device for saving the analysis results and generated explanations.

[1231] A "user terminal" is a device used to notify the analysis results and generated explanatory text, and typically refers to a smartphone or tablet.

[1232] "Urgency" refers to the result of evaluating the seriousness of detected symptoms or abnormalities, and serves as a criterion for determining whether immediate action is required.

[1233] "Medical response" refers to medical treatment or intervention based on the urgency of the situation, such as dispatching an ambulance.

[1234] "Security response" refers to crime prevention measures taken when an abnormality is detected within the home, including issuing emergency notifications and warnings.

[1235] System Overview

[1236] The system for implementing this invention is composed of a video device, a central processing unit, a user terminal, and software for linking these devices installed in the home. The system is designed to simultaneously monitor the user's health and the safety of the home, and to respond immediately if an abnormality is detected.

[1237] Hardware and software used

[1238] Hardware:

[1239] Video device (camera): Continuously captures video from within the home.

[1240] Central processing unit (server): Receives captured video data and performs analysis.

[1241] User device (smartphone / tablet / smart glasses): A device that notifies the analysis results.

[1242] software:

[1243] OpenCV: A library for camera operation and image processing.

[1244] TensorFlow: A library for analyzing video data using generative artificial intelligence models.

[1245] smtplib: A library for sending analysis results and notifications by email.

[1246] Program processing

[1247] The server periodically acquires video data from cameras installed in the home and analyzes the data using a central processing unit. During the analysis, a generative artificial intelligence model is used to extract abnormal features (e.g., falling, changes in facial color). The extracted features are converted into explanatory text using natural language processing technology, and the analysis results are saved on a recording medium.

[1248] Program execution procedure

[1249] 1. Acquisition of video data:

[1250] The video device continuously monitors a specific area in the home (e.g., living room, bedroom) and acquires video data.

[1251] 2. Upload data:

[1252] The captured video data is uploaded to the central processing unit via the user terminal.

[1253] 3. Data Analysis:

[1254] The central processing unit analyzes the uploaded video data using AI libraries such as TensorFlow.

[1255] Based on the analysis, feature quantities of symptoms and abnormalities are extracted.

[1256] 4. Description generation:

[1257] Based on the extracted features, a generative artificial intelligence model generates a natural language explanation.

[1258] 5. Data Retention and Notification:

[1259] The generated explanation and analysis results are stored on a recording medium, and a notification is sent to the user terminal according to the level of urgency.

[1260] Specific examples

[1261] For example, if an elderly person collapses at night, the camera detects the movement and sends the video data to a central processing unit. The central processing unit extracts features such as the collapse movement and changes in facial color, and generates a description such as "The elderly person suddenly collapsed, their face turned pale, and their breathing became heavy." This result is saved in the central processing unit, and if the emergency is deemed high, medical and security measures are immediately implemented. A notification is immediately sent to the user's device, and an emergency vehicle may be dispatched, and an alert may also be sent to the family.

[1262] Prompt Sentence Examples

[1263] "We are building a new system to monitor family members. We would like the system to use a camera to detect abnormalities such as elderly or sick members collapsing or becoming pale, and then take appropriate action. What kind of prompt would be appropriate?"

[1264] In this way, the system can take appropriate measures quickly and effectively support health and safety monitoring within the home.

[1265] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1266] Step 1:

[1267] A video device (camera) in the home captures video data. The camera continuously monitors a specific area, such as the living room or bedroom, and collects video data at a constant frame rate. The input is real-time video from the camera, and the output is the captured video frames.

[1268] Step 2:

[1269] The terminal uploads the video data acquired from the camera to a central processing unit (server). The terminal buffers video frames at regular intervals and transmits them to the central processing unit via the network. The input is the video data from the video device, and the output is the video data uploaded to the server.

[1270] Step 3:

[1271] The server receives the uploaded video data and begins analysis. The server divides the video data into frames and inputs them into a generative AI model. The input is the uploaded video data, and the output is image data for each frame to perform analysis.

[1272] Step 4:

[1273] The server uses a generative AI model to analyze the video data and extract features of symptoms and abnormalities. Typically, an AI library such as TensorFlow is used, and the model detects abnormal behavior and changes from the video data. The input is frame image data for analysis, and the output is feature data (e.g., the behavior of falling, changes in facial color).

[1274] Step 5:

[1275] The server uses a generative AI model to generate a description of the symptoms or abnormalities based on the extracted features. The generated description is created using natural language processing (NLP) technology. The input is feature data, and the output is a description in a format that is easy for humans to understand.

[1276] Step 6:

[1277] The server stores the generated explanation and analysis results on a recording medium. The server stores the generated explanation and related analysis results in a database. The input is the generated explanation and analysis results, and the output is the data stored in the database.

[1278] Step 7:

[1279] The server evaluates the urgency and sends a notification to the user device if it is determined to be highly urgent. The urgency evaluation is performed using an AI model or rule-based system. The input is the generated explanation and analysis results, and the output is notification data sent to the user device.

[1280] Step 8:

[1281] The server sends a notification to the user's smart device (e.g., smartphone, tablet, smart glasses). The user device responds based on the received notification, such as issuing an audio warning. The input is the result of the urgency assessment and notification data, and the output is a real-time notification and audio warning.

[1282] Step 9:

[1283] The user checks the notification and takes additional action as necessary (e.g., contacting a medical institution or reporting to a security agency). The input is the notification sent to the user terminal, and the output is the user's response action.

[1284] Step 10:

[1285] The server automatically triggers appropriate actions (e.g. dispatching emergency vehicles, notifying security). In the case of a high-urgency situation, the system automatically executes triggers to ensure a prompt response. The input is the result of the urgency assessment, and the output is the automatic trigger action.

[1286] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1287] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. This system analyzes changes in the user's medical condition and emotional state in real time, enabling appropriate medical response.

[1288] System configuration

[1289] The system consists of the following main components:

[1290] Camera device: Installed in a specific location in the home, it constantly captures video data.

[1291] Terminal: Manages video data and uploads it to the server. The terminal is also a device that receives notifications from the server.

[1292] Server: Receives and analyzes video data, stores the results in a database, and takes emergency action based on the analysis results.

[1293] Generative AI model: An artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these.

[1294] Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis.

[1295] Program processing

[1296] Recording and uploading video

[1297] The user constantly monitors video data using a camera device installed in their home. When a symptom occurs, the camera device automatically records the video. The recorded video is automatically uploaded to the server by the terminal. The server receives the uploaded video file and adds it to the analysis queue.

[1298] Video analysis and symptom feature extraction

[1299] The server passes the uploaded video data to the AI ​​model, which then begins analysis. The AI ​​model analyzes the video data frame by frame and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). These features are the basic elements for describing specific symptoms.

[1300] Further analysis of the emotion engine

[1301] Furthermore, the emotion engine analyzes the user's emotional state from the video and audio data. The emotion engine uses, for example, facial expression analysis and audio analysis to identify the user's emotions (e.g., surprise, fear, anger, etc.). This emotional state information is integrated with the symptom analysis results to provide comprehensive information for judgment.

[1302] Symptom description generation

[1303] Based on the extracted features and the analysis results of the emotion engine, the generative AI model uses natural language processing technology to generate a specific description of the symptoms. This description is written in a format that is easy for users and doctors to understand. The server stores the generated description in a database.

[1304] Urgency assessment and notification

[1305] The server evaluates the generated analysis results and the symptom description to determine the level of urgency. The results of the emotion engine are also taken into account here. If the level of urgency is high, the server immediately sends an instruction to the device to issue a voice warning and, if necessary, automatically dispatch an ambulance. It also notifies the user's family members about the symptoms that have occurred.

[1306] Specific examples

[1307] For example, if an elderly user suddenly collapses at home in the middle of the night, the camera device automatically records the video and uploads it to the server via the device. The server receives the video data and begins analysis using the generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "The elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[1308] This system will enable users to respond quickly and appropriately to emergencies and provide accurate information to doctors, and is expected to effectively support daily disease management and monitoring.

[1309] The processing flow will be explained below.

[1310] Step 1:

[1311] User: A camera installed in the user's home constantly monitors video. When a symptom occurs, the camera automatically starts recording.

[1312] Step 2:

[1313] Terminal: After recording is complete, the terminal starts communication to upload the recorded video data to the server. The upload program imports the video file and establishes communication with the server.

[1314] Step 3:

[1315] Server: Receives upload requests, saves video data to storage, and adds the saved video data to the analysis queue to start the analysis process.

[1316] Step 4:

[1317] Server: Takes video data from the analysis queue and prepares it for analysis by the AI ​​model. Divides the video data into frames and passes them to the analysis module.

[1318] Step 5:

[1319] Generative AI model: Analyzes each frame of video and extracts symptom features (e.g., abnormal movements, changes in facial expression, changes in skin color, etc.). The extracted features are recorded as data.

[1320] Step 6:

[1321] Server: The extracted features are formatted for storage in a database, making them possible to integrate with existing symptom data.

[1322] Step 7:

[1323] Emotion engine: Analyzes the user's emotional state from video and audio data. For example, facial expression analysis and audio analysis are used to identify emotions such as surprise, fear, and anger. Emotional state information is integrated with symptom analysis results.

[1324] Step 8:

[1325] Generative AI model: Based on the extracted features and the analysis results of the emotion engine, a specific description of the symptoms is generated using natural language processing technology. This description is constructed in a format that is easy for users to understand.

[1326] Step 9:

[1327] Server: Stores the generated descriptions and analysis data in a database, where the information is indexed and made available for future reference and analysis.

[1328] Step 10:

[1329] Server: After confirming that the data has been saved, it sends a notification to the user device, which includes a summary of the analysis results and a generated explanation.

[1330] Step 11:

[1331] Terminal: The user terminal receives the notification and displays the alert to the user, who can then check the alert and present it to a doctor if necessary.

[1332] Step 12:

[1333] Server: Evaluates the urgency of symptoms based on the saved analysis data. Taking into account the results of the emotion engine, if the urgency is evaluated as high, sets an emergency response flag.

[1334] Step 13:

[1335] Server: If the emergency response flag is set, the emergency response program issues instructions to trigger a voice alert and dispatch an ambulance.

[1336] Step 14:

[1337] Device: The device issues a voice alert, announcing that an ambulance has been called and will arrive shortly. At the same time, a notification is sent to the user's family, informing them that symptoms have been observed and that emergency response is required.

[1338] This detailed processing step allows for real-time monitoring of the user's medical condition and emotional state, enabling prompt and accurate responses, and the stored data can then be used by doctors for more accurate and effective treatment.

[1339] Example 2

[1340] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1341] When the condition of an elderly person or a sick person suddenly changes at home, it is difficult to quickly and accurately monitor and judge the change and take appropriate measures promptly. In particular, for elderly people living alone, there is a lack of means to grasp changes in the condition in real time and take emergency measures, so a system that solves this problem is needed.

[1342] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server via a terminal, means for analyzing the video data uploaded to the server and extracting symptom features, means for using a generative AI model to generate a symptom description based on the extracted features, means for analyzing an emotional state from video data and audio data, means for storing the generated description and analysis results in a database, means for notifying a user terminal of the stored data, and means for triggering an emergency response based on the urgency. This makes it possible to monitor changes in the condition of elderly people and sick people in real time and take prompt and appropriate medical action.

[1343] A "symptom" is a sign or phenomenon that indicates an abnormal physical or mental condition or malfunction.

[1344] A "camera device" is a hardware device for acquiring video data.

[1345] "Video data" refers to digital data containing visual information captured by a camera device.

[1346] A "terminal" is a relay device for uploading video data to a server.

[1347] "Server" means a computer system on a network that receives, analyzes, and stores video data.

[1348] "Features" refer to important attributes or elements that make up symptoms and are extracted from the subject of analysis.

[1349] A "generative AI model" is an artificial intelligence that generates symptom descriptions based on extracted features.

[1350] "Emotional state" refers to the mental state and emotions of the user that are analyzed from video data and audio data.

[1351] "Database" refers to an information storage system for storing generated descriptions and analysis results.

[1352] A "user terminal" is a device that receives information stored in a database and notifies the user.

[1353] "Urgency" is an indicator that shows the seriousness of the symptoms and the need for response.

[1354] "Emergency response" refers to the actions and means to provide prompt medical treatment according to the degree of urgency.

[1355] The present invention is a system for monitoring the medical and emotional state of elderly and sick people and for responding quickly in emergencies. The system consists of the following main components:

[1356] 1. Camera device: A device that is installed in a specific location in the home and captures video data at all times. In this invention, the general term camera device is used.

[1357] 2. Device: A device that manages video data and uploads it to the server. The device also receives notifications from the server. Examples of devices that fall under this category include, but are not limited to, the Google Pixel and the Apple iPad.

[1358] 3. Server: This is a system that receives and analyzes video data and stores the results in a database. As a specific example, we use an AWS EC2 instance.

[1359] 4. Generative AI model: This is an artificial intelligence model that extracts symptom features from video data and generates symptom descriptions based on these. A TensorFlow-based model is used as the AI ​​model.

[1360] 5. Emotion engine: An engine that analyzes the user's emotional state from video and audio data, and outputs the results in conjunction with medical condition analysis. An example is IBM Watson's emotion analysis engine.

[1361] 6. Database: This is an information storage system for saving the generated explanations and analysis results. Database systems such as MySQL or MongoDB are used.

[1362] 7. User terminal: A device that receives information stored in the database and notifies the user. Devices such as Raspberry Pi fall into this category.

[1363] Operation overview

[1364] A user constantly monitors video data using a camera device installed in the home. For example, when the camera device detects an elderly person falling, it records the video.

[1365] The device automatically uploads the recorded video to the server. For example, if the device is a Google Pixel smartphone, the video data is stored from the device in an AWS S3 bucket and then sent to an AWS EC2 instance for analysis.

[1366] When the server receives the uploaded video data, it begins analyzing it using an AI model. For example, a TensorFlow-based AI model analyzes the video data frame by frame and extracts features such as abnormal behavior and changes in facial color.

[1367] The emotion engine further analyzes the emotional state based on the extracted features. For example, IBM Watson's emotion analysis engine can be used to identify emotions such as "pain" or "fear" from the user's facial expressions and voice.

[1368] Based on these features and the analysis results of the emotion engine, the generative AI model generates a specific description of the symptoms, such as, "An elderly person suddenly collapsed, their face was pale, and their breathing was heavy. They appeared to be in pain."

[1369] The server stores the generated descriptions in a database, for example a MySQL database.

[1370] The user device receives the stored data and notifies the user or their family. For example, the Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio.

[1371] Prompt Sentence Examples

[1372] "If the user is elderly and suddenly collapses in the middle of the night, please explain in detail the process of detecting abnormal movements and changes in facial expressions from video data recorded by a camera device, identifying the emotion of pain using an emotion analysis engine, and dispatching an ambulance based on the results."

[1373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1374] Step 1:

[1375] Users constantly monitor their own situation using cameras installed in their homes. If the camera detects an abnormality, it starts recording. For example, if the user suddenly collapses, the video of the incident is automatically recorded. This video data becomes the input.

[1376] Step 2:

[1377] The device automatically uploads the recorded video to the server. During this process, the video data is temporarily stored on the device and then sent to the server via the network. For example, devices such as the Google Pixel and Apple iPad are used. The device inputs the video data (mp4 format) and uploads it to the server.

[1378] Step 3:

[1379] The server receives the uploaded video data. A dedicated script running on an AWS EC2 instance adds the received video data to the analysis queue. The server receives the video data as input and processes it by passing it to the AI ​​model.

[1380] Step 4:

[1381] The server passes the received video data to a TensorFlow-based AI model and begins analysis. The AI ​​model analyzes each video frame and extracts features such as abnormal behavior and changes in facial color. In this step, the video data is input and features (abnormal behavior, changes in facial color, etc.) are obtained as output. Specifically, the AI ​​model analyzes each frame and detects abnormal patterns.

[1382] Step 5:

[1383] The server passes the features to IBM Watson's emotion analysis engine for additional analysis. The emotion analysis engine analyzes the user's emotional state (e.g., pain, fear) from the video and audio data. In this step, the features are the input and the emotional state is obtained as the output. Specifically, the emotion analysis engine identifies changes in facial expressions and voice to analyze the user's emotions.

[1384] Step 6:

[1385] The server uses a generative AI model to generate a symptom description based on the results obtained from the AI ​​model and the emotion analysis engine. For example, a description such as "An elderly person suddenly collapsed, his face was pale, his breathing was rough, and he was in pain" is generated. In this step, the features and emotional state are input, and a symptom description is obtained as the output. Specifically, the generative AI model constructs the description using natural language processing technology.

[1386] Step 7:

[1387] The server stores the generated explanations and analysis results in a database. For example, this information is stored in MySQL or MongoDB. In this step, the symptom explanations and analysis results are input, and the output is stored in the database.

[1388] Step 8:

[1389] The user device receives the stored data and notifies the user and their family. For example, a Raspberry Pi device can issue a voice alert to the family and automatically dispatch an ambulance using Twilio. Specifically, notifications from the database are input, and based on that, voice alerts and dispatch of an ambulance are output.

[1390] In this way, each processing step works closely together to realize a system that monitors the user's condition and responds to emergencies quickly and appropriately.

[1391] (Application example 2)

[1392] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1393] Conventional home monitoring systems have problems with insufficient monitoring of the condition of elderly and sick people, making it difficult to respond quickly and appropriately when an emergency occurs. Furthermore, because the level of urgency is determined solely by symptom analysis without taking into account the user's emotional state, appropriate responses may be delayed. Furthermore, there is a lack of a way to present the analysis results in an easy-to-understand format, making it difficult to provide accurate information to family members and doctors.

[1394] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1395] In this invention, the server includes means for acquiring video data from a camera device installed in the home to monitor symptoms, means for uploading the acquired video data to the server, means for analyzing the video data uploaded to the server and extracting features of the symptoms, means for using an artificial intelligence model to generate a description of the symptoms based on the extracted features, means for saving the generated description and analysis results in a database, means for notifying a user terminal of the saved data, means for triggering an emergency response based on the level of urgency, means for analyzing an emotional state from the acquired video data and audio data using an emotion engine, and means for integrating the analysis results of the emotional state with the analysis results of the symptoms. This enables the user's medical condition and emotional state to be analyzed in real time, enabling prompt and appropriate response in emergencies.

[1396] A "camera device" is a device that is installed in the home and is used to acquire video data.

[1397] A "server" is a central processing unit that receives acquired video data, performs analysis, and stores and notifies the results.

[1398] "Video data" is a collection of continuous image information captured by a camera device.

[1399] "Upload" refers to the process of sending video data acquired from a camera device to a server.

[1400] "Analysis" is the process of extracting specific information from the acquired video data and organizing the detailed information using machine learning models and artificial intelligence technology.

[1401] A "feature" is a specific pattern or indicator of data extracted through analysis, and is an element that represents symptoms or emotional states.

[1402] The "explanation" is a detailed document describing the patient's condition and emotional state in natural language, generated based on the extracted features.

[1403] An "artificial intelligence model" is a set of algorithms and programs used to analyze acquired data, extract features, and generate explanatory text.

[1404] A "database" is a system for storing and managing analysis results and generated explanatory text.

[1405] A "user terminal" is a terminal device for receiving and displaying notifications of stored data.

[1406] The "emotion engine" is a software engine that analyzes the user's emotional state from video and audio data and outputs the results.

[1407] "Urgency" refers to the level of necessity for emergency response, determined based on the analysis results.

[1408] "Emergency response" is a process of arranging for an ambulance and notifying family members based on the level of urgency.

[1409] This invention is a system that combines a camera device and an emotion engine installed in the home to monitor the medical condition and emotional state of sick people and the elderly, and responds quickly in emergencies. A specific embodiment based on an application called "Safeguard Watch" will be described.

[1410] The server acquires video data from a camera device installed in the home. This camera device continuously monitors the user's daily life in real time and acquires important video data. It also has an audio input device, so it can acquire the user's voice data.

[1411] Video and audio data are uploaded to a server via a device installed in the home. The server receives this data and prepares it for storage. The server then analyzes the uploaded video and audio data. A generative AI model and an emotion engine are used for the analysis. The generative AI model extracts symptom features and emotional states from the video and audio data. For example, abnormal movements, changes in facial color, changes in facial expression, and trembling voice are analyzed as features. The emotion engine also analyzes the user's emotional state (e.g., surprise, fear, anger, etc.).

[1412] Based on the extracted features and emotional state, the generative AI model generates a specific description of the symptoms. The generated description is presented in a format that is easy for users and doctors to understand. This description and the analysis results are stored in a database on the server. The saved data is then sent to the user's device. The user's device can receive the notification and issue a warning to the user. It can also trigger an emergency response based on the level of urgency. If the level of urgency is high, the server immediately issues an audio warning and automatically dispatches an ambulance. The system also notifies the user's family members about the symptoms that have occurred.

[1413] This system allows users to respond quickly and appropriately to emergency situations. It also enables doctors to receive accurate information. It is expected to effectively support daily disease management and monitoring. Specific examples are shown below.

[1414] If an elderly person suddenly collapses at home in the middle of the night, the camera automatically records the footage and uploads it to a server via a device. The server receives the video data and begins analysis using a generative AI model and emotion engine. The AI ​​model extracts features such as the falling motion, changes in facial color, and abnormal breathing, and generates a description such as "An elderly person suddenly collapsed, their face was pale, and their breathing was rough." The emotion engine reads emotions such as fear and pain from the elderly person's facial expressions and voice and adds them to the analysis results. This comprehensive analysis result is stored in a database, and if the emergency is deemed high, the server will issue an audio warning, dispatch an ambulance, and notify family members.

[1415] Prompt Sentence Examples

[1416] "Please analyze the user's falling motion and changes in facial expressions from multiple angles to identify changes in the patient's condition in real time."

[1417] "Analyze the behavioral patterns of elderly people when their complexion suddenly changes and suggest appropriate emergency responses."

[1418] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1419] Step 1:

[1420] The terminal acquires video data in real time from a camera device installed in the home, and this video data is immediately transmitted to the terminal along with audio data collected by an audio input device installed in the home.

[1421] input:

[1422] Real-time video and audio data from home camera devices and audio input devices.

[1423] output:

[1424] Capture of acquired video and audio data.

[1425] Step 2:

[1426] The device uploads the captured video and audio data to the server, where it undergoes format conversion and compression before being sent in a format that the server can use for analysis.

[1427] input:

[1428] Video data and audio data acquired from a camera device and an audio input device.

[1429] output:

[1430] Video and audio data that has been converted and compressed into an analyzable format.

[1431] Step 3:

[1432] The server receives the uploaded video and audio data and passes it to the generative AI model and emotion engine. The generative AI model extracts symptom features from the video data, and the emotion engine analyzes the emotional state from the audio data.

[1433] input:

[1434] Video and audio data uploaded to the server.

[1435] output:

[1436] Emotional state data analyzed from symptom features of video data and audio data.

[1437] Specific behavior:

[1438] It uses a generative AI model to detect abnormalities in behavior and facial expressions in each frame of video data, and an emotion engine to identify emotional states from audio data.

[1439] Step 4:

[1440] The server generates a specific description of the symptoms based on the analysis results obtained from the generative AI model and the emotion engine. The generative AI model documents the description using natural language processing (NLP) techniques.

[1441] input:

[1442] Symptom features from the generative AI model and emotional state data from the emotion engine.

[1443] output:

[1444] A detailed description of the symptoms.

[1445] Specific behavior:

[1446] Using NLP technology, a description such as "An elderly person suddenly collapsed, his face was pale, and his breathing was rough" was generated based on the analysis results.

[1447] Step 5:

[1448] The server stores the generated descriptions and analysis results in a database for future reference and emergency response decisions.

[1449] input:

[1450] Generated explanations and analysis results.

[1451] output:

[1452] Descriptions and analysis results stored in a database.

[1453] Specific behavior:

[1454] The data is saved to the database and managed with a timestamp.

[1455] Step 6:

[1456] The server notifies the user device of the stored data, which then receives the notification, issues a warning to the user, and triggers an emergency response based on the urgency level.

[1457] input:

[1458] Descriptions and analysis results stored in a database.

[1459] output:

[1460] Notifications to user devices and emergency response triggers.

[1461] Specific behavior:

[1462] A push notification is sent to the user's device. If the emergency is severe, an audio warning is issued and an ambulance is automatically dispatched. The user's family is also notified of the situation.

[1463] The above steps enable real-time monitoring of the medical conditions and emotional states of elderly and sick people at home, and enable prompt and appropriate response to emergencies.

[1464] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1465] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1466] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1467] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1468] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1469] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1470] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1471] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1472] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1473] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1474] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1475] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1476] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1477] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1478] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1479] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1480] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1481] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1482] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1483] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1484] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1485] The following is further disclosed regarding the above embodiment.

[1486] (Claim 1)

[1487] means for acquiring video data from a camera device installed in the home to monitor symptoms;

[1488] means for uploading the acquired video data to a server;

[1489] A means for analyzing the video data uploaded to the server and extracting symptom features;

[1490] a means for using an artificial intelligence model to generate symptom descriptions based on the extracted features;

[1491] a means for storing the generated descriptions and analysis results in a database;

[1492] means for notifying a user terminal of the stored data;

[1493] a means for triggering an emergency response based on the urgency;

[1494] A system including:

[1495] (Claim 2)

[1496] means for assessing the severity and urgency of symptoms from the acquired video data;

[1497] means for issuing an audio warning based on the evaluation result;

[1498] A means of dispatching an ambulance;

[1499] The system of claim 1 further comprising:

[1500] (Claim 3)

[1501] 2. The system according to claim 1, further comprising means for presenting the analysis results notified to the user terminal to a doctor at the time of consultation.

[1502] "Example 1"

[1503] (Claim 1)

[1504] means for acquiring video data from a video recording device installed in the home for symptom monitoring;

[1505] means for uploading the captured video data to a central processing unit;

[1506] means for analyzing the video data uploaded to the central processing unit and extracting symptom features;

[1507] using an artificial intelligence system to generate symptom descriptions based on the extracted features;

[1508] a means for storing the generated explanation and analysis results in a data storage system;

[1509] means for notifying a user terminal of the stored data;

[1510] a means for triggering an emergency response based on the urgency;

[1511] A means for executing the processes of each stage in a coordinated manner;

[1512] A system including:

[1513] (Claim 2)

[1514] a means for assessing the severity and urgency of symptoms from the acquired video data;

[1515] means for issuing an audio warning based on the evaluation result;

[1516] Arranging for emergency medical evacuation;

[1517] The system of claim 1 further comprising:

[1518] (Claim 3)

[1519] 10. The system according to claim 1, further comprising means for presenting the analysis results notified to the user terminal to a healthcare provider at the time of consultation.

[1520] "Application Example 1"

[1521] (Claim 1)

[1522] means for acquiring video data from a video device installed in the home to monitor symptoms and abnormalities;

[1523] means for uploading the captured video data to a central processing unit;

[1524] means for analyzing the video data uploaded to the central processing unit and extracting symptom and abnormality feature quantities;

[1525] a means for using a generative artificial intelligence model to generate a narrative based on the extracted features;

[1526] a means for storing the generated explanatory text and analysis results in a recording medium;

[1527] means for notifying a user terminal of the stored data;

[1528] a means for triggering medical and security responses based on urgency;

[1529] A system including:

[1530] (Claim 2)

[1531] means for assessing the severity and urgency of symptoms and abnormalities from the acquired video data;

[1532] means for issuing an audio warning based on the evaluation result;

[1533] A means of dispatching emergency vehicles;

[1534] The system of claim 1 further comprising:

[1535] (Claim 3)

[1536] 2. The system according to claim 1, further comprising means for presenting the analysis results notified to the user terminal at the time of consultation by a specialist.

[1537] "Example 2: Combining Emotion Engines"

[1538] (Claim 1)

[1539] means for acquiring video data from a camera device installed in the home to monitor symptoms;

[1540] means for uploading the acquired video data to a server via the terminal;

[1541] A means for analyzing the video data uploaded to the server and extracting symptom features;

[1542] a means for using a generative AI model to generate symptom descriptions based on the extracted features;

[1543] means for analyzing an emotional state from video data and audio data;

[1544] a means for storing the generated descriptions and analysis results in a database;

[1545] means for notifying a user terminal of the stored data;

[1546] a means for triggering an emergency response based on the urgency;

[1547] A system including:

[1548] (Claim 2)

[1549] means for assessing the severity and urgency of symptoms from the acquired video data;

[1550] means for issuing an audio warning based on the evaluation result;

[1551] A means of dispatching an ambulance;

[1552] The system of claim 1 further comprising:

[1553] (Claim 3)

[1554] 2. The system according to claim 1, further comprising means for presenting the analysis results notified to the user terminal to a doctor at the time of consultation.

[1555] "Application example 2 when combining emotion engines"

[1556] (Claim 1)

[1557] means for acquiring video data from a camera device installed in the home to monitor symptoms;

[1558] means for uploading the acquired video data to a server;

[1559] A means for analyzing the video data uploaded to the server and extracting symptom features;

[1560] a means for using an artificial intelligence model to generate symptom descriptions based on the extracted features;

[1561] a means for storing the generated descriptions and analysis results in a database;

[1562] means for notifying a user terminal of the stored data;

[1563] a means for triggering an emergency response based on the urgency;

[1564] means for analyzing an emotional state from the acquired video data and audio data using an emotion engine;

[1565] means for integrating the emotional state analysis with the symptom analysis;

[1566] A system including:

[1567] (Claim 2)

[1568] means for assessing the severity and urgency of symptoms from the acquired video data;

[1569] means for issuing an audio warning based on the evaluation result;

[1570] A means of dispatching an ambulance;

[1571] a means for adjusting the urgency level in consideration of the analyzed emotional state information;

[1572] The system of claim 1 further comprising:

[1573] (Claim 3)

[1574] The method further includes a means for presenting the analysis results notified to the user terminal at the time of a doctor's consultation,

[1575] 10. The system of claim 1, further comprising means for presenting the generated narrative in natural language. [Explanation of symbols]

[1576] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring video data from a camera device installed in the home to monitor symptoms; means for uploading the acquired video data to a server; A means for analyzing the video data uploaded to the server and extracting symptom features; a means for using an artificial intelligence model to generate symptom descriptions based on the extracted features; a means for storing the generated descriptions and analysis results in a database; means for notifying a user terminal of the stored data; a means for triggering an emergency response based on the urgency; A system including:

2. means for assessing the severity and urgency of symptoms from the acquired video data; means for issuing an audio warning based on the evaluation result; A means of dispatching an ambulance; The system of claim 1 further comprising:

3. The system according to claim 1 , further comprising means for presenting the analysis results notified to the user terminal to a doctor at the time of consultation.

Images