Smart mirror system using sensor and artificial intelligence

Abstract

A smart mirror system utilizing sensors and artificial intelligence is disclosed. According to one embodiment of the present invention, the smart mirror system utilizing sensors and artificial intelligence comprises: a mirror unit that functions as a mirror reflecting an image of the front and includes a function as a display for displaying guidance information and images; a sensor unit that includes a camera sensor, a laser sensor, and a LiDAR sensor, and detects a user's biosignals and abnormal movement in a non-contact manner; a control unit that controls the smart mirror system; and a communication unit for transmitting signals to the outside. In the sensor unit, the camera sensor recognizes changes in the user's face, the laser sensor irradiates a laser onto the user's chest to detect micro-vibrations corresponding to breathing and heartbeats, and the LiDAR sensor generates point cloud data to scan the user's posture and walking pattern. The control unit is characterized by including a daily assistance unit for assisting the user's daily life; a digital care unit that provides digital treatment for the user; and an emergency notification unit that provides an emergency notification when an emergency situation occurs for the user.

Description

Technology Field

[0001] The present invention relates to a smart mirror system utilizing sensors and artificial intelligence, and more specifically, to a smart mirror system that integrates non-contact biosignal measurement, facial and voice recognition, provision of customized digital therapeutics, and real-time emergency situation monitoring through a mirror placed in a user's daily living space. Background Technology

[0002] As the population ages rapidly in modern society, the number of elderly people living alone is increasing sharply. Accordingly, various healthcare devices and smart home technologies have been developed to monitor the health of the elderly in their daily lives and respond quickly to emergencies.

[0003] Conventional healthcare monitoring systems primarily rely on wearable devices in the form of smartwatches or patches. However, there are limitations in the continuity of data collection as the elderly tend to forget to wear the devices or find them cumbersome.

[0004] Furthermore, camera-based motion detection systems are difficult to install in private spaces such as bathrooms or bedrooms due to privacy concerns, and face limitations in seamlessly detecting emergency situations such as falls.

[0005] Furthermore, there is still a lack of emotional care systems that go beyond simple physical health checks to alleviate depression, cognitive decline, and emotional isolation in the elderly within their daily lives and provide therapeutic interventions.

[0006] Therefore, there is an urgent need for the development of an integrated smart mirror system that can monitor physical and emotional states in an active and non-contact manner within everyday environments (e.g., mirrors) without users wearing separate devices, and automatically respond in emergency situations.

[0007] The technology forming the background of the present invention is described in Korean Published Patent Application No. 10-2021-0063188 (June 1, 2021). Prior art literature

[0008] (Patent Document 0001) KR 10-2021-0063188 A The problem to be solved

[0009] The present invention aims to solve the problems of the aforementioned prior art, and the first problem that the present invention aims to solve is to provide a smart mirror system that can measure biosignals (pulse, respiration, heart rate variability, etc.) non-contactually through a sensor equipped in the smart mirror system without the user wearing a separate wearable device, and detect falls and abnormal movements in real time.

[0010] The second objective of this institute is to analyze users' emotions and language to detect early warning signs of depression and cognitive decline, and to implement practical digital therapeutic (DTx) interventions through personalized reminiscence content and conversational agents.

[0011] The third objective that this institute aims to address is to process video data on-device to minimize privacy infringement controversies, and to perform precise, malfunction-free notifications and smart home control by fusing multi-sensor data in the event of an emergency.

[0012] However, the technical problems that the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem

[0013] As a technical means for achieving the above-mentioned technical problem, a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention comprises: a mirror unit that functions as a mirror reflecting an image of the front and includes a function as a display for displaying guidance information and images; a sensor unit that includes at least one of a camera sensor, a laser sensor, and a LiDAR sensor and detects a user's biosignals and abnormal movement in a non-contact manner; a control unit that controls the smart mirror system; and a communication unit for transmitting signals to the outside, wherein in the sensor unit, the camera sensor recognizes changes in the user's face, the laser sensor irradiates a laser onto the user's chest to detect micro-vibrations corresponding to breathing and heartbeats, and the LiDAR sensor generates point cloud data to scan the user's posture and walking pattern; and the control unit comprises a living assistance unit for assisting the user's daily life; a digital care unit for providing digital treatment to the user; and an emergency notification unit for providing an emergency notification when an emergency situation occurs to the user.

[0014] Additionally, the digital care unit may include a biosignal analysis unit that extracts cardiovascular information by capturing changes in the user's skin color recognized by the camera sensor, and analyzes the user's biosignals by analyzing changes in light reflectance of the user's face surface by the lens of the camera sensor to measure heart rate, respiratory rate, oxygen saturation, and blood pressure in real time in a non-contact manner.

[0015] Additionally, the digital care unit may further include a behavior pattern detection unit that detects the user's behavior pattern using major joint points of the user's body tracked in real time through the camera sensor; and an emotion and language analysis unit that determines the user's emotions visually by analyzing the user's facial movements and determines the user's emotions audibly by analyzing the user's voice data.

[0016] In addition, the digital care unit may further include a treatment intervention unit that performs digital treatment for the user based on the judgment results of the biosignal analysis unit, the behavior pattern detection unit, and the emotion and language analysis unit.

[0017] Additionally, the therapeutic intervention unit may include a vector database storing data on major past life events and preferences of the user; and an intelligent conversation unit that designs an intelligent conversation by understanding the user's intentions, wherein the intelligent conversation unit may further include a personalized RAG (Retrieval-Augmented Generation) module that prevents hallucination by generating a response by searching for relevant data in the vector database based on the user's utterance context; and an agent AI module that performs at least one of sentiment analysis, recall strategy, and information exchange based on the user's state obtained from the sensor unit.

[0018] Additionally, the therapeutic intervention unit further includes a past recall unit, and the past recall unit further includes an I2V (Image-to-Video) generation module that converts the user's past photos into video; and a lip-sync module that synthesizes the restored voice of an acquaintance in real time to match the lip shape based on information stored in the vector database, and the past recall unit can provide interactive reminiscence therapy content through the mirror unit according to a P-TTS-based past recall program.

[0019] Additionally, the above-mentioned life assistance unit may further include a medication guidance unit that outputs an image of a medicine bag and voice guidance at a set medication time and records whether the medication has been taken based on the user's voice response; and a biosignal notification unit that notifies the user's guardian if there are abnormal signs based on the pulse, respiratory rate, and stress index measured while the user is standing in front of the mirror unit.

[0020] In addition, the smart mirror system may operate using an on-device AI method that, for the protection of the user's privacy, does not transmit collected video data to an external cloud but instead directly analyzes it within a computing device inside the smart mirror system and transmits only the quantified result value.

[0021] Additionally, the emergency notification unit may further include an emergency situation determination unit that determines whether the user is in an emergency situation; an indoor warning unit that notifies the area around the user's residence that the user is in an emergency situation if the user is in an emergency situation; and an emergency center connection unit that notifies an emergency center of the emergency situation if the user is in an emergency situation.

[0022] In addition, the above emergency notification unit further includes a smart home control unit that controls the smart home system of the user's residence upon determining an emergency situation, and the smart home control unit may be linked with the smart door lock of the user's residence to automatically open the front door in the event of an emergency.

[0023] The means for solving the problem described above are merely exemplary and should not be interpreted as intended to limit the present invention. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and the detailed description of the invention. Effects of the invention

[0024] According to the means for solving the problem of the present invention described above, a smart mirror system can be provided that measures various biosignals non-contactually through sensors equipped in the smart mirror system without wearing a wearable device, detects falls and abnormal movements in real time, and provides emergency notifications quickly in emergency situations for the user.

[0025] In addition, according to the solution provided by the present invention, a smart mirror system capable of effectively performing digital therapeutic (DTx) interventions through personalized past recollection content and interactive agents can be provided.

[0026] In addition, according to the solution to the problem of the present invention, a smart mirror system can be provided that protects the user's sensitive information by processing video data on-device and facilitates the user's emergency rescue operations through smart home control in the event of an emergency.

[0027] However, the effects obtainable from this invention are not limited to those described above, and other effects may exist. Brief explanation of the drawing

[0028] FIG. 1 is a diagram showing the schematic configuration of a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention. FIG. 2 is a control configuration diagram showing the internal module division structure of the control unit in a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention. FIG. 3 is a control configuration diagram showing the configuration of a living assistance unit inside a control unit in more detail in a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention. FIG. 4 is a control configuration diagram showing the configuration of a digital care unit inside a control unit in more detail in a smart mirror system utilizing a sensor and artificial intelligence according to one embodiment of the present invention. FIG. 5 is a control configuration diagram showing the configuration of an emergency notification unit inside a control unit in more detail in a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention. Specific details for implementing the invention

[0029] Embodiments of the present invention are described below with reference to the attached drawings to enable those skilled in the art to easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0030] Throughout this specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" or "indirectly connected" with other elements interposed between them.

[0031] Throughout the entire specification, when a component is described as being located "on," "on top," "on top," "under," "on bottom," or "on bottom" of another component, this includes not only cases where the component is in contact with the other component but also cases where another component exists between the two components.

[0032] Throughout this specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0033] The present invention relates to a smart mirror system utilizing sensors and artificial intelligence. FIG. 1 is a diagram showing the schematic configuration of a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention. FIG. 2 is a control configuration diagram showing the internal module division structure of a control unit in a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention.

[0034] Referring to FIG. 1 of the present invention, a smart mirror system (1000) utilizing sensors and artificial intelligence according to one embodiment of the present invention may include: a mirror unit (100) which acts as a mirror that reflects an image of the front and has a function as a display that displays guidance information and an image; a sensor unit (300) which includes at least one of a camera sensor (320), a laser sensor (330), and a LiDAR sensor (340) and detects a user's biosignal and movement abnormalities in a non-contact manner; a control unit (200) that controls the smart mirror system (1000); and a communication unit (500) for transmitting a signal to the outside.

[0035] In the sensor unit (300) above, the camera sensor (320) recognizes changes in the user's face, the laser sensor (330) irradiates a laser onto the user's chest to detect micro-vibrations according to breathing and heartbeats, and the lidar sensor (340) can generate point cloud data to scan the user's posture and walking pattern.

[0036] Additionally, referring to FIG. 2, the control unit (200) may include a living assistance unit (210) for assisting the daily life of the user; a digital care unit (220) for providing digital treatment to the user; and an emergency notification unit (230) for providing an emergency notification when an emergency situation occurs to the user.

[0037] Referring again to FIG. 1, the mirror portion (100) acts as a mirror that reflects an image of the front, and can have the shape of a mirror that a user (e.g., an elderly person) frequently looks at to check their own appearance.

[0038] For example, in order to determine the user's condition by performing facial recognition or iris recognition using a camera sensor, the user's eyes need to look at the system. By providing a mirror unit (100) in this way, the user can frequently look at it to check their appearance, and thereby the smart mirror system can operate effectively.

[0039] Meanwhile, the mirror unit (100) can also function as a display unit. For example, a display may be placed behind the reflective mirror surface to provide visual information through the reflective mirror surface. By providing a mirror unit (100) with such a display function, the user can make video calls or watch videos through a mirror familiar to the user, thereby facilitating the user's use of the system. Meanwhile, a switch (110) capable of turning the display function ON / OFF may be installed at the bottom of the mirror unit (100). The switch (110) may be in the form of an input panel.

[0040] In one embodiment of the present invention, the control unit (200) may serve to control the smart mirror system (1000) as a whole. For example, the control unit (200) may include a CPU, a GPU, and RAM.

[0041] A sensor unit (300) according to one embodiment of the present invention may include a camera sensor (320), a laser sensor (330), and a LiDAR sensor (340), and may additionally include a gas (fire) detection sensor (310).

[0042] The gas (fire) detection sensor (310) may be intended to detect a fire occurring inside the user's residence. Alternatively, the gas (fire) detection sensor (310) may be intended to detect a gas leak. With such a configuration of the gas detection sensor (310), the user may be warned of exposure to danger and may also be notified of an emergency situation to the outside.

[0043] Additionally, the camera sensor (320) can perform the role of recognizing the user's face or iris. The camera sensor (320) may be, for example, a multimodal camera sensor that integrates various types of information. With such a multimodal camera sensor, a normal color image (multimodal visual information) can be captured during the day, and the iris can be recognized through infrared pixels when authentication is required.

[0044] In addition, through iris recognition by the camera sensor (320), it is possible to check the user's hypertension and vascular health, diabetic complications, liver function and jaundice, cholesterol levels, nervous system and stress (using eye tracking function), and through facial recognition by the camera sensor, it is possible to estimate the user's heart rate, respiratory rate, and blood pressure, detect stroke and nervous system abnormalities, analyze nutritional and dehydration status, and monitor pain and emotions. Furthermore, the camera sensor (320) according to the embodiment of the present invention can switch the software filter of a single lens in a time-division manner to alternate between iris recognition mode and expression / motion detection mode.

[0045] In addition, the laser sensor (330) may enable the detection of micro-vibrations (LPD: Laser Photo-Detector) and the analysis of vascular elasticity and pulse waves. Specifically, a laser can be directed at the user's chest to measure minute chest movements caused by breathing and heartbeats. By using the laser sensor (330), measurements can be taken even while wearing clothes, and there is an advantage in accurately calculating heart rate, respiratory rate, and heart rate variability (HRV) during the short time spent standing in front of the smart mirror.

[0046] In addition, point cloud data can be generated through a LiDAR sensor (340) to scan the user's posture and walking pattern, and, for example, changes in the user's height can be scanned in real time to detect falls. Furthermore, since point cloud data is used, there is an advantage in that fall accidents can be detected without infringing on privacy even when the user is undressed (e.g., in a bathroom). Additionally, by scanning the user's walking pattern with the LiDAR sensor (340), the walking speed as the user approaches the mirror, irregularity of stride length, and body tilt can be scanned to detect early changes in movement caused by sarcopenia or Parkinson's disease.

[0047] Meanwhile, a smart mirror system (1000) according to one embodiment of the present invention may further include a data storage unit (400). This data storage unit (400) can store various data such as the user's personal information and past photos.

[0048] Additionally, a smart mirror system (1000) according to one embodiment of the present invention may further include a communication unit (500). The communication unit (500) may support sending an emergency notification to the outside when an emergency notification is required due to an emergency situation of the user.

[0049] The communication unit (500) can select the optimal communication method according to the nature of the data. For example, when an emergency occurs to a user, MQTT (Message Queuing Telemetry Transport), WebSockets, RESTful API (HTTPS), etc. can be used as a protocol to connect the emergency situation data detected by the smart mirror system to an emergency center.

[0050] MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol that operates even at low bandwidths and is optimized for continuously transmitting smart mirror 'status information (Heartbeat)' or minute 'heart rate variability values' to the control server. WebSockets supports bidirectional real-time communication between the server and the mirror and can be utilized as a streaming channel for controllers to remotely deliver voice guidance via the mirror's speaker in emergency situations, or for guardians to view real-time video. RESTful API (HTTPS) can be used as a standard for transmitting 'incident reports (JSON format)' immediately after an incident occurs, or for sending static data, such as user medication records stored in the RAG system, to the control system.

[0051] In addition, data standard protocols used for the compatibility of medical and care information must comply with international standard specifications, such as HL7 FHIR (Fast Healthcare Interoperability Resources) and Matter standards.

[0052] HL7 FHIR is a global standard for medical information exchange, and when iris data, heart rate, fall history, etc. detected by smart mirror systems are converted into FHIR resource format and transmitted, they have the advantage of being immediately usable as patient history in hospitals or emergency medical systems without separate conversion. The Matter standard is a modern standard that ensures interoperability between smart home devices, and has the advantage of enabling fire detectors, smart door locks, and smart mirror systems to simultaneously execute 'emergency mode' scenarios (door opening, lighting turning on, etc.) regardless of the manufacturer.

[0053] Meanwhile, data packets transmitted to the control system must include key information necessary for the structure, such as location information (GPS / LBS), severity level, and multimodal evidence.

[0054] In addition, to protect sensitive information, the highest level of security must be maintained for users' biometric and image information. Protocols for this purpose may include OAuth 2.0 / OpenID Connect, TLS 1.3 encryption, and on-device metadata processing.

[0055] OAuth 2.0 / OpenID Connect is a procedure that authenticates whether the control system has legitimate authority when accessing data from the smart mirror, and prevents interception in transit by encrypting data throughout the entire journey from the mirror to the control server using TLS 1.3 encryption. Additionally, on-device metadata processing transmits only the text "fall occurred" and extracted biometric values ​​instead of the original video, and adopts a method where the actual video is streamed temporarily only when explicitly approved by a guardian or control operator.

[0056] Meanwhile, as means for the effective interoperability of such communication methods, edge-cloud collaboration models, public API integration, and log integrity may be used.

[0057] For example, through an edge-cloud collaboration model, an initial decision can be made within the mirror (Edge), and a detailed analysis can be requested from the cloud control server only when there is uncertainty. Additionally, through the integration of public APIs, a system can be established to immediately transmit a user's medical history information to rescue workers. Furthermore, if all emergency call attempts and response processes are recorded on a blockchain-based distributed ledger, they can serve as reliable data for post-mortem analysis and determining liability.

[0058] Referring again to FIG. 1, a smart mirror system (1000) utilizing sensors and artificial intelligence according to one embodiment of the present invention may include speakers (600A, 600B) and microphones (700A, 700B). Due to the configuration of these speakers and microphones, notifications (e.g., medication reminders) can be provided to the user, and voice sentiment analysis and / or language analysis can be performed by recognizing the user's voice.

[0059] Additionally, referring to FIG. 1, a smart mirror system (1000) utilizing a sensor and artificial intelligence according to one embodiment of the present invention may further include a frame (800), and such a frame (800) may serve to stably support the entire system. Additionally, the frame (800) may be equipped with an LED light, thereby enabling accurate detection of the user's condition even when the room is dark.

[0060] Next, we will examine the configuration of the control unit (200) of the present invention in more detail with reference to FIGS. 3 to 5.

[0061] FIG. 3 is a control configuration diagram showing the configuration of a living assistance unit inside a control unit in more detail in a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention.

[0062] Referring to FIG. 3, the life assistance unit (210) included in the control unit (200) may include a voice recognition control unit (211). For example, for users who are not good at operating devices, it may recognize natural language commands such as "What is the weather like today?" and provide information necessary for daily life.

[0063] Additionally, the living assistance unit (210) may include an AI avatar generation unit (212). For example, the mirror unit (100) may generate an AI avatar to perform the role of a conversational companion for the user. The AI ​​avatar may provide emotional support, such as answering the user's questions or asking about their well-being first.

[0064] Additionally, the living assistance unit (210) may include a video call control unit (213). For example, the video call control unit (213) can control the user (elderly person) to make a real-time video call with a guardian or medical staff through the mirror unit (100) via a simple command such as “connect video call.”

[0065] Additionally, the living aid unit (210) may include a medication notification unit (214). The medication notification unit (214) can control the notification to output a voice guidance and a phrase such as "It is time to take your blood pressure medication" along with an image of a medicine bag to the mirror unit (100) at a set time. Additionally, the medication notification unit (214) can record whether the medication was taken based on the user's voice response to the voice guidance.

[0066] Additionally, the living assistance unit (210) may include a biosignal notification unit (215). The biosignal notification unit (215) may notify the guardian of the user (elderly person) if there are abnormal signs based on the pulse, respiratory rate, and stress index measured while the user is standing in front of the mirror unit. In this way, information can be provided so that the guardian can take appropriate measures in advance.

[0067] Additionally, the living aid unit (210) may include a rehabilitation guide unit (216). For example, the rehabilitation guide unit (216) may allow the user to follow gymnastics or rehabilitation movements while watching a guide video in the mirror unit (100), and provide real-time feedback such as “raise your arm a little higher” through the speakers (600A, 600B).

[0068] Additionally, the living assistance unit (210) may include a living pattern notification unit (217). For example, the living pattern notification unit (217) may first speak to the user through the speaker (600A, 600B) to check the status if the user does not appear in front of the mirror for a long time or if a pattern different from usual is detected, and if there is no response from the user, it may determine that it is a dangerous situation and notify the guardian.

[0069] Additionally, the living assistance unit (210) may include a checklist notification unit (218). For example, the checklist notification unit (218) can serve to complement cognitive functions by providing voice guidance such as “Did you turn off the gas?” or “It is going to rain today, so please take an umbrella” when the user looks at the mirror unit (mirror) and prepares to go out.

[0070] Additionally, the living aid unit (210) may include an audiovisual aid unit (219). For example, the audiovisual aid unit (219) may output time or major announcements to the mirror unit (100) in large, clear fonts and high-contrast colors for users with low vision, and may output clear voice that can be heard better by adjusting the frequency band of the speakers (600A, 600B).

[0071] In this way, the living assistance unit (210) performs the function of assisting the user's life and can provide convenience functions that allow the user to conveniently lead their daily life in various aspects.

[0072] Additionally, according to the living assistance unit (210) of the embodiment of the present invention, the smart mirror system can also perform the following functions.

[0073] It analyzes the user's body type, personal color, and wardrobe, as well as the day's weather and schedule, to advise on optimal styling just like a personal stylist (Customized Styling Coordinator (Fashion & Lifestyle)).

[0074] In addition, by detecting subtle changes in the user through high-resolution cameras and AI vision technology, it can enhance practicality by adjusting makeup details before going out or notifying the user when periodic care is needed (Ultra-close Beauty & Detail Advisor).

[0075] FIG. 4 is a control configuration diagram showing the configuration of a digital care unit inside a control unit in more detail in a smart mirror system utilizing a sensor and artificial intelligence according to one embodiment of the present invention.

[0076] Referring to FIG. 4, the digital care unit (220) may include a biosignal analysis unit (221) that extracts cardiovascular information by capturing changes in the user's skin color recognized by a camera sensor, and analyzes the user's biosignals by analyzing changes in light reflectance of the user's face surface by the lens of the camera sensor to measure heart rate, respiratory rate, oxygen saturation, and blood pressure in real time in a non-contact manner.

[0077] Specifically, the camera sensor can capture subtle changes in the user's skin color through a high-resolution lens and extract cardiovascular information from it. Additionally, the camera lens analyzes changes in light reflectivity on the face surface to measure heart rate, respiratory rate, oxygen saturation, and blood pressure in real-time using a non-contact method. This technology is called rPPG (remote photoplethysmography). This may be a suitable measurement method for users who find it cumbersome to wear wearable devices such as smartwatches (e.g., the elderly).

[0078] In addition, the measured heart rate data can be used to determine the state of the autonomic nervous system, from which the user's stress index or fatigue level can be calculated.

[0079] The biosignal analysis unit (221) can analyze the user's current physical condition based on the biosignal by analyzing the user's biosignal measured by the sensor unit in this way.

[0080] The digital care unit (220) may further include a behavior pattern detection unit (222) that detects the user's behavior patterns using major joint points of the user's body tracked in real time through a camera sensor. By this behavior pattern detection unit (222), abnormalities in the user's movement can be determined.

[0081] Meanwhile, the determination of abnormal movement by the behavior pattern detection unit (222) can be performed by skeleton tracking. According to skeleton tracking, the AI ​​tracks the major joint points of the user's body in real time, thereby enabling immediate identification of falls, sudden staggering, or a 'stationary state' where there is no movement for a long time.

[0082] In addition, the behavior pattern detection unit (222) can determine whether there is a decrease in physical strength or signs of musculoskeletal disorders such as Parkinson's disease by converting the user's normal walking speed or agility of sitting and standing movements into data.

[0083] Additionally, the digital care unit (220) may further include an emotion and language analysis unit (223) that analyzes the user's facial movements to visually determine the user's emotions and analyzes the user's voice data to auditorily determine the user's emotions. Through the emotion and language analysis unit (223), the user's (elderly person's) psychological state can be analyzed by synthesizing the user's facial expressions, tone of voice, and conversation content.

[0084] For example, the emotion and language analysis unit (223) can identify emotions such as joy, sadness, and anger by analyzing the fine movements of the user's facial muscles, and can quantify and determine feelings of depression or helplessness, which are particularly important for the elderly.

[0085] Additionally, the emotion and language analysis unit (223) can detect the degree of psychological anxiety through the trembling, speed, and intensity of the user's voice. Furthermore, through natural language analysis, if the user uses negative keywords such as "lonely" or "sick" during a conversation, the frequency can be tracked to determine the state of emotional isolation.

[0086] Additionally, when the user looks at the mirror unit, if the user's facial expression appears dark or the stress level appears high, the emotion and language analysis unit (223) can analyze the user's emotions and change the color temperature of the lighting to warm through the smart home system, and, under the control of the treatment intervention unit (224) to be explained later, can attempt to have a conversation that helps with emotional stability through the speaker of the smart mirror system, such as, "You must have had a very tiring day today, right? Shall I play some music you like while you have a warm cup of tea?" (Health & Safety monitoring and psychological / safety care).

[0087] Meanwhile, the digital care unit (220) may further include a treatment intervention unit (224) that performs digital treatment for the user based on the judgment results of the biosignal analysis unit (221), the behavioral pattern detection unit (222), and the emotion and language analysis unit (223).

[0088] The treatment intervention unit (224) may include a vector database (224A) that stores data on major past life events and preferences of the user; and an intelligent conversation unit (224B) that designs an intelligent conversation by understanding the user's intentions.

[0089] The vector database (224A) may be a database in which the user's past diary, family relationship diagram, preferences, and major life events are stored in advance. Additionally, the vector database may store the user's past video information or voice information (for example, video information or voice information of the user's acquaintances). Furthermore, the vector database may also store the user's past information collected by the sensor unit. Due to the configuration of the vector database (224A) in this way, when a question is entered, the relevant data can be searched in the vector database (224A) to generate an answer in the intelligent conversation unit (224B).

[0090] The intelligent conversation unit (224B) may further include a personalized RAG (Retrieval-Augmented Generation) module (224B-1) that prevents hallucination by generating an answer by searching for relevant data in a vector database (224A) based on the user's speech context, and an agent AI module (224B-2) that performs at least one of sentiment analysis, recall strategy, and information exchange based on the user's state obtained from the sensor unit.

[0091] The RAG module (224B-1) can prevent hallucination and generate personalized answers by searching for relevant data in the vector database (224A) when a question is entered by the user and generating answers based on the user's past diary, family tree, preferences, and major life events. Additionally, the vector database (224A) may additionally store specialized medical knowledge, thereby providing reliable answers by combining the user's personal context with the 'specialized medical knowledge base (Ontology).'

[0092] Additionally, the agent AI module (224B-2) can perform at least one of emotional analysis, recall strategy, and information exchange based on the user's state obtained from the sensor unit. Specifically, the agent AI module (224B-2) may include an emotional analysis agent that detects depression or anxiety from the user's voice tone and word choice, a recall strategy agent that determines the direction of conversation, such as whether to recall "happy memories from the past" or resolve "current discomfort," depending on the detected emotional state, and an information exchange agent that accurately searches for and delivers health information or community information when the user needs it.

[0093] The therapeutic intervention unit (224) may further include a past recollection unit (224C). The past recollection unit (224C) may further include an I2V (Image-to-Video) generation module (224C-1) that converts the user's past photos into video; and a lip-sync module (224C-2) that synthesizes the restored voice of an acquaintance in real-time to match the lip shape based on information stored in a vector database (224A). The past recollection unit (224C) may provide interactive recollection therapy content through the mirror unit (100) according to a past recollection program based on P-TTS (Personalized-Text To Speech).

[0094] Looking more specifically at P-TTS-based reminiscence programs, for example, they can be trained to learn the voices of a deceased spouse, a child living far away, or the elderly person themselves from their younger days to enable recall of the past through voice replication and restoration. Furthermore, by delivering questions such as "Do you remember that time?" in a familiar voice using the replicated voice, they can simultaneously provide cognitive stimulation and psychological stability (reminiscence therapy). In addition, during TTS generation, instead of simple recitation, emotional parameters such as joy and calmness can be adjusted in real-time according to the conversation content to enhance delivery (variable emotion projection).

[0095] In addition, the I2V generation module (224C-1) that converts the user's past photos into video can use a high-performance model, such as Runway Gen-3, to convert the user's past photos into naturally moving video, and to generate actions such as the person in the photo waving their hand or smiling, going beyond simply blinking eyes, thereby creating a vivid digital twin.

[0096] In addition, in the lip-sync module (224C-2), when applying P-TTS voice to the generated video, the mouth shape can be precisely matched using the SadTalker or HeyGen API, etc., to create an interactive experience as if a person from the past were speaking to the mirror.

[0097] According to these intelligent conversation unit (224B) and past recollection unit (224C), in response to a user's speech or a system's preemptive question, the context of the conversation is determined, and related photos and episodes are extracted as RAG to generate video (Runway) and voice (P-TTS), and by merging them with a lip-sync module, a natural conversation can be output through the mirror unit.

[0098] In addition, user reactions occurring during the conversation can be used as input values ​​to intervene in digital therapy through an agent AI according to the user's current state. Specifically, a feedback loop logic based on the user's reactions (facial expressions, voice) can be applied, and if the user makes a sad face, the agent can say comforting words or change the mood of the content. Through this, a significant effect can be achieved in that effective digital therapy can be performed by further enhancing interaction with the user.

[0099] Meanwhile, the smart mirror system in the embodiment of the present invention may operate using an on-device AI method to protect user privacy, in which collected video data is not transmitted to an external cloud but is directly analyzed by a computing device within the smart mirror system and only numerical result values ​​are transmitted. For example, to further enhance the security of privacy protection, the on-device AI method may involve processing data within a 'Trusted Execution Environment (TEE)' or transmitting only numerical information after 'data de-identification processing.'

[0100] FIG. 5 is a control configuration diagram showing the configuration of an emergency notification unit inside a control unit in more detail in a smart mirror system utilizing sensors and artificial intelligence according to one embodiment of the present invention.

[0101] An emergency notification unit (230) according to an embodiment of the present invention may further include: an emergency situation determination unit (231) for determining whether the user is in an emergency situation; an indoor warning unit (232) for notifying the area around the user's residence that the user is in an emergency situation when the user is in an emergency situation; and an emergency center connection unit (233) for notifying an emergency center of the emergency situation when the user is in an emergency situation.

[0102] For example, the emergency situation determination unit (231) can determine that the user is in an emergency situation by analyzing information such as the user's facial recognition, iris recognition information, heart rate, respiratory rate, heart rate variability (HRV), fall detection, or gait pattern detection from the sensors of the sensor unit. Additionally, the emergency situation determination unit (231) may determine that the user is in an emergency situation if a dangerous situation, such as a fire, occurs through a gas detection sensor.

[0103] Additionally, the indoor warning unit (232) can provide a warning to the surroundings through the speaker of the smart mirror system or through the smart home system.

[0104] Additionally, the emergency center connection unit (233) can be controlled to communicate with the emergency center through the communication unit (500) to notify the emergency situation when an emergency occurs to the user.

[0105] Additionally, the emergency notification unit (230) may further include a location information determination unit (234) for accurately determining the user's location and a sensitive information protection unit (235) for maintaining security regarding the user's biometric and image information when communicating with an emergency center or the outside.

[0106] The location information determination unit (234) can specifically determine the user's location information through indoor positioning technology and provide information so that rescue personnel at the emergency center can determine the user's exact location in the event of an emergency. The sensitive information protection unit (235) can control communication in the communication unit (500) to prevent privacy infringement by ensuring thorough protection of personal information.

[0107] Additionally, the emergency notification unit (230) may further include a smart home control unit (236) that controls the smart home system of the user's residence when an emergency situation is determined.

[0108] The smart home control unit (236) above is linked with the smart door lock of the user's residence and can control the front door to open automatically in an emergency situation.

[0109] In this way, a significant effect can be achieved in enabling effective rescue operations by allowing rescue workers to easily enter the user's residence in the event that the user is in a very critical situation.

[0110] According to the embodiments of the present invention as described above, a smart mirror system utilizing sensors and artificial intelligence can be provided that measures various biosignals of a user through a non-contact sensor without wearing a wearable device, detects falls and abnormal movements in real time, and provides emergency notifications quickly in critical situations for the user.

[0111] In addition, according to an embodiment of the present invention, a significant effect can be achieved in providing personalized intelligent conversation and past reminiscence therapy based on the user's personal biometric information, behavioral patterns, emotions and language, and personal experiences stored in a database.

[0112] Furthermore, according to an embodiment of the present invention, a significant effect can be achieved in that personal information can be thoroughly protected by adopting an on-device AI method that analyzes video data directly on an internal computing device without sending it to an external cloud.

[0113] The smart mirror system according to the embodiment of the present invention also has the following effects. It eliminates user resistance through the most familiar medium in daily life (mirror), which is unconsciously encountered at least once a day by people of all ages and genders, and comprehensively improves the quality of life, ranging from safety to beauty. By simultaneously providing enjoyable daily experiences of beauty and self-expression (coordi / beauty) alongside the potentially heavy objectives of treatment and surveillance (emergency / falls), it reduces resistance to the technology and doubles the enjoyment and necessity. Users naturally gain psychological comfort in the process of pleasantly interacting with the mirror every day, and experience truly human-centered technology that guarantees safety from unexpected emergency situations.

[0114] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical concept or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0115] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0116] 1000: Smart mirror system utilizing sensors and artificial intelligence 100: Mirror section 200: Control unit 210: Living Assistance Department 220: Digital Care Department 221: Biosignal Analysis Unit 222: Behavior pattern detection unit 223: Emotion and Language Analysis Department 224: Treatment Intervention Section 230: Emergency Notification Department 300: Sensor section 400: Data storage unit 500: Communications Department 600A, 600B: Speakers 700A, 700B: Microphone 800: Frame

Claims

Claim 1 A smart mirror system utilizing sensors and artificial intelligence comprises: a mirror unit that functions as a mirror reflecting an image of the front and includes a function as a display for displaying guidance information and images; a sensor unit that includes at least one of a camera sensor, a laser sensor, and a LiDAR sensor, and detects a user's biosignals and abnormal movement in a non-contact manner; a control unit that controls the smart mirror system; and a communication unit for transmitting signals to the outside, wherein in the sensor unit, the camera sensor recognizes changes in the user's face, the laser sensor irradiates a laser onto the user's chest to detect micro-vibrations corresponding to breathing and heartbeats, and the LiDAR sensor generates point cloud data to scan the user's posture and gait patterns; and the control unit includes a living assistance unit for assisting the user's daily life; and a digital care unit for providing digital treatment to the user. and includes an emergency notification unit that provides an emergency notification when an emergency situation occurs for the user, and the digital care unit includes a biosignal analysis unit that extracts cardiovascular information by capturing changes in the user's skin color recognized by the camera sensor, and analyzes the user's biosignals by measuring heart rate, respiratory rate, oxygen saturation, and blood pressure in real-time in a non-contact manner by analyzing changes in light reflectance of the user's face surface by the lens of the camera sensor, and the digital care unit includes a behavior pattern detection unit that detects the user's behavior patterns using major joint points of the user's body tracked in real-time through the camera sensor; and an emotion and language analysis unit that determines the user's emotions visually by analyzing the user's facial movements and determines the user's emotions auditorily by analyzing the user's voice data.A smart mirror system further comprising a therapeutic intervention unit that performs digital treatment of the user based on the judgment results of the biosignal analysis unit, the behavioral pattern detection unit, and the emotion and language analysis unit, wherein the therapeutic intervention unit includes: a vector database that stores data on major past life events and preferences of the user; and an intelligent conversation unit that designs an intelligent conversation by understanding the user's intention, wherein the intelligent conversation unit further comprises: a personalized RAG (Retrieval-Augmented Generation) module that prevents hallucination by generating an answer by searching for relevant data in the vector database based on the user's utterance context; and an agent AI module that performs at least one of emotion analysis, recall strategy, and information exchange based on the user's state obtained from the sensor unit. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 A smart mirror system according to claim 1, wherein the therapeutic intervention unit further includes a past recall unit, and the past recall unit further includes an I2V (Image-to-Video) generation module that converts the user's past photos into video; and a lip-sync module that synthesizes the restored voice of an acquaintance in real-time to match the lip shape based on information stored in the vector database, and wherein the past recall unit provides interactive reminiscence therapy content through the mirror unit according to a P-TTS-based past recall program. Claim 7 A smart mirror system according to claim 1, wherein the living assistance unit further comprises: a medication guidance unit that outputs an image of a medicine bag and voice guidance at a set medication time and records whether the medication has been taken based on the user's voice response; and a biosignal notification unit that notifies the user's guardian of any abnormal signs based on the pulse, respiratory rate, and stress index measured while the user is standing in front of the mirror unit. Claim 8 A smart mirror system characterized by operating in an on-device AI manner, wherein, in any one of claims 1 and 6 to 7, collected video data is not transmitted to an external cloud for the protection of the user's privacy, but is directly analyzed by a computing device inside the smart mirror system and only the quantified result is transmitted. Claim 9 A smart mirror system according to any one of claims 1 and 6 to 7, wherein the emergency notification unit further comprises: an emergency situation determination unit that determines whether the user is in an emergency situation; an indoor warning unit that notifies the area around the user's residence that the user is in an emergency situation when the user is in an emergency situation; and an emergency center connection unit that notifies an emergency center of the emergency situation when the user is in an emergency situation. Claim 10 In claim 9, the emergency notification unit further includes a smart home control unit that controls the smart home system of the user's residence upon determining an emergency situation, and the smart home control unit is linked with the smart door lock of the user's residence to automatically open the front door in the event of an emergency.

Images