System

Abstract

A system is provided.SOLUTION: A smart mirror to be installed in a house, the smart mirror comprising a camera for capturing an image of a facial expression of a family member, the smart mirror comprising: means for temporarily storing facial expression data captured by the camera; means for transmitting the facial expression data to a cloud server; means for analyzing, by the cloud server, the received facial expression data and identifying an emotional state of the family member; and means for analyzing, by the cloud server, a communication pattern between a plurality of family members, A system comprising: a unit configured to detect occurrence of a conflict; a unit configured to propose a message or an activity for promoting relationship repair based on past behavior data of a family member when the occurrence of the conflict is detected; and a unit including a display configured to display the message or the activity.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern families, conflicts and discord are more likely to occur between family members due to lack of communication and misunderstandings. In particular, busy lifestyles and the widespread use of digital devices have led to less dialogue between family members and fewer opportunities to understand each other's feelings. As a result, conflicts can drag on and relationships within the family can deteriorate. Traditional methods lack effective means to resolve these issues, and there is a need for a system that facilitates communication between family members and supports reconciliation after arguments. [Means for solving the problem]

[0005] The present invention provides a smart mirror to be installed in the home, which solves the aforementioned problems by the following means. The smart mirror has a camera that captures the facial expressions of family members and includes means for temporarily saving facial expression data captured by the camera. It also has means for transmitting the facial expression data to a cloud server. The cloud server has means for analyzing the received facial expression data and identifying the emotional states of the family members, and further has means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict. The smart mirror also has means for, when the occurrence of conflict is detected, suggesting messages or activities that promote relationship repair based on the family members' past behavioral data, and a display for displaying these suggestions, thereby facilitating communication between family members and supporting reconciliation.

[0006] A "smart mirror" is a mirror with a display that captures the facial expressions of family members and analyzes their communication status.

[0007] "Family members" refer to people who live together in the home, and are the subjects of emotion analysis and communication in this system.

[0008] The "camera" is a device that captures the facial expressions of family members and is built into the smart mirror.

[0009] "Facial expression data" refers to image or video data of the faces of family members, and this data is used for emotion analysis.

[0010] A "cloud server" is a server that can be accessed remotely via the Internet and analyzes facial expression data and generates suggestions.

[0011] "Emotional state" refers to the emotions of family members (e.g., happiness, anger, sadness, etc.) analyzed from facial expression data.

[0012] "Communication patterns" refer to records of conversations and behaviors between family members and the results of their analysis, and are used to determine the occurrence of conflicts.

[0013] "Conflict" means any disagreement or conflict between family members, which is detected by the system.

[0014] "Past behavioral data" refers to records of past behaviors and activities of family members, and is the basis for generating suggestions.

[0015] "Message" refers to a written or written notice presented to family members to facilitate relationship repair.

[0016] "Activities" refer to actions or events suggested to family members to promote reconciliation after a conflict.

[0017] "Display" refers to a display device that displays suggested messages and activities and is built into the smart mirror.

[0018] "Reaction" refers to a family member's reaction or behavior to a proposed message or activity. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after a fight. Specific embodiments of the system are described below.

[0041] System Configuration

[0042] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[0043] Program processing

[0044] Facial Capture

[0045] 1. Terminal

[0046] The camera built into the smart mirror captures the faces of family members at regular intervals (e.g., once per second). The captured facial expression data is temporarily stored locally.

[0047] 2. Terminal

[0048] Every certain time (for example, every minute), the captured facial expression data is sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[0049] facial expression analysis

[0050] 3. Server

[0051] The transmitted facial expression data is received and subjected to emotion analysis using AI. The analysis results indicate the emotional state of the family user (e.g., happiness, sadness, anger, etc.). This data is stored in a database.

[0052] Conflict Detection

[0053] 4. Server

[0054] The system analyzes communication patterns between family members from the accumulated facial expression data and behavior logs. When a specific pattern is detected, such as repeated angry expressions or prolonged silence, it recognizes the occurrence of a conflict.

[0055] 5. Server

[0056] When a conflict is detected, past behavioral data is used to generate suggestions to promote relationship repair, such as messages or activities like "Would you like to make dinner together?" or "Would you like to watch a movie together?"

[0057] Suggestion display

[0058] 6. Terminal

[0059] Suggested messages and activities from the cloud server are displayed on the screen, allowing the user to check them and take appropriate action.

[0060] Specific examples

[0061] Example 1: Resolving conflicts in the morning

[0062] Device: In the morning, the camera detects that Person A looks unhappy.

[0063] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[0064] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[0065] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[0066] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[0067] Example 2: Resolving conflicts during the night

[0068] Device: At night, the camera detects that B and C are arguing.

[0069] Server: Through analysis, it is determined that both B and C are feeling angry.

[0070] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[0071] Device: Show this suggestion in the mirror.

[0072] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[0073] In this way, the present invention is a system that monitors communication within the home and proposes reconciliation at appropriate times, thereby maintaining smooth relationships between family members.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[0077] Step 2:

[0078] Device: The captured facial image data is sent to the cloud server as a batch at regular intervals (e.g., every minute). The batch also includes a user identifier and a timestamp.

[0079] Step 3:

[0080] Server: The received facial image data is input into the AI ​​model to analyze the emotional state of the family user. The analysis results are stored in an emotional state database. Emotional states are classified as happiness, anger, sadness, etc.

[0081] Step 4:

[0082] Server: Analyzes communication patterns between multiple family members in real time based on data stored in the emotional state database. When a specific pattern is detected (e.g., continuous angry facial expressions or silence), it recognizes this as a sign of conflict.

[0083] Step 5:

[0084] Server: If signs of conflict are detected, AI refers to the family members' past behavioral data and generates messages and activity suggestions to promote relationship repair. For example, "Would you like to watch a movie together?"

[0085] Step 6:

[0086] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display. The suggestions are presented in an intuitive way for the user.

[0087] Step 7:

[0088] User: Review suggested messages and activities displayed on the mirror and act on them if they feel it is appropriate. For example, in response to the suggested message "Want to watch a movie together?", they act on the suggestion to watch a movie together.

[0089] Step 8:

[0090] Device: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[0091] Step 9:

[0092] Server: Analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[0093] Through these steps, the smart mirror monitors communication between family members and makes reconciliation suggestions at the appropriate time, thereby maintaining smooth relationships within the home.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Conflicts and arguments between family members are inevitable in domestic communication. However, if such conflicts are left unaddressed, family relationships are likely to deteriorate. In modern living environments, there is a need for a method to grasp family members' emotional states in real time and promote relationship repair at an appropriate time when conflicts occur. However, the lack of an automated system for detecting conflicts and proposing relationship repair makes it difficult to solve problems in individual households.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for analyzing the received facial expression data and identifying the emotional states of family members, means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, and means for suggesting messages or activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected. This makes it possible to quickly detect conflicts within the family and make suggestions to promote relationship repair at an appropriate time.

[0099] A "smart mirror" is a device that is installed in the home, equipped with a camera and a display, and used to capture and display the facial expressions and movements of family members.

[0100] The "photography device" is a camera built into the smart mirror, and is a means for capturing facial expression data of family members.

[0101] "Facial expression data" refers to data including facial features and expressions of family members captured by a photographing device.

[0102] A "cloud server" is a remote data center accessible via the Internet that provides analysis and storage functions for received data.

[0103] "Emotional state" refers to an individual's psychological state or mood (e.g., happiness, sadness, anger, etc.) analyzed based on facial expression data.

[0104] "Communication patterns" are data that indicate tendencies and regularities in the dialogue and behavior of family members.

[0105] "Conflict" is a state of discord caused by disputes or differences of opinion that arise between family members.

[0106] A "relationship repair message" is a message that includes suggestions or instructions for actions to promote the improvement of relationships between family members when the occurrence of a conflict is detected.

[0107] A "display device" is a display that is integrated into the smart mirror to visually present messages and activities to family members.

[0108] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates appropriate suggestions and responses to specific questions or requests.

[0109] A "prompt" is an instruction that is input into a generative AI model to generate an appropriate response or suggestion.

[0110] The present invention is a system that utilizes a smart mirror used at home to facilitate communication between family members and support the repair of relationships after conflicts or arguments. Specific embodiments of the system are described below.

[0111] System Configuration

[0112] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[0113] Hardware and software used

[0114] 1. Hardware

[0115] Smart mirror (with built-in camera and display)

[0116] Local Storage

[0117] 2. Software

[0118] Facial expression analysis AI (Google Face API)

[0119] Cloud infrastructure (AWS)

[0120] Database (MySQL)

[0121] Relationship repair suggestion AI model (OpenAI GPT-3)

[0122] Periodic job execution environment (Python)

[0123] Program processing

[0124] Facial Capture

[0125] When a user stands in front of the smart mirror, the device uses its built-in camera to capture a picture of the family user's face. The picture is taken at regular intervals (e.g., once per second), and the resulting facial expression data is temporarily stored in local storage. This facial expression data includes the user's identifier and a timestamp of when the picture was taken. Every regular interval (e.g., every minute), the device sends this facial expression data in batch format to a cloud server.

[0126] facial expression analysis

[0127] The server receives the facial expression data sent from the device. This data is securely received using cloud infrastructure such as AWS. The server performs emotion analysis on the received facial expression data using an AI model such as Google Face API to obtain the emotional state of each family user. The analysis results are stored in a database such as MySQL, and the data is organized by user and chronological order.

[0128] Conflict Detection

[0129] The server periodically analyzes the accumulated facial expression data and behavior logs in the database. This analysis uses Python to detect patterns of consecutive angry expressions or silence. If certain conditions are met (for example, five consecutive angry expressions), the server recognizes the conflict.

[0130] Proposal Generation and Display

[0131] When a conflict is detected, the server generates suggestions to promote relationship repair based on past behavioral data. These suggestions are generated using generative AI models such as OpenAI's GPT-3, and include specific activities (e.g., "Would you like to cook dinner together?" or "Would you like to watch a movie together?"). The generated suggestion message is sent to the device via the cloud, where it can be displayed on the device's display. The user can review the suggestion and take appropriate action.

[0132] Specific examples

[0133] Resolving conflicts in the morning

[0134] User: Person A stands in front of the smart mirror with a displeased look on his face.

[0135] Device: The camera captures A's facial expression, and the device sends the data to the cloud.

[0136] Server: Analyzes the received data using AI and detects that Person A is feeling stressed.

[0137] Server: Generates a suggestion, "Would you like to go for a morning walk together?" and sends it to the smart mirror.

[0138] Terminal: Show suggestions on the display.

[0139] User: Person A reviews the suggestion and goes for a walk with other family members.

[0140] Resolving conflicts during the night

[0141] User: The camera detects that B and C are arguing.

[0142] Device: The camera captures facial expression data and sends it to the cloud.

[0143] Server: Through data analysis, it detects that both B and C are angry.

[0144] Server: Generate a suggestion such as "Would you like to make a new recipe for dinner together?" and send it to the device.

[0145] Terminal: Show suggestions on the display.

[0146] User: B and C accept the suggestion and start making dinner together.

[0147] Example prompts for generative AI models

[0148] "Please suggest ways to improve communication between family members. Specifically, what actions should users take when they are feeling stressed?"

[0149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0150] Step 1:

[0151] The user stands in front of the smart mirror, and the smart mirror's built-in camera captures the user's face.

[0152] Input: Face image of a family member

[0153] Output: Photographed facial expression data

[0154] Step 2:

[0155] The terminal uses a camera to take pictures of the faces of family users at regular intervals (for example, once per second) and acquires facial expression data.

[0156] Specific operation: The captured facial expression data is temporarily stored in local storage. The data includes the user's identifier and a timestamp.

[0157] Input: Face image of a family member

[0158] Output: Facial expression data temporarily saved in local storage

[0159] Step 3:

[0160] The device sends the temporarily stored facial expression data to the cloud server in batch format at regular intervals (for example, every minute).

[0161] Specific operation: The facial expression data is assigned a user identifier and a timestamp and sent to the cloud server.

[0162] Input: Facial expression data stored in local storage

[0163] Output: Facial expression data sent to the cloud server

[0164] Step 4:

[0165] The server receives the facial expression data transmitted from the terminal.

[0166] Specific operation: Facial expression data is securely received using cloud infrastructure such as AWS.

[0167] Input: Facial expression data sent from the device

[0168] Output: Received facial expression data

[0169] Step 5:

[0170] The server uses an AI model (e.g., Google Face API) to analyze the emotion of the received facial expression data.

[0171] Specific operation: Analyzes facial expression data and obtains the emotional state of family users (e.g., happiness, sadness, anger, etc.).

[0172] Input: Received facial expression data

[0173] Output: Parsed emotional state data

[0174] Step 6:

[0175] The server stores the analysis results in a database (e.g., MySQL).

[0176] Specific operation: Analyzed emotional state data is stored in a database for each user and for each time series.

[0177] Input: Parsed emotional state data

[0178] Output: Emotional state data stored in a database

[0179] Step 7:

[0180] The server periodically analyzes the facial expression data and behavior logs stored in the database.

[0181] What it does: A periodic job using Python detects patterns of successive angry expressions and silences.

[0182] Input: Facial expression data stored in a database

[0183] Output: Detected communication patterns

[0184] Step 8:

[0185] The server recognizes the occurrence of a conflict when certain conditions are met (for example, five consecutive angry facial expressions).

[0186] Specific behavior: Determine whether a conflict has occurred based on detected communication patterns.

[0187] Input: Detected communication patterns

[0188] Output: Conflict occurrence determination

[0189] Step 9:

[0190] When conflict is detected, the server generates suggestions based on past behavioral data to promote relationship repair.

[0191] Specific actions: Use a generative AI model (e.g., OpenAI's GPT-3) to generate messages that involve specific activities (e.g., "Would you like to make dinner together?" or "Would you like to watch a movie together?").

[0192] Input: Conflict occurrence determination, past behavior data

[0193] Output: Proposal message

[0194] Step 10:

[0195] The server sends the generated proposal message to the terminal via the cloud.

[0196] Specific operation: A proposal message is sent to the device via the cloud infrastructure.

[0197] Input: Proposal message

[0198] Output: Proposal message sent to the terminal

[0199] Step 11:

[0200] The terminal displays the proposal message received from the cloud server on the display.

[0201] Specific behavior: Suggestions are displayed in large letters and icons for easy viewing by family users.

[0202] Input: The proposal message received

[0203] Output: Suggestion message shown on the display

[0204] Step 12:

[0205] The user checks the suggestion message and takes appropriate action. For example, if they want to cook together, the user starts a conversation in the kitchen.

[0206] Specific Action: Act on the suggested activities to improve communication.

[0207] Input: Suggestion message shown on the display

[0208] Output: Improved communication through appropriate actions

[0209] (Application example 1)

[0210] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] Conventional home communication support systems lack the ability to accurately detect emotions and conflicts among family members and propose appropriate solutions, which can lead to unnecessary friction and misunderstandings. Furthermore, the system's overall accuracy and effectiveness are limited because it is unable to effectively collect information on how family members respond to proposed activities and messages and reflect this information in future proposals.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0213] In this invention, the server includes a device that generates specific execution examples in response to suggestions displayed on a home communication terminal, a smart mirror installed in the home and having an imaging device that captures the facial expressions of family members and a device that temporarily stores the facial expression data captured by the imaging device, a device that transmits the facial expression data to a remote server, means for analyzing the facial expression data received by the remote server and identifying the emotional states of the family members, a device that analyzes communication patterns between multiple family members using the remote server and detects the occurrence of conflict, a device that generates suggestions and activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected, a system having a display device that displays the suggestions and activities, and a device that shares suggestion information with the home communication terminal and generates specific execution examples in response to the suggestions displayed on the home communication terminal.

[0214] This makes it possible to suggest specific activities and messages at the appropriate time when conflicts arise within the family, and to collect responses to these suggestions in order to improve the accuracy of future suggestions.

[0215] The "imaging device" is a device for capturing images of the facial expressions of family members.

[0216] The "remote server" is a server that analyzes data sent from smart mirrors installed in the home and identifies the emotional states of family members.

[0217] "Facial expression data" refers to data on the facial expressions of family members captured by an imaging device installed in the smart mirror.

[0218] "Communication patterns" refer to patterns of conversation and behavior between multiple family members, and are data that can be analyzed to detect the occurrence of conflicts.

[0219] "Conflict" refers to a fight or disagreement between family members and is important for analysis to detect this occurrence.

[0220] "Relationship repair" refers to the process of resolving conflicts and misunderstandings between family members and rebuilding positive relationships.

[0221] "Suggestions and actions" are specific actions and messages to family members to promote relationship repair, which are generated by the remote server when a conflict is detected.

[0222] "Display Device" refers to a display installed on the smart mirror to visually communicate suggestions and activities to family members.

[0223] "Home communication terminal" refers to a device such as a smartphone used by a family member, which works in conjunction with the smart mirror to share suggested information.

[0224] The "specific execution examples" are detailed instructions or examples for actually encouraging actions in response to the suggestions displayed on the home communication terminal by the remote server.

[0225] This invention uses a system that combines a smart mirror installed in the home, a cloud server, and a home communication terminal to analyze communication between family members and support relationship repair.

[0226] System Configuration

[0227] The system consists of the following equipment:

[0228] 1. Smart mirror terminal:

[0229] Image capture device: Captures the facial expressions of family members and temporarily stores the facial expression data.

[0230] Display device: Displays the suggestions and activities sent from the cloud server.

[0231] 2. Cloud Server:

[0232] Receives facial expression data and performs emotion analysis.

[0233] Analyze communication patterns between multiple family members to detect conflicts.

[0234] When conflict is detected, past behavioral data is used to generate suggestions and actions to promote relationship repair.

[0235] The proposal information is sent to a home communication terminal, and a specific execution example is generated.

[0236] 3. Home communication devices:

[0237] The proposed information transmitted from the cloud server is received and displayed on the home communication terminal.

[0238] Data Processing

[0239] 1. Server:

[0240] Facial expression data captured by an imaging device is analyzed to identify the emotional states of family members.

[0241] The communication patterns between multiple family members are analyzed based on the received facial expression data and past data.

[0242] Detect specific patterns and recognize conflicts as they occur.

[0243] 2. Terminal:

[0244] Suggestions and activity information sent from the cloud server are displayed on the smart mirror's display device.

[0245] Based on the specific execution examples displayed on the home communication terminal, the user acts in accordance with the suggestions.

[0246] Specific examples

[0247] Below is a specific example of how this system can be used.

[0248] Example 1: Morning conflict resolution

[0249] Device: In the morning, the camera detects that Mr. A has a sullen expression.

[0250] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[0251] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[0252] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[0253] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[0254] Example 2: Nighttime conflict resolution

[0255] Device: At night, the imaging device detects that Mr. B and Mr. C are arguing.

[0256] Server: Through analysis, it is determined that both B and C are feeling angry.

[0257] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[0258] Device: Show this suggestion in the mirror.

[0259] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[0260] Example prompts for generative AI models

[0261] "Analyze the emotional state of family members and make reconciliation suggestions at the appropriate time. Generate specific suggestions based on the following information:

[0262] 1. User facial expression data

[0263] 2. User Timestamp

[0264] 3. Past communication patterns

[0265] Examples of suggestions: 'Would you like to go for a morning walk with me?' 'Would you like to make a new recipe for dinner with me?'

[0266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0267] Step 1:

[0268] The imaging device of the smart mirror terminal captures the facial expressions of family members at regular intervals (for example, once per second). This captured data is temporarily saved locally. The input is image data captured in real time, and the output is a batch file containing the captured image data. The specific operation of this step is that the imaging device recognizes faces and captures the data as images.

[0269] Step 2:

[0270] The device sends the temporarily stored facial expression data to the cloud server in batch format. The user identifier and timestamp are attached to the transmission. The input is the batch file generated in step 1, and the output is the data sent to the cloud server. The specific operation of this step is to periodically trigger batch processing and send the facial expression data to the cloud server via an HTTP request.

[0271] Step 3:

[0272] The server receives the transmitted facial expression data and performs emotion analysis using a generative AI model. The input is a batch file containing facial expression data of family members, and the output is data indicating their emotional state. Specifically, in this step, the AI ​​model analyzes each image and generates an emotion label.

[0273] Step 4:

[0274] The server stores the analyzed emotional data in a database and further analyzes the communication patterns between family members. The input is the emotional data generated in step 3, and the output is the analysis results of the communication patterns. The specific operation of this step is for the server to compare past data with current data and detect specific conflict patterns.

[0275] Step 5:

[0276] When the server detects the occurrence of a conflict, it generates proposals and activities to promote relationship repair based on past behavioral data. The input is the analysis results of communication patterns and past behavioral data, and the output is a message of the proposal or activity. The specific operation of this step is that the server executes a proposal example generation algorithm to create appropriate proposals.

[0277] Step 6:

[0278] The proposal or activity is sent from the cloud server to the home communication device and displayed on the device. The input is the proposal or activity message generated in step 5, and the output is the proposal or activity displayed on the home communication device. The specific operation of this step is that the proposal message is displayed on the screen of the home communication device and the user is notified.

[0279] Step 7:

[0280] The user performs the action suggested, and their reaction is captured again by the smart mirror's imaging device. The input is the user's action and facial expression data at that time, and the output is new facial expression data sent to the cloud server. Specifically, this step involves capturing the user's facial expression again after the action and sending it to the server for analysis.

[0281] These steps enable the system to detect conflicts within the family and make timely suggestions to repair the relationship.

[0282] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0283] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments, and by combining it with an emotion engine that recognizes the user's emotional state, it makes more accurate suggestions. Specific embodiments of the system are described below.

[0284] System Configuration

[0285] This system consists of a smart mirror device, a cloud server, an emotion engine, and family users. The smart mirror device is equipped with a camera and a display, and works in conjunction with the emotion engine and the cloud server.

[0286] Program processing

[0287] Facial Expression Capture and Emotion Recognition

[0288] 1. Terminal

[0289] The camera built into the smart mirror takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[0290] 2. Terminal

[0291] The captured facial image data is temporarily stored, and the emotion engine is also activated. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[0292] 3. Terminal

[0293] Every certain time (e.g., every minute), the captured facial image data and emotional state data are sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[0294] Facial Expression Analysis and Conflict Detection

[0295] 4. Server

[0296] The facial image data and emotional state data are received and analyzed using an AI model. The analysis results provide detailed information about the emotional state of the family user. This data is then stored in a database.

[0297] 5. Server

[0298] Based on the accumulated data, the system analyzes communication patterns between multiple family members in real time. When certain patterns are detected, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[0299] 6. Server

[0300] If signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair by referencing the family user's past behavioral data and emotional history. For example, specific suggestions such as "Would you like to go see a movie together?" are generated.

[0301] Suggestion display and user response

[0302] 7. Terminal

[0303] Suggested messages and activities received from the cloud server are displayed on the mirror's display, where the suggestions are presented in a way that is intuitive to the user.

[0304] 8. Users

[0305] Review suggested messages and activities displayed in the mirror and act on them if you feel it is appropriate. For example, in response to the suggestion "Would you like to watch a movie together?", you can take the action of watching a movie together.

[0306] Reactions and Feedback

[0307] 9. Terminal

[0308] The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The emotion engine also analyzes the data and records the emotional data.

[0309] 10. Server

[0310] The received behavioral data, reactions, and emotional data are analyzed, and the results are used to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[0311] Specific examples

[0312] Example 1: Resolving conflicts in the morning

[0313] Device: In the morning, the camera detects that Person A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[0314] Server: Analyzes facial expression data and emotional state to detect that Person A is feeling stressed.

[0315] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[0316] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[0317] User: Mr. A and the rest of the family accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[0318] Example 2: Resolving conflicts during the night

[0319] Device: At night, the camera detects that B and C are arguing, and the emotion engine analyzes this to identify anger.

[0320] Server: Analyzes facial image data and emotional states and detects that Person B and Person C are feeling angry.

[0321] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[0322] Device: Show this suggestion in the mirror.

[0323] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[0324] In this way, by combining an emotion engine, the present invention is a system that can monitor communication within the home more precisely and make personalized reconciliation suggestions at the appropriate time.

[0325] The processing flow will be explained below.

[0326] Step 1:

[0327] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[0328] Step 2:

[0329] Device: Facial image data captured by the camera is sent in real time to the emotion engine, which analyzes this data and identifies the user's emotional state.

[0330] Step 3:

[0331] Device: Temporarily stores the user's emotional state data (e.g., happiness, anger, sadness, etc.) obtained from the emotion engine locally.

[0332] Step 4:

[0333] Device: At regular intervals (e.g., every minute), the device sends facial image data and emotional state data in batches to the cloud server. Each batch contains a user identifier and a timestamp.

[0334] Step 5:

[0335] Server: Receives the facial image data and emotional state data and analyzes them again using an AI model. As a result of the analysis, detailed emotional states are obtained and stored in a database.

[0336] Step 6:

[0337] Server: Analyzes communication patterns between multiple family members in real time based on the emotional state database. When certain patterns are detected (e.g., repeated angry facial expressions or prolonged silence), they are recognized as signs of conflict.

[0338] Step 7:

[0339] Server: When signs of conflict are detected, the AI ​​refers to the family user's past behavioral and emotional history data and generates messages and activity suggestions to promote relationship repair, such as "Would you like to go see a movie together?"

[0340] Step 8:

[0341] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display in a format that is intuitive and easy for the user to understand.

[0342] Step 9:

[0343] User: Review the suggested messages and activities displayed on the mirror and act on the suggestions that they feel are appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", they act on the suggestion to watch a movie together.

[0344] Step 10:

[0345] On the device: The user's behavioral data and reactions are captured again by the camera and analyzed by the emotion engine. Emotional changes are tracked and data is generated.

[0346] Step 11:

[0347] Device: Captured and analyzed reaction and emotion data is sent to a cloud server, allowing for real-time updates and analysis.

[0348] Step 12:

[0349] Server: Analyzes the received behavioral data, reaction and emotional data, and uses the results to improve the accuracy of future suggestions, resulting in more personalized suggestions.

[0350] Through these steps, the smart mirror monitors communication between family members and provides personalized reconciliation suggestions at the right time, thereby maintaining smooth relationships within the home.

[0351] Example 2

[0352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] There is a demand for smart mirror systems as a means of facilitating smoother communication within the home. However, current technology lacks the ability to adequately recognize emotions and analyze communication patterns, making it difficult to detect conflicts early or propose appropriate reconciliation strategies. Furthermore, feedback functions to improve the accuracy of suggested messages are not adequately implemented, making it difficult to provide optimal suggestions tailored to each individual household. As a result, repairing relationships and improving communication within the home has not been fully achieved.

[0354] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0355] In this invention, the server includes a means for transmitting emotional state data to a cloud server at regular intervals, a means for displaying suggested messages and activities received from the cloud server to the user, and a means for capturing the user's behavioral data and reactions to the suggestions and feeding them back to the cloud server. This enables the user's emotional state and communication patterns to be analyzed in real time, making it possible to provide timely and personalized suggestions. Furthermore, the accuracy of suggestions from the next time onwards can be improved based on the user's feedback, thereby more effectively repairing relationships and improving communication within the home.

[0356] A "smart mirror" is an electrical device that is installed in the home, has a built-in camera and display that captures the user's facial expressions, and processes facial image data and emotional state data.

[0357] The "camera" is a device built into the smart mirror that takes pictures of the user's face at regular intervals and acquires facial image data.

[0358] A "cloud server" is a remote computing system that receives and analyzes data sent from smart mirrors.

[0359] "Facial expression data" refers to image data of the user's face captured by a camera, and includes facial expression features.

[0360] "Emotional state data" is data that indicates the emotional state of the user analyzed by the emotion engine.

[0361] "Communication patterns" refer to a continuous flow of dialogue and behavior among multiple family members, including specific emotional and behavioral changes.

[0362] "Conflict" refers to the disagreements and discord that arise in the interactions and actions of family members.

[0363] "Messages and activities that promote relationship repair" are suggestions and action plans aimed at resolving conflicts and improving communication between family members.

[0364] "Displaying a suggested message" refers to the act of displaying a suggested message or activity received from the cloud server on the smart mirror's display.

[0365] "Behavioral data" is data indicating the actions that a user actually takes in response to a suggestion.

[0366] "Feedback" refers to sending the user's reaction and behavioral data to a cloud server to provide information to improve the accuracy of future suggestions.

[0367] An "emotion engine" is software or algorithm that analyzes facial image data to identify a user's emotional state in real time.

[0368] This invention is a system that uses a smart mirror installed in the home to facilitate communication between family members, detect conflicts, and support reconciliation. The system mainly consists of a smart mirror terminal, a cloud server, an emotion engine, and family users.

[0369] System Configuration

[0370] Smart mirror device: The smart mirror has a built-in camera and display. The camera captures facial expressions of family members and collects facial data. The display shows messages and activities received from the cloud server.

[0371] Cloud Server: The cloud server receives facial expression and emotional state data sent from the smart mirror and analyzes it using an AI model. The cloud server identifies the emotional state of family members, analyzes their communication patterns, and, if a conflict is detected, suggests messages or activities to promote relationship repair.

[0372] Emotion Engine: The emotion engine is software that analyzes facial image data captured by the camera in real time and identifies the emotional state. The identified emotional state data is temporarily stored locally and periodically sent to a cloud server.

[0373] Family users: Family users can review the suggested messages and activities displayed on their smart mirror and act on the suggestions if they feel it is appropriate.

[0374] Program processing

[0375] Facial Expression Capture and Emotion Recognition

[0376] 1. The camera on the smart mirror device takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[0377] 2. The emotion engine analyzes the facial image data and identifies the emotional state. This emotional state data is temporarily stored locally.

[0378] 3. At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server.

[0379] Facial Expression Analysis and Conflict Detection

[0380] 1. The cloud server analyzes the received facial image data and emotional state data. As a result of the analysis, detailed emotional states of family members are obtained.

[0381] 2. The server detects certain patterns, such as repeated angry facial expressions or long periods of silence, and identifies them as signs of conflict.

[0382] 3. If signs of conflict are detected, AI will dynamically tailor and generate messages and activities that promote relationship repair based on past behavioral data and emotional history.

[0383] Suggestion display and user response

[0384] 1. Display suggested messages and activities received from the cloud server on the smart mirror display.

[0385] 2. The user reviews the suggested messages and activities and acts on the suggestions if they feel appropriate.

[0386] Reactions and Feedback

[0387] 1. The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[0388] 2. The cloud server analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions.

[0389] Specific examples

[0390] Example 1: Resolving conflicts in the morning

[0391] 1. In the morning, the camera detects that Person A has a bad mood, and the emotion engine analyzes this to identify stress.

[0392] 2. The cloud server analyzes the facial expression data and emotional state and detects that Person A is feeling stressed.

[0393] 3. Based on past data, it is understood that Person A prefers the calm of the morning hours, and a message is generated saying, "Would you like to take a morning walk with me?"

[0394] 4. This message will be displayed on the smart mirror's display and suggested to Mr. A and other family members.

[0395] 5. Mr. A and the other family members accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[0396] Example 2: Resolving conflicts during the night

[0397] 1. At night, the camera detects that Person B and Person C are arguing, and the emotion engine analyzes this to identify anger.

[0398] 2. The cloud server analyzes the facial image data and emotional state and detects that both B and C are feeling angry.

[0399] 3. Recognizing that Mr. B and Mr. C share a common hobby of cooking, the system generates a suggestion: "Why don't we make a new recipe for dinner together?"

[0400] 4. The suggestion will appear on the smart mirror's display.

[0401] 5. When B and C accept the suggestion and make dinner together, the argument ends and their relationship is repaired.

[0402] In this way, the present invention is a system that effectively combines an emotion engine and an AI model to monitor communication within the home and make timely and personalized reconciliation suggestions.

[0403] Example prompts to input to the generative AI model

[0404] "Describe a system that uses a smart mirror to recognize family members' emotions and generate appropriate suggestions to facilitate communication."

[0405] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0406] Program processing flow

[0407] Facial Expression Capture and Emotion Recognition

[0408] Step 1:

[0409] Device: The camera built into the smart mirror takes pictures of the faces of family members. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data. The input is the face of the family member, and the output is facial image data. Specifically, the camera shutter operates and an image is captured.

[0410] Step 2:

[0411] Device: The captured facial image data is temporarily stored, and the emotion engine runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. The input is facial image data, and the output is emotional state data. Specifically, the emotion engine runs an image analysis algorithm to identify the emotional state.

[0412] Step 3:

[0413] Terminal: At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server. The input is the facial image data and emotional state data, and the output is the batch data sent to the server. Specifically, the network module operates and sends the data to the cloud server.

[0414] Facial Expression Analysis and Conflict Detection

[0415] Step 4:

[0416] Server: The cloud server analyzes the received facial image data and emotional state data. An AI model is used to obtain detailed emotional states of family members. The input is the batch data sent to the cloud server, and the output is the analyzed emotional state data. Specifically, the AI ​​model processes the data and identifies the emotional state.

[0417] Step 5:

[0418] Server: Based on the accumulated data, it analyzes communication patterns between multiple family members in real time. If a specific pattern is detected (such as repeated angry facial expressions or periods of silence), it recognizes this as a sign of conflict. The input is the analyzed emotional state data, and the output is conflict sign data. Specifically, it queries data from the database and executes a pattern matching algorithm.

[0419] Step 6:

[0420] Server: When signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities that promote relationship repair by referencing past behavioral data and emotional history. The input is conflict sign data and past behavioral data, and the output is a suggested message. Specifically, the suggestion generation algorithm is executed to create the suggested message.

[0421] Suggestion display and user response

[0422] Step 7:

[0423] Terminal: The smart mirror displays the suggested messages and activities received from the cloud server. The input is the suggested message, and the output is the suggested message displayed on the display. Specifically, the display controller operates and the message is displayed.

[0424] Step 8:

[0425] User: Checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. The input is the suggested message displayed, and the output is the user's actions. Specific actions include the user following the suggestions and taking actions such as watching a movie or going for a walk.

[0426] Reactions and Feedback

[0427] Step 9:

[0428] Terminal: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The input is the user's behavior, and the output is the behavioral data sent to the cloud server. Specifically, the camera operates again and the behavior is captured.

[0429] Step 10:

[0430] Server: Analyzes the received behavioral data, reaction, and emotional data, and uses the results to improve the accuracy of future proposals. The input is behavioral data and reaction data, and the output is the analysis results. Specifically, the data analysis algorithm is executed, and the results are stored in a database.

[0431] (Application example 2)

[0432] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0433] There is a demand for technology that can facilitate smooth communication within the home, especially to support reconciliation after arguments. Furthermore, there is also a demand for technology that can grasp customer emotions in real time and improve the quality of service in brick-and-mortar stores. However, to meet these needs, an advanced system that integrates emotion recognition technology with an appropriate recommendation system is required.

[0434] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0435] In this invention, the server includes a means for capturing facial expressions of family members using a smart mirror installed in the home and analyzing the data to identify their emotional states, a means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, a means for suggesting messages or activities to promote relationship repair when the occurrence of conflict is detected, and a means for staff wearing smart glasses in a physical store to capture customer facial expressions, analyze their emotional states, and display appropriate suggestions on the smart glasses, thereby improving communication within the home and customer satisfaction in physical stores.

[0436] A "smart mirror" is a mirror-shaped interactive device installed in the home that has a built-in camera and display, captures the user's facial expressions and movements, and displays information.

[0437] A "cloud server" is a server that stores and processes data via the Internet, and is a computer system that aggregates and analyzes data from multiple users.

[0438] An "emotion engine" is a software or hardware component for recognizing and analyzing a user's emotional state from captured image data or other data.

[0439] "Conflict occurrence" refers to friction or conflict in communication that arises between multiple members, and particularly refers to a state in which negative emotions such as anger or bad mood persist.

[0440] "Messages and activities that promote relationship repair" refer to suggestions and activities to improve communication and repair relationships based on emotional state and past behavioral data.

[0441] "Smart glasses" are wearable devices that incorporate a display and camera, allowing the wearer to acquire and display information in real time.

[0442] The "proposal system" is a system that dynamically generates optimal messages and activities for users based on captured data and analysis results.

[0443] The "service improvement system for physical stores" is a system that aims to analyze the emotional state of customers in physical stores in real time and provide services accordingly.

[0444] A "camera" is a photographing device for capturing images or videos of a user, and is used to identify facial expressions, etc.

[0445] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments. Another application example is a customer service improvement system that uses smart glasses worn by store staff in a brick-and-mortar store. The system of the present invention is composed of a smart mirror terminal, a cloud server, an emotion engine, smart glasses, and family users or store staff.

[0446] System Configuration

[0447] The smart mirror terminal has a built-in camera and display that captures the facial expressions of the user (family member or customer) and sends the data to a cloud server, while the smart glasses also have a built-in camera and display that transmits the captured facial expression data to a cloud server and displays service suggestions based on the analysis results.

[0448] Hardware and software used

[0449] Hardware: smart mirrors, cameras, smart glasses, webcams

[0450] Software: Python, OpenCV, data analysis software for cloud servers, local emotion recognition AI models (e.g., Flask servers)

[0451] Data processing and calculation

[0452] The server analyzes facial expression data sent from the smart mirror device or smart glasses to identify the user's emotional state. An emotion engine performs this analysis and identifies emotions such as confusion, satisfaction, or displeasure. Based on the identified emotions, the server then dynamically generates appropriate suggested messages and activities for family members or customers and displays them on the device's display.

[0453] Specifically, the cloud server performs the following data calculations:

[0454] 1. Facial expression data analysis: Using an AI model, captured facial expression data is analyzed in real time to identify the user's emotional state.

[0455] 2. Analysis of communication patterns: Analyze communication patterns between multiple family members to detect the occurrence of conflicts.

[0456] 3. Suggestion Generation: When a conflict is detected, messages and activities that promote relationship repair are dynamically generated based on the past behavioral data of family members. In physical stores, suggestions based on the customer's emotional state are displayed to store clerks wearing smart glasses.

[0457] Specific examples

[0458] Example 1: Conflict resolution within the family

[0459] Device: In the morning, the camera detects that family member A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[0460] Server: Analyzes facial expression data and emotional state to detect that A is feeling stressed.

[0461] Server: Based on past data, it understands that A prefers calm mornings and generates a suggestion: "Would you like to go for a morning walk together?"

[0462] Terminal: Display this message on the mirror's display and suggest it to A and other family members.

[0463] Example 2: Customer service in a brick-and-mortar store

[0464] Terminal: In the store, the camera in the smart glasses detects that Customer B has a confused expression.

[0465] Server: Analyzes facial image data and emotional state to detect that B is confused.

[0466] Server: Based on past data and analysis results, generate the optimal suggestion for B, "Do you need more information about this product?", and display it on the smart glasses.

[0467] Example of input prompt for generative AI model

[0468] "Do you need more information about this product?"

[0469] "Would you like to join me for a morning walk?"

[0470] This will facilitate smooth communication within homes and in physical stores, making it possible to build better relationships.

[0471] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0472] Step 1:

[0473] The device uses the smart mirror's built-in camera to capture images of family members' faces. Images are taken at regular intervals (e.g., once per second) and temporarily saved as facial image data.

[0474] Input: Facial expressions of family members

[0475] Output: Face image data

[0476] Step 2:

[0477] The device temporarily stores the captured facial image data, and the emotion engine also runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[0478] Input: Face image data

[0479] Output: Emotional state data

[0480] Step 3:

[0481] The device sends the captured facial image data and emotional state data to the cloud server in batch format at regular intervals (e.g., every minute). Each batch includes a user identifier and a timestamp.

[0482] Input: Facial image data, emotional state data

[0483] Output: Batch data sent to the cloud server

[0484] Step 4:

[0485] The server receives the facial image data and emotional state data and analyzes them using an AI model. The analysis results provide detailed emotional states of family members. This data is then stored in a database.

[0486] Input: Batch data

[0487] Output: Detailed emotional state data, stored in a database

[0488] Step 5:

[0489] The server analyzes communication patterns between multiple family members in real time based on the accumulated data, and when it detects certain patterns, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[0490] Input: Emotional state data stored in a database

[0491] Output: Conflict symptom data

[0492] Step 6:

[0493] If signs of conflict are detected, the server references the family members' past behavioral data and emotional history, and the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair.

[0494] Input: Conflict symptom data, past behavior data, emotional history

[0495] Output: Suggested messages and activities

[0496] Step 7:

[0497] The device receives suggested messages and activities from the cloud server and displays them on the mirror's display, where the suggestions are presented in an intuitive way for family members to understand.

[0498] Input: Suggested message or activity

[0499] Output: Messages and activity shown on the display

[0500] Step 8:

[0501] The user checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", the user takes the action of watching a movie together.

[0502] Input: Messages and activity shown on the display

[0503] Output: Family member behavior

[0504] Step 9:

[0505] The device again captures the user's behavioral data and reactions with its camera and sends them to the cloud server, where the emotion engine also analyzes them and records the emotional data.

[0506] Input: Family member behavior data, reactions

[0507] Output: Behavioral and emotional data sent to the cloud server

[0508] Step 10:

[0509] The server analyzes the received behavioral data, reactions, and emotional data, and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[0510] Input: Behavioral data, emotion data

[0511] Output: Data to improve the accuracy of proposals from next time onwards

[0512] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0513] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0514] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0515] [Second embodiment]

[0516] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0517] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0518] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0519] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0520] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0521] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0522] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0523] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0524] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0525] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0526] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0527] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0528] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after a fight. Specific embodiments of the system are described below.

[0529] System Configuration

[0530] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[0531] Program processing

[0532] Facial Capture

[0533] 1. Terminal

[0534] The camera built into the smart mirror captures the faces of family members at regular intervals (e.g., once per second). The captured facial expression data is temporarily stored locally.

[0535] 2. Terminal

[0536] Every certain time (for example, every minute), the captured facial expression data is sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[0537] facial expression analysis

[0538] 3. Server

[0539] The transmitted facial expression data is received and subjected to emotion analysis using AI. The analysis results indicate the emotional state of the family user (e.g., happiness, sadness, anger, etc.). This data is stored in a database.

[0540] Conflict Detection

[0541] 4. Server

[0542] The system analyzes communication patterns between family members from the accumulated facial expression data and behavior logs. When a specific pattern is detected, such as repeated angry expressions or prolonged silence, it recognizes the occurrence of a conflict.

[0543] 5. Server

[0544] When a conflict is detected, past behavioral data is used to generate suggestions to promote relationship repair, such as messages or activities like "Would you like to make dinner together?" or "Would you like to watch a movie together?"

[0545] Suggestion display

[0546] 6. Terminal

[0547] Suggested messages and activities from the cloud server are displayed on the screen, allowing the user to check them and take appropriate action.

[0548] Specific examples

[0549] Example 1: Resolving conflicts in the morning

[0550] Device: In the morning, the camera detects that Person A looks unhappy.

[0551] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[0552] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[0553] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[0554] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[0555] Example 2: Resolving conflicts during the night

[0556] Device: At night, the camera detects that B and C are arguing.

[0557] Server: Through analysis, it is determined that both B and C are feeling angry.

[0558] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[0559] Device: Show this suggestion in the mirror.

[0560] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[0561] In this way, the present invention is a system that monitors communication within the home and proposes reconciliation at appropriate times, thereby maintaining smooth relationships between family members.

[0562] The processing flow will be explained below.

[0563] Step 1:

[0564] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[0565] Step 2:

[0566] Device: The captured facial image data is sent to the cloud server as a batch at regular intervals (e.g., every minute). The batch also includes a user identifier and a timestamp.

[0567] Step 3:

[0568] Server: The received facial image data is input into the AI ​​model to analyze the emotional state of the family user. The analysis results are stored in an emotional state database. Emotional states are classified as happiness, anger, sadness, etc.

[0569] Step 4:

[0570] Server: Analyzes communication patterns between multiple family members in real time based on data stored in the emotional state database. When a specific pattern is detected (e.g., continuous angry facial expressions or silence), it recognizes this as a sign of conflict.

[0571] Step 5:

[0572] Server: If signs of conflict are detected, AI refers to the family members' past behavioral data and generates messages and activity suggestions to promote relationship repair. For example, "Would you like to watch a movie together?"

[0573] Step 6:

[0574] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display. The suggestions are presented in an intuitive way for the user.

[0575] Step 7:

[0576] User: Review suggested messages and activities displayed on the mirror and act on them if they feel it is appropriate. For example, in response to the suggested message "Want to watch a movie together?", they act on the suggestion to watch a movie together.

[0577] Step 8:

[0578] Device: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[0579] Step 9:

[0580] Server: Analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[0581] Through these steps, the smart mirror monitors communication between family members and makes reconciliation suggestions at the appropriate time, thereby maintaining smooth relationships within the home.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] Conflicts and arguments between family members are inevitable in domestic communication. However, if such conflicts are left unaddressed, family relationships are likely to deteriorate. In modern living environments, there is a need for a method to grasp family members' emotional states in real time and promote relationship repair at an appropriate time when conflicts occur. However, the lack of an automated system for detecting conflicts and proposing relationship repair makes it difficult to solve problems in individual households.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes means for analyzing the received facial expression data and identifying the emotional states of family members, means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, and means for suggesting messages or activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected. This makes it possible to quickly detect conflicts within the family and make suggestions to promote relationship repair at an appropriate time.

[0587] A "smart mirror" is a device that is installed in the home, equipped with a camera and a display, and used to capture and display the facial expressions and movements of family members.

[0588] The "photography device" is a camera built into the smart mirror, and is a means for capturing facial expression data of family members.

[0589] "Facial expression data" refers to data including facial features and expressions of family members captured by a photographing device.

[0590] A "cloud server" is a remote data center accessible via the Internet that provides analysis and storage functions for received data.

[0591] "Emotional state" refers to an individual's psychological state or mood (e.g., happiness, sadness, anger, etc.) analyzed based on facial expression data.

[0592] "Communication patterns" are data that indicate tendencies and regularities in the dialogue and behavior of family members.

[0593] "Conflict" is a state of discord caused by disputes or differences of opinion that arise between family members.

[0594] A "relationship repair message" is a message that includes suggestions or instructions for actions to promote the improvement of relationships between family members when the occurrence of a conflict is detected.

[0595] A "display device" is a display that is integrated into the smart mirror to visually present messages and activities to family members.

[0596] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates appropriate suggestions and responses to specific questions or requests.

[0597] A "prompt" is an instruction that is input into a generative AI model to generate an appropriate response or suggestion.

[0598] The present invention is a system that utilizes a smart mirror used at home to facilitate communication between family members and support the repair of relationships after conflicts or arguments. Specific embodiments of the system are described below.

[0599] System Configuration

[0600] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[0601] Hardware and software used

[0602] 1. Hardware

[0603] Smart mirror (with built-in camera and display)

[0604] Local Storage

[0605] 2. Software

[0606] Facial expression analysis AI (Google Face API)

[0607] Cloud infrastructure (AWS)

[0608] Database (MySQL)

[0609] Relationship repair suggestion AI model (OpenAI GPT-3)

[0610] Periodic job execution environment (Python)

[0611] Program processing

[0612] Facial Capture

[0613] When a user stands in front of the smart mirror, the device uses its built-in camera to capture a picture of the family user's face. The picture is taken at regular intervals (e.g., once per second), and the resulting facial expression data is temporarily stored in local storage. This facial expression data includes the user's identifier and a timestamp of when the picture was taken. Every regular interval (e.g., every minute), the device sends this facial expression data in batch format to a cloud server.

[0614] facial expression analysis

[0615] The server receives the facial expression data sent from the device. This data is securely received using cloud infrastructure such as AWS. The server performs emotion analysis on the received facial expression data using an AI model such as Google Face API to obtain the emotional state of each family user. The analysis results are stored in a database such as MySQL, and the data is organized by user and chronological order.

[0616] Conflict Detection

[0617] The server periodically analyzes the accumulated facial expression data and behavior logs in the database. This analysis uses Python to detect patterns of consecutive angry expressions or silence. If certain conditions are met (for example, five consecutive angry expressions), the server recognizes the conflict.

[0618] Proposal Generation and Display

[0619] When a conflict is detected, the server generates suggestions to promote relationship repair based on past behavioral data. These suggestions are generated using generative AI models such as OpenAI's GPT-3, and include specific activities (e.g., "Would you like to cook dinner together?" or "Would you like to watch a movie together?"). The generated suggestion message is sent to the device via the cloud, where it can be displayed on the device's display. The user can review the suggestion and take appropriate action.

[0620] Specific examples

[0621] Resolving conflicts in the morning

[0622] User: Person A stands in front of the smart mirror with a displeased look on his face.

[0623] Device: The camera captures A's facial expression, and the device sends the data to the cloud.

[0624] Server: Analyzes the received data using AI and detects that Person A is feeling stressed.

[0625] Server: Generates a suggestion, "Would you like to go for a morning walk together?" and sends it to the smart mirror.

[0626] Terminal: Show suggestions on the display.

[0627] User: Person A reviews the suggestion and goes for a walk with other family members.

[0628] Resolving conflicts during the night

[0629] User: The camera detects that B and C are arguing.

[0630] Device: The camera captures facial expression data and sends it to the cloud.

[0631] Server: Through data analysis, it detects that both B and C are angry.

[0632] Server: Generate a suggestion such as "Would you like to make a new recipe for dinner together?" and send it to the device.

[0633] Terminal: Show suggestions on the display.

[0634] User: B and C accept the suggestion and start making dinner together.

[0635] Example prompts for generative AI models

[0636] "Please suggest ways to improve communication between family members. Specifically, what actions should users take when they are feeling stressed?"

[0637] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0638] Step 1:

[0639] The user stands in front of the smart mirror, and the smart mirror's built-in camera captures the user's face.

[0640] Input: Face image of a family member

[0641] Output: Photographed facial expression data

[0642] Step 2:

[0643] The terminal uses a camera to take pictures of the faces of family users at regular intervals (for example, once per second) and acquires facial expression data.

[0644] Specific operation: The captured facial expression data is temporarily stored in local storage. The data includes the user's identifier and a timestamp.

[0645] Input: Face image of a family member

[0646] Output: Facial expression data temporarily saved in local storage

[0647] Step 3:

[0648] The device sends the temporarily stored facial expression data to the cloud server in batch format at regular intervals (for example, every minute).

[0649] Specific operation: The facial expression data is assigned a user identifier and a timestamp and sent to the cloud server.

[0650] Input: Facial expression data stored in local storage

[0651] Output: Facial expression data sent to the cloud server

[0652] Step 4:

[0653] The server receives the facial expression data transmitted from the terminal.

[0654] Specific operation: Facial expression data is securely received using cloud infrastructure such as AWS.

[0655] Input: Facial expression data sent from the device

[0656] Output: Received facial expression data

[0657] Step 5:

[0658] The server uses an AI model (e.g., Google Face API) to analyze the emotion of the received facial expression data.

[0659] Specific operation: Analyzes facial expression data and obtains the emotional state of family users (e.g., happiness, sadness, anger, etc.).

[0660] Input: Received facial expression data

[0661] Output: Parsed emotional state data

[0662] Step 6:

[0663] The server stores the analysis results in a database (e.g., MySQL).

[0664] Specific operation: Analyzed emotional state data is stored in a database for each user and for each time series.

[0665] Input: Parsed emotional state data

[0666] Output: Emotional state data stored in a database

[0667] Step 7:

[0668] The server periodically analyzes the facial expression data and behavior logs stored in the database.

[0669] What it does: A periodic job using Python detects patterns of successive angry expressions and silences.

[0670] Input: Facial expression data stored in a database

[0671] Output: Detected communication patterns

[0672] Step 8:

[0673] The server recognizes the occurrence of a conflict when certain conditions are met (for example, five consecutive angry facial expressions).

[0674] Specific behavior: Determine whether a conflict has occurred based on detected communication patterns.

[0675] Input: Detected communication patterns

[0676] Output: Conflict occurrence determination

[0677] Step 9:

[0678] When conflict is detected, the server generates suggestions based on past behavioral data to promote relationship repair.

[0679] Specific actions: Use a generative AI model (e.g., OpenAI's GPT-3) to generate messages that involve specific activities (e.g., "Would you like to make dinner together?" or "Would you like to watch a movie together?").

[0680] Input: Conflict occurrence determination, past behavior data

[0681] Output: Proposal message

[0682] Step 10:

[0683] The server sends the generated proposal message to the terminal via the cloud.

[0684] Specific operation: A proposal message is sent to the device via the cloud infrastructure.

[0685] Input: Proposal message

[0686] Output: Proposal message sent to the terminal

[0687] Step 11:

[0688] The terminal displays the proposal message received from the cloud server on the display.

[0689] Specific behavior: Suggestions are displayed in large letters and icons for easy viewing by family users.

[0690] Input: The proposal message received

[0691] Output: Suggestion message shown on the display

[0692] Step 12:

[0693] The user checks the suggestion message and takes appropriate action. For example, if they want to cook together, the user starts a conversation in the kitchen.

[0694] Specific Action: Act on the suggested activities to improve communication.

[0695] Input: Suggestion message shown on the display

[0696] Output: Improved communication through appropriate actions

[0697] (Application example 1)

[0698] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] Conventional home communication support systems lack the ability to accurately detect emotions and conflicts among family members and propose appropriate solutions, which can lead to unnecessary friction and misunderstandings. Furthermore, the system's overall accuracy and effectiveness are limited because it is unable to effectively collect information on how family members respond to proposed activities and messages and reflect this information in future proposals.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0701] In this invention, the server includes a device that generates specific execution examples in response to suggestions displayed on a home communication terminal, a smart mirror installed in the home and having an imaging device that captures the facial expressions of family members and a device that temporarily stores the facial expression data captured by the imaging device, a device that transmits the facial expression data to a remote server, means for analyzing the facial expression data received by the remote server and identifying the emotional states of the family members, a device that analyzes communication patterns between multiple family members using the remote server and detects the occurrence of conflict, a device that generates suggestions and activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected, a system having a display device that displays the suggestions and activities, and a device that shares suggestion information with the home communication terminal and generates specific execution examples in response to the suggestions displayed on the home communication terminal.

[0702] This makes it possible to suggest specific activities and messages at the appropriate time when conflicts arise within the family, and to collect responses to these suggestions in order to improve the accuracy of future suggestions.

[0703] The "imaging device" is a device for capturing images of the facial expressions of family members.

[0704] The "remote server" is a server that analyzes data sent from smart mirrors installed in the home and identifies the emotional states of family members.

[0705] "Facial expression data" refers to data on the facial expressions of family members captured by an imaging device installed in the smart mirror.

[0706] "Communication patterns" refer to patterns of conversation and behavior between multiple family members, and are data that can be analyzed to detect the occurrence of conflicts.

[0707] "Conflict" refers to a fight or disagreement between family members and is important for analysis to detect this occurrence.

[0708] "Relationship repair" refers to the process of resolving conflicts and misunderstandings between family members and rebuilding positive relationships.

[0709] "Suggestions and actions" are specific actions and messages to family members to promote relationship repair, which are generated by the remote server when a conflict is detected.

[0710] "Display Device" refers to a display installed on the smart mirror to visually communicate suggestions and activities to family members.

[0711] "Home communication terminal" refers to a device such as a smartphone used by a family member, which works in conjunction with the smart mirror to share suggested information.

[0712] The "specific execution examples" are detailed instructions or examples for actually encouraging actions in response to the suggestions displayed on the home communication terminal by the remote server.

[0713] This invention uses a system that combines a smart mirror installed in the home, a cloud server, and a home communication terminal to analyze communication between family members and support relationship repair.

[0714] System Configuration

[0715] The system consists of the following equipment:

[0716] 1. Smart mirror terminal:

[0717] Image capture device: Captures the facial expressions of family members and temporarily stores the facial expression data.

[0718] Display device: Displays the suggestions and activities sent from the cloud server.

[0719] 2. Cloud Server:

[0720] Receives facial expression data and performs emotion analysis.

[0721] Analyze communication patterns between multiple family members to detect conflicts.

[0722] When conflict is detected, past behavioral data is used to generate suggestions and actions to promote relationship repair.

[0723] The proposal information is sent to a home communication terminal, and a specific execution example is generated.

[0724] 3. Home communication devices:

[0725] The proposed information transmitted from the cloud server is received and displayed on the home communication terminal.

[0726] Data Processing

[0727] 1. Server:

[0728] Facial expression data captured by an imaging device is analyzed to identify the emotional states of family members.

[0729] The communication patterns between multiple family members are analyzed based on the received facial expression data and past data.

[0730] Detect specific patterns and recognize conflicts as they occur.

[0731] 2. Terminal:

[0732] Suggestions and activity information sent from the cloud server are displayed on the smart mirror's display device.

[0733] Based on the specific execution examples displayed on the home communication terminal, the user acts in accordance with the suggestions.

[0734] Specific examples

[0735] Below is a specific example of how this system can be used.

[0736] Example 1: Morning conflict resolution

[0737] Device: In the morning, the camera detects that Mr. A has a sullen expression.

[0738] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[0739] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[0740] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[0741] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[0742] Example 2: Nighttime conflict resolution

[0743] Device: At night, the imaging device detects that Mr. B and Mr. C are arguing.

[0744] Server: Through analysis, it is determined that both B and C are feeling angry.

[0745] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[0746] Device: Show this suggestion in the mirror.

[0747] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[0748] Example prompts for generative AI models

[0749] "Analyze the emotional state of family members and make reconciliation suggestions at the appropriate time. Generate specific suggestions based on the following information:

[0750] 1. User facial expression data

[0751] 2. User Timestamp

[0752] 3. Past communication patterns

[0753] Examples of suggestions: 'Would you like to go for a morning walk with me?' 'Would you like to make a new recipe for dinner with me?'

[0754] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0755] Step 1:

[0756] The imaging device of the smart mirror terminal captures the facial expressions of family members at regular intervals (for example, once per second). This captured data is temporarily saved locally. The input is image data captured in real time, and the output is a batch file containing the captured image data. The specific operation of this step is that the imaging device recognizes faces and captures the data as images.

[0757] Step 2:

[0758] The device sends the temporarily stored facial expression data to the cloud server in batch format. The user identifier and timestamp are attached to the transmission. The input is the batch file generated in step 1, and the output is the data sent to the cloud server. The specific operation of this step is to periodically trigger batch processing and send the facial expression data to the cloud server via an HTTP request.

[0759] Step 3:

[0760] The server receives the transmitted facial expression data and performs emotion analysis using a generative AI model. The input is a batch file containing facial expression data of family members, and the output is data indicating their emotional state. Specifically, in this step, the AI ​​model analyzes each image and generates an emotion label.

[0761] Step 4:

[0762] The server stores the analyzed emotional data in a database and further analyzes the communication patterns between family members. The input is the emotional data generated in step 3, and the output is the analysis results of the communication patterns. The specific operation of this step is for the server to compare past data with current data and detect specific conflict patterns.

[0763] Step 5:

[0764] When the server detects the occurrence of a conflict, it generates proposals and activities to promote relationship repair based on past behavioral data. The input is the analysis results of communication patterns and past behavioral data, and the output is a message of the proposal or activity. The specific operation of this step is that the server executes a proposal example generation algorithm to create appropriate proposals.

[0765] Step 6:

[0766] The proposal or activity is sent from the cloud server to the home communication device and displayed on the device. The input is the proposal or activity message generated in step 5, and the output is the proposal or activity displayed on the home communication device. The specific operation of this step is that the proposal message is displayed on the screen of the home communication device and the user is notified.

[0767] Step 7:

[0768] The user performs the action suggested, and their reaction is captured again by the smart mirror's imaging device. The input is the user's action and facial expression data at that time, and the output is new facial expression data sent to the cloud server. Specifically, this step involves capturing the user's facial expression again after the action and sending it to the server for analysis.

[0769] These steps enable the system to detect conflicts within the family and make timely suggestions to repair the relationship.

[0770] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0771] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments, and by combining it with an emotion engine that recognizes the user's emotional state, it makes more accurate suggestions. Specific embodiments of the system are described below.

[0772] System Configuration

[0773] This system consists of a smart mirror device, a cloud server, an emotion engine, and family users. The smart mirror device is equipped with a camera and a display, and works in conjunction with the emotion engine and the cloud server.

[0774] Program processing

[0775] Facial Expression Capture and Emotion Recognition

[0776] 1. Terminal

[0777] The camera built into the smart mirror takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[0778] 2. Terminal

[0779] The captured facial image data is temporarily stored, and the emotion engine is also activated. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[0780] 3. Terminal

[0781] Every certain time (e.g., every minute), the captured facial image data and emotional state data are sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[0782] Facial Expression Analysis and Conflict Detection

[0783] 4. Server

[0784] The facial image data and emotional state data are received and analyzed using an AI model. The analysis results provide detailed information about the emotional state of the family user. This data is then stored in a database.

[0785] 5. Server

[0786] Based on the accumulated data, the system analyzes communication patterns between multiple family members in real time. When certain patterns are detected, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[0787] 6. Server

[0788] If signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair by referencing the family user's past behavioral data and emotional history. For example, specific suggestions such as "Would you like to go see a movie together?" are generated.

[0789] Suggestion display and user response

[0790] 7. Terminal

[0791] Suggested messages and activities received from the cloud server are displayed on the mirror's display, where the suggestions are presented in a way that is intuitive to the user.

[0792] 8. Users

[0793] Review suggested messages and activities displayed in the mirror and act on them if you feel it is appropriate. For example, in response to the suggestion "Would you like to watch a movie together?", you can take the action of watching a movie together.

[0794] Reactions and Feedback

[0795] 9. Terminal

[0796] The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The emotion engine also analyzes the data and records the emotional data.

[0797] 10. Server

[0798] The received behavioral data, reactions, and emotional data are analyzed, and the results are used to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[0799] Specific examples

[0800] Example 1: Resolving conflicts in the morning

[0801] Device: In the morning, the camera detects that Person A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[0802] Server: Analyzes facial expression data and emotional state to detect that Person A is feeling stressed.

[0803] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[0804] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[0805] User: Mr. A and the rest of the family accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[0806] Example 2: Resolving conflicts during the night

[0807] Device: At night, the camera detects that B and C are arguing, and the emotion engine analyzes this to identify anger.

[0808] Server: Analyzes facial image data and emotional states and detects that Person B and Person C are feeling angry.

[0809] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[0810] Device: Show this suggestion in the mirror.

[0811] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[0812] In this way, by combining an emotion engine, the present invention is a system that can monitor communication within the home more precisely and make personalized reconciliation suggestions at the appropriate time.

[0813] The processing flow will be explained below.

[0814] Step 1:

[0815] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[0816] Step 2:

[0817] Device: Facial image data captured by the camera is sent in real time to the emotion engine, which analyzes this data and identifies the user's emotional state.

[0818] Step 3:

[0819] Device: Temporarily stores the user's emotional state data (e.g., happiness, anger, sadness, etc.) obtained from the emotion engine locally.

[0820] Step 4:

[0821] Device: At regular intervals (e.g., every minute), the device sends facial image data and emotional state data in batches to the cloud server. Each batch contains a user identifier and a timestamp.

[0822] Step 5:

[0823] Server: Receives the facial image data and emotional state data and analyzes them again using an AI model. As a result of the analysis, detailed emotional states are obtained and stored in a database.

[0824] Step 6:

[0825] Server: Analyzes communication patterns between multiple family members in real time based on the emotional state database. When certain patterns are detected (e.g., repeated angry facial expressions or prolonged silence), they are recognized as signs of conflict.

[0826] Step 7:

[0827] Server: When signs of conflict are detected, the AI ​​refers to the family user's past behavioral and emotional history data and generates messages and activity suggestions to promote relationship repair, such as "Would you like to go see a movie together?"

[0828] Step 8:

[0829] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display in a format that is intuitive and easy for the user to understand.

[0830] Step 9:

[0831] User: Review the suggested messages and activities displayed on the mirror and act on the suggestions that they feel are appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", they act on the suggestion to watch a movie together.

[0832] Step 10:

[0833] On the device: The user's behavioral data and reactions are captured again by the camera and analyzed by the emotion engine. Emotional changes are tracked and data is generated.

[0834] Step 11:

[0835] Device: Captured and analyzed reaction and emotion data is sent to a cloud server, allowing for real-time updates and analysis.

[0836] Step 12:

[0837] Server: Analyzes the received behavioral data, reaction and emotional data, and uses the results to improve the accuracy of future suggestions, resulting in more personalized suggestions.

[0838] Through these steps, the smart mirror monitors communication between family members and provides personalized reconciliation suggestions at the right time, thereby maintaining smooth relationships within the home.

[0839] Example 2

[0840] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0841] There is a demand for smart mirror systems as a means of facilitating smoother communication within the home. However, current technology lacks the ability to adequately recognize emotions and analyze communication patterns, making it difficult to detect conflicts early or propose appropriate reconciliation strategies. Furthermore, feedback functions to improve the accuracy of suggested messages are not adequately implemented, making it difficult to provide optimal suggestions tailored to each individual household. As a result, repairing relationships and improving communication within the home has not been fully achieved.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0843] In this invention, the server includes a means for transmitting emotional state data to a cloud server at regular intervals, a means for displaying suggested messages and activities received from the cloud server to the user, and a means for capturing the user's behavioral data and reactions to the suggestions and feeding them back to the cloud server. This enables the user's emotional state and communication patterns to be analyzed in real time, making it possible to provide timely and personalized suggestions. Furthermore, the accuracy of suggestions from the next time onwards can be improved based on the user's feedback, thereby more effectively repairing relationships and improving communication within the home.

[0844] A "smart mirror" is an electrical device that is installed in the home, has a built-in camera and display that captures the user's facial expressions, and processes facial image data and emotional state data.

[0845] The "camera" is a device built into the smart mirror that takes pictures of the user's face at regular intervals and acquires facial image data.

[0846] A "cloud server" is a remote computing system that receives and analyzes data sent from smart mirrors.

[0847] "Facial expression data" refers to image data of the user's face captured by a camera, and includes facial expression features.

[0848] "Emotional state data" is data that indicates the emotional state of the user analyzed by the emotion engine.

[0849] "Communication patterns" refer to a continuous flow of dialogue and behavior among multiple family members, including specific emotional and behavioral changes.

[0850] "Conflict" refers to the disagreements and discord that arise in the interactions and actions of family members.

[0851] "Messages and activities that promote relationship repair" are suggestions and action plans aimed at resolving conflicts and improving communication between family members.

[0852] "Displaying a suggested message" refers to the act of displaying a suggested message or activity received from the cloud server on the smart mirror's display.

[0853] "Behavioral data" is data indicating the actions that a user actually takes in response to a suggestion.

[0854] "Feedback" refers to sending the user's reaction and behavioral data to a cloud server to provide information to improve the accuracy of future suggestions.

[0855] An "emotion engine" is software or algorithm that analyzes facial image data to identify a user's emotional state in real time.

[0856] This invention is a system that uses a smart mirror installed in the home to facilitate communication between family members, detect conflicts, and support reconciliation. The system mainly consists of a smart mirror terminal, a cloud server, an emotion engine, and family users.

[0857] System Configuration

[0858] Smart mirror device: The smart mirror has a built-in camera and display. The camera captures facial expressions of family members and collects facial data. The display shows messages and activities received from the cloud server.

[0859] Cloud Server: The cloud server receives facial expression and emotional state data sent from the smart mirror and analyzes it using an AI model. The cloud server identifies the emotional state of family members, analyzes their communication patterns, and, if a conflict is detected, suggests messages or activities to promote relationship repair.

[0860] Emotion Engine: The emotion engine is software that analyzes facial image data captured by the camera in real time and identifies the emotional state. The identified emotional state data is temporarily stored locally and periodically sent to a cloud server.

[0861] Family users: Family users can review the suggested messages and activities displayed on their smart mirror and act on the suggestions if they feel it is appropriate.

[0862] Program processing

[0863] Facial Expression Capture and Emotion Recognition

[0864] 1. The camera on the smart mirror device takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[0865] 2. The emotion engine analyzes the facial image data and identifies the emotional state. This emotional state data is temporarily stored locally.

[0866] 3. At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server.

[0867] Facial Expression Analysis and Conflict Detection

[0868] 1. The cloud server analyzes the received facial image data and emotional state data. As a result of the analysis, detailed emotional states of family members are obtained.

[0869] 2. The server detects certain patterns, such as repeated angry facial expressions or long periods of silence, and identifies them as signs of conflict.

[0870] 3. If signs of conflict are detected, AI will dynamically tailor and generate messages and activities that promote relationship repair based on past behavioral data and emotional history.

[0871] Suggestion display and user response

[0872] 1. Display suggested messages and activities received from the cloud server on the smart mirror display.

[0873] 2. The user reviews the suggested messages and activities and acts on the suggestions if they feel appropriate.

[0874] Reactions and Feedback

[0875] 1. The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[0876] 2. The cloud server analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions.

[0877] Specific examples

[0878] Example 1: Resolving conflicts in the morning

[0879] 1. In the morning, the camera detects that Person A has a bad mood, and the emotion engine analyzes this to identify stress.

[0880] 2. The cloud server analyzes the facial expression data and emotional state and detects that Person A is feeling stressed.

[0881] 3. Based on past data, it is understood that Person A prefers the calm of the morning hours, and a message is generated saying, "Would you like to take a morning walk with me?"

[0882] 4. This message will be displayed on the smart mirror's display and suggested to Mr. A and other family members.

[0883] 5. Mr. A and the other family members accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[0884] Example 2: Resolving conflicts during the night

[0885] 1. At night, the camera detects that Person B and Person C are arguing, and the emotion engine analyzes this to identify anger.

[0886] 2. The cloud server analyzes the facial image data and emotional state and detects that both B and C are feeling angry.

[0887] 3. Recognizing that Mr. B and Mr. C share a common hobby of cooking, the system generates a suggestion: "Why don't we make a new recipe for dinner together?"

[0888] 4. The suggestion will appear on the smart mirror's display.

[0889] 5. When B and C accept the suggestion and make dinner together, the argument ends and their relationship is repaired.

[0890] In this way, the present invention is a system that effectively combines an emotion engine and an AI model to monitor communication within the home and make timely and personalized reconciliation suggestions.

[0891] Example prompts to input to the generative AI model

[0892] "Describe a system that uses a smart mirror to recognize family members' emotions and generate appropriate suggestions to facilitate communication."

[0893] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0894] Program processing flow

[0895] Facial Expression Capture and Emotion Recognition

[0896] Step 1:

[0897] Device: The camera built into the smart mirror takes pictures of the faces of family members. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data. The input is the face of the family member, and the output is facial image data. Specifically, the camera shutter operates and an image is captured.

[0898] Step 2:

[0899] Device: The captured facial image data is temporarily stored, and the emotion engine runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. The input is facial image data, and the output is emotional state data. Specifically, the emotion engine runs an image analysis algorithm to identify the emotional state.

[0900] Step 3:

[0901] Terminal: At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server. The input is the facial image data and emotional state data, and the output is the batch data sent to the server. Specifically, the network module operates and sends the data to the cloud server.

[0902] Facial Expression Analysis and Conflict Detection

[0903] Step 4:

[0904] Server: The cloud server analyzes the received facial image data and emotional state data. An AI model is used to obtain detailed emotional states of family members. The input is the batch data sent to the cloud server, and the output is the analyzed emotional state data. Specifically, the AI ​​model processes the data and identifies the emotional state.

[0905] Step 5:

[0906] Server: Based on the accumulated data, it analyzes communication patterns between multiple family members in real time. If a specific pattern is detected (such as repeated angry facial expressions or periods of silence), it recognizes this as a sign of conflict. The input is the analyzed emotional state data, and the output is conflict sign data. Specifically, it queries data from the database and executes a pattern matching algorithm.

[0907] Step 6:

[0908] Server: When signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities that promote relationship repair by referencing past behavioral data and emotional history. The input is conflict sign data and past behavioral data, and the output is a suggested message. Specifically, the suggestion generation algorithm is executed to create the suggested message.

[0909] Suggestion display and user response

[0910] Step 7:

[0911] Terminal: The smart mirror displays the suggested messages and activities received from the cloud server. The input is the suggested message, and the output is the suggested message displayed on the display. Specifically, the display controller operates and the message is displayed.

[0912] Step 8:

[0913] User: Checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. The input is the suggested message displayed, and the output is the user's actions. Specific actions include the user following the suggestions and taking actions such as watching a movie or going for a walk.

[0914] Reactions and Feedback

[0915] Step 9:

[0916] Terminal: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The input is the user's behavior, and the output is the behavioral data sent to the cloud server. Specifically, the camera operates again and the behavior is captured.

[0917] Step 10:

[0918] Server: Analyzes the received behavioral data, reaction, and emotional data, and uses the results to improve the accuracy of future proposals. The input is behavioral data and reaction data, and the output is the analysis results. Specifically, the data analysis algorithm is executed, and the results are stored in a database.

[0919] (Application example 2)

[0920] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0921] There is a demand for technology that can facilitate smooth communication within the home, especially to support reconciliation after arguments. Furthermore, there is also a demand for technology that can grasp customer emotions in real time and improve the quality of service in brick-and-mortar stores. However, to meet these needs, an advanced system that integrates emotion recognition technology with an appropriate recommendation system is required.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0923] In this invention, the server includes a means for capturing facial expressions of family members using a smart mirror installed in the home and analyzing the data to identify their emotional states, a means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, a means for suggesting messages or activities to promote relationship repair when the occurrence of conflict is detected, and a means for staff wearing smart glasses in a physical store to capture customer facial expressions, analyze their emotional states, and display appropriate suggestions on the smart glasses, thereby improving communication within the home and customer satisfaction in physical stores.

[0924] A "smart mirror" is a mirror-shaped interactive device installed in the home that has a built-in camera and display, captures the user's facial expressions and movements, and displays information.

[0925] A "cloud server" is a server that stores and processes data via the Internet, and is a computer system that aggregates and analyzes data from multiple users.

[0926] An "emotion engine" is a software or hardware component for recognizing and analyzing a user's emotional state from captured image data or other data.

[0927] "Conflict occurrence" refers to friction or conflict in communication that arises between multiple members, and particularly refers to a state in which negative emotions such as anger or bad mood persist.

[0928] "Messages and activities that promote relationship repair" refer to suggestions and activities to improve communication and repair relationships based on emotional state and past behavioral data.

[0929] "Smart glasses" are wearable devices that incorporate a display and camera, allowing the wearer to acquire and display information in real time.

[0930] The "proposal system" is a system that dynamically generates optimal messages and activities for users based on captured data and analysis results.

[0931] The "service improvement system for physical stores" is a system that aims to analyze the emotional state of customers in physical stores in real time and provide services accordingly.

[0932] A "camera" is a photographing device for capturing images or videos of a user, and is used to identify facial expressions, etc.

[0933] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments. Another application example is a customer service improvement system that uses smart glasses worn by store staff in a brick-and-mortar store. The system of the present invention is composed of a smart mirror terminal, a cloud server, an emotion engine, smart glasses, and family users or store staff.

[0934] System Configuration

[0935] The smart mirror terminal has a built-in camera and display that captures the facial expressions of the user (family member or customer) and sends the data to a cloud server, while the smart glasses also have a built-in camera and display that transmits the captured facial expression data to a cloud server and displays service suggestions based on the analysis results.

[0936] Hardware and software used

[0937] Hardware: smart mirrors, cameras, smart glasses, webcams

[0938] Software: Python, OpenCV, data analysis software for cloud servers, local emotion recognition AI models (e.g., Flask servers)

[0939] Data processing and calculation

[0940] The server analyzes facial expression data sent from the smart mirror device or smart glasses to identify the user's emotional state. An emotion engine performs this analysis and identifies emotions such as confusion, satisfaction, or displeasure. Based on the identified emotions, the server then dynamically generates appropriate suggested messages and activities for family members or customers and displays them on the device's display.

[0941] Specifically, the cloud server performs the following data calculations:

[0942] 1. Facial expression data analysis: Using an AI model, captured facial expression data is analyzed in real time to identify the user's emotional state.

[0943] 2. Analysis of communication patterns: Analyze communication patterns between multiple family members to detect the occurrence of conflicts.

[0944] 3. Suggestion Generation: When a conflict is detected, messages and activities that promote relationship repair are dynamically generated based on the past behavioral data of family members. In physical stores, suggestions based on the customer's emotional state are displayed to store clerks wearing smart glasses.

[0945] Specific examples

[0946] Example 1: Conflict resolution within the family

[0947] Device: In the morning, the camera detects that family member A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[0948] Server: Analyzes facial expression data and emotional state to detect that A is feeling stressed.

[0949] Server: Based on past data, it understands that A prefers calm mornings and generates a suggestion: "Would you like to go for a morning walk together?"

[0950] Terminal: Display this message on the mirror's display and suggest it to A and other family members.

[0951] Example 2: Customer service in a brick-and-mortar store

[0952] Terminal: In the store, the camera in the smart glasses detects that Customer B has a confused expression.

[0953] Server: Analyzes facial image data and emotional state to detect that B is confused.

[0954] Server: Based on past data and analysis results, generate the optimal suggestion for B, "Do you need more information about this product?", and display it on the smart glasses.

[0955] Example of input prompt for generative AI model

[0956] "Do you need more information about this product?"

[0957] "Would you like to join me for a morning walk?"

[0958] This will facilitate smooth communication within homes and in physical stores, making it possible to build better relationships.

[0959] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0960] Step 1:

[0961] The device uses the smart mirror's built-in camera to capture images of family members' faces. Images are taken at regular intervals (e.g., once per second) and temporarily saved as facial image data.

[0962] Input: Facial expressions of family members

[0963] Output: Face image data

[0964] Step 2:

[0965] The device temporarily stores the captured facial image data, and the emotion engine also runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[0966] Input: Face image data

[0967] Output: Emotional state data

[0968] Step 3:

[0969] The device sends the captured facial image data and emotional state data to the cloud server in batch format at regular intervals (e.g., every minute). Each batch includes a user identifier and a timestamp.

[0970] Input: Facial image data, emotional state data

[0971] Output: Batch data sent to the cloud server

[0972] Step 4:

[0973] The server receives the facial image data and emotional state data and analyzes them using an AI model. The analysis results provide detailed emotional states of family members. This data is then stored in a database.

[0974] Input: Batch data

[0975] Output: Detailed emotional state data, stored in a database

[0976] Step 5:

[0977] The server analyzes communication patterns between multiple family members in real time based on the accumulated data, and when it detects certain patterns, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[0978] Input: Emotional state data stored in a database

[0979] Output: Conflict symptom data

[0980] Step 6:

[0981] If signs of conflict are detected, the server references the family members' past behavioral data and emotional history, and the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair.

[0982] Input: Conflict symptom data, past behavior data, emotional history

[0983] Output: Suggested messages and activities

[0984] Step 7:

[0985] The device receives suggested messages and activities from the cloud server and displays them on the mirror's display, where the suggestions are presented in an intuitive way for family members to understand.

[0986] Input: Suggested message or activity

[0987] Output: Messages and activity shown on the display

[0988] Step 8:

[0989] The user checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", the user takes the action of watching a movie together.

[0990] Input: Messages and activity shown on the display

[0991] Output: Family member behavior

[0992] Step 9:

[0993] The device again captures the user's behavioral data and reactions with its camera and sends them to the cloud server, where the emotion engine also analyzes them and records the emotional data.

[0994] Input: Family member behavior data, reactions

[0995] Output: Behavioral and emotional data sent to the cloud server

[0996] Step 10:

[0997] The server analyzes the received behavioral data, reactions, and emotional data, and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[0998] Input: Behavioral data, emotion data

[0999] Output: Data to improve the accuracy of proposals from next time onwards

[1000] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1001] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1002] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1003] [Third embodiment]

[1004] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1005] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1006] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1007] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1008] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1009] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1010] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1011] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1012] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1013] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1014] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1015] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1016] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after a fight. Specific embodiments of the system are described below.

[1017] System Configuration

[1018] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[1019] Program processing

[1020] Facial Capture

[1021] 1. Terminal

[1022] The camera built into the smart mirror captures the faces of family members at regular intervals (e.g., once per second). The captured facial expression data is temporarily stored locally.

[1023] 2. Terminal

[1024] Every certain time (for example, every minute), the captured facial expression data is sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[1025] facial expression analysis

[1026] 3. Server

[1027] The transmitted facial expression data is received and subjected to emotion analysis using AI. The analysis results indicate the emotional state of the family user (e.g., happiness, sadness, anger, etc.). This data is stored in a database.

[1028] Conflict Detection

[1029] 4. Server

[1030] The system analyzes communication patterns between family members from the accumulated facial expression data and behavior logs. When a specific pattern is detected, such as repeated angry expressions or prolonged silence, it recognizes the occurrence of a conflict.

[1031] 5. Server

[1032] When a conflict is detected, past behavioral data is used to generate suggestions to promote relationship repair, such as messages or activities like "Would you like to make dinner together?" or "Would you like to watch a movie together?"

[1033] Suggestion display

[1034] 6. Terminal

[1035] Suggested messages and activities from the cloud server are displayed on the screen, allowing the user to check them and take appropriate action.

[1036] Specific examples

[1037] Example 1: Resolving conflicts in the morning

[1038] Device: In the morning, the camera detects that Person A looks unhappy.

[1039] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[1040] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[1041] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[1042] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[1043] Example 2: Resolving conflicts during the night

[1044] Device: At night, the camera detects that B and C are arguing.

[1045] Server: Through analysis, it is determined that both B and C are feeling angry.

[1046] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[1047] Device: Show this suggestion in the mirror.

[1048] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[1049] In this way, the present invention is a system that monitors communication within the home and proposes reconciliation at appropriate times, thereby maintaining smooth relationships between family members.

[1050] The processing flow will be explained below.

[1051] Step 1:

[1052] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[1053] Step 2:

[1054] Device: The captured facial image data is sent to the cloud server as a batch at regular intervals (e.g., every minute). The batch also includes a user identifier and a timestamp.

[1055] Step 3:

[1056] Server: The received facial image data is input into the AI ​​model to analyze the emotional state of the family user. The analysis results are stored in an emotional state database. Emotional states are classified as happiness, anger, sadness, etc.

[1057] Step 4:

[1058] Server: Analyzes communication patterns between multiple family members in real time based on data stored in the emotional state database. When a specific pattern is detected (e.g., continuous angry facial expressions or silence), it recognizes this as a sign of conflict.

[1059] Step 5:

[1060] Server: If signs of conflict are detected, AI refers to the family members' past behavioral data and generates messages and activity suggestions to promote relationship repair. For example, "Would you like to watch a movie together?"

[1061] Step 6:

[1062] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display. The suggestions are presented in an intuitive way for the user.

[1063] Step 7:

[1064] User: Review suggested messages and activities displayed on the mirror and act on them if they feel it is appropriate. For example, in response to the suggested message "Want to watch a movie together?", they act on the suggestion to watch a movie together.

[1065] Step 8:

[1066] Device: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[1067] Step 9:

[1068] Server: Analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[1069] Through these steps, the smart mirror monitors communication between family members and makes reconciliation suggestions at the appropriate time, thereby maintaining smooth relationships within the home.

[1070] Example 1

[1071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] Conflicts and arguments between family members are inevitable in domestic communication. However, if such conflicts are left unaddressed, family relationships are likely to deteriorate. In modern living environments, there is a need for a method to grasp family members' emotional states in real time and promote relationship repair at an appropriate time when conflicts occur. However, the lack of an automated system for detecting conflicts and proposing relationship repair makes it difficult to solve problems in individual households.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1074] In this invention, the server includes means for analyzing the received facial expression data and identifying the emotional states of family members, means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, and means for suggesting messages or activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected. This makes it possible to quickly detect conflicts within the family and make suggestions to promote relationship repair at an appropriate time.

[1075] A "smart mirror" is a device that is installed in the home, equipped with a camera and a display, and used to capture and display the facial expressions and movements of family members.

[1076] The "photography device" is a camera built into the smart mirror, and is a means for capturing facial expression data of family members.

[1077] "Facial expression data" refers to data including facial features and expressions of family members captured by a photographing device.

[1078] A "cloud server" is a remote data center accessible via the Internet that provides analysis and storage functions for received data.

[1079] "Emotional state" refers to an individual's psychological state or mood (e.g., happiness, sadness, anger, etc.) analyzed based on facial expression data.

[1080] "Communication patterns" are data that indicate tendencies and regularities in the dialogue and behavior of family members.

[1081] "Conflict" is a state of discord caused by disputes or differences of opinion that arise between family members.

[1082] A "relationship repair message" is a message that includes suggestions or instructions for actions to promote the improvement of relationships between family members when the occurrence of a conflict is detected.

[1083] A "display device" is a display that is integrated into the smart mirror to visually present messages and activities to family members.

[1084] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates appropriate suggestions and responses to specific questions or requests.

[1085] A "prompt" is an instruction that is input into a generative AI model to generate an appropriate response or suggestion.

[1086] The present invention is a system that utilizes a smart mirror used at home to facilitate communication between family members and support the repair of relationships after conflicts or arguments. Specific embodiments of the system are described below.

[1087] System Configuration

[1088] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[1089] Hardware and software used

[1090] 1. Hardware

[1091] Smart mirror (with built-in camera and display)

[1092] Local Storage

[1093] 2. Software

[1094] Facial expression analysis AI (Google Face API)

[1095] Cloud infrastructure (AWS)

[1096] Database (MySQL)

[1097] Relationship repair suggestion AI model (OpenAI GPT-3)

[1098] Periodic job execution environment (Python)

[1099] Program processing

[1100] Facial Capture

[1101] When a user stands in front of the smart mirror, the device uses its built-in camera to capture a picture of the family user's face. The picture is taken at regular intervals (e.g., once per second), and the resulting facial expression data is temporarily stored in local storage. This facial expression data includes the user's identifier and a timestamp of when the picture was taken. Every regular interval (e.g., every minute), the device sends this facial expression data in batch format to a cloud server.

[1102] facial expression analysis

[1103] The server receives the facial expression data sent from the device. This data is securely received using cloud infrastructure such as AWS. The server performs emotion analysis on the received facial expression data using an AI model such as Google Face API to obtain the emotional state of each family user. The analysis results are stored in a database such as MySQL, and the data is organized by user and chronological order.

[1104] Conflict Detection

[1105] The server periodically analyzes the accumulated facial expression data and behavior logs in the database. This analysis uses Python to detect patterns of consecutive angry expressions or silence. If certain conditions are met (for example, five consecutive angry expressions), the server recognizes the conflict.

[1106] Proposal Generation and Display

[1107] When a conflict is detected, the server generates suggestions to promote relationship repair based on past behavioral data. These suggestions are generated using generative AI models such as OpenAI's GPT-3, and include specific activities (e.g., "Would you like to cook dinner together?" or "Would you like to watch a movie together?"). The generated suggestion message is sent to the device via the cloud, where it can be displayed on the device's display. The user can review the suggestion and take appropriate action.

[1108] Specific examples

[1109] Resolving conflicts in the morning

[1110] User: Person A stands in front of the smart mirror with a displeased look on his face.

[1111] Device: The camera captures A's facial expression, and the device sends the data to the cloud.

[1112] Server: Analyzes the received data using AI and detects that Person A is feeling stressed.

[1113] Server: Generates a suggestion, "Would you like to go for a morning walk together?" and sends it to the smart mirror.

[1114] Terminal: Show suggestions on the display.

[1115] User: Person A reviews the suggestion and goes for a walk with other family members.

[1116] Resolving conflicts during the night

[1117] User: The camera detects that B and C are arguing.

[1118] Device: The camera captures facial expression data and sends it to the cloud.

[1119] Server: Through data analysis, it detects that both B and C are angry.

[1120] Server: Generate a suggestion such as "Would you like to make a new recipe for dinner together?" and send it to the device.

[1121] Terminal: Show suggestions on the display.

[1122] User: B and C accept the suggestion and start making dinner together.

[1123] Example prompts for generative AI models

[1124] "Please suggest ways to improve communication between family members. Specifically, what actions should users take when they are feeling stressed?"

[1125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1126] Step 1:

[1127] The user stands in front of the smart mirror, and the smart mirror's built-in camera captures the user's face.

[1128] Input: Face image of a family member

[1129] Output: Photographed facial expression data

[1130] Step 2:

[1131] The terminal uses a camera to take pictures of the faces of family users at regular intervals (for example, once per second) and acquires facial expression data.

[1132] Specific operation: The captured facial expression data is temporarily stored in local storage. The data includes the user's identifier and a timestamp.

[1133] Input: Face image of a family member

[1134] Output: Facial expression data temporarily saved in local storage

[1135] Step 3:

[1136] The device sends the temporarily stored facial expression data to the cloud server in batch format at regular intervals (for example, every minute).

[1137] Specific operation: The facial expression data is assigned a user identifier and a timestamp and sent to the cloud server.

[1138] Input: Facial expression data stored in local storage

[1139] Output: Facial expression data sent to the cloud server

[1140] Step 4:

[1141] The server receives the facial expression data transmitted from the terminal.

[1142] Specific operation: Facial expression data is securely received using cloud infrastructure such as AWS.

[1143] Input: Facial expression data sent from the device

[1144] Output: Received facial expression data

[1145] Step 5:

[1146] The server uses an AI model (e.g., Google Face API) to analyze the emotion of the received facial expression data.

[1147] Specific operation: Analyzes facial expression data and obtains the emotional state of family users (e.g., happiness, sadness, anger, etc.).

[1148] Input: Received facial expression data

[1149] Output: Parsed emotional state data

[1150] Step 6:

[1151] The server stores the analysis results in a database (e.g., MySQL).

[1152] Specific operation: Analyzed emotional state data is stored in a database for each user and for each time series.

[1153] Input: Parsed emotional state data

[1154] Output: Emotional state data stored in a database

[1155] Step 7:

[1156] The server periodically analyzes the facial expression data and behavior logs stored in the database.

[1157] What it does: A periodic job using Python detects patterns of successive angry expressions and silences.

[1158] Input: Facial expression data stored in a database

[1159] Output: Detected communication patterns

[1160] Step 8:

[1161] The server recognizes the occurrence of a conflict when certain conditions are met (for example, five consecutive angry facial expressions).

[1162] Specific behavior: Determine whether a conflict has occurred based on detected communication patterns.

[1163] Input: Detected communication patterns

[1164] Output: Conflict occurrence determination

[1165] Step 9:

[1166] When conflict is detected, the server generates suggestions based on past behavioral data to promote relationship repair.

[1167] Specific actions: Use a generative AI model (e.g., OpenAI's GPT-3) to generate messages that involve specific activities (e.g., "Would you like to make dinner together?" or "Would you like to watch a movie together?").

[1168] Input: Conflict occurrence determination, past behavior data

[1169] Output: Proposal message

[1170] Step 10:

[1171] The server sends the generated proposal message to the terminal via the cloud.

[1172] Specific operation: A proposal message is sent to the device via the cloud infrastructure.

[1173] Input: Proposal message

[1174] Output: Proposal message sent to the terminal

[1175] Step 11:

[1176] The terminal displays the proposal message received from the cloud server on the display.

[1177] Specific behavior: Suggestions are displayed in large letters and icons for easy viewing by family users.

[1178] Input: The proposal message received

[1179] Output: Suggestion message shown on the display

[1180] Step 12:

[1181] The user checks the suggestion message and takes appropriate action. For example, if they want to cook together, the user starts a conversation in the kitchen.

[1182] Specific Action: Act on the suggested activities to improve communication.

[1183] Input: Suggestion message shown on the display

[1184] Output: Improved communication through appropriate actions

[1185] (Application example 1)

[1186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1187] Conventional home communication support systems lack the ability to accurately detect emotions and conflicts among family members and propose appropriate solutions, which can lead to unnecessary friction and misunderstandings. Furthermore, the system's overall accuracy and effectiveness are limited because it is unable to effectively collect information on how family members respond to proposed activities and messages and reflect this information in future proposals.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1189] In this invention, the server includes a device that generates specific execution examples in response to suggestions displayed on a home communication terminal, a smart mirror installed in the home and having an imaging device that captures the facial expressions of family members and a device that temporarily stores the facial expression data captured by the imaging device, a device that transmits the facial expression data to a remote server, means for analyzing the facial expression data received by the remote server and identifying the emotional states of the family members, a device that analyzes communication patterns between multiple family members using the remote server and detects the occurrence of conflict, a device that generates suggestions and activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected, a system having a display device that displays the suggestions and activities, and a device that shares suggestion information with the home communication terminal and generates specific execution examples in response to the suggestions displayed on the home communication terminal.

[1190] This makes it possible to suggest specific activities and messages at the appropriate time when conflicts arise within the family, and to collect responses to these suggestions in order to improve the accuracy of future suggestions.

[1191] The "imaging device" is a device for capturing images of the facial expressions of family members.

[1192] The "remote server" is a server that analyzes data sent from smart mirrors installed in the home and identifies the emotional states of family members.

[1193] "Facial expression data" refers to data on the facial expressions of family members captured by an imaging device installed in the smart mirror.

[1194] "Communication patterns" refer to patterns of conversation and behavior between multiple family members, and are data that can be analyzed to detect the occurrence of conflicts.

[1195] "Conflict" refers to a fight or disagreement between family members and is important for analysis to detect this occurrence.

[1196] "Relationship repair" refers to the process of resolving conflicts and misunderstandings between family members and rebuilding positive relationships.

[1197] "Suggestions and actions" are specific actions and messages to family members to promote relationship repair, which are generated by the remote server when a conflict is detected.

[1198] "Display Device" refers to a display installed on the smart mirror to visually communicate suggestions and activities to family members.

[1199] "Home communication terminal" refers to a device such as a smartphone used by a family member, which works in conjunction with the smart mirror to share suggested information.

[1200] The "specific execution examples" are detailed instructions or examples for actually encouraging actions in response to the suggestions displayed on the home communication terminal by the remote server.

[1201] This invention uses a system that combines a smart mirror installed in the home, a cloud server, and a home communication terminal to analyze communication between family members and support relationship repair.

[1202] System Configuration

[1203] The system consists of the following equipment:

[1204] 1. Smart mirror terminal:

[1205] Image capture device: Captures the facial expressions of family members and temporarily stores the facial expression data.

[1206] Display device: Displays the suggestions and activities sent from the cloud server.

[1207] 2. Cloud Server:

[1208] Receives facial expression data and performs emotion analysis.

[1209] Analyze communication patterns between multiple family members to detect conflicts.

[1210] When conflict is detected, past behavioral data is used to generate suggestions and actions to promote relationship repair.

[1211] The proposal information is sent to a home communication terminal, and a specific execution example is generated.

[1212] 3. Home communication devices:

[1213] The proposed information transmitted from the cloud server is received and displayed on the home communication terminal.

[1214] Data Processing

[1215] 1. Server:

[1216] Facial expression data captured by an imaging device is analyzed to identify the emotional states of family members.

[1217] The communication patterns between multiple family members are analyzed based on the received facial expression data and past data.

[1218] Detect specific patterns and recognize conflicts as they occur.

[1219] 2. Terminal:

[1220] Suggestions and activity information sent from the cloud server are displayed on the smart mirror's display device.

[1221] Based on the specific execution examples displayed on the home communication terminal, the user acts in accordance with the suggestions.

[1222] Specific examples

[1223] Below is a specific example of how this system can be used.

[1224] Example 1: Morning conflict resolution

[1225] Device: In the morning, the camera detects that Mr. A has a sullen expression.

[1226] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[1227] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[1228] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[1229] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[1230] Example 2: Nighttime conflict resolution

[1231] Device: At night, the imaging device detects that Mr. B and Mr. C are arguing.

[1232] Server: Through analysis, it is determined that both B and C are feeling angry.

[1233] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[1234] Device: Show this suggestion in the mirror.

[1235] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[1236] Example prompts for generative AI models

[1237] "Analyze the emotional state of family members and make reconciliation suggestions at the appropriate time. Generate specific suggestions based on the following information:

[1238] 1. User facial expression data

[1239] 2. User Timestamp

[1240] 3. Past communication patterns

[1241] Examples of suggestions: 'Would you like to go for a morning walk with me?' 'Would you like to make a new recipe for dinner with me?'

[1242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1243] Step 1:

[1244] The imaging device of the smart mirror terminal captures the facial expressions of family members at regular intervals (for example, once per second). This captured data is temporarily saved locally. The input is image data captured in real time, and the output is a batch file containing the captured image data. The specific operation of this step is that the imaging device recognizes faces and captures the data as images.

[1245] Step 2:

[1246] The device sends the temporarily stored facial expression data to the cloud server in batch format. The user identifier and timestamp are attached to the transmission. The input is the batch file generated in step 1, and the output is the data sent to the cloud server. The specific operation of this step is to periodically trigger batch processing and send the facial expression data to the cloud server via an HTTP request.

[1247] Step 3:

[1248] The server receives the transmitted facial expression data and performs emotion analysis using a generative AI model. The input is a batch file containing facial expression data of family members, and the output is data indicating their emotional state. Specifically, in this step, the AI ​​model analyzes each image and generates an emotion label.

[1249] Step 4:

[1250] The server stores the analyzed emotional data in a database and further analyzes the communication patterns between family members. The input is the emotional data generated in step 3, and the output is the analysis results of the communication patterns. The specific operation of this step is for the server to compare past data with current data and detect specific conflict patterns.

[1251] Step 5:

[1252] When the server detects the occurrence of a conflict, it generates proposals and activities to promote relationship repair based on past behavioral data. The input is the analysis results of communication patterns and past behavioral data, and the output is a message of the proposal or activity. The specific operation of this step is that the server executes a proposal example generation algorithm to create appropriate proposals.

[1253] Step 6:

[1254] The proposal or activity is sent from the cloud server to the home communication device and displayed on the device. The input is the proposal or activity message generated in step 5, and the output is the proposal or activity displayed on the home communication device. The specific operation of this step is that the proposal message is displayed on the screen of the home communication device and the user is notified.

[1255] Step 7:

[1256] The user performs the action suggested, and their reaction is captured again by the smart mirror's imaging device. The input is the user's action and facial expression data at that time, and the output is new facial expression data sent to the cloud server. Specifically, this step involves capturing the user's facial expression again after the action and sending it to the server for analysis.

[1257] These steps enable the system to detect conflicts within the family and make timely suggestions to repair the relationship.

[1258] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1259] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments, and by combining it with an emotion engine that recognizes the user's emotional state, it makes more accurate suggestions. Specific embodiments of the system are described below.

[1260] System Configuration

[1261] This system consists of a smart mirror device, a cloud server, an emotion engine, and family users. The smart mirror device is equipped with a camera and a display, and works in conjunction with the emotion engine and the cloud server.

[1262] Program processing

[1263] Facial Expression Capture and Emotion Recognition

[1264] 1. Terminal

[1265] The camera built into the smart mirror takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[1266] 2. Terminal

[1267] The captured facial image data is temporarily stored, and the emotion engine is also activated. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[1268] 3. Terminal

[1269] Every certain time (e.g., every minute), the captured facial image data and emotional state data are sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[1270] Facial Expression Analysis and Conflict Detection

[1271] 4. Server

[1272] The facial image data and emotional state data are received and analyzed using an AI model. The analysis results provide detailed information about the emotional state of the family user. This data is then stored in a database.

[1273] 5. Server

[1274] Based on the accumulated data, the system analyzes communication patterns between multiple family members in real time. When certain patterns are detected, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[1275] 6. Server

[1276] If signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair by referencing the family user's past behavioral data and emotional history. For example, specific suggestions such as "Would you like to go see a movie together?" are generated.

[1277] Suggestion display and user response

[1278] 7. Terminal

[1279] Suggested messages and activities received from the cloud server are displayed on the mirror's display, where the suggestions are presented in a way that is intuitive to the user.

[1280] 8. Users

[1281] Review suggested messages and activities displayed in the mirror and act on them if you feel it is appropriate. For example, in response to the suggestion "Would you like to watch a movie together?", you can take the action of watching a movie together.

[1282] Reactions and Feedback

[1283] 9. Terminal

[1284] The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The emotion engine also analyzes the data and records the emotional data.

[1285] 10. Server

[1286] The received behavioral data, reactions, and emotional data are analyzed, and the results are used to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[1287] Specific examples

[1288] Example 1: Resolving conflicts in the morning

[1289] Device: In the morning, the camera detects that Person A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[1290] Server: Analyzes facial expression data and emotional state to detect that Person A is feeling stressed.

[1291] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[1292] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[1293] User: Mr. A and the rest of the family accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[1294] Example 2: Resolving conflicts during the night

[1295] Device: At night, the camera detects that B and C are arguing, and the emotion engine analyzes this to identify anger.

[1296] Server: Analyzes facial image data and emotional states and detects that Person B and Person C are feeling angry.

[1297] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[1298] Device: Show this suggestion in the mirror.

[1299] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[1300] In this way, by combining an emotion engine, the present invention is a system that can monitor communication within the home more precisely and make personalized reconciliation suggestions at the appropriate time.

[1301] The processing flow will be explained below.

[1302] Step 1:

[1303] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[1304] Step 2:

[1305] Device: Facial image data captured by the camera is sent in real time to the emotion engine, which analyzes this data and identifies the user's emotional state.

[1306] Step 3:

[1307] Device: Temporarily stores the user's emotional state data (e.g., happiness, anger, sadness, etc.) obtained from the emotion engine locally.

[1308] Step 4:

[1309] Device: At regular intervals (e.g., every minute), the device sends facial image data and emotional state data in batches to the cloud server. Each batch contains a user identifier and a timestamp.

[1310] Step 5:

[1311] Server: Receives the facial image data and emotional state data and analyzes them again using an AI model. As a result of the analysis, detailed emotional states are obtained and stored in a database.

[1312] Step 6:

[1313] Server: Analyzes communication patterns between multiple family members in real time based on the emotional state database. When certain patterns are detected (e.g., repeated angry facial expressions or prolonged silence), they are recognized as signs of conflict.

[1314] Step 7:

[1315] Server: When signs of conflict are detected, the AI ​​refers to the family user's past behavioral and emotional history data and generates messages and activity suggestions to promote relationship repair, such as "Would you like to go see a movie together?"

[1316] Step 8:

[1317] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display in a format that is intuitive and easy for the user to understand.

[1318] Step 9:

[1319] User: Review the suggested messages and activities displayed on the mirror and act on the suggestions that they feel are appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", they act on the suggestion to watch a movie together.

[1320] Step 10:

[1321] On the device: The user's behavioral data and reactions are captured again by the camera and analyzed by the emotion engine. Emotional changes are tracked and data is generated.

[1322] Step 11:

[1323] Device: Captured and analyzed reaction and emotion data is sent to a cloud server, allowing for real-time updates and analysis.

[1324] Step 12:

[1325] Server: Analyzes the received behavioral data, reaction and emotional data, and uses the results to improve the accuracy of future suggestions, resulting in more personalized suggestions.

[1326] Through these steps, the smart mirror monitors communication between family members and provides personalized reconciliation suggestions at the right time, thereby maintaining smooth relationships within the home.

[1327] Example 2

[1328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1329] There is a demand for smart mirror systems as a means of facilitating smoother communication within the home. However, current technology lacks the ability to adequately recognize emotions and analyze communication patterns, making it difficult to detect conflicts early or propose appropriate reconciliation strategies. Furthermore, feedback functions to improve the accuracy of suggested messages are not adequately implemented, making it difficult to provide optimal suggestions tailored to each individual household. As a result, repairing relationships and improving communication within the home has not been fully achieved.

[1330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1331] In this invention, the server includes a means for transmitting emotional state data to a cloud server at regular intervals, a means for displaying suggested messages and activities received from the cloud server to the user, and a means for capturing the user's behavioral data and reactions to the suggestions and feeding them back to the cloud server. This enables the user's emotional state and communication patterns to be analyzed in real time, making it possible to provide timely and personalized suggestions. Furthermore, the accuracy of suggestions from the next time onwards can be improved based on the user's feedback, thereby more effectively repairing relationships and improving communication within the home.

[1332] A "smart mirror" is an electrical device that is installed in the home, has a built-in camera and display that captures the user's facial expressions, and processes facial image data and emotional state data.

[1333] The "camera" is a device built into the smart mirror that takes pictures of the user's face at regular intervals and acquires facial image data.

[1334] A "cloud server" is a remote computing system that receives and analyzes data sent from smart mirrors.

[1335] "Facial expression data" refers to image data of the user's face captured by a camera, and includes facial expression features.

[1336] "Emotional state data" is data that indicates the emotional state of the user analyzed by the emotion engine.

[1337] "Communication patterns" refer to a continuous flow of dialogue and behavior among multiple family members, including specific emotional and behavioral changes.

[1338] "Conflict" refers to the disagreements and discord that arise in the interactions and actions of family members.

[1339] "Messages and activities that promote relationship repair" are suggestions and action plans aimed at resolving conflicts and improving communication between family members.

[1340] "Displaying a suggested message" refers to the act of displaying a suggested message or activity received from the cloud server on the smart mirror's display.

[1341] "Behavioral data" is data indicating the actions that a user actually takes in response to a suggestion.

[1342] "Feedback" refers to sending the user's reaction and behavioral data to a cloud server to provide information to improve the accuracy of future suggestions.

[1343] An "emotion engine" is software or algorithm that analyzes facial image data to identify a user's emotional state in real time.

[1344] This invention is a system that uses a smart mirror installed in the home to facilitate communication between family members, detect conflicts, and support reconciliation. The system mainly consists of a smart mirror terminal, a cloud server, an emotion engine, and family users.

[1345] System Configuration

[1346] Smart mirror device: The smart mirror has a built-in camera and display. The camera captures facial expressions of family members and collects facial data. The display shows messages and activities received from the cloud server.

[1347] Cloud Server: The cloud server receives facial expression and emotional state data sent from the smart mirror and analyzes it using an AI model. The cloud server identifies the emotional state of family members, analyzes their communication patterns, and, if a conflict is detected, suggests messages or activities to promote relationship repair.

[1348] Emotion Engine: The emotion engine is software that analyzes facial image data captured by the camera in real time and identifies the emotional state. The identified emotional state data is temporarily stored locally and periodically sent to a cloud server.

[1349] Family users: Family users can review the suggested messages and activities displayed on their smart mirror and act on the suggestions if they feel it is appropriate.

[1350] Program processing

[1351] Facial Expression Capture and Emotion Recognition

[1352] 1. The camera on the smart mirror device takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[1353] 2. The emotion engine analyzes the facial image data and identifies the emotional state. This emotional state data is temporarily stored locally.

[1354] 3. At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server.

[1355] Facial Expression Analysis and Conflict Detection

[1356] 1. The cloud server analyzes the received facial image data and emotional state data. As a result of the analysis, detailed emotional states of family members are obtained.

[1357] 2. The server detects certain patterns, such as repeated angry facial expressions or long periods of silence, and identifies them as signs of conflict.

[1358] 3. If signs of conflict are detected, AI will dynamically tailor and generate messages and activities that promote relationship repair based on past behavioral data and emotional history.

[1359] Suggestion display and user response

[1360] 1. Display suggested messages and activities received from the cloud server on the smart mirror display.

[1361] 2. The user reviews the suggested messages and activities and acts on the suggestions if they feel appropriate.

[1362] Reactions and Feedback

[1363] 1. The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[1364] 2. The cloud server analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions.

[1365] Specific examples

[1366] Example 1: Resolving conflicts in the morning

[1367] 1. In the morning, the camera detects that Person A has a bad mood, and the emotion engine analyzes this to identify stress.

[1368] 2. The cloud server analyzes the facial expression data and emotional state and detects that Person A is feeling stressed.

[1369] 3. Based on past data, it is understood that Person A prefers the calm of the morning hours, and a message is generated saying, "Would you like to take a morning walk with me?"

[1370] 4. This message will be displayed on the smart mirror's display and suggested to Mr. A and other family members.

[1371] 5. Mr. A and the other family members accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[1372] Example 2: Resolving conflicts during the night

[1373] 1. At night, the camera detects that Person B and Person C are arguing, and the emotion engine analyzes this to identify anger.

[1374] 2. The cloud server analyzes the facial image data and emotional state and detects that both B and C are feeling angry.

[1375] 3. Recognizing that Mr. B and Mr. C share a common hobby of cooking, the system generates a suggestion: "Why don't we make a new recipe for dinner together?"

[1376] 4. The suggestion will appear on the smart mirror's display.

[1377] 5. When B and C accept the suggestion and make dinner together, the argument ends and their relationship is repaired.

[1378] In this way, the present invention is a system that effectively combines an emotion engine and an AI model to monitor communication within the home and make timely and personalized reconciliation suggestions.

[1379] Example prompts to input to the generative AI model

[1380] "Describe a system that uses a smart mirror to recognize family members' emotions and generate appropriate suggestions to facilitate communication."

[1381] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1382] Program processing flow

[1383] Facial Expression Capture and Emotion Recognition

[1384] Step 1:

[1385] Device: The camera built into the smart mirror takes pictures of the faces of family members. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data. The input is the face of the family member, and the output is facial image data. Specifically, the camera shutter operates and an image is captured.

[1386] Step 2:

[1387] Device: The captured facial image data is temporarily stored, and the emotion engine runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. The input is facial image data, and the output is emotional state data. Specifically, the emotion engine runs an image analysis algorithm to identify the emotional state.

[1388] Step 3:

[1389] Terminal: At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server. The input is the facial image data and emotional state data, and the output is the batch data sent to the server. Specifically, the network module operates and sends the data to the cloud server.

[1390] Facial Expression Analysis and Conflict Detection

[1391] Step 4:

[1392] Server: The cloud server analyzes the received facial image data and emotional state data. An AI model is used to obtain detailed emotional states of family members. The input is the batch data sent to the cloud server, and the output is the analyzed emotional state data. Specifically, the AI ​​model processes the data and identifies the emotional state.

[1393] Step 5:

[1394] Server: Based on the accumulated data, it analyzes communication patterns between multiple family members in real time. If a specific pattern is detected (such as repeated angry facial expressions or periods of silence), it recognizes this as a sign of conflict. The input is the analyzed emotional state data, and the output is conflict sign data. Specifically, it queries data from the database and executes a pattern matching algorithm.

[1395] Step 6:

[1396] Server: When signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities that promote relationship repair by referencing past behavioral data and emotional history. The input is conflict sign data and past behavioral data, and the output is a suggested message. Specifically, the suggestion generation algorithm is executed to create the suggested message.

[1397] Suggestion display and user response

[1398] Step 7:

[1399] Terminal: The smart mirror displays the suggested messages and activities received from the cloud server. The input is the suggested message, and the output is the suggested message displayed on the display. Specifically, the display controller operates and the message is displayed.

[1400] Step 8:

[1401] User: Checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. The input is the suggested message displayed, and the output is the user's actions. Specific actions include the user following the suggestions and taking actions such as watching a movie or going for a walk.

[1402] Reactions and Feedback

[1403] Step 9:

[1404] Terminal: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The input is the user's behavior, and the output is the behavioral data sent to the cloud server. Specifically, the camera operates again and the behavior is captured.

[1405] Step 10:

[1406] Server: Analyzes the received behavioral data, reaction, and emotional data, and uses the results to improve the accuracy of future proposals. The input is behavioral data and reaction data, and the output is the analysis results. Specifically, the data analysis algorithm is executed, and the results are stored in a database.

[1407] (Application example 2)

[1408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] There is a demand for technology that can facilitate smooth communication within the home, especially to support reconciliation after arguments. Furthermore, there is also a demand for technology that can grasp customer emotions in real time and improve the quality of service in brick-and-mortar stores. However, to meet these needs, an advanced system that integrates emotion recognition technology with an appropriate recommendation system is required.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1411] In this invention, the server includes a means for capturing facial expressions of family members using a smart mirror installed in the home and analyzing the data to identify their emotional states, a means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, a means for suggesting messages or activities to promote relationship repair when the occurrence of conflict is detected, and a means for staff wearing smart glasses in a physical store to capture customer facial expressions, analyze their emotional states, and display appropriate suggestions on the smart glasses, thereby improving communication within the home and customer satisfaction in physical stores.

[1412] A "smart mirror" is a mirror-shaped interactive device installed in the home that has a built-in camera and display, captures the user's facial expressions and movements, and displays information.

[1413] A "cloud server" is a server that stores and processes data via the Internet, and is a computer system that aggregates and analyzes data from multiple users.

[1414] An "emotion engine" is a software or hardware component for recognizing and analyzing a user's emotional state from captured image data or other data.

[1415] "Conflict occurrence" refers to friction or conflict in communication that arises between multiple members, and particularly refers to a state in which negative emotions such as anger or bad mood persist.

[1416] "Messages and activities that promote relationship repair" refer to suggestions and activities to improve communication and repair relationships based on emotional state and past behavioral data.

[1417] "Smart glasses" are wearable devices that incorporate a display and camera, allowing the wearer to acquire and display information in real time.

[1418] The "proposal system" is a system that dynamically generates optimal messages and activities for users based on captured data and analysis results.

[1419] The "service improvement system for physical stores" is a system that aims to analyze the emotional state of customers in physical stores in real time and provide services accordingly.

[1420] A "camera" is a photographing device for capturing images or videos of a user, and is used to identify facial expressions, etc.

[1421] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments. Another application example is a customer service improvement system that uses smart glasses worn by store staff in a brick-and-mortar store. The system of the present invention is composed of a smart mirror terminal, a cloud server, an emotion engine, smart glasses, and family users or store staff.

[1422] System Configuration

[1423] The smart mirror terminal has a built-in camera and display that captures the facial expressions of the user (family member or customer) and sends the data to a cloud server, while the smart glasses also have a built-in camera and display that transmits the captured facial expression data to a cloud server and displays service suggestions based on the analysis results.

[1424] Hardware and software used

[1425] Hardware: smart mirrors, cameras, smart glasses, webcams

[1426] Software: Python, OpenCV, data analysis software for cloud servers, local emotion recognition AI models (e.g., Flask servers)

[1427] Data processing and calculation

[1428] The server analyzes facial expression data sent from the smart mirror device or smart glasses to identify the user's emotional state. An emotion engine performs this analysis and identifies emotions such as confusion, satisfaction, or displeasure. Based on the identified emotions, the server then dynamically generates appropriate suggested messages and activities for family members or customers and displays them on the device's display.

[1429] Specifically, the cloud server performs the following data calculations:

[1430] 1. Facial expression data analysis: Using an AI model, captured facial expression data is analyzed in real time to identify the user's emotional state.

[1431] 2. Analysis of communication patterns: Analyze communication patterns between multiple family members to detect the occurrence of conflicts.

[1432] 3. Suggestion Generation: When a conflict is detected, messages and activities that promote relationship repair are dynamically generated based on the past behavioral data of family members. In physical stores, suggestions based on the customer's emotional state are displayed to store clerks wearing smart glasses.

[1433] Specific examples

[1434] Example 1: Conflict resolution within the family

[1435] Device: In the morning, the camera detects that family member A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[1436] Server: Analyzes facial expression data and emotional state to detect that A is feeling stressed.

[1437] Server: Based on past data, it understands that A prefers calm mornings and generates a suggestion: "Would you like to go for a morning walk together?"

[1438] Terminal: Display this message on the mirror's display and suggest it to A and other family members.

[1439] Example 2: Customer service in a brick-and-mortar store

[1440] Terminal: In the store, the camera in the smart glasses detects that Customer B has a confused expression.

[1441] Server: Analyzes facial image data and emotional state to detect that B is confused.

[1442] Server: Based on past data and analysis results, generate the optimal suggestion for B, "Do you need more information about this product?", and display it on the smart glasses.

[1443] Example of input prompt for generative AI model

[1444] "Do you need more information about this product?"

[1445] "Would you like to join me for a morning walk?"

[1446] This will facilitate smooth communication within homes and in physical stores, making it possible to build better relationships.

[1447] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1448] Step 1:

[1449] The device uses the smart mirror's built-in camera to capture images of family members' faces. Images are taken at regular intervals (e.g., once per second) and temporarily saved as facial image data.

[1450] Input: Facial expressions of family members

[1451] Output: Face image data

[1452] Step 2:

[1453] The device temporarily stores the captured facial image data, and the emotion engine also runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[1454] Input: Face image data

[1455] Output: Emotional state data

[1456] Step 3:

[1457] The device sends the captured facial image data and emotional state data to the cloud server in batch format at regular intervals (e.g., every minute). Each batch includes a user identifier and a timestamp.

[1458] Input: Facial image data, emotional state data

[1459] Output: Batch data sent to the cloud server

[1460] Step 4:

[1461] The server receives the facial image data and emotional state data and analyzes them using an AI model. The analysis results provide detailed emotional states of family members. This data is then stored in a database.

[1462] Input: Batch data

[1463] Output: Detailed emotional state data, stored in a database

[1464] Step 5:

[1465] The server analyzes communication patterns between multiple family members in real time based on the accumulated data, and when it detects certain patterns, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[1466] Input: Emotional state data stored in a database

[1467] Output: Conflict symptom data

[1468] Step 6:

[1469] If signs of conflict are detected, the server references the family members' past behavioral data and emotional history, and the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair.

[1470] Input: Conflict symptom data, past behavior data, emotional history

[1471] Output: Suggested messages and activities

[1472] Step 7:

[1473] The device receives suggested messages and activities from the cloud server and displays them on the mirror's display, where the suggestions are presented in an intuitive way for family members to understand.

[1474] Input: Suggested message or activity

[1475] Output: Messages and activity shown on the display

[1476] Step 8:

[1477] The user checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", the user takes the action of watching a movie together.

[1478] Input: Messages and activity shown on the display

[1479] Output: Family member behavior

[1480] Step 9:

[1481] The device again captures the user's behavioral data and reactions with its camera and sends them to the cloud server, where the emotion engine also analyzes them and records the emotional data.

[1482] Input: Family member behavior data, reactions

[1483] Output: Behavioral and emotional data sent to the cloud server

[1484] Step 10:

[1485] The server analyzes the received behavioral data, reactions, and emotional data, and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[1486] Input: Behavioral data, emotion data

[1487] Output: Data to improve the accuracy of proposals from next time onwards

[1488] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1489] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1490] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1491] [Fourth embodiment]

[1492] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1493] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1494] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1495] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1496] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1497] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1498] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1499] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1500] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1503] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1504] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1505] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after a fight. Specific embodiments of the system are described below.

[1506] System Configuration

[1507] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[1508] Program processing

[1509] Facial Capture

[1510] 1. Terminal

[1511] The camera built into the smart mirror captures the faces of family members at regular intervals (e.g., once per second). The captured facial expression data is temporarily stored locally.

[1512] 2. Terminal

[1513] Every certain time (for example, every minute), the captured facial expression data is sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[1514] facial expression analysis

[1515] 3. Server

[1516] The transmitted facial expression data is received and subjected to emotion analysis using AI. The analysis results indicate the emotional state of the family user (e.g., happiness, sadness, anger, etc.). This data is stored in a database.

[1517] Conflict Detection

[1518] 4. Server

[1519] The system analyzes communication patterns between family members from the accumulated facial expression data and behavior logs. When a specific pattern is detected, such as repeated angry expressions or prolonged silence, it recognizes the occurrence of a conflict.

[1520] 5. Server

[1521] When a conflict is detected, past behavioral data is used to generate suggestions to promote relationship repair, such as messages or activities like "Would you like to make dinner together?" or "Would you like to watch a movie together?"

[1522] Suggestion display

[1523] 6. Terminal

[1524] Suggested messages and activities from the cloud server are displayed on the screen, allowing the user to check them and take appropriate action.

[1525] Specific examples

[1526] Example 1: Resolving conflicts in the morning

[1527] Device: In the morning, the camera detects that Person A looks unhappy.

[1528] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[1529] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[1530] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[1531] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[1532] Example 2: Resolving conflicts during the night

[1533] Device: At night, the camera detects that B and C are arguing.

[1534] Server: Through analysis, it is determined that both B and C are feeling angry.

[1535] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[1536] Device: Show this suggestion in the mirror.

[1537] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[1538] In this way, the present invention is a system that monitors communication within the home and proposes reconciliation at appropriate times, thereby maintaining smooth relationships between family members.

[1539] The processing flow will be explained below.

[1540] Step 1:

[1541] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[1542] Step 2:

[1543] Device: The captured facial image data is sent to the cloud server as a batch at regular intervals (e.g., every minute). The batch also includes a user identifier and a timestamp.

[1544] Step 3:

[1545] Server: The received facial image data is input into the AI ​​model to analyze the emotional state of the family user. The analysis results are stored in an emotional state database. Emotional states are classified as happiness, anger, sadness, etc.

[1546] Step 4:

[1547] Server: Analyzes communication patterns between multiple family members in real time based on data stored in the emotional state database. When a specific pattern is detected (e.g., continuous angry facial expressions or silence), it recognizes this as a sign of conflict.

[1548] Step 5:

[1549] Server: If signs of conflict are detected, AI refers to the family members' past behavioral data and generates messages and activity suggestions to promote relationship repair. For example, "Would you like to watch a movie together?"

[1550] Step 6:

[1551] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display. The suggestions are presented in an intuitive way for the user.

[1552] Step 7:

[1553] User: Review suggested messages and activities displayed on the mirror and act on them if they feel it is appropriate. For example, in response to the suggested message "Want to watch a movie together?", they act on the suggestion to watch a movie together.

[1554] Step 8:

[1555] Device: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[1556] Step 9:

[1557] Server: Analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[1558] Through these steps, the smart mirror monitors communication between family members and makes reconciliation suggestions at the appropriate time, thereby maintaining smooth relationships within the home.

[1559] Example 1

[1560] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] Conflicts and arguments between family members are inevitable in domestic communication. However, if such conflicts are left unaddressed, family relationships are likely to deteriorate. In modern living environments, there is a need for a method to grasp family members' emotional states in real time and promote relationship repair at an appropriate time when conflicts occur. However, the lack of an automated system for detecting conflicts and proposing relationship repair makes it difficult to solve problems in individual households.

[1562] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1563] In this invention, the server includes means for analyzing the received facial expression data and identifying the emotional states of family members, means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, and means for suggesting messages or activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected. This makes it possible to quickly detect conflicts within the family and make suggestions to promote relationship repair at an appropriate time.

[1564] A "smart mirror" is a device that is installed in the home, equipped with a camera and a display, and used to capture and display the facial expressions and movements of family members.

[1565] The "photography device" is a camera built into the smart mirror, and is a means for capturing facial expression data of family members.

[1566] "Facial expression data" refers to data including facial features and expressions of family members captured by a photographing device.

[1567] A "cloud server" is a remote data center accessible via the Internet that provides analysis and storage functions for received data.

[1568] "Emotional state" refers to an individual's psychological state or mood (e.g., happiness, sadness, anger, etc.) analyzed based on facial expression data.

[1569] "Communication patterns" are data that indicate tendencies and regularities in the dialogue and behavior of family members.

[1570] "Conflict" is a state of discord caused by disputes or differences of opinion that arise between family members.

[1571] A "relationship repair message" is a message that includes suggestions or instructions for actions to promote the improvement of relationships between family members when the occurrence of a conflict is detected.

[1572] A "display device" is a display that is integrated into the smart mirror to visually present messages and activities to family members.

[1573] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and generates appropriate suggestions and responses to specific questions or requests.

[1574] A "prompt" is an instruction that is input into a generative AI model to generate an appropriate response or suggestion.

[1575] The present invention is a system that utilizes a smart mirror used at home to facilitate communication between family members and support the repair of relationships after conflicts or arguments. Specific embodiments of the system are described below.

[1576] System Configuration

[1577] This system consists of a smart mirror device, a cloud server, and family users. The smart mirror device is equipped with a camera and a display, and operates in conjunction with the cloud server.

[1578] Hardware and software used

[1579] 1. Hardware

[1580] Smart mirror (with built-in camera and display)

[1581] Local Storage

[1582] 2. Software

[1583] Facial expression analysis AI (Google Face API)

[1584] Cloud infrastructure (AWS)

[1585] Database (MySQL)

[1586] Relationship repair suggestion AI model (OpenAI GPT-3)

[1587] Periodic job execution environment (Python)

[1588] Program processing

[1589] Facial Capture

[1590] When a user stands in front of the smart mirror, the device uses its built-in camera to capture a picture of the family user's face. The picture is taken at regular intervals (e.g., once per second), and the resulting facial expression data is temporarily stored in local storage. This facial expression data includes the user's identifier and a timestamp of when the picture was taken. Every regular interval (e.g., every minute), the device sends this facial expression data in batch format to a cloud server.

[1591] facial expression analysis

[1592] The server receives the facial expression data sent from the device. This data is securely received using cloud infrastructure such as AWS. The server performs emotion analysis on the received facial expression data using an AI model such as Google Face API to obtain the emotional state of each family user. The analysis results are stored in a database such as MySQL, and the data is organized by user and chronological order.

[1593] Conflict Detection

[1594] The server periodically analyzes the accumulated facial expression data and behavior logs in the database. This analysis uses Python to detect patterns of consecutive angry expressions or silence. If certain conditions are met (for example, five consecutive angry expressions), the server recognizes the conflict.

[1595] Proposal Generation and Display

[1596] When a conflict is detected, the server generates suggestions to promote relationship repair based on past behavioral data. These suggestions are generated using generative AI models such as OpenAI's GPT-3, and include specific activities (e.g., "Would you like to cook dinner together?" or "Would you like to watch a movie together?"). The generated suggestion message is sent to the device via the cloud, where it can be displayed on the device's display. The user can review the suggestion and take appropriate action.

[1597] Specific examples

[1598] Resolving conflicts in the morning

[1599] User: Person A stands in front of the smart mirror with a displeased look on his face.

[1600] Device: The camera captures A's facial expression, and the device sends the data to the cloud.

[1601] Server: Analyzes the received data using AI and detects that Person A is feeling stressed.

[1602] Server: Generates a suggestion, "Would you like to go for a morning walk together?" and sends it to the smart mirror.

[1603] Terminal: Show suggestions on the display.

[1604] User: Person A reviews the suggestion and goes for a walk with other family members.

[1605] Resolving conflicts during the night

[1606] User: The camera detects that B and C are arguing.

[1607] Device: The camera captures facial expression data and sends it to the cloud.

[1608] Server: Through data analysis, it detects that both B and C are angry.

[1609] Server: Generate a suggestion such as "Would you like to make a new recipe for dinner together?" and send it to the device.

[1610] Terminal: Show suggestions on the display.

[1611] User: B and C accept the suggestion and start making dinner together.

[1612] Example prompts for generative AI models

[1613] "Please suggest ways to improve communication between family members. Specifically, what actions should users take when they are feeling stressed?"

[1614] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1615] Step 1:

[1616] The user stands in front of the smart mirror, and the smart mirror's built-in camera captures the user's face.

[1617] Input: Face image of a family member

[1618] Output: Photographed facial expression data

[1619] Step 2:

[1620] The terminal uses a camera to take pictures of the faces of family users at regular intervals (for example, once per second) and acquires facial expression data.

[1621] Specific operation: The captured facial expression data is temporarily stored in local storage. The data includes the user's identifier and a timestamp.

[1622] Input: Face image of a family member

[1623] Output: Facial expression data temporarily saved in local storage

[1624] Step 3:

[1625] The device sends the temporarily stored facial expression data to the cloud server in batch format at regular intervals (for example, every minute).

[1626] Specific operation: The facial expression data is assigned a user identifier and a timestamp and sent to the cloud server.

[1627] Input: Facial expression data stored in local storage

[1628] Output: Facial expression data sent to the cloud server

[1629] Step 4:

[1630] The server receives the facial expression data transmitted from the terminal.

[1631] Specific operation: Facial expression data is securely received using cloud infrastructure such as AWS.

[1632] Input: Facial expression data sent from the device

[1633] Output: Received facial expression data

[1634] Step 5:

[1635] The server uses an AI model (e.g., Google Face API) to analyze the emotion of the received facial expression data.

[1636] Specific operation: Analyzes facial expression data and obtains the emotional state of family users (e.g., happiness, sadness, anger, etc.).

[1637] Input: Received facial expression data

[1638] Output: Parsed emotional state data

[1639] Step 6:

[1640] The server stores the analysis results in a database (e.g., MySQL).

[1641] Specific operation: Analyzed emotional state data is stored in a database for each user and for each time series.

[1642] Input: Parsed emotional state data

[1643] Output: Emotional state data stored in a database

[1644] Step 7:

[1645] The server periodically analyzes the facial expression data and behavior logs stored in the database.

[1646] What it does: A periodic job using Python detects patterns of successive angry expressions and silences.

[1647] Input: Facial expression data stored in a database

[1648] Output: Detected communication patterns

[1649] Step 8:

[1650] The server recognizes the occurrence of a conflict when certain conditions are met (for example, five consecutive angry facial expressions).

[1651] Specific behavior: Determine whether a conflict has occurred based on detected communication patterns.

[1652] Input: Detected communication patterns

[1653] Output: Conflict occurrence determination

[1654] Step 9:

[1655] When conflict is detected, the server generates suggestions based on past behavioral data to promote relationship repair.

[1656] Specific actions: Use a generative AI model (e.g., OpenAI's GPT-3) to generate messages that involve specific activities (e.g., "Would you like to make dinner together?" or "Would you like to watch a movie together?").

[1657] Input: Conflict occurrence determination, past behavior data

[1658] Output: Proposal message

[1659] Step 10:

[1660] The server sends the generated proposal message to the terminal via the cloud.

[1661] Specific operation: A proposal message is sent to the device via the cloud infrastructure.

[1662] Input: Proposal message

[1663] Output: Proposal message sent to the terminal

[1664] Step 11:

[1665] The terminal displays the proposal message received from the cloud server on the display.

[1666] Specific behavior: Suggestions are displayed in large letters and icons for easy viewing by family users.

[1667] Input: The proposal message received

[1668] Output: Suggestion message shown on the display

[1669] Step 12:

[1670] The user checks the suggestion message and takes appropriate action. For example, if they want to cook together, the user starts a conversation in the kitchen.

[1671] Specific Action: Act on the suggested activities to improve communication.

[1672] Input: Suggestion message shown on the display

[1673] Output: Improved communication through appropriate actions

[1674] (Application example 1)

[1675] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1676] Conventional home communication support systems lack the ability to accurately detect emotions and conflicts among family members and propose appropriate solutions, which can lead to unnecessary friction and misunderstandings. Furthermore, the system's overall accuracy and effectiveness are limited because it is unable to effectively collect information on how family members respond to proposed activities and messages and reflect this information in future proposals.

[1677] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1678] In this invention, the server includes a device that generates specific execution examples in response to suggestions displayed on a home communication terminal, a smart mirror installed in the home and having an imaging device that captures the facial expressions of family members and a device that temporarily stores the facial expression data captured by the imaging device, a device that transmits the facial expression data to a remote server, means for analyzing the facial expression data received by the remote server and identifying the emotional states of the family members, a device that analyzes communication patterns between multiple family members using the remote server and detects the occurrence of conflict, a device that generates suggestions and activities to promote relationship repair based on past behavioral data of the family members when the occurrence of conflict is detected, a system having a display device that displays the suggestions and activities, and a device that shares suggestion information with the home communication terminal and generates specific execution examples in response to the suggestions displayed on the home communication terminal.

[1679] This makes it possible to suggest specific activities and messages at the appropriate time when conflicts arise within the family, and to collect responses to these suggestions in order to improve the accuracy of future suggestions.

[1680] The "imaging device" is a device for capturing images of the facial expressions of family members.

[1681] The "remote server" is a server that analyzes data sent from smart mirrors installed in the home and identifies the emotional states of family members.

[1682] "Facial expression data" refers to data on the facial expressions of family members captured by an imaging device installed in the smart mirror.

[1683] "Communication patterns" refer to patterns of conversation and behavior between multiple family members, and are data that can be analyzed to detect the occurrence of conflicts.

[1684] "Conflict" refers to a fight or disagreement between family members and is important for analysis to detect this occurrence.

[1685] "Relationship repair" refers to the process of resolving conflicts and misunderstandings between family members and rebuilding positive relationships.

[1686] "Suggestions and actions" are specific actions and messages to family members to promote relationship repair, which are generated by the remote server when a conflict is detected.

[1687] "Display Device" refers to a display installed on the smart mirror to visually communicate suggestions and activities to family members.

[1688] "Home communication terminal" refers to a device such as a smartphone used by a family member, which works in conjunction with the smart mirror to share suggested information.

[1689] The "specific execution examples" are detailed instructions or examples for actually encouraging actions in response to the suggestions displayed on the home communication terminal by the remote server.

[1690] This invention uses a system that combines a smart mirror installed in the home, a cloud server, and a home communication terminal to analyze communication between family members and support relationship repair.

[1691] System Configuration

[1692] The system consists of the following equipment:

[1693] 1. Smart mirror terminal:

[1694] Image capture device: Captures the facial expressions of family members and temporarily stores the facial expression data.

[1695] Display device: Displays the suggestions and activities sent from the cloud server.

[1696] 2. Cloud Server:

[1697] Receives facial expression data and performs emotion analysis.

[1698] Analyze communication patterns between multiple family members to detect conflicts.

[1699] When conflict is detected, past behavioral data is used to generate suggestions and actions to promote relationship repair.

[1700] The proposal information is sent to a home communication terminal, and a specific execution example is generated.

[1701] 3. Home communication devices:

[1702] The proposed information transmitted from the cloud server is received and displayed on the home communication terminal.

[1703] Data Processing

[1704] 1. Server:

[1705] Facial expression data captured by an imaging device is analyzed to identify the emotional states of family members.

[1706] The communication patterns between multiple family members are analyzed based on the received facial expression data and past data.

[1707] Detect specific patterns and recognize conflicts as they occur.

[1708] 2. Terminal:

[1709] Suggestions and activity information sent from the cloud server are displayed on the smart mirror's display device.

[1710] Based on the specific execution examples displayed on the home communication terminal, the user acts in accordance with the suggestions.

[1711] Specific examples

[1712] Below is a specific example of how this system can be used.

[1713] Example 1: Morning conflict resolution

[1714] Device: In the morning, the camera detects that Mr. A has a sullen expression.

[1715] Server: Analyzes facial expression data and detects that Person A is feeling stressed.

[1716] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[1717] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[1718] User: If A and the rest of the family accept the suggestion and go for a morning walk together, stress will decrease and communication will improve.

[1719] Example 2: Nighttime conflict resolution

[1720] Device: At night, the imaging device detects that Mr. B and Mr. C are arguing.

[1721] Server: Through analysis, it is determined that both B and C are feeling angry.

[1722] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[1723] Device: Show this suggestion in the mirror.

[1724] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[1725] Example prompts for generative AI models

[1726] "Analyze the emotional state of family members and make reconciliation suggestions at the appropriate time. Generate specific suggestions based on the following information:

[1727] 1. User facial expression data

[1728] 2. User Timestamp

[1729] 3. Past communication patterns

[1730] Examples of suggestions: 'Would you like to go for a morning walk with me?' 'Would you like to make a new recipe for dinner with me?'

[1731] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1732] Step 1:

[1733] The imaging device of the smart mirror terminal captures the facial expressions of family members at regular intervals (for example, once per second). This captured data is temporarily saved locally. The input is image data captured in real time, and the output is a batch file containing the captured image data. The specific operation of this step is that the imaging device recognizes faces and captures the data as images.

[1734] Step 2:

[1735] The device sends the temporarily stored facial expression data to the cloud server in batch format. The user identifier and timestamp are attached to the transmission. The input is the batch file generated in step 1, and the output is the data sent to the cloud server. The specific operation of this step is to periodically trigger batch processing and send the facial expression data to the cloud server via an HTTP request.

[1736] Step 3:

[1737] The server receives the transmitted facial expression data and performs emotion analysis using a generative AI model. The input is a batch file containing facial expression data of family members, and the output is data indicating their emotional state. Specifically, in this step, the AI ​​model analyzes each image and generates an emotion label.

[1738] Step 4:

[1739] The server stores the analyzed emotional data in a database and further analyzes the communication patterns between family members. The input is the emotional data generated in step 3, and the output is the analysis results of the communication patterns. The specific operation of this step is for the server to compare past data with current data and detect specific conflict patterns.

[1740] Step 5:

[1741] When the server detects the occurrence of a conflict, it generates proposals and activities to promote relationship repair based on past behavioral data. The input is the analysis results of communication patterns and past behavioral data, and the output is a message of the proposal or activity. The specific operation of this step is that the server executes a proposal example generation algorithm to create appropriate proposals.

[1742] Step 6:

[1743] The proposal or activity is sent from the cloud server to the home communication device and displayed on the device. The input is the proposal or activity message generated in step 5, and the output is the proposal or activity displayed on the home communication device. The specific operation of this step is that the proposal message is displayed on the screen of the home communication device and the user is notified.

[1744] Step 7:

[1745] The user performs the action suggested, and their reaction is captured again by the smart mirror's imaging device. The input is the user's action and facial expression data at that time, and the output is new facial expression data sent to the cloud server. Specifically, this step involves capturing the user's facial expression again after the action and sending it to the server for analysis.

[1746] These steps enable the system to detect conflicts within the family and make timely suggestions to repair the relationship.

[1747] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1748] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments, and by combining it with an emotion engine that recognizes the user's emotional state, it makes more accurate suggestions. Specific embodiments of the system are described below.

[1749] System Configuration

[1750] This system consists of a smart mirror device, a cloud server, an emotion engine, and family users. The smart mirror device is equipped with a camera and a display, and works in conjunction with the emotion engine and the cloud server.

[1751] Program processing

[1752] Facial Expression Capture and Emotion Recognition

[1753] 1. Terminal

[1754] The camera built into the smart mirror takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[1755] 2. Terminal

[1756] The captured facial image data is temporarily stored, and the emotion engine is also activated. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[1757] 3. Terminal

[1758] Every certain time (e.g., every minute), the captured facial image data and emotional state data are sent to the cloud server in a batch format, which includes a user identifier and a timestamp.

[1759] Facial Expression Analysis and Conflict Detection

[1760] 4. Server

[1761] The facial image data and emotional state data are received and analyzed using an AI model. The analysis results provide detailed information about the emotional state of the family user. This data is then stored in a database.

[1762] 5. Server

[1763] Based on the accumulated data, the system analyzes communication patterns between multiple family members in real time. When certain patterns are detected, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[1764] 6. Server

[1765] If signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair by referencing the family user's past behavioral data and emotional history. For example, specific suggestions such as "Would you like to go see a movie together?" are generated.

[1766] Suggestion display and user response

[1767] 7. Terminal

[1768] Suggested messages and activities received from the cloud server are displayed on the mirror's display, where the suggestions are presented in a way that is intuitive to the user.

[1769] 8. Users

[1770] Review suggested messages and activities displayed in the mirror and act on them if you feel it is appropriate. For example, in response to the suggestion "Would you like to watch a movie together?", you can take the action of watching a movie together.

[1771] Reactions and Feedback

[1772] 9. Terminal

[1773] The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The emotion engine also analyzes the data and records the emotional data.

[1774] 10. Server

[1775] The received behavioral data, reactions, and emotional data are analyzed, and the results are used to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[1776] Specific examples

[1777] Example 1: Resolving conflicts in the morning

[1778] Device: In the morning, the camera detects that Person A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[1779] Server: Analyzes facial expression data and emotional state to detect that Person A is feeling stressed.

[1780] Server: Based on past data, it understands that Person A prefers the calm of the morning, and generates a message saying, "Would you like to take a morning walk with me?"

[1781] Device: Display this message on the mirror and suggest it to Mr. A and other family members.

[1782] User: Mr. A and the rest of the family accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[1783] Example 2: Resolving conflicts during the night

[1784] Device: At night, the camera detects that B and C are arguing, and the emotion engine analyzes this to identify anger.

[1785] Server: Analyzes facial image data and emotional states and detects that Person B and Person C are feeling angry.

[1786] Server: Recognizes that Persons B and C share a common hobby of cooking and generates a suggestion: "Would you like to make a new recipe for dinner together?"

[1787] Device: Show this suggestion in the mirror.

[1788] User: B and C accept the suggestion and make dinner together, which ends the argument and repairs their relationship.

[1789] In this way, by combining an emotion engine, the present invention is a system that can monitor communication within the home more precisely and make personalized reconciliation suggestions at the appropriate time.

[1790] The processing flow will be explained below.

[1791] Step 1:

[1792] Device: The built-in camera takes pictures of the faces of family members at home. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data.

[1793] Step 2:

[1794] Device: Facial image data captured by the camera is sent in real time to the emotion engine, which analyzes this data and identifies the user's emotional state.

[1795] Step 3:

[1796] Device: Temporarily stores the user's emotional state data (e.g., happiness, anger, sadness, etc.) obtained from the emotion engine locally.

[1797] Step 4:

[1798] Device: At regular intervals (e.g., every minute), the device sends facial image data and emotional state data in batches to the cloud server. Each batch contains a user identifier and a timestamp.

[1799] Step 5:

[1800] Server: Receives the facial image data and emotional state data and analyzes them again using an AI model. As a result of the analysis, detailed emotional states are obtained and stored in a database.

[1801] Step 6:

[1802] Server: Analyzes communication patterns between multiple family members in real time based on the emotional state database. When certain patterns are detected (e.g., repeated angry facial expressions or prolonged silence), they are recognized as signs of conflict.

[1803] Step 7:

[1804] Server: When signs of conflict are detected, the AI ​​refers to the family user's past behavioral and emotional history data and generates messages and activity suggestions to promote relationship repair, such as "Would you like to go see a movie together?"

[1805] Step 8:

[1806] Device: Suggested messages and activities received from the cloud server are displayed on the mirror's display in a format that is intuitive and easy for the user to understand.

[1807] Step 9:

[1808] User: Review the suggested messages and activities displayed on the mirror and act on the suggestions that they feel are appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", they act on the suggestion to watch a movie together.

[1809] Step 10:

[1810] On the device: The user's behavioral data and reactions are captured again by the camera and analyzed by the emotion engine. Emotional changes are tracked and data is generated.

[1811] Step 11:

[1812] Device: Captured and analyzed reaction and emotion data is sent to a cloud server, allowing for real-time updates and analysis.

[1813] Step 12:

[1814] Server: Analyzes the received behavioral data, reaction and emotional data, and uses the results to improve the accuracy of future suggestions, resulting in more personalized suggestions.

[1815] Through these steps, the smart mirror monitors communication between family members and provides personalized reconciliation suggestions at the right time, thereby maintaining smooth relationships within the home.

[1816] Example 2

[1817] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1818] There is a demand for smart mirror systems as a means of facilitating smoother communication within the home. However, current technology lacks the ability to adequately recognize emotions and analyze communication patterns, making it difficult to detect conflicts early or propose appropriate reconciliation strategies. Furthermore, feedback functions to improve the accuracy of suggested messages are not adequately implemented, making it difficult to provide optimal suggestions tailored to each individual household. As a result, repairing relationships and improving communication within the home has not been fully achieved.

[1819] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1820] In this invention, the server includes a means for transmitting emotional state data to a cloud server at regular intervals, a means for displaying suggested messages and activities received from the cloud server to the user, and a means for capturing the user's behavioral data and reactions to the suggestions and feeding them back to the cloud server. This enables the user's emotional state and communication patterns to be analyzed in real time, making it possible to provide timely and personalized suggestions. Furthermore, the accuracy of suggestions from the next time onwards can be improved based on the user's feedback, thereby more effectively repairing relationships and improving communication within the home.

[1821] A "smart mirror" is an electrical device that is installed in the home, has a built-in camera and display that captures the user's facial expressions, and processes facial image data and emotional state data.

[1822] The "camera" is a device built into the smart mirror that takes pictures of the user's face at regular intervals and acquires facial image data.

[1823] A "cloud server" is a remote computing system that receives and analyzes data sent from smart mirrors.

[1824] "Facial expression data" refers to image data of the user's face captured by a camera, and includes facial expression features.

[1825] "Emotional state data" is data that indicates the emotional state of the user analyzed by the emotion engine.

[1826] "Communication patterns" refer to a continuous flow of dialogue and behavior among multiple family members, including specific emotional and behavioral changes.

[1827] "Conflict" refers to the disagreements and discord that arise in the interactions and actions of family members.

[1828] "Messages and activities that promote relationship repair" are suggestions and action plans aimed at resolving conflicts and improving communication between family members.

[1829] "Displaying a suggested message" refers to the act of displaying a suggested message or activity received from the cloud server on the smart mirror's display.

[1830] "Behavioral data" is data indicating the actions that a user actually takes in response to a suggestion.

[1831] "Feedback" refers to sending the user's reaction and behavioral data to a cloud server to provide information to improve the accuracy of future suggestions.

[1832] An "emotion engine" is software or algorithm that analyzes facial image data to identify a user's emotional state in real time.

[1833] This invention is a system that uses a smart mirror installed in the home to facilitate communication between family members, detect conflicts, and support reconciliation. The system mainly consists of a smart mirror terminal, a cloud server, an emotion engine, and family users.

[1834] System Configuration

[1835] Smart mirror device: The smart mirror has a built-in camera and display. The camera captures facial expressions of family members and collects facial data. The display shows messages and activities received from the cloud server.

[1836] Cloud Server: The cloud server receives facial expression and emotional state data sent from the smart mirror and analyzes it using an AI model. The cloud server identifies the emotional state of family members, analyzes their communication patterns, and, if a conflict is detected, suggests messages or activities to promote relationship repair.

[1837] Emotion Engine: The emotion engine is software that analyzes facial image data captured by the camera in real time and identifies the emotional state. The identified emotional state data is temporarily stored locally and periodically sent to a cloud server.

[1838] Family users: Family users can review the suggested messages and activities displayed on their smart mirror and act on the suggestions if they feel it is appropriate.

[1839] Program processing

[1840] Facial Expression Capture and Emotion Recognition

[1841] 1. The camera on the smart mirror device takes pictures of the faces of family members at regular intervals (e.g., once per second) and saves the images as facial image data.

[1842] 2. The emotion engine analyzes the facial image data and identifies the emotional state. This emotional state data is temporarily stored locally.

[1843] 3. At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server.

[1844] Facial Expression Analysis and Conflict Detection

[1845] 1. The cloud server analyzes the received facial image data and emotional state data. As a result of the analysis, detailed emotional states of family members are obtained.

[1846] 2. The server detects certain patterns, such as repeated angry facial expressions or long periods of silence, and identifies them as signs of conflict.

[1847] 3. If signs of conflict are detected, AI will dynamically tailor and generate messages and activities that promote relationship repair based on past behavioral data and emotional history.

[1848] Suggestion display and user response

[1849] 1. Display suggested messages and activities received from the cloud server on the smart mirror display.

[1850] 2. The user reviews the suggested messages and activities and acts on the suggestions if they feel appropriate.

[1851] Reactions and Feedback

[1852] 1. The user's behavioral data and reactions are captured again by the camera and sent to the cloud server.

[1853] 2. The cloud server analyzes the received behavioral data and reactions and uses the results to improve the accuracy of future suggestions.

[1854] Specific examples

[1855] Example 1: Resolving conflicts in the morning

[1856] 1. In the morning, the camera detects that Person A has a bad mood, and the emotion engine analyzes this to identify stress.

[1857] 2. The cloud server analyzes the facial expression data and emotional state and detects that Person A is feeling stressed.

[1858] 3. Based on past data, it is understood that Person A prefers the calm of the morning hours, and a message is generated saying, "Would you like to take a morning walk with me?"

[1859] 4. This message will be displayed on the smart mirror's display and suggested to Mr. A and other family members.

[1860] 5. Mr. A and the other family members accept the suggestion and go for a morning walk together, which reduces stress and improves communication.

[1861] Example 2: Resolving conflicts during the night

[1862] 1. At night, the camera detects that Person B and Person C are arguing, and the emotion engine analyzes this to identify anger.

[1863] 2. The cloud server analyzes the facial image data and emotional state and detects that both B and C are feeling angry.

[1864] 3. Recognizing that Mr. B and Mr. C share a common hobby of cooking, the system generates a suggestion: "Why don't we make a new recipe for dinner together?"

[1865] 4. The suggestion will appear on the smart mirror's display.

[1866] 5. When B and C accept the suggestion and make dinner together, the argument ends and their relationship is repaired.

[1867] In this way, the present invention is a system that effectively combines an emotion engine and an AI model to monitor communication within the home and make timely and personalized reconciliation suggestions.

[1868] Example prompts to input to the generative AI model

[1869] "Describe a system that uses a smart mirror to recognize family members' emotions and generate appropriate suggestions to facilitate communication."

[1870] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1871] Program processing flow

[1872] Facial Expression Capture and Emotion Recognition

[1873] Step 1:

[1874] Device: The camera built into the smart mirror takes pictures of the faces of family members. The pictures are taken at regular intervals (e.g., once per second) and saved as facial image data. The input is the face of the family member, and the output is facial image data. Specifically, the camera shutter operates and an image is captured.

[1875] Step 2:

[1876] Device: The captured facial image data is temporarily stored, and the emotion engine runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. The input is facial image data, and the output is emotional state data. Specifically, the emotion engine runs an image analysis algorithm to identify the emotional state.

[1877] Step 3:

[1878] Terminal: At regular intervals (e.g., every minute), the captured facial image data and emotional state data are sent in batch format to the cloud server. The input is the facial image data and emotional state data, and the output is the batch data sent to the server. Specifically, the network module operates and sends the data to the cloud server.

[1879] Facial Expression Analysis and Conflict Detection

[1880] Step 4:

[1881] Server: The cloud server analyzes the received facial image data and emotional state data. An AI model is used to obtain detailed emotional states of family members. The input is the batch data sent to the cloud server, and the output is the analyzed emotional state data. Specifically, the AI ​​model processes the data and identifies the emotional state.

[1882] Step 5:

[1883] Server: Based on the accumulated data, it analyzes communication patterns between multiple family members in real time. If a specific pattern is detected (such as repeated angry facial expressions or periods of silence), it recognizes this as a sign of conflict. The input is the analyzed emotional state data, and the output is conflict sign data. Specifically, it queries data from the database and executes a pattern matching algorithm.

[1884] Step 6:

[1885] Server: When signs of conflict are detected, the AI ​​dynamically adjusts and generates suggested messages and activities that promote relationship repair by referencing past behavioral data and emotional history. The input is conflict sign data and past behavioral data, and the output is a suggested message. Specifically, the suggestion generation algorithm is executed to create the suggested message.

[1886] Suggestion display and user response

[1887] Step 7:

[1888] Terminal: The smart mirror displays the suggested messages and activities received from the cloud server. The input is the suggested message, and the output is the suggested message displayed on the display. Specifically, the display controller operates and the message is displayed.

[1889] Step 8:

[1890] User: Checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. The input is the suggested message displayed, and the output is the user's actions. Specific actions include the user following the suggestions and taking actions such as watching a movie or going for a walk.

[1891] Reactions and Feedback

[1892] Step 9:

[1893] Terminal: The user's behavioral data and reactions are captured again by the camera and sent to the cloud server. The input is the user's behavior, and the output is the behavioral data sent to the cloud server. Specifically, the camera operates again and the behavior is captured.

[1894] Step 10:

[1895] Server: Analyzes the received behavioral data, reaction, and emotional data, and uses the results to improve the accuracy of future proposals. The input is behavioral data and reaction data, and the output is the analysis results. Specifically, the data analysis algorithm is executed, and the results are stored in a database.

[1896] (Application example 2)

[1897] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1898] There is a demand for technology that can facilitate smooth communication within the home, especially to support reconciliation after arguments. Furthermore, there is also a demand for technology that can grasp customer emotions in real time and improve the quality of service in brick-and-mortar stores. However, to meet these needs, an advanced system that integrates emotion recognition technology with an appropriate recommendation system is required.

[1899] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1900] In this invention, the server includes a means for capturing facial expressions of family members using a smart mirror installed in the home and analyzing the data to identify their emotional states, a means for analyzing communication patterns between multiple family members and detecting the occurrence of conflict, a means for suggesting messages or activities to promote relationship repair when the occurrence of conflict is detected, and a means for staff wearing smart glasses in a physical store to capture customer facial expressions, analyze their emotional states, and display appropriate suggestions on the smart glasses, thereby improving communication within the home and customer satisfaction in physical stores.

[1901] A "smart mirror" is a mirror-shaped interactive device installed in the home that has a built-in camera and display, captures the user's facial expressions and movements, and displays information.

[1902] A "cloud server" is a server that stores and processes data via the Internet, and is a computer system that aggregates and analyzes data from multiple users.

[1903] An "emotion engine" is a software or hardware component for recognizing and analyzing a user's emotional state from captured image data or other data.

[1904] "Conflict occurrence" refers to friction or conflict in communication that arises between multiple members, and particularly refers to a state in which negative emotions such as anger or bad mood persist.

[1905] "Messages and activities that promote relationship repair" refer to suggestions and activities to improve communication and repair relationships based on emotional state and past behavioral data.

[1906] "Smart glasses" are wearable devices that incorporate a display and camera, allowing the wearer to acquire and display information in real time.

[1907] The "proposal system" is a system that dynamically generates optimal messages and activities for users based on captured data and analysis results.

[1908] The "service improvement system for physical stores" is a system that aims to analyze the emotional state of customers in physical stores in real time and provide services accordingly.

[1909] A "camera" is a photographing device for capturing images or videos of a user, and is used to identify facial expressions, etc.

[1910] The present invention is a system that uses a smart mirror installed in the home to facilitate communication between family members and support reconciliation after arguments. Another application example is a customer service improvement system that uses smart glasses worn by store staff in a brick-and-mortar store. The system of the present invention is composed of a smart mirror terminal, a cloud server, an emotion engine, smart glasses, and family users or store staff.

[1911] System Configuration

[1912] The smart mirror terminal has a built-in camera and display that captures the facial expressions of the user (family member or customer) and sends the data to a cloud server, while the smart glasses also have a built-in camera and display that transmits the captured facial expression data to a cloud server and displays service suggestions based on the analysis results.

[1913] Hardware and software used

[1914] Hardware: smart mirrors, cameras, smart glasses, webcams

[1915] Software: Python, OpenCV, data analysis software for cloud servers, local emotion recognition AI models (e.g., Flask servers)

[1916] Data processing and calculation

[1917] The server analyzes facial expression data sent from the smart mirror device or smart glasses to identify the user's emotional state. An emotion engine performs this analysis and identifies emotions such as confusion, satisfaction, or displeasure. Based on the identified emotions, the server then dynamically generates appropriate suggested messages and activities for family members or customers and displays them on the device's display.

[1918] Specifically, the cloud server performs the following data calculations:

[1919] 1. Facial expression data analysis: Using an AI model, captured facial expression data is analyzed in real time to identify the user's emotional state.

[1920] 2. Analysis of communication patterns: Analyze communication patterns between multiple family members to detect the occurrence of conflicts.

[1921] 3. Suggestion Generation: When a conflict is detected, messages and activities that promote relationship repair are dynamically generated based on the past behavioral data of family members. In physical stores, suggestions based on the customer's emotional state are displayed to store clerks wearing smart glasses.

[1922] Specific examples

[1923] Example 1: Conflict resolution within the family

[1924] Device: In the morning, the camera detects that family member A has a grumpy expression, and the emotion engine analyzes this to identify stress.

[1925] Server: Analyzes facial expression data and emotional state to detect that A is feeling stressed.

[1926] Server: Based on past data, it understands that A prefers calm mornings and generates a suggestion: "Would you like to go for a morning walk together?"

[1927] Terminal: Display this message on the mirror's display and suggest it to A and other family members.

[1928] Example 2: Customer service in a brick-and-mortar store

[1929] Terminal: In the store, the camera in the smart glasses detects that Customer B has a confused expression.

[1930] Server: Analyzes facial image data and emotional state to detect that B is confused.

[1931] Server: Based on past data and analysis results, generate the optimal suggestion for B, "Do you need more information about this product?", and display it on the smart glasses.

[1932] Example of input prompt for generative AI model

[1933] "Do you need more information about this product?"

[1934] "Would you like to join me for a morning walk?"

[1935] This will facilitate smooth communication within homes and in physical stores, making it possible to build better relationships.

[1936] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1937] Step 1:

[1938] The device uses the smart mirror's built-in camera to capture images of family members' faces. Images are taken at regular intervals (e.g., once per second) and temporarily saved as facial image data.

[1939] Input: Facial expressions of family members

[1940] Output: Face image data

[1941] Step 2:

[1942] The device temporarily stores the captured facial image data, and the emotion engine also runs simultaneously. The emotion engine analyzes facial expressions in real time and identifies the user's emotional state. This emotional state data is temporarily stored locally.

[1943] Input: Face image data

[1944] Output: Emotional state data

[1945] Step 3:

[1946] The device sends the captured facial image data and emotional state data to the cloud server in batch format at regular intervals (e.g., every minute). Each batch includes a user identifier and a timestamp.

[1947] Input: Facial image data, emotional state data

[1948] Output: Batch data sent to the cloud server

[1949] Step 4:

[1950] The server receives the facial image data and emotional state data and analyzes them using an AI model. The analysis results provide detailed emotional states of family members. This data is then stored in a database.

[1951] Input: Batch data

[1952] Output: Detailed emotional state data, stored in a database

[1953] Step 5:

[1954] The server analyzes communication patterns between multiple family members in real time based on the accumulated data, and when it detects certain patterns, such as repeated angry expressions or periods of silence, it recognizes these as signs of conflict.

[1955] Input: Emotional state data stored in a database

[1956] Output: Conflict symptom data

[1957] Step 6:

[1958] If signs of conflict are detected, the server references the family members' past behavioral data and emotional history, and the AI ​​dynamically adjusts and generates suggested messages and activities to promote relationship repair.

[1959] Input: Conflict symptom data, past behavior data, emotional history

[1960] Output: Suggested messages and activities

[1961] Step 7:

[1962] The device receives suggested messages and activities from the cloud server and displays them on the mirror's display, where the suggestions are presented in an intuitive way for family members to understand.

[1963] Input: Suggested message or activity

[1964] Output: Messages and activity shown on the display

[1965] Step 8:

[1966] The user checks the suggested messages and activities displayed on the mirror and acts accordingly if they feel it is appropriate. For example, in response to the suggested message "Would you like to watch a movie together?", the user takes the action of watching a movie together.

[1967] Input: Messages and activity shown on the display

[1968] Output: Family member behavior

[1969] Step 9:

[1970] The device again captures the user's behavioral data and reactions with its camera and sends them to the cloud server, where the emotion engine also analyzes them and records the emotional data.

[1971] Input: Family member behavior data, reactions

[1972] Output: Behavioral and emotional data sent to the cloud server

[1973] Step 10:

[1974] The server analyzes the received behavioral data, reactions, and emotional data, and uses the results to improve the accuracy of future suggestions. Based on the accumulated data, more personalized suggestions are possible.

[1975] Input: Behavioral data, emotion data

[1976] Output: Data to improve the accuracy of proposals from next time onwards

[1977] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1978] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1979] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1980] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1981] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1982] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1983] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1984] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1985] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1986] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1987] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1988] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1989] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1990] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1991] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1992] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1993] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1994] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1995] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1996] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1997] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1998] The following is further disclosed regarding the above embodiment.

[1999] (Claim 1)

[2000] A smart mirror installed in a home, having a camera for capturing facial expressions of family members, and means for temporarily storing facial expression data captured by the camera;

[2001] means for transmitting the facial expression data to a cloud server;

[2002] A means for analyzing the received facial expression data by the cloud server and identifying the emotional states of the family members;

[2003] means for analyzing communication patterns between a plurality of family members by the cloud server and detecting occurrence of conflicts;

[2004] a means for suggesting messages or activities that promote relationship repair based on past behavioral data of family members when the occurrence of the conflict is detected;

[2005] means for displaying said messages and activities;

[2006] A system including:

[2007] (Claim 2)

[2008] The system according to claim 1, further comprising means for capturing again the reactions and actions of family members who responded to the suggestions by the cloud server, and using the captured reactions and actions to improve the accuracy of subsequent suggestions.

[2009] (Claim 3)

[2010] The system according to claim 1, further comprising means for transmitting the facial expression data in batches to a cloud server at regular intervals by the camera.

[2011] "Example 1"

[2012] (Claim 1)

[2013] A smart mirror installed in a home has a photographing device that photographs facial expressions of family members, and a means for temporarily storing facial expression data photographed by the photographing device;

[2014] means for transmitting the facial expression data to a cloud server;

[2015] A means for analyzing the received facial expression data by the cloud server and identifying the emotional states of the family members;

[2016] means for analyzing communication patterns between a plurality of family members and detecting occurrence of conflicts by the cloud server;

[2017] a means for suggesting messages or activities that promote relationship repair based on past behavioral data of family members when the occurrence of the conflict is detected;

[2018] means for displaying said messages and activities;

[2019] A means for using generative AI models to analyze facial expression data; and

[2020] A system including:

[2021] (Claim 2)

[2022] The system according to claim 1, further comprising means for capturing again the reactions and actions of family members who responded to the suggestions by the cloud server, and using the captured reactions and actions to improve the accuracy of subsequent suggestions.

[2023] (Claim 3)

[2024] 2. The system according to claim 1, further comprising means for transmitting the facial expression data in batches to a cloud server at regular intervals by the photographing device.

[2025] "Application Example 1"

[2026] (Claim 1)

[2027] A smart mirror installed in a home has an imaging device that captures facial expressions of family members and a device that temporarily stores facial expression data captured by the imaging device;

[2028] a device for transmitting the facial expression data to a remote server;

[2029] means for analyzing the received facial expression data by the remote server to identify the emotional states of family members;

[2030] a device for analyzing communication patterns among a plurality of family members and detecting occurrence of conflicts by the remote server;

[2031] a device that...

Claims

1. A smart mirror installed in a home, having a camera for capturing facial expressions of family members, and means for temporarily storing facial expression data captured by the camera; means for transmitting the facial expression data to a cloud server; A means for analyzing the received facial expression data by the cloud server and identifying the emotional states of the family members; means for analyzing communication patterns between a plurality of family members by the cloud server and detecting occurrence of conflicts; a means for suggesting messages or activities that promote relationship repair based on past behavioral data of family members when the occurrence of the conflict is detected; means for displaying said messages and activities; A system including:

2. The system according to claim 1 , further comprising means for capturing again the reactions and actions of family members who responded to the suggestions by the cloud server, and using the captured reactions and actions to improve the accuracy of subsequent suggestions.

3. The system according to claim 1 , further comprising means for transmitting the facial expression data in batches to a cloud server at regular intervals by the camera.

Images