System

Abstract

A system is provided.SOLUTION: A system for learning user behavior patterns and providing assistance to the user, comprising: means for collecting user behavior data; means for analyzing user behavior data and detecting behavior patterns; means for predicting the user's next behavior based on the detected behavior patterns; means for operating home appliances based on the prediction; means for recognizing user voice input and generating appropriate voice responses to engage in natural conversation with the user; and means for controlling cameras and sensors for security and surveillance in the home.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern homes, there is a growing need for security and monitoring, along with the need to streamline housework and home office work. However, existing technology is inadequate for systems that can efficiently learn and predict user behavior and provide support. In particular, there is no means to provide integrated functions such as natural conversation, automatic control of home appliances, crime prevention, and monitoring. The purpose of this invention is to solve these problems and provide a comprehensive, in-home system that comprehensively supports users' lives. [Means for solving the problem]

[0005] The present invention provides a system including means for collecting user behavior data, means for analyzing the user behavior data and detecting behavioral patterns, means for predicting the user's next behavior based on the detected behavioral patterns, means for operating home appliances based on the prediction, means for recognizing user voice input and generating appropriate voice responses to engage in natural conversation with the user, and means for controlling cameras and sensors for home security and monitoring. This system tracks the user's behavior in real time, provides optimal support through operation of home appliances and natural conversation, and comprehensively realizes security and monitoring functions. Furthermore, by further including a user interface that allows the user to set specific conditions, the system can flexibly respond to user needs.

[0006] "User behavior data" refers to information about users' behavior within the home, their movement history, and the usage of home appliances.

[0007] "Analysis" means extracting certain patterns and characteristics from collected data and performing a process to interpret their meaning.

[0008] "Behavioral patterns" refer to the tendencies and habits of actions that a user repeatedly performs at specific times or in specific situations.

[0009] "Predicting" means estimating future situations or actions from past data.

[0010] "Home appliances" refers to all electrical equipment used in the home, including lighting, televisions, air conditioners, etc.

[0011] "Natural conversation" refers to a smooth exchange between the user and the robot that feels natural to both parties.

[0012] "Voice input" refers to the collection and recognition of user-uttered speech as data.

[0013] "Voice response" refers to the system generating a voice message to the user.

[0014] "Crime prevention" means monitoring and vigilance to protect the home from unauthorized access and intruders.

[0015] "Monitoring" refers to monitoring and supporting the activities of people, especially children and the elderly, to ensure their safety.

[0016] A "camera" is a device that takes pictures or videos and records the data.

[0017] A "sensor" is a device that detects physical phenomena and outputs them as data.

[0018] "User interface means" refers to a method or device by which a user provides input to a system. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[0041] 1. Collecting user behavior data

[0042] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[0043] 2. Analyzing user behavior data and detecting patterns

[0044] The collected user behavior data is sent to a server. The server analyzes this data and detects certain behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This pattern information is used to predict the user's next action.

[0045] 3. Predict your next move and operate your home appliances

[0046] The server predicts the user's next action based on the detected behavioral patterns. This prediction result is then sent back to the device. For example, if the user has a pattern of watching TV every night at 10 p.m., the device will automatically turn on the TV at that time.

[0047] 4. Realizing natural conversation

[0048] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. If the user asks, "What's the weather like tomorrow?", the device analyzes this voice input, and the server's generation AI generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees." The device then conveys this to the user via voice.

[0049] 5. Security and monitoring functions

[0050] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0051] Specific examples

[0052] Morning support:

[0053] If the device learns that the user wakes up at 7:00 every morning, it will turn on the lights and open the curtains at 7:00. It will also provide weather information via voice, along with a greeting such as "Good morning. It's sunny today."

[0054] Remote work support:

[0055] When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[0056] Through the above-described specific processes and functions, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[0057] The processing flow will be explained below.

[0058] Specific flow of program processing

[0059] User behavior pattern learning

[0060] Step 1:

[0061] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[0062] Step 2:

[0063] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[0064] Step 3:

[0065] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[0066] Step 4:

[0067] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[0068] Step 5:

[0069] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[0070] Behavior prediction and home appliance operation

[0071] Step 1:

[0072] The server predicts the user's next action based on the analyzed behavioral patterns, for example, predicting that the user has a habit of watching TV at 10 p.m.

[0073] Step 2:

[0074] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[0075] Step 3:

[0076] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[0077] Realizing natural conversation

[0078] Step 1:

[0079] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[0080] Step 2:

[0081] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[0082] Step 3:

[0083] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[0084] Step 4:

[0085] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[0086] Security and monitoring functions

[0087] Step 1:

[0088] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[0089] Step 2:

[0090] The device turns off home appliances such as lights, televisions, and air conditioners.

[0091] Step 3:

[0092] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[0093] Step 4:

[0094] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[0095] Step 5:

[0096] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[0097] Through the above steps, learning of user behavior patterns, predictive operation of home appliances, natural conversation, crime prevention and monitoring functions can be effectively realized.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] In modern homes, systems that learn user behavior patterns and automatically operate and respond to the necessary home appliances are needed to make users' daily lives more efficient and comfortable. Functions such as monitoring elderly people and children and crime prevention are also required. However, existing systems often have separate functions and do not provide comprehensive support. This creates the complexity of having to install multiple systems and operate them individually.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes means for collecting user behavior data in real time using multiple sensors, cameras, and microphones, means for transmitting the collected behavior data to the server, and means for analyzing the user behavior data and detecting behavior patterns in the server, thereby enabling more efficient support for users' daily lives and improving comfort and safety in the home.

[0103] A "sensor" is a device that detects physical phenomena (temperature, humidity, movement, etc.) and provides that information as data.

[0104] A "camera" is a device used to capture images and videos and to visually record a user's actions and situations.

[0105] A "microphone" is a device that picks up sound and converts it into an electrical signal, and is used to obtain a user's voice input.

[0106] A "server" is a computing device for receiving, storing, analyzing data, and generating a response.

[0107] "Behavioral data" refers to data that includes information related to a user's daily life, such as the user's specific movements, location, and voice.

[0108] "Analysis" is the process of extracting meaning and patterns from collected data.

[0109] "Behavioral patterns" refer to repetition of a user's behavior or tendencies in behavior at a specific time.

[0110] A "machine learning algorithm" is a method of learning specific trends and patterns based on large amounts of data and making predictions and classifications for new data.

[0111] A "generative AI model" is an artificial intelligence model that generates responses or results that are meaningful to humans based on given input data.

[0112] "Speech recognition means" is a technology that converts voice input into text data.

[0113] "Crime prevention" refers to activities aimed at protecting physical spaces from unauthorized intrusion and crime.

[0114] "Monitoring" involves monitoring elderly people, children, and others to ensure their safety, and taking appropriate action if any abnormalities are detected.

[0115] "Encryption" is a technology that converts data using a specific algorithm to make it difficult for third parties to decipher.

[0116] "Real-time" means that processing or response occurs immediately after data is collected, and refers to operations or processes that require immediacy.

[0117] "Home comfort" refers to the physical and psychological satisfaction that users feel when spending time at home.

[0118] A "user interface" is the means or screen layout by which a user directly interacts with a system or device.

[0119] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[0120] Hardware and Software Configuration

[0121] Terminal: Infrared sensor, environmental sensor, IP camera, microphone

[0122] Server: A computer device for data analysis. Software used includes Python, Pandas, scikit-learn, and TENSORFLOW (registered trademark).

[0123] Network: Use secure protocols such as HTTPS for data transmission.

[0124] Speech Recognition: Speech recognition technologies such as Google® Cloud Speech-to-Text.

[0125] Text-to-speech: Text-to-speech technologies such as Google Cloud Text-to-Speech.

[0126] Collecting user behavior data

[0127] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[0128] Example: If a user is listening to music in their living room, the device's camera captures their movements, the microphone captures audio data, and the environmental sensors record the temperature and humidity in the living room.

[0129] Data transmission and analysis

[0130] The collected user behavior data is sent from the device to a server, which receives the data and stores it in a database. It then uses machine learning algorithms to analyze the behavioral patterns and detect specific behavioral patterns.

[0131] Example: Based on data from the past week, the server detects a pattern: "The user wakes up at 7am every morning and has breakfast at 8am."

[0132] Behavior prediction and automatic operation of home appliances

[0133] The server predicts the user's next action based on the detected behavioral patterns, and the prediction results are sent back to the device, allowing the appliance to prepare ahead of the user's actions.

[0134] Example: The server predicts that the user will have breakfast at 8:00 and sends that information to the device. The device then turns on the kitchen lights and starts the coffee maker at 8:00. It also adjusts the air conditioner temperature appropriately.

[0135] Realizing natural conversation

[0136] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3®) to generate an appropriate response, which is then sent back to the device and conveyed to the user using speech synthesis software.

[0137] Example: When a user asks, "What's the weather going to be like tomorrow?", the device converts the speech to text and sends it to a server. The server generates "It's going to be sunny tomorrow," which the device then replies to the user by voice.

[0138] Security and monitoring functions

[0139] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary.

[0140] Example: If the motion sensor in the room detects abnormal activity, for example, that a suspicious person has entered through the window, the device will report the situation to the server, and the server will immediately send an alert to the user's smartphone.

[0141] Examples of concrete examples and prompts

[0142] Wake-up support: If the device learns that the user wakes up at 7:00 a.m. every morning, it will turn on the lights and open the curtains at 7:00 a.m. It will also provide weather information by voice, along with a greeting such as "Good morning. It's sunny today."

[0143] Example prompt: "When the user wakes up at 7:00, turn on the lights and give them the weather report."

[0144] Support for remote work: When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[0145] Example prompt: "Please notify me 10 minutes before a meeting starts and adjust the air conditioning temperature when users are working remotely."

[0146] As described above, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] Collecting user behavior data

[0150] The device collects user behavior data in real time using infrared sensors, environmental sensors, IP cameras, and microphones. Specifically, the device's camera captures the user's movements, and the microphone captures audio data. The environmental sensors collect data such as temperature and humidity. This data indicates which room the user is in and which appliances they are using. For example, if a user is listening to music in the living room, the camera records the user's movements and the microphone captures their audio.

[0151] Input: Data from infrared sensors, environmental sensors, IP cameras, microphones

[0152] Output: User behavior data (location, movement, audio, environmental data)

[0153] Step 2:

[0154] Sending behavioral data to the server

[0155] The device encrypts the collected user behavior data and sends it to a server via the Internet. Specifically, the device batches the collected data and sends it to the server using a secure protocol such as HTTPS. For example, data on the user's presence in the living room is sent to the server every hour.

[0156] Input: Collected user behavior data

[0157] Output: Data sent to the server

[0158] Step 3:

[0159] Behavioral data analysis and pattern detection

[0160] The server stores the received user behavior data in a database and analyzes the data using analytical software (e.g., Python, Pandas). The server then analyzes the data using machine learning algorithms (e.g., scikit-learn, TensorFlow) to detect specific behavioral patterns. For example, the server might detect a pattern such as "the user wakes up at 7:00 every morning and eats breakfast at 8:00."

[0161] Input: User behavior data sent to the server

[0162] Output: Detected behavioral patterns

[0163] Step 4:

[0164] Predicting next actions

[0165] The server predicts the user's next action based on the detected behavioral patterns. The server uses a machine learning model to predict the user's next action based on past behavioral data. The prediction result is then sent back to the device. For example, a prediction that "the user will have breakfast at 8 o'clock" is sent to the device.

[0166] Input: Detected behavioral patterns

[0167] Output: Predicted data of next action

[0168] Step 5:

[0169] Automatic operation of home appliances

[0170] The device automatically controls home appliances based on the next action prediction received from the server. Specifically, the device controls home appliances through smart plugs and smart remote controls. For example, the device might turn on the kitchen lights and start the coffee maker at 8:00 a.m., and also adjust the air conditioner temperature appropriately.

[0171] Input: Next action prediction data received from the server

[0172] Output: Operation instructions for home appliances

[0173] Step 6:

[0174] Realizing natural conversation

[0175] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response. The generated response is then sent back to the device, where it is communicated to the user using speech synthesis software (e.g., Google Cloud Text-to-Speech). For example, if a user asks, "What's the weather like tomorrow?" the device converts the speech into text and sends it to the server. The server generates a response, which the device then returns to the user via voice.

[0176] Input: User voice input

[0177] Output: The generated audio response

[0178] Step 7:

[0179] Security and monitoring functions

[0180] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary. For example, if the device's motion sensor detects abnormal activity in a room, it will report the situation to the server, stating that a "suspicious person has entered through a window," and the server will immediately send an alert to the user's smartphone.

[0181] Input: Anomaly detection data from cameras and sensors

[0182] Output: User notification

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Conventional home support systems have the ability to learn the user's behavioral patterns and operate home appliances, but they lack the ability to automatically switch security modes or notify users of suspicious activity in real time. This makes it difficult for users to leave their home in the care of the system while they are out. To solve this issue, there is a need for enhanced home security and monitoring functions and for real-time notification when suspicious activity is detected.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes means for collecting user behavior data to learn user behavior patterns and provide support to the user, means for analyzing user behavior data and detecting behavior patterns, means for predicting the user's next action based on the detected behavior patterns, means for operating home appliances based on the prediction, means for recognizing user voice input and generating appropriate voice responses to have natural conversations with the user, means for controlling cameras and sensors for home security and surveillance, and means for automatically switching to security mode when the user goes out and for notifying in real time if suspicious activity is detected, allowing the user to leave their home in the care of the appliances while they are out with peace of mind.

[0188] "User behavior data" is data that records a user's daily behavior and lifestyle patterns.

[0189] "Behavioral patterns" refer to specific actions or habits that a user repeatedly performs.

[0190] "User voice input" refers to words or commands that a user speaks to a robot.

[0191] "Cameras and sensors" is a general term for optical and electronic devices used to detect images, sounds, and motion.

[0192] "Security mode" is an operating mode for strengthening security in the home when the user is away.

[0193] "Real-time notification" refers to notifying users immediately when abnormal or suspicious behavior is detected.

[0194] This invention is a specialized domestic butler robot system that learns the user's behavioral patterns and supports the user's daily life. This system is realized by combining various technologies, including the collection, analysis, and prediction of user behavior data, security functions, and natural conversation functions.

[0195] 1. Collecting user behavior data

[0196] The system is equipped with multiple sensors and cameras that monitor and record users' behavior in real time. These data provide a detailed record of users' daily movements, allowing the system to know which room the user is in and which appliances they are using.

[0197] 2. Analyzing user behavior data

[0198] The server analyzes the collected behavioral data to detect user behavior patterns. This process involves using a generative AI model to analyze patterns and predict the user's next action. To do this, the collected data is stored in a database and the AI ​​model learns from it.

[0199] 3. Controlling home appliances

[0200] Based on the user's behavioral patterns, the server predicts the user's next actions and operates the necessary home appliances. For example, the server can set the living room lights and TV to turn off at 10 p.m. every night. This allows the user to automatically operate the home appliances and provide a comfortable living environment.

[0201] 4. Realizing natural conversation

[0202] The device recognizes voice input from the user and generates an appropriate response. For example, if a user asks, "What's the weather like tomorrow?", the server-based generative AI model generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees," and conveys this to the user via voice. This process uses natural language processing technology to analyze the user's intent.

[0203] 5. Security features

[0204] When the user leaves the house, the system automatically switches to security mode and monitors the home using cameras and sensors for any unusual activity. If any suspicious activity is detected, the system sends a real-time notification to the user's smartphone, urging immediate action. If an abnormality is detected, an alarm will sound and the system can also notify security authorities.

[0205] 6. Program Processing

[0206] The server and device collect user behavior data in real time and store it in a database. Generative AI models are used to analyze and predict the data, and the results are used to automatically operate home appliances. Voice recognition software and natural language processing technology are combined to enable natural conversations with users. For security purposes, the system analyzes data detected by cameras and sensors, and notifies users in real time if an abnormality is detected.

[0207] As a concrete example, if a user asks, "What time is a good time to leave the house today?", they can input the following prompt into the generative AI model:

[0208] "The system learns the pattern of when users leave home at 8 o'clock every morning and automatically switches on security mode. This learning is based on simple rules, but in the future, we will be able to predict behavior with greater accuracy by introducing reinforcement learning using generative AI. Specifically, we will input the following prompt into the generative AI: 'Please tell me how I can learn the patterns of when users often leave home and automatically switch on security mode according to that time of day.'"

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] Collecting user behavior data

[0212] The device uses multiple sensors and cameras installed in the home to collect user behavior data in real time. Specifically, it obtains information on which room the user is in and which home appliances they are using. This behavior data is sent from the device to a server and stored in a database.

[0213] Input: User behavior data from sensors and cameras.

[0214] Output: The collected user behavior data is sent to the server and stored in a database.

[0215] Step 2:

[0216] Analyzing user behavior data

[0217] The server analyzes the user behavior data stored in the database and detects specific behavioral patterns. Using a generative AI model, it learns from this data and discovers user behavior patterns. For example, it discovers that the user wakes up at 7am every morning and watches TV in the living room.

[0218] Input: User behavior data stored in a database.

[0219] Output: Analyzed by a generative AI model and detected as behavioral patterns.

[0220] Step 3:

[0221] Predicting next actions based on behavioral patterns

[0222] The server predicts the user's next action based on the detected user behavior patterns. The generative AI model makes predictions based on past data, and the server generates predictions based on the user's lifestyle habits. For example, it predicts that the TV will be turned off at 10 p.m.

[0223] Input: Detected behavioral patterns.

[0224] Output: Predicted result of next action.

[0225] Step 4:

[0226] Automatic operation of home appliances

[0227] The server automatically controls household appliances based on the prediction results. For example, if the prediction is to "turn off the TV at 10 o'clock," the server will issue a command to turn off the TV at that time. This provides a comfortable living environment for the user.

[0228] Input: Predicted outcome of the action.

[0229] Output: Status of automatically operated appliances.

[0230] Step 5:

[0231] Achieving natural conversations with users

[0232] The device uses a voice recognition system to receive voice input from the user, and the server uses a generative AI model to generate appropriate responses to the user's questions and commands, which are then spoken to the user via the device.

[0233] Input: User's voice input.

[0234] Output: Natural-sounding conversational responses generated by the generative AI model.

[0235] Step 6:

[0236] Security features implemented

[0237] When the device detects that the user has left the house, it automatically switches to security mode. Cameras and sensors monitor for suspicious activity, and if the server detects an abnormality, it sends a real-time notification to the user's smartphone.

[0238] Input: External environment data from cameras and sensors.

[0239] Output: Real-time notification of suspicious behavior.

[0240] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0241] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[0242] 1. Collecting user behavior data

[0243] The device uses cameras, microphones, and sensors installed in the home to collect real-time user behavior data. For example, by recording which room the user is in and which home appliances they are using, the device can understand the user's daily behavioral patterns. This data includes the user's movements, movement history, and usage of home appliances.

[0244] 2. Analyzing user behavior data and detecting patterns

[0245] The collected user behavior data is sent to a server, which analyzes it and detects specific behavioral patterns. For example, the server can determine that the user wakes up at 7am every morning and turns on the living room light. This provides the basis for predicting the user's next behavior.

[0246] 3. Emotion Recognition by Emotion Engine

[0247] The device recognizes the user's voice and facial expressions and records them as emotional data. For example, if the user looks tired, the emotion engine will recognize this as a "fatigue state." This emotional data is also sent to the server along with behavioral data for analysis.

[0248] 4. Behavior prediction and home appliance operation

[0249] The server predicts the user's next action based on the detected behavioral patterns and emotional data. This prediction result is then sent back to the device. For example, if the user has a habit of watching TV at 10 p.m. and the emotional data for that day indicates that they want to relax, the device will dim the lights and automatically turn on the TV at that time.

[0250] 5. Realizing natural conversation

[0251] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. For example, if the user says, "I'm tired today," the server's AI will respond, "Thank you for your hard work. I'll play some relaxing music." This response is generated taking into account the user's emotional state.

[0252] 6. Security and monitoring functions

[0253] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0254] Specific examples

[0255] Morning support:

[0256] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[0257] Remote work support:

[0258] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[0259] Through the above-described specific processing and functions, the specialized domestic butler robot of the present invention comprehensively learns the user's behavior and emotions, provides more effective support, and provides a comfortable home environment.

[0260] The processing flow will be explained below.

[0261] Specific flow of program processing

[0262] User behavior pattern learning

[0263] Step 1:

[0264] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[0265] Step 2:

[0266] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[0267] Step 3:

[0268] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[0269] Step 4:

[0270] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[0271] Step 5:

[0272] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[0273] Emotion data collection using an emotion engine

[0274] Step 1:

[0275] The user interacts with the device and performs everyday activities. For example, the user might say, "I'm tired today."

[0276] Step 2:

[0277] The device detects the user's voice and facial expressions, and analyzes this data using an emotion engine. The device recognizes the user's emotional state (e.g., "fatigue," "comfort," etc.) from the voice data and facial image data.

[0278] Step 3:

[0279] The device sends the recognized emotion data to the server, where it is integrated with the behavioral data and analyzed.

[0280] Behavior prediction and home appliance operation

[0281] Step 1:

[0282] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. For example, if the user has a habit of watching TV at 10 p.m., it predicts the user's emotional state for that day as "relaxed."

[0283] Step 2:

[0284] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[0285] Step 3:

[0286] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[0287] Realizing natural conversation

[0288] Step 1:

[0289] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[0290] Step 2:

[0291] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[0292] Step 3:

[0293] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[0294] Step 4:

[0295] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[0296] Security and monitoring functions

[0297] Step 1:

[0298] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[0299] Step 2:

[0300] The device turns off home appliances such as lights, televisions, and air conditioners.

[0301] Step 3:

[0302] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[0303] Step 4:

[0304] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[0305] Step 5:

[0306] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[0307] Specific examples

[0308] Morning support:

[0309] Step 1:

[0310] The system learns that the user wakes up at 7am every morning.

[0311] Step 2:

[0312] The emotion engine recognizes your emotional state from the past morning as "refreshed."

[0313] Step 3:

[0314] The device will turn on the lights at 7:00 and open the curtains, and will also provide a voice greeting such as, "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[0315] Remote work support:

[0316] Step 1:

[0317] The emotion engine recognizes that users are feeling stressed while working from home.

[0318] Step 2:

[0319] The device plays relaxing music before the meeting starts.

[0320] Step 3:

[0321] During the meeting, the device will adjust the air conditioning temperature appropriately and start the projector.

[0322] These steps will enable a comprehensive realization of learning user behavior patterns, collecting and analyzing emotional data, predictive home appliance operation, natural conversation, and crime prevention and monitoring functions.

[0323] Example 2

[0324] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] In modern homes, it is difficult to accurately grasp a user's behavioral patterns and emotional state and provide support based on that. Furthermore, while there is a demand for effectively combining security and monitoring functions, it is difficult to build a comprehensive system using existing technology. To address these issues, the present invention provides a system that learns a user's behavioral patterns and emotional state and provides optimal support.

[0326] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting user behavior data in real time using cameras, microphones, and sensors installed in the home; means for transmitting the collected user behavior data to the server and analyzing it to detect behavioral patterns; means for recognizing the user's voice and facial expressions and recording emotional data; means for transmitting the recorded emotional data to the server and using it for analysis; means for predicting the user's next behavior based on the detected behavioral patterns and emotional data; means for operating home appliances based on the prediction; means for recognizing voice input from the user and generating an appropriate voice response using a generative AI model; and means for detecting abnormal behavior in the home and controlling the cameras and sensors to provide security and monitoring functions. This makes it possible to comprehensively learn the user's behavior and emotions, provide effective support, and provide a comfortable and safe home environment.

[0327] A "camera" is a photographic device for capturing visual information.

[0328] A "microphone" is an acoustic transducer for collecting sound.

[0329] A "sensor" is a device that collects physical environmental information and has the ability to detect temperature, movement, light, etc.

[0330] "Behavioral data" refers to data that indicates information about a user's actions, movements, and the equipment they use.

[0331] A "server" is a computer system that analyzes, stores, and communicates data.

[0332] "Behavioral patterns" are data that show regularities and trends extracted from a series of user actions.

[0333] "Emotion data" refers to data that indicates the emotional state of a user that can be inferred from their speech and facial expressions.

[0334] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses or predictions from input data.

[0335] "Security" is a function that aims to protect the safety of the home by detecting intrusions by suspicious individuals and abnormal situations.

[0336] "Monitoring" is a function that checks the safety of children, elderly people, etc. and notifies you if any abnormal behavior occurs.

[0337] "Real-time" means that processing and reaction occur immediately in accordance with actual time.

[0338] "Voice input" refers to data that the system captures and processes from the user's speech.

[0339] "Speech recognition" is the technology that analyzes voice input and understands what it means.

[0340] A "voice response" is a voice message that the system responds to a user's voice input.

[0341] This invention relates to a specialized in-home butler robot system that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot system collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[0342] Collecting user behavior data

[0343] The device collects user behavior data in real time using cameras, microphones, and sensors installed in the home. This includes cameras and sensors installed in each room of the home. This allows the device to record which room the user is in and which home appliances they are using. This behavioral data includes the user's movements, movement history, and usage status of home appliances.

[0344] Analyzing user behavior data and detecting patterns

[0345] The collected behavioral data is sent from the device to a server, which analyzes the data to detect specific behavioral patterns. For example, the server may detect that the user wakes up at 7am every morning and turns on the living room light. This analysis is performed using a data analysis algorithm.

[0346] Emotion recognition by emotion engine

[0347] The device recognizes the user's voice and facial expressions in real time and records them as emotional data. For example, if a user says, "I'm tired today," the device's microphone captures this voice and converts it into text using voice recognition software. The emotion engine analyzes this and recognizes it as "fatigue." This emotional data is also sent to the server for further analysis.

[0348] Behavior prediction and home appliance operation

[0349] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. One example of this prediction is when a user has a habit of watching TV at 10pm, and the server determines from that day's emotional data that they "want to relax." The server then sends the device an instruction to "dim the lights and automatically turn on the TV at 10pm." The device receives this instruction and operates the home appliances.

[0350] Realizing natural conversation

[0351] To achieve natural conversation with the user, the device receives voice input and generates an appropriate response using a generative AI model. For example, if the user says, "I'm tired today," the server's generative AI model generates a response such as, "Thank you for your hard work. I'll play some relaxing music for you," and sends it to the device. The device then conveys this as voice to the user.

[0352] Security and monitoring functions

[0353] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0354] Specific examples

[0355] Morning support:

[0356] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[0357] Remote work support:

[0358] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[0359] Prompt Sentence Examples

[0360] When a user inputs "I'm tired today" into the generative AI model, the model responds, "Thank you for your hard work. I'll play some relaxing music right now."

[0361] Through these specific processing steps, the specialized domestic butler robot can comprehensively learn the user's behavior and emotions, provide more effective support, and provide a comfortable and safe home environment.

[0362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0363] Step 1: Collect user behavior data

[0364] The device uses cameras, microphones, and sensors installed in the home to collect user behavior data in real time. Inputs include camera footage, audio data, and sensor data. This input data is processed and output as user location information and behavior data. Specifically, the camera detects the user's movements, the microphone records speech, and the sensors detect temperature and movement.

[0365] Step 2: Analyze user behavior data and detect patterns

[0366] The collected user behavior data is sent from the device to a server. The server analyzes this data and detects behavioral patterns. The input is the collected behavioral data. A data analysis algorithm is used to analyze regularity and frequency, and the detected behavioral patterns are output. Specifically, the server cross-references the user's behavior history and detects a pattern of waking up at 7am every morning and turning on the lights.

[0367] Step 3: Emotion recognition by the emotion engine

[0368] The device recognizes the user's voice and facial expressions and records them as emotional data. The input is the user's voice and facial expression data. Speech recognition software is used to convert the voice into text, and an emotional engine analyzes the emotional state. The analyzed emotional data is sent to the server. For example, if the user says, "I'm tired today," the voice input is output as "fatigue state."

[0369] Step 4: Behavior prediction and home appliance operation

[0370] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. The input is behavioral pattern data and emotional data. The server uses a prediction algorithm to predict the user's next action and sends the prediction result to the device. The device then executes instructions to operate home appliances. For example, if it predicts that the user will watch TV at 10 p.m., it dims the lights and turns on the TV.

[0371] Step 5: Achieving natural conversation

[0372] The device recognizes voice input from the user and uses a generative AI model to generate an appropriate voice response. The input is the user's voice data. This is converted into text data through speech recognition software, and the generative AI model generates a response. The device outputs the generated voice response. For example, if the user says, "I'm tired today," the generative AI will respond, "Thank you for your hard work. I'll play some relaxing music."

[0373] Step 6: Security and monitoring functions

[0374] When the user goes out, the device switches to security mode. The input is the away mode setting. The camera and sensors installed on the device monitor abnormal movements and detect the intrusion of suspicious individuals. Any detected abnormalities are sent to the server, which then sends an alert to the user in real time. For example, if a suspicious individual is detected, an alert notification is output to the user's smartphone.

[0375] (Application example 2)

[0376] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0377] The present invention relates to a system that learns user behavior patterns in the home, recognizes emotions, and provides optimal home appliance operation, natural conversation, security, and monitoring functions. However, conventional technologies have had challenges in detecting customer movements and emotions in real time within stores and economic activity areas, and in using this information to make appropriate product recommendations and improve operational efficiency. While conventional technology systems are suitable for the specific environment of the home, they are not suitable for use in a wide range of economic activity areas, making it difficult to simultaneously improve customer satisfaction and store operational efficiency. Furthermore, recognizing customer emotions in real time and dynamically responding based on this information also present technical challenges.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user behavior data, means for analyzing the behavior data and detecting behavior patterns, means for predicting next behavior based on the behavior patterns, means for operating home appliances, means for recognizing voice input and generating voice responses, means for controlling cameras and sensors for security and surveillance, means for detecting customer movements and emotions within an economic activity site in real time, means for providing optimal product recommendations and location guidance to customers based on the analysis results, and means for using a generative AI model including a staffing suggestion function to improve operational efficiency. As a result, customer behavior and emotions can be recognized and analyzed in real time not only at home but also at economic activity sites, and optimal responses can be taken, thereby improving customer satisfaction and streamlining store operations.

[0379] "User behavior patterns" refers to trends in user behavior, movement, and home appliance usage.

[0380] "Behavioral data" is data based on the user's movements, location information, voice, facial expressions, etc.

[0381] "Behavioral pattern analysis" is the process of detecting specific behavioral tendencies and patterns of users based on collected behavioral data.

[0382] "Predicting next behavior" means estimating what behavior a user will take in the future based on analyzed behavioral patterns.

[0383] "Operating home appliances" means controlling household appliances such as air conditioners, lights, and televisions.

[0384] "User voice input" refers to speech uttered by a user that is analyzed by a voice recognition system.

[0385] "Generating a voice response" is the process of generating an appropriate response to a voice input.

[0386] "Crime prevention and surveillance" refers to ensuring safety within homes and stores and monitoring for abnormal situations.

[0387] "Cameras and sensors" is a general term for devices that detect images, sounds, and motion.

[0388] "Points of economic activity" refer to places where goods and services are provided, such as physical stores and commercial facilities.

[0389] "Real-time detection" is the process of collecting data instantly and analyzing it near-simultaneously.

[0390] "Product suggestion" refers to recommending products based on a customer's preferences and purchasing history.

[0391] "Location guidance" refers to providing guidance to the optimal route and location based on the customer's current location and purpose.

[0392] "Improved operational efficiency" is the process of optimizing store and facility operations and increasing productivity.

[0393] The "staffing suggestion function" refers to a function that suggests appropriate staffing according to location and time of day.

[0394] A "generative AI model" is an artificial intelligence model that automatically generates responses and predictions based on data.

[0395] The present invention relates to a system for detecting and analyzing a user's movements and emotions in real time and providing optimal support. This system operates to effectively support users and customers in the home or at points of economic activity. Specific embodiments are described below.

[0396] Hardware and Software

[0397] 1. Hardware Configuration

[0398] Camera: A device installed within an economic activity point to capture the movements and expressions of users and customers. For example, HD cameras and infrared cameras are used.

[0399] Microphone: A device used to collect voice input and recognize user or customer speech.

[0400] Sensors: These are devices installed to collect movement data and environmental data. These include temperature sensors, motion sensors, etc.

[0401] Server: A central device for data collection, analysis, and control. It is desirable to have a high-performance processor and a large amount of memory.

[0402] 2. Software Configuration

[0403] Movement and Emotion Detection Software: Detects movements and facial expressions from camera footage using Python and OpenCV.

[0404] Speech recognition system: Analyzes voice data and converts it into text in real time. Google Speech-to-Text API is used.

[0405] Emotion recognition engine: Using a deep learning model with TensorFlow and Keras, emotions are recognized from collected voice and facial expression data.

[0406] Server-side processing: Back-end processing and data management are performed using Node.js and Express.

[0407] Database management system: MongoDB is used to store and query the collected data.

[0408] Processing flow

[0409] 1. Data collection

[0410] The server collects user and customer behavioral and voice data through cameras and microphones.

[0411] Data is also collected from sensors, which transmit location and environmental data of users and customers to a server.

[0412] 2. Data Analysis

[0413] The server uses Python scripts to analyze video data from the cameras and detect customer movements and behavior patterns.

[0414] An emotion recognition engine using TensorFlow and Keras analyzes voice and facial expression data to identify the emotional state of users and customers in real time.

[0415] 3. Creating Actions

[0416] A server-side Node.js application uses the analyzed data to predict the next action and, based on the results, optimally operate home appliances and make product suggestions.

[0417] The server uses a generative AI model to generate appropriate responses to voice input.

[0418] 4. Examples and Prompts

[0419] Example: If a customer spends too long at a particular shelf and their facial expression indicates confusion, the server will notify a nearby staff member and instruct them to explain the product. An audio guide will also ask, "Can I help you with this product?"

[0420] Example prompt sentence:

[0421] "Explain how you can use customer facial expression data to estimate their current emotional state and provide a pleasant in-store experience."

[0422] "Please show us an algorithm that analyzes customer movement data and purchase history and recommends products that are likely to be purchased again."

[0423] This will realize a system that can comprehensively recognize and analyze the behavior and emotions of users and customers at home and at points of economic activity, and respond appropriately. This system is designed to improve customer satisfaction and streamline store operations.

[0424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0425] Step 1:

[0426] The device collects user movements and voice in real time from cameras and microphones installed in the physical store. As input, it acquires camera video data and audio data. The camera data is broken down into video frames, and the audio data is converted into a frequency spectrum. As output, the acquired frame data and audio spectrum data are sent to the server.

[0427] Step 2:

[0428] The server uses Python and OpenCV to analyze the user's movements from the received frame data. Camera image data is used as input. Data processing involves extracting movement data using background subtraction and object detection algorithms. User movement data is generated as output.

[0429] Step 3:

[0430] The server uses TensorFlow and Keras to analyze the user's emotions from voice data and facial expression data. Voice spectrum data and facial frame data are used as input. Data calculations are performed to classify the emotional state using a voice emotion recognition model and a facial expression recognition model. Emotional state data is generated as output.

[0431] Step 4:

[0432] The server predicts the user's next behavior based on the analyzed behavioral and emotional data. The inputs are movement data and emotional data. The data is calculated using a time series analysis method to build a behavioral pattern model, and the next behavior is estimated using a prediction algorithm. The output is next behavior prediction data.

[0433] Step 5:

[0434] The server uses a generative AI model to generate optimal home appliance operations and product suggestions based on the predicted behavior and emotion data. Behavior prediction data and emotion data are used as input. Data processing involves selecting appropriate responses and actions based on pre-set scenarios and rules. Specific operation commands and suggestions are generated as output.

[0435] Step 6:

[0436] The server sends the generated operation commands and suggestions to terminals in the store to control the appliances and provide product information. The operation commands and suggestions from the generative AI model are used as input. As data calculations, the terminal executes the commands and feeds the results back to the server. As output, data on the status of the appliances and customer reactions is generated.

[0437] Step 7:

[0438] The server updates the generative AI model based on the feedback data, improving accuracy from the next time onwards. Execution result data and customer response data are used as input. Data processing involves retraining the model using a machine learning algorithm. An updated generative AI model is generated as output.

[0439] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0440] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0441] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0442] [Second embodiment]

[0443] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0444] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0445] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0446] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0447] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0448] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0449] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0450] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0451] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0452] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0453] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0454] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0455] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[0456] 1. Collecting user behavior data

[0457] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[0458] 2. Analyzing user behavior data and detecting patterns

[0459] The collected user behavior data is sent to a server. The server analyzes this data and detects certain behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This pattern information is used to predict the user's next action.

[0460] 3. Predict your next move and operate your home appliances

[0461] The server predicts the user's next action based on the detected behavioral patterns. This prediction result is then sent back to the device. For example, if the user has a pattern of watching TV every night at 10 p.m., the device will automatically turn on the TV at that time.

[0462] 4. Realizing natural conversation

[0463] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. If the user asks, "What's the weather like tomorrow?", the device analyzes this voice input, and the server's generation AI generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees." The device then conveys this to the user via voice.

[0464] 5. Security and monitoring functions

[0465] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0466] Specific examples

[0467] Morning support:

[0468] If the device learns that the user wakes up at 7:00 every morning, it will turn on the lights and open the curtains at 7:00. It will also provide weather information via voice, along with a greeting such as "Good morning. It's sunny today."

[0469] Remote work support:

[0470] When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[0471] Through the above-described specific processes and functions, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[0472] The processing flow will be explained below.

[0473] Specific flow of program processing

[0474] User behavior pattern learning

[0475] Step 1:

[0476] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[0477] Step 2:

[0478] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[0479] Step 3:

[0480] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[0481] Step 4:

[0482] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[0483] Step 5:

[0484] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[0485] Behavior prediction and home appliance operation

[0486] Step 1:

[0487] The server predicts the user's next action based on the analyzed behavioral patterns, for example, predicting that the user has a habit of watching TV at 10 p.m.

[0488] Step 2:

[0489] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[0490] Step 3:

[0491] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[0492] Realizing natural conversation

[0493] Step 1:

[0494] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[0495] Step 2:

[0496] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[0497] Step 3:

[0498] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[0499] Step 4:

[0500] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[0501] Security and monitoring functions

[0502] Step 1:

[0503] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[0504] Step 2:

[0505] The device turns off home appliances such as lights, televisions, and air conditioners.

[0506] Step 3:

[0507] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[0508] Step 4:

[0509] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[0510] Step 5:

[0511] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[0512] Through the above steps, learning of user behavior patterns, predictive operation of home appliances, natural conversation, crime prevention and monitoring functions can be effectively realized.

[0513] Example 1

[0514] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0515] In modern homes, systems that learn user behavior patterns and automatically operate and respond to the necessary home appliances are needed to make users' daily lives more efficient and comfortable. Functions such as monitoring elderly people and children and crime prevention are also required. However, existing systems often have separate functions and do not provide comprehensive support. This creates the complexity of having to install multiple systems and operate them individually.

[0516] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0517] In this invention, the server includes means for collecting user behavior data in real time using multiple sensors, cameras, and microphones, means for transmitting the collected behavior data to the server, and means for analyzing the user behavior data and detecting behavior patterns in the server, thereby enabling more efficient support for users' daily lives and improving comfort and safety in the home.

[0518] A "sensor" is a device that detects physical phenomena (temperature, humidity, movement, etc.) and provides that information as data.

[0519] A "camera" is a device used to capture images and videos and to visually record a user's actions and situations.

[0520] A "microphone" is a device that picks up sound and converts it into an electrical signal, and is used to obtain a user's voice input.

[0521] A "server" is a computing device for receiving, storing, analyzing data, and generating a response.

[0522] "Behavioral data" refers to data that includes information related to a user's daily life, such as the user's specific movements, location, and voice.

[0523] "Analysis" is the process of extracting meaning and patterns from collected data.

[0524] "Behavioral patterns" refer to repetition of a user's behavior or tendencies in behavior at a specific time.

[0525] A "machine learning algorithm" is a method of learning specific trends and patterns based on large amounts of data and making predictions and classifications for new data.

[0526] A "generative AI model" is an artificial intelligence model that generates responses or results that are meaningful to humans based on given input data.

[0527] "Speech recognition means" is a technology that converts voice input into text data.

[0528] "Crime prevention" refers to activities aimed at protecting physical spaces from unauthorized intrusion and crime.

[0529] "Monitoring" involves monitoring elderly people, children, and others to ensure their safety, and taking appropriate action if any abnormalities are detected.

[0530] "Encryption" is a technology that converts data using a specific algorithm to make it difficult for third parties to decipher.

[0531] "Real-time" means that processing or response occurs immediately after data is collected, and refers to operations or processes that require immediacy.

[0532] "Home comfort" refers to the physical and psychological satisfaction that users feel when spending time at home.

[0533] A "user interface" is the means or screen layout by which a user directly interacts with a system or device.

[0534] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[0535] Hardware and Software Configuration

[0536] Terminal: Infrared sensor, environmental sensor, IP camera, microphone

[0537] Server: A computer device for data analysis. Software used includes Python, Pandas, scikit-learn, TensorFlow, etc.

[0538] Network: Use secure protocols such as HTTPS for data transmission.

[0539] Speech Recognition: Speech recognition technologies such as Google Cloud Speech-to-Text.

[0540] Text-to-speech: Text-to-speech technologies such as Google Cloud Text-to-Speech.

[0541] Collecting user behavior data

[0542] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[0543] Example: If a user is listening to music in their living room, the device's camera captures their movements, the microphone captures audio data, and the environmental sensors record the temperature and humidity in the living room.

[0544] Data transmission and analysis

[0545] The collected user behavior data is sent from the device to a server, which receives the data and stores it in a database. It then uses machine learning algorithms to analyze the behavioral patterns and detect specific behavioral patterns.

[0546] Example: Based on data from the past week, the server detects a pattern: "The user wakes up at 7am every morning and has breakfast at 8am."

[0547] Behavior prediction and automatic operation of home appliances

[0548] The server predicts the user's next action based on the detected behavioral patterns, and the prediction results are sent back to the device, allowing the appliance to prepare ahead of the user's actions.

[0549] Example: The server predicts that the user will have breakfast at 8:00 and sends that information to the device. The device then turns on the kitchen lights and starts the coffee maker at 8:00. It also adjusts the air conditioner temperature appropriately.

[0550] Realizing natural conversation

[0551] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response, which is then sent back to the device and conveyed to the user using speech synthesis software.

[0552] Example: When a user asks, "What's the weather going to be like tomorrow?", the device converts the speech to text and sends it to a server. The server generates "It's going to be sunny tomorrow," which the device then replies to the user by voice.

[0553] Security and monitoring functions

[0554] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary.

[0555] Example: If the motion sensor in the room detects abnormal activity, for example, that a suspicious person has entered through the window, the device reports the situation to the server, and the server immediately sends an alert to the user's smartphone.

[0556] Examples of specific examples and prompts

[0557] Wake-up support: If the device learns that the user wakes up at 7:00 a.m. every morning, it will turn on the lights and open the curtains at 7:00 a.m. It will also provide weather information by voice, along with a greeting such as "Good morning. It's sunny today."

[0558] Example prompt: "When the user wakes up at 7:00, turn on the lights and give them the weather report."

[0559] Support for remote work: When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[0560] Example prompt: "Please notify me 10 minutes before a meeting starts and adjust the air conditioning temperature when users are working remotely."

[0561] As described above, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[0562] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0563] Step 1:

[0564] Collecting user behavior data

[0565] The device collects user behavior data in real time using infrared sensors, environmental sensors, IP cameras, and microphones. Specifically, the device's camera captures the user's movements, and the microphone captures audio data. The environmental sensors collect data such as temperature and humidity. This data indicates which room the user is in and which appliances they are using. For example, if a user is listening to music in the living room, the camera records the user's movements and the microphone captures their audio.

[0566] Input: Data from infrared sensors, environmental sensors, IP cameras, microphones

[0567] Output: User behavior data (location, movement, audio, environmental data)

[0568] Step 2:

[0569] Sending behavioral data to the server

[0570] The device encrypts the collected user behavior data and sends it to a server via the Internet. Specifically, the device batches the collected data and sends it to the server using a secure protocol such as HTTPS. For example, data on the user's presence in the living room is sent to the server every hour.

[0571] Input: Collected user behavior data

[0572] Output: Data sent to the server

[0573] Step 3:

[0574] Behavioral data analysis and pattern detection

[0575] The server stores the received user behavior data in a database and analyzes the data using analytical software (e.g., Python, Pandas). The server then analyzes the data using machine learning algorithms (e.g., scikit-learn, TensorFlow) to detect specific behavioral patterns. For example, the server might detect a pattern such as "the user wakes up at 7:00 every morning and eats breakfast at 8:00."

[0576] Input: User behavior data sent to the server

[0577] Output: Detected behavioral patterns

[0578] Step 4:

[0579] Predicting next actions

[0580] The server predicts the user's next action based on the detected behavioral patterns. The server uses a machine learning model to predict the user's next action based on past behavioral data. The prediction result is then sent back to the device. For example, a prediction that "the user will have breakfast at 8 o'clock" is sent to the device.

[0581] Input: Detected behavioral patterns

[0582] Output: Predicted data of next action

[0583] Step 5:

[0584] Automatic operation of home appliances

[0585] The device automatically controls home appliances based on the next action prediction received from the server. Specifically, the device controls home appliances through smart plugs and smart remote controls. For example, the device might turn on the kitchen lights and start the coffee maker at 8:00 a.m., and also adjust the air conditioner temperature appropriately.

[0586] Input: Next action prediction data received from the server

[0587] Output: Operation instructions for home appliances

[0588] Step 6:

[0589] Realizing natural conversation

[0590] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response. The generated response is then sent back to the device, where it is communicated to the user using speech synthesis software (e.g., Google Cloud Text-to-Speech). For example, if a user asks, "What's the weather like tomorrow?" the device converts the speech into text and sends it to the server. The server generates a response, which the device then returns to the user via voice.

[0591] Input: User voice input

[0592] Output: The generated audio response

[0593] Step 7:

[0594] Security and monitoring functions

[0595] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary. For example, if the device's motion sensor detects abnormal activity in a room, it will report the situation to the server, stating that a "suspicious person has entered through a window," and the server will immediately send an alert to the user's smartphone.

[0596] Input: Anomaly detection data from cameras and sensors

[0597] Output: User notification

[0598] (Application example 1)

[0599] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Conventional home support systems have the ability to learn the user's behavioral patterns and operate home appliances, but they lack the ability to automatically switch security modes or notify users of suspicious activity in real time. This makes it difficult for users to leave their home in the care of the system while they are out. To solve this issue, there is a need for enhanced home security and monitoring functions and for real-time notification when suspicious activity is detected.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0602] In this invention, the server includes means for collecting user behavior data to learn user behavior patterns and provide support to the user, means for analyzing user behavior data and detecting behavior patterns, means for predicting the user's next action based on the detected behavior patterns, means for operating home appliances based on the prediction, means for recognizing user voice input and generating appropriate voice responses to have natural conversations with the user, means for controlling cameras and sensors for home security and surveillance, and means for automatically switching to security mode when the user goes out and for notifying in real time if suspicious activity is detected, allowing the user to leave their home in the care of the appliances while they are out with peace of mind.

[0603] "User behavior data" is data that records a user's daily behavior and lifestyle patterns.

[0604] "Behavioral patterns" refer to specific actions or habits that a user repeatedly performs.

[0605] "User voice input" refers to words or commands that a user speaks to a robot.

[0606] "Cameras and sensors" is a general term for optical and electronic devices used to detect images, sounds, and motion.

[0607] "Security mode" is an operating mode for strengthening security in the home when the user is away.

[0608] "Real-time notification" refers to notifying users immediately when abnormal or suspicious behavior is detected.

[0609] This invention is a specialized domestic butler robot system that learns the user's behavioral patterns and supports the user's daily life. This system is realized by combining various technologies, including the collection, analysis, and prediction of user behavior data, security functions, and natural conversation functions.

[0610] 1. Collecting user behavior data

[0611] The system is equipped with multiple sensors and cameras that monitor and record users' behavior in real time. These data provide a detailed record of users' daily movements, allowing the system to know which room the user is in and which appliances they are using.

[0612] 2. Analyzing user behavior data

[0613] The server analyzes the collected behavioral data to detect user behavior patterns. This process involves using a generative AI model to analyze patterns and predict the user's next action. To do this, the collected data is stored in a database and the AI ​​model learns from it.

[0614] 3. Controlling home appliances

[0615] Based on the user's behavioral patterns, the server predicts the user's next actions and operates the necessary home appliances. For example, the server can set the living room lights and TV to turn off at 10 p.m. every night. This allows the user to automatically operate the home appliances and provide a comfortable living environment.

[0616] 4. Realizing natural conversation

[0617] The device recognizes voice input from the user and generates an appropriate response. For example, if a user asks, "What's the weather like tomorrow?", the server-based generative AI model generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees," and conveys this to the user via voice. This process uses natural language processing technology to analyze the user's intent.

[0618] 5. Security features

[0619] When the user leaves the house, the system automatically switches to security mode and monitors the home using cameras and sensors for any unusual activity. If any suspicious activity is detected, the system sends a real-time notification to the user's smartphone, urging immediate action. If an abnormality is detected, an alarm will sound and the system can also notify security authorities.

[0620] 6. Program Processing

[0621] The server and device collect user behavior data in real time and store it in a database. Generative AI models are used to analyze and predict the data, and the results are used to automatically operate home appliances. Voice recognition software and natural language processing technology are combined to enable natural conversations with users. For security purposes, the system analyzes data detected by cameras and sensors, and notifies users in real time if an abnormality is detected.

[0622] As a concrete example, if a user asks, "What time is a good time to leave the house today?", they can input the following prompt into the generative AI model:

[0623] "The system learns the pattern of when users leave home at 8 o'clock every morning and automatically switches on security mode. This learning is based on simple rules, but in the future, we will be able to predict behavior with greater accuracy by introducing reinforcement learning using generative AI. Specifically, we will input the following prompt into the generative AI: 'Please tell me how I can learn the patterns of when users often leave home and automatically switch on security mode according to that time of day.'"

[0624] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0625] Step 1:

[0626] Collecting user behavior data

[0627] The device uses multiple sensors and cameras installed in the home to collect user behavior data in real time. Specifically, it obtains information on which room the user is in and which home appliances they are using. This behavior data is sent from the device to a server and stored in a database.

[0628] Input: User behavior data from sensors and cameras.

[0629] Output: The collected user behavior data is sent to the server and stored in a database.

[0630] Step 2:

[0631] Analyzing user behavior data

[0632] The server analyzes the user behavior data stored in the database and detects specific behavioral patterns. Using a generative AI model, it learns from this data and discovers user behavioral patterns. For example, it discovers that the user wakes up at 7am every morning and watches TV in the living room.

[0633] Input: User behavior data stored in a database.

[0634] Output: Analyzed by a generative AI model and detected as a behavioral pattern.

[0635] Step 3:

[0636] Predicting next actions based on behavioral patterns

[0637] The server predicts the user's next action based on the detected user behavior patterns. The generative AI model makes predictions based on past data, and the server generates predictions based on the user's lifestyle habits. For example, it predicts that the TV will be turned off at 10 p.m.

[0638] Input: Detected behavioral patterns.

[0639] Output: Predicted result of next action.

[0640] Step 4:

[0641] Automatic operation of home appliances

[0642] The server automatically controls household appliances based on the prediction results. For example, if the prediction is to "turn off the TV at 10 o'clock," the server will issue a command to turn off the TV at that time. This provides a comfortable living environment for the user.

[0643] Input: Predicted outcome of the action.

[0644] Output: Status of automatically operated appliances.

[0645] Step 5:

[0646] Achieving natural conversations with users

[0647] The device uses a voice recognition system to receive voice input from the user, and the server uses a generative AI model to generate appropriate responses to the user's questions and commands, which are then spoken to the user via the device.

[0648] Input: User's voice input.

[0649] Output: Natural-sounding conversational responses generated by the generative AI model.

[0650] Step 6:

[0651] Security features implemented

[0652] When the device detects that the user has left the house, it automatically switches to security mode. Cameras and sensors monitor for suspicious activity, and if the server detects an abnormality, it sends a real-time notification to the user's smartphone.

[0653] Input: External environment data from cameras and sensors.

[0654] Output: Real-time notification of suspicious behavior.

[0655] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0656] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[0657] 1. Collecting user behavior data

[0658] The device uses cameras, microphones, and sensors installed in the home to collect real-time user behavior data. For example, by recording which room the user is in and which home appliances they are using, the device can understand the user's daily behavioral patterns. This data includes the user's movements, movement history, and usage of home appliances.

[0659] 2. Analyzing user behavior data and detecting patterns

[0660] The collected user behavior data is sent to a server, which analyzes it and detects specific behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This provides the basis for predicting the user's next behavior.

[0661] 3. Emotion Recognition by Emotion Engine

[0662] The device recognizes the user's voice and facial expressions and records them as emotional data. For example, if the user looks tired, the emotion engine will recognize this as a "fatigue state." This emotional data is also sent to the server along with behavioral data for analysis.

[0663] 4. Behavior prediction and home appliance operation

[0664] The server predicts the user's next action based on the detected behavioral patterns and emotional data. This prediction result is then sent back to the device. For example, if the user has a habit of watching TV at 10 p.m. and the emotional data for that day indicates that they want to relax, the device will dim the lights and automatically turn on the TV at that time.

[0665] 5. Realizing natural conversation

[0666] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. For example, if the user says, "I'm tired today," the server's AI will respond, "Thank you for your hard work. I'll play some relaxing music." This response is generated taking into account the user's emotional state.

[0667] 6. Security and monitoring functions

[0668] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0669] Specific examples

[0670] Morning support:

[0671] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[0672] Remote work support:

[0673] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[0674] Through the above-described specific processing and functions, the specialized domestic butler robot of the present invention comprehensively learns the user's behavior and emotions, provides more effective support, and provides a comfortable home environment.

[0675] The processing flow will be explained below.

[0676] Specific flow of program processing

[0677] User behavior pattern learning

[0678] Step 1:

[0679] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[0680] Step 2:

[0681] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[0682] Step 3:

[0683] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[0684] Step 4:

[0685] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[0686] Step 5:

[0687] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[0688] Emotion data collection using an emotion engine

[0689] Step 1:

[0690] The user interacts with the device and performs everyday activities. For example, the user might say, "I'm tired today."

[0691] Step 2:

[0692] The device detects the user's voice and facial expressions, and analyzes this data using an emotion engine. The device recognizes the user's emotional state (e.g., "fatigue," "comfort," etc.) from the voice data and facial image data.

[0693] Step 3:

[0694] The device sends the recognized emotion data to the server, where it is integrated with the behavioral data and analyzed.

[0695] Behavior prediction and home appliance operation

[0696] Step 1:

[0697] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. For example, if the user has a habit of watching TV at 10 p.m., it predicts the user's emotional state for that day as "relaxed."

[0698] Step 2:

[0699] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[0700] Step 3:

[0701] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[0702] Realizing natural conversation

[0703] Step 1:

[0704] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[0705] Step 2:

[0706] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[0707] Step 3:

[0708] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[0709] Step 4:

[0710] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[0711] Security and monitoring functions

[0712] Step 1:

[0713] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[0714] Step 2:

[0715] The device turns off home appliances such as lights, televisions, and air conditioners.

[0716] Step 3:

[0717] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[0718] Step 4:

[0719] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[0720] Step 5:

[0721] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[0722] Specific examples

[0723] Morning support:

[0724] Step 1:

[0725] The system learns that the user wakes up at 7am every morning.

[0726] Step 2:

[0727] The emotion engine recognizes your emotional state from the past morning as "refreshed."

[0728] Step 3:

[0729] The device will turn on the lights at 7:00 and open the curtains, and will also provide a voice greeting such as, "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[0730] Remote work support:

[0731] Step 1:

[0732] The emotion engine recognizes that users are feeling stressed while working from home.

[0733] Step 2:

[0734] The device plays relaxing music before the meeting starts.

[0735] Step 3:

[0736] During the meeting, the device will adjust the air conditioning temperature appropriately and start the projector.

[0737] These steps will enable a comprehensive realization of learning user behavior patterns, collecting and analyzing emotional data, predictive home appliance operation, natural conversation, and crime prevention and monitoring functions.

[0738] Example 2

[0739] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0740] In modern homes, it is difficult to accurately grasp a user's behavioral patterns and emotional state and provide support based on that. Furthermore, while there is a demand for effectively combining security and monitoring functions, it is difficult to build a comprehensive system using existing technology. To address these issues, the present invention provides a system that learns a user's behavioral patterns and emotional state and provides optimal support.

[0741] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting user behavior data in real time using cameras, microphones, and sensors installed in the home; means for transmitting the collected user behavior data to the server and analyzing it to detect behavioral patterns; means for recognizing the user's voice and facial expressions and recording emotional data; means for transmitting the recorded emotional data to the server and using it for analysis; means for predicting the user's next behavior based on the detected behavioral patterns and emotional data; means for operating home appliances based on the prediction; means for recognizing voice input from the user and generating an appropriate voice response using a generative AI model; and means for detecting abnormal behavior in the home and controlling the cameras and sensors to provide security and monitoring functions. This makes it possible to comprehensively learn the user's behavior and emotions, provide effective support, and provide a comfortable and safe home environment.

[0742] A "camera" is a photographic device for capturing visual information.

[0743] A "microphone" is an acoustic transducer for collecting sound.

[0744] A "sensor" is a device that collects physical environmental information and has the ability to detect temperature, movement, light, etc.

[0745] "Behavioral data" refers to data that indicates information about a user's actions, movements, and the equipment they use.

[0746] A "server" is a computer system that analyzes, stores, and communicates data.

[0747] "Behavioral patterns" are data that show regularities and trends extracted from a series of user actions.

[0748] "Emotion data" refers to data that indicates the emotional state of a user that can be inferred from their speech and facial expressions.

[0749] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses or predictions from input data.

[0750] "Security" is a function that aims to protect the safety of the home by detecting intrusions by suspicious individuals and abnormal situations.

[0751] "Monitoring" is a function that checks the safety of children, elderly people, etc. and notifies you if any abnormal behavior occurs.

[0752] "Real-time" means that processing and reaction occur immediately in accordance with actual time.

[0753] "Voice input" refers to data that the system captures and processes from the user's speech.

[0754] "Speech recognition" is the technology that analyzes voice input and understands what it means.

[0755] A "voice response" is a voice message that the system responds to a user's voice input.

[0756] This invention relates to a specialized in-home butler robot system that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot system collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[0757] Collecting user behavior data

[0758] The device collects user behavior data in real time using cameras, microphones, and sensors installed in the home. This includes cameras and sensors installed in each room of the home. This allows the device to record which room the user is in and which home appliances they are using. This behavioral data includes the user's movements, movement history, and usage status of home appliances.

[0759] Analyzing user behavior data and detecting patterns

[0760] The collected behavioral data is sent from the device to a server, which analyzes the data to detect specific behavioral patterns. For example, the server may detect that the user wakes up at 7am every morning and turns on the living room light. This analysis is performed using a data analysis algorithm.

[0761] Emotion recognition by emotion engine

[0762] The device recognizes the user's voice and facial expressions in real time and records them as emotional data. For example, if a user says, "I'm tired today," the device's microphone captures this voice and converts it into text using voice recognition software. The emotion engine analyzes this and recognizes it as "fatigue." This emotional data is also sent to the server for further analysis.

[0763] Behavior prediction and home appliance operation

[0764] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. One example of this prediction is when a user has a habit of watching TV at 10pm, and the server determines from that day's emotional data that they "want to relax." The server then sends the device an instruction to "dim the lights and automatically turn on the TV at 10pm." The device receives this instruction and operates the home appliances.

[0765] Realizing natural conversation

[0766] To achieve natural conversation with the user, the device receives voice input and generates an appropriate response using a generative AI model. For example, if the user says, "I'm tired today," the server's generative AI model generates a response such as, "Thank you for your hard work. I'll play some relaxing music for you," and sends it to the device. The device then conveys this as voice to the user.

[0767] Security and monitoring functions

[0768] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0769] Specific examples

[0770] Morning support:

[0771] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[0772] Remote work support:

[0773] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[0774] Prompt Sentence Examples

[0775] When a user inputs "I'm tired today" into the generative AI model, the model responds, "Thank you for your hard work. I'll play some relaxing music right now."

[0776] Through these specific processing steps, the specialized domestic butler robot can comprehensively learn the user's behavior and emotions, provide more effective support, and provide a comfortable and safe home environment.

[0777] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0778] Step 1: Collect user behavior data

[0779] The device uses cameras, microphones, and sensors installed in the home to collect user behavior data in real time. Inputs include camera footage, audio data, and sensor data. This input data is processed and output as user location information and behavior data. Specifically, the camera detects the user's movements, the microphone records speech, and the sensors detect temperature and movement.

[0780] Step 2: Analyze user behavior data and detect patterns

[0781] The collected user behavior data is sent from the device to a server. The server analyzes this data and detects behavioral patterns. The input is the collected behavioral data. A data analysis algorithm is used to analyze regularity and frequency, and the detected behavioral patterns are output. Specifically, the server cross-references the user's behavior history and detects a pattern of waking up at 7am every morning and turning on the lights.

[0782] Step 3: Emotion recognition by the emotion engine

[0783] The device recognizes the user's voice and facial expressions and records them as emotional data. The input is the user's voice and facial expression data. Speech recognition software is used to convert the voice into text, and an emotional engine analyzes the emotional state. The analyzed emotional data is sent to the server. For example, if the user says, "I'm tired today," the voice input is output as "fatigue state."

[0784] Step 4: Behavior prediction and home appliance operation

[0785] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. The input is behavioral pattern data and emotional data. The server uses a prediction algorithm to predict the user's next action and sends the prediction result to the device. The device then executes instructions to operate home appliances. For example, if it predicts that the user will watch TV at 10 p.m., it dims the lights and turns on the TV.

[0786] Step 5: Achieving natural conversation

[0787] The device recognizes voice input from the user and uses a generative AI model to generate an appropriate voice response. The input is the user's voice data. This is converted into text data through speech recognition software, and the generative AI model generates a response. The device outputs the generated voice response. For example, if the user says, "I'm tired today," the generative AI will respond, "Thank you for your hard work. I'll play some relaxing music."

[0788] Step 6: Security and monitoring functions

[0789] When the user goes out, the device switches to security mode. The input is the away mode setting. The camera and sensors installed on the device monitor abnormal movements and detect the intrusion of suspicious individuals. Any detected abnormalities are sent to the server, which then sends an alert to the user in real time. For example, if a suspicious individual is detected, an alert notification is output to the user's smartphone.

[0790] (Application example 2)

[0791] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0792] The present invention relates to a system that learns user behavior patterns in the home, recognizes emotions, and provides optimal home appliance operation, natural conversation, security, and monitoring functions. However, conventional technologies have had challenges in detecting customer movements and emotions in real time within stores and economic activity areas, and in using this information to make appropriate product recommendations and improve operational efficiency. While conventional technology systems are suitable for the specific environment of the home, they are not suitable for use in a wide range of economic activity areas, making it difficult to simultaneously improve customer satisfaction and store operational efficiency. Furthermore, recognizing customer emotions in real time and dynamically responding based on this information also present technical challenges.

[0793] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user behavior data, means for analyzing the behavior data and detecting behavior patterns, means for predicting next behavior based on the behavior patterns, means for operating home appliances, means for recognizing voice input and generating voice responses, means for controlling cameras and sensors for security and surveillance, means for detecting customer movements and emotions within an economic activity site in real time, means for providing optimal product recommendations and location guidance to customers based on the analysis results, and means for using a generative AI model including a staffing suggestion function to improve operational efficiency. As a result, customer behavior and emotions can be recognized and analyzed in real time not only at home but also at economic activity sites, and optimal responses can be taken, thereby improving customer satisfaction and streamlining store operations.

[0794] "User behavior patterns" refers to trends in user behavior, movement, and home appliance usage.

[0795] "Behavioral data" is data based on the user's movements, location information, voice, facial expressions, etc.

[0796] "Behavioral pattern analysis" is the process of detecting specific behavioral tendencies and patterns of users based on collected behavioral data.

[0797] "Predicting next behavior" means estimating what behavior a user will take in the future based on analyzed behavioral patterns.

[0798] "Operating home appliances" means controlling household appliances such as air conditioners, lights, and televisions.

[0799] "User voice input" refers to speech uttered by a user that is analyzed by a voice recognition system.

[0800] "Generating a voice response" is the process of generating an appropriate response to a voice input.

[0801] "Crime prevention and surveillance" refers to ensuring safety within homes and stores and monitoring for abnormal situations.

[0802] "Cameras and sensors" is a general term for devices that detect images, sounds, and motion.

[0803] "Points of economic activity" refer to places where goods and services are provided, such as physical stores and commercial facilities.

[0804] "Real-time detection" is the process of collecting data instantly and analyzing it near-simultaneously.

[0805] "Product suggestion" refers to recommending products based on a customer's preferences and purchasing history.

[0806] "Location guidance" refers to providing guidance to the optimal route and location based on the customer's current location and purpose.

[0807] "Operational efficiency improvement" is the process of optimizing store and facility operations and increasing productivity.

[0808] The "staffing suggestion function" refers to a function that suggests appropriate staffing according to location and time of day.

[0809] A "generative AI model" is an artificial intelligence model that automatically generates responses and predictions based on data.

[0810] The present invention relates to a system for detecting and analyzing a user's movements and emotions in real time and providing optimal support. This system operates to effectively support users and customers at home or at points of economic activity. Specific embodiments are described below.

[0811] Hardware and Software

[0812] 1. Hardware Configuration

[0813] Camera: A device installed within an economic activity point to capture the movements and expressions of users and customers. For example, HD cameras and infrared cameras are used.

[0814] Microphone: A device used to collect voice input and recognize user or customer speech.

[0815] Sensors: Devices installed to collect movement data and environmental data. These include temperature sensors, motion sensors, etc.

[0816] Server: A central device for data collection, analysis, and control. It is desirable to have a high-performance processor and a large amount of memory.

[0817] 2. Software Configuration

[0818] Movement and Emotion Detection Software: Detects movements and facial expressions from camera footage using Python and OpenCV.

[0819] Speech recognition system: Analyzes voice data and converts it into text in real time. Google Speech-to-Text API is used.

[0820] Emotion recognition engine: Using a deep learning model with TensorFlow and Keras, emotions are recognized from collected voice and facial expression data.

[0821] Server-side processing: Back-end processing and data management are performed using Node.js and Express.

[0822] Database management system: MongoDB is used to store and query the collected data.

[0823] Processing flow

[0824] 1. Data collection

[0825] The server collects user and customer behavioral and voice data through cameras and microphones.

[0826] Data is also collected from sensors, which transmit location and environmental data of users and customers to a server.

[0827] 2. Data Analysis

[0828] The server uses Python scripts to analyze video data from the cameras and detect customer movements and behavior patterns.

[0829] An emotion recognition engine using TensorFlow and Keras analyzes voice and facial expression data to identify the emotional state of users and customers in real time.

[0830] 3. Creating Actions

[0831] A server-side Node.js application uses the analyzed data to predict the next action and, based on the results, optimally operate home appliances and make product suggestions.

[0832] The server uses a generative AI model to generate appropriate responses to voice input.

[0833] 4. Examples and Prompts

[0834] Example: If a customer spends too long at a particular shelf and their facial expression indicates confusion, the server will notify a nearby staff member and instruct them to explain the product. An audio guide will also ask, "Can I help you with this product?"

[0835] Example prompt sentence:

[0836] "Explain how you can use customer facial expression data to estimate their current emotional state and provide a pleasant in-store experience."

[0837] "Please show us an algorithm that analyzes customer movement data and purchase history and recommends products that are likely to be purchased again."

[0838] This will realize a system that can comprehensively recognize and analyze the behavior and emotions of users and customers at home and at points of economic activity, and respond appropriately. This system is designed to improve customer satisfaction and streamline store operations.

[0839] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0840] Step 1:

[0841] The device collects user movements and voice in real time from cameras and microphones installed in the physical store. As input, it acquires camera video data and audio data. The camera data is broken down into video frames, and the audio data is converted into a frequency spectrum. As output, the acquired frame data and audio spectrum data are sent to the server.

[0842] Step 2:

[0843] The server uses Python and OpenCV to analyze the user's movements from the received frame data. Camera image data is used as input. Data processing involves extracting movement data using background subtraction and object detection algorithms. User movement data is generated as output.

[0844] Step 3:

[0845] The server uses TensorFlow and Keras to analyze the user's emotions from voice data and facial expression data. Voice spectrum data and facial frame data are used as input. Data calculations are performed to classify the emotional state using a voice emotion recognition model and a facial expression recognition model. Emotional state data is generated as output.

[0846] Step 4:

[0847] The server predicts the user's next behavior based on the analyzed behavioral and emotional data. The inputs are movement data and emotional data. The data is calculated using a time series analysis method to build a behavioral pattern model, and the next behavior is estimated using a prediction algorithm. The output is next behavior prediction data.

[0848] Step 5:

[0849] The server uses a generative AI model to generate optimal home appliance operations and product suggestions based on the predicted behavior and emotion data. Behavior prediction data and emotion data are used as input. Data processing involves selecting appropriate responses and actions based on pre-set scenarios and rules. Specific operation commands and suggestions are generated as output.

[0850] Step 6:

[0851] The server sends the generated operation commands and suggestions to terminals in the store to control the appliances and provide product information. The operation commands and suggestions from the generative AI model are used as input. As data calculations, the terminal executes the commands and feeds the results back to the server. As output, data on the status of the appliances and customer reactions is generated.

[0852] Step 7:

[0853] The server updates the generative AI model based on the feedback data, improving accuracy from the next time onwards. Execution result data and customer response data are used as input. Data processing involves retraining the model using a machine learning algorithm. An updated generative AI model is generated as output.

[0854] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0855] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0856] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0857] [Third embodiment]

[0858] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0859] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0860] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0861] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0862] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0863] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0864] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0865] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0866] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0867] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0868] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0869] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0870] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[0871] 1. Collecting user behavior data

[0872] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[0873] 2. Analyzing user behavior data and detecting patterns

[0874] The collected user behavior data is sent to a server. The server analyzes this data and detects certain behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This pattern information is used to predict the user's next action.

[0875] 3. Predict your next move and operate your home appliances

[0876] The server predicts the user's next action based on the detected behavioral patterns. This prediction result is then sent back to the device. For example, if the user has a pattern of watching TV every night at 10 p.m., the device will automatically turn on the TV at that time.

[0877] 4. Realizing natural conversation

[0878] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. If the user asks, "What's the weather like tomorrow?", the device analyzes this voice input, and the server's generation AI generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees." The device then conveys this to the user via voice.

[0879] 5. Security and monitoring functions

[0880] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[0881] Specific examples

[0882] Morning support:

[0883] If the device learns that the user wakes up at 7:00 every morning, it will turn on the lights and open the curtains at 7:00. It will also provide weather information via voice, along with a greeting such as "Good morning. It's sunny today."

[0884] Remote work support:

[0885] When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[0886] Through the above-described specific processes and functions, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[0887] The processing flow will be explained below.

[0888] Specific flow of program processing

[0889] User behavior pattern learning

[0890] Step 1:

[0891] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[0892] Step 2:

[0893] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[0894] Step 3:

[0895] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[0896] Step 4:

[0897] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[0898] Step 5:

[0899] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[0900] Behavior prediction and home appliance operation

[0901] Step 1:

[0902] The server predicts the user's next action based on the analyzed behavioral patterns, for example, predicting that the user has a habit of watching TV at 10 p.m.

[0903] Step 2:

[0904] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[0905] Step 3:

[0906] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[0907] Realizing natural conversation

[0908] Step 1:

[0909] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[0910] Step 2:

[0911] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[0912] Step 3:

[0913] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[0914] Step 4:

[0915] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[0916] Security and monitoring functions

[0917] Step 1:

[0918] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[0919] Step 2:

[0920] The device turns off home appliances such as lights, televisions, and air conditioners.

[0921] Step 3:

[0922] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[0923] Step 4:

[0924] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[0925] Step 5:

[0926] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[0927] Through the above steps, learning of user behavior patterns, predictive operation of home appliances, natural conversation, crime prevention and monitoring functions can be effectively realized.

[0928] Example 1

[0929] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0930] In modern homes, systems that learn user behavior patterns and automatically operate and respond to the necessary home appliances are needed to make users' daily lives more efficient and comfortable. Functions such as monitoring elderly people and children and crime prevention are also required. However, existing systems often have separate functions and do not provide comprehensive support. This creates the complexity of having to install multiple systems and operate them individually.

[0931] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0932] In this invention, the server includes means for collecting user behavior data in real time using multiple sensors, cameras, and microphones, means for transmitting the collected behavior data to the server, and means for analyzing the user behavior data and detecting behavior patterns in the server, thereby enabling more efficient support for users' daily lives and improving comfort and safety in the home.

[0933] A "sensor" is a device that detects physical phenomena (temperature, humidity, movement, etc.) and provides that information as data.

[0934] A "camera" is a device used to capture images and videos and to visually record a user's actions and situations.

[0935] A "microphone" is a device that picks up sound and converts it into an electrical signal, and is used to obtain a user's voice input.

[0936] A "server" is a computing device for receiving, storing, analyzing data, and generating a response.

[0937] "Behavioral data" refers to data that includes information related to a user's daily life, such as the user's specific movements, location, and voice.

[0938] "Analysis" is the process of extracting meaning and patterns from collected data.

[0939] "Behavioral patterns" refer to repetition of a user's behavior or tendencies in behavior at a specific time.

[0940] A "machine learning algorithm" is a method of learning specific trends and patterns based on large amounts of data and making predictions and classifications for new data.

[0941] A "generative AI model" is an artificial intelligence model that generates responses or results that are meaningful to humans based on given input data.

[0942] "Speech recognition means" is a technology that converts voice input into text data.

[0943] "Crime prevention" refers to activities aimed at protecting physical spaces from unauthorized intrusion and crime.

[0944] "Monitoring" involves monitoring elderly people, children, and others to ensure their safety, and taking appropriate action if any abnormalities are detected.

[0945] "Encryption" is a technology that converts data using a specific algorithm to make it difficult for third parties to decipher.

[0946] "Real-time" means that processing or response occurs immediately after data is collected, and refers to operations or processes that require immediacy.

[0947] "Home comfort" refers to the physical and psychological satisfaction that users feel when spending time at home.

[0948] A "user interface" is the means or screen layout by which a user directly interacts with a system or device.

[0949] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[0950] Hardware and Software Configuration

[0951] Terminal: Infrared sensor, environmental sensor, IP camera, microphone

[0952] Server: A computer device for data analysis. Software used includes Python, Pandas, scikit-learn, TensorFlow, etc.

[0953] Network: Use secure protocols such as HTTPS for data transmission.

[0954] Speech Recognition: Speech recognition technologies such as Google Cloud Speech-to-Text.

[0955] Text-to-speech: Text-to-speech technologies such as Google Cloud Text-to-Speech.

[0956] Collecting user behavior data

[0957] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[0958] Example: If a user is listening to music in their living room, the device's camera captures their movements, the microphone captures audio data, and the environmental sensors record the temperature and humidity in the living room.

[0959] Data transmission and analysis

[0960] The collected user behavior data is sent from the device to a server, which receives the data and stores it in a database. It then uses machine learning algorithms to analyze the behavioral patterns and detect specific behavioral patterns.

[0961] Example: Based on data from the past week, the server detects a pattern: "The user wakes up at 7am every morning and has breakfast at 8am."

[0962] Behavior prediction and automatic operation of home appliances

[0963] The server predicts the user's next action based on the detected behavioral patterns, and the prediction results are sent back to the device, allowing the appliance to prepare ahead of the user's actions.

[0964] Example: The server predicts that the user will have breakfast at 8:00 and sends that information to the device. The device then turns on the kitchen lights and starts the coffee maker at 8:00. It also adjusts the air conditioner temperature appropriately.

[0965] Realizing natural conversation

[0966] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response, which is then sent back to the device and conveyed to the user using speech synthesis software.

[0967] Example: When a user asks, "What's the weather going to be like tomorrow?", the device converts the speech to text and sends it to a server. The server generates "It's going to be sunny tomorrow," which the device then replies to the user by voice.

[0968] Security and monitoring functions

[0969] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary.

[0970] Example: If the motion sensor in the room detects abnormal activity, for example, that a suspicious person has entered through the window, the device reports the situation to the server, and the server immediately sends an alert to the user's smartphone.

[0971] Examples of specific examples and prompts

[0972] Wake-up support: If the device learns that the user wakes up at 7:00 a.m. every morning, it will turn on the lights and open the curtains at 7:00 a.m. It will also provide weather information by voice, along with a greeting such as "Good morning. It's sunny today."

[0973] Example prompt: "When the user wakes up at 7:00, turn on the lights and give them the weather report."

[0974] Support for remote work: When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[0975] Example prompt: "Please notify me 10 minutes before a meeting starts and adjust the air conditioning temperature when users are working remotely."

[0976] As described above, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[0977] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0978] Step 1:

[0979] Collecting user behavior data

[0980] The device collects user behavior data in real time using infrared sensors, environmental sensors, IP cameras, and microphones. Specifically, the device's camera captures the user's movements, and the microphone captures audio data. The environmental sensors collect data such as temperature and humidity. This data indicates which room the user is in and which appliances they are using. For example, if a user is listening to music in the living room, the camera records the user's movements and the microphone captures their audio.

[0981] Input: Data from infrared sensors, environmental sensors, IP cameras, microphones

[0982] Output: User behavior data (location, movement, audio, environmental data)

[0983] Step 2:

[0984] Sending behavioral data to the server

[0985] The device encrypts the collected user behavior data and sends it to a server via the Internet. Specifically, the device batches the collected data and sends it to the server using a secure protocol such as HTTPS. For example, data on the user's presence in the living room is sent to the server every hour.

[0986] Input: Collected user behavior data

[0987] Output: Data sent to the server

[0988] Step 3:

[0989] Behavioral data analysis and pattern detection

[0990] The server stores the received user behavior data in a database and analyzes the data using analytical software (e.g., Python, Pandas). The server then analyzes the data using machine learning algorithms (e.g., scikit-learn, TensorFlow) to detect specific behavioral patterns. For example, the server might detect a pattern such as "the user wakes up at 7:00 every morning and eats breakfast at 8:00."

[0991] Input: User behavior data sent to the server

[0992] Output: Detected behavioral patterns

[0993] Step 4:

[0994] Predicting next actions

[0995] The server predicts the user's next action based on the detected behavioral patterns. The server uses a machine learning model to predict the user's next action based on past behavioral data. The prediction result is then sent back to the device. For example, a prediction that "the user will have breakfast at 8 o'clock" is sent to the device.

[0996] Input: Detected behavioral patterns

[0997] Output: Predicted data of next action

[0998] Step 5:

[0999] Automatic operation of home appliances

[1000] The device automatically controls home appliances based on the next action prediction received from the server. Specifically, the device controls home appliances through smart plugs and smart remote controls. For example, the device might turn on the kitchen lights and start the coffee maker at 8:00 a.m., and also adjust the air conditioner temperature appropriately.

[1001] Input: Next action prediction data received from the server

[1002] Output: Operation instructions for home appliances

[1003] Step 6:

[1004] Realizing natural conversation

[1005] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response. The generated response is then sent back to the device, where it is communicated to the user using speech synthesis software (e.g., Google Cloud Text-to-Speech). For example, if a user asks, "What's the weather like tomorrow?" the device converts the speech into text and sends it to the server. The server generates a response, which the device then returns to the user via voice.

[1006] Input: User voice input

[1007] Output: The generated audio response

[1008] Step 7:

[1009] Security and monitoring functions

[1010] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary. For example, if the device's motion sensor detects abnormal activity in a room, it will report the situation to the server, stating that a "suspicious person has entered through a window," and the server will immediately send an alert to the user's smartphone.

[1011] Input: Anomaly detection data from cameras and sensors

[1012] Output: User notification

[1013] (Application example 1)

[1014] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] Conventional home support systems have the ability to learn the user's behavioral patterns and operate home appliances, but they lack the ability to automatically switch security modes or notify users of suspicious activity in real time. This makes it difficult for users to leave their home in the care of the system while they are out. To solve this issue, there is a need for enhanced home security and monitoring functions and for real-time notification when suspicious activity is detected.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1017] In this invention, the server includes means for collecting user behavior data to learn user behavior patterns and provide support to the user, means for analyzing user behavior data and detecting behavior patterns, means for predicting the user's next action based on the detected behavior patterns, means for operating home appliances based on the prediction, means for recognizing user voice input and generating appropriate voice responses to have natural conversations with the user, means for controlling cameras and sensors for home security and surveillance, and means for automatically switching to security mode when the user goes out and for notifying in real time if suspicious activity is detected, allowing the user to leave their home in the care of the appliances while they are out with peace of mind.

[1018] "User behavior data" is data that records a user's daily behavior and lifestyle patterns.

[1019] "Behavioral patterns" refer to specific actions or habits that a user repeatedly performs.

[1020] "User voice input" refers to words or commands that a user speaks to a robot.

[1021] "Cameras and sensors" is a general term for optical and electronic devices used to detect images, sounds, and motion.

[1022] "Security mode" is an operating mode for strengthening security in the home when the user is away.

[1023] "Real-time notification" refers to notifying users immediately when abnormal or suspicious behavior is detected.

[1024] This invention is a specialized domestic butler robot system that learns the user's behavioral patterns and supports the user's daily life. This system is realized by combining various technologies, including the collection, analysis, and prediction of user behavior data, security functions, and natural conversation functions.

[1025] 1. Collecting user behavior data

[1026] The system is equipped with multiple sensors and cameras that monitor and record users' behavior in real time. These data provide a detailed record of users' daily movements, allowing the system to know which room the user is in and which appliances they are using.

[1027] 2. Analyzing user behavior data

[1028] The server analyzes the collected behavioral data to detect user behavior patterns. This process involves using a generative AI model to analyze patterns and predict the user's next action. To do this, the collected data is stored in a database and the AI ​​model learns from it.

[1029] 3. Controlling home appliances

[1030] Based on the user's behavioral patterns, the server predicts the user's next actions and operates the necessary home appliances. For example, the server can set the living room lights and TV to turn off at 10 p.m. every night. This allows the user to automatically operate the home appliances and provide a comfortable living environment.

[1031] 4. Realizing natural conversation

[1032] The device recognizes voice input from the user and generates an appropriate response. For example, if a user asks, "What's the weather like tomorrow?", the server-based generative AI model generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees," and conveys this to the user via voice. This process uses natural language processing technology to analyze the user's intent.

[1033] 5. Security features

[1034] When the user leaves the house, the system automatically switches to security mode and monitors the home using cameras and sensors for any unusual activity. If any suspicious activity is detected, the system sends a real-time notification to the user's smartphone, urging immediate action. If an abnormality is detected, an alarm will sound and the system can also notify security authorities.

[1035] 6. Program Processing

[1036] The server and device collect user behavior data in real time and store it in a database. Generative AI models are used to analyze and predict the data, and the results are used to automatically operate home appliances. Voice recognition software and natural language processing technology are combined to enable natural conversations with users. For security purposes, the system analyzes data detected by cameras and sensors, and notifies users in real time if an abnormality is detected.

[1037] As a concrete example, if a user asks, "What time is a good time to leave the house today?", they can input the following prompt into the generative AI model:

[1038] "The system learns the pattern of when users leave home at 8 o'clock every morning and automatically switches on security mode. This learning is based on simple rules, but in the future, we will be able to predict behavior with greater accuracy by introducing reinforcement learning using generative AI. Specifically, we will input the following prompt into the generative AI: 'Please tell me how I can learn the patterns of when users often leave home and automatically switch on security mode according to that time of day.'"

[1039] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1040] Step 1:

[1041] Collecting user behavior data

[1042] The device uses multiple sensors and cameras installed in the home to collect user behavior data in real time. Specifically, it obtains information on which room the user is in and which home appliances they are using. This behavior data is sent from the device to a server and stored in a database.

[1043] Input: User behavior data from sensors and cameras.

[1044] Output: The collected user behavior data is sent to the server and stored in a database.

[1045] Step 2:

[1046] Analyzing user behavior data

[1047] The server analyzes the user behavior data stored in the database and detects specific behavioral patterns. Using a generative AI model, it learns from this data and discovers user behavioral patterns. For example, it discovers that the user wakes up at 7am every morning and watches TV in the living room.

[1048] Input: User behavior data stored in a database.

[1049] Output: Analyzed by a generative AI model and detected as a behavioral pattern.

[1050] Step 3:

[1051] Predicting next actions based on behavioral patterns

[1052] The server predicts the user's next action based on the detected user behavior patterns. The generative AI model makes predictions based on past data, and the server generates predictions based on the user's lifestyle habits. For example, it predicts that the TV will be turned off at 10 p.m.

[1053] Input: Detected behavioral patterns.

[1054] Output: Predicted result of next action.

[1055] Step 4:

[1056] Automatic operation of home appliances

[1057] The server automatically controls household appliances based on the prediction results. For example, if the prediction is to "turn off the TV at 10 o'clock," the server will issue a command to turn off the TV at that time. This provides a comfortable living environment for the user.

[1058] Input: Predicted outcome of the action.

[1059] Output: Status of automatically operated appliances.

[1060] Step 5:

[1061] Achieving natural conversations with users

[1062] The device uses a voice recognition system to receive voice input from the user, and the server uses a generative AI model to generate appropriate responses to the user's questions and commands, which are then spoken to the user via the device.

[1063] Input: User's voice input.

[1064] Output: Natural-sounding conversational responses generated by the generative AI model.

[1065] Step 6:

[1066] Security features implemented

[1067] When the device detects that the user has left the house, it automatically switches to security mode. Cameras and sensors monitor for suspicious activity, and if the server detects an abnormality, it sends a real-time notification to the user's smartphone.

[1068] Input: External environment data from cameras and sensors.

[1069] Output: Real-time notification of suspicious behavior.

[1070] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1071] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[1072] 1. Collecting user behavior data

[1073] The device uses cameras, microphones, and sensors installed in the home to collect real-time user behavior data. For example, by recording which room the user is in and which home appliances they are using, the device can understand the user's daily behavioral patterns. This data includes the user's movements, movement history, and usage of home appliances.

[1074] 2. Analyzing user behavior data and detecting patterns

[1075] The collected user behavior data is sent to a server, which analyzes it and detects specific behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This provides the basis for predicting the user's next behavior.

[1076] 3. Emotion Recognition by Emotion Engine

[1077] The device recognizes the user's voice and facial expressions and records them as emotional data. For example, if the user looks tired, the emotion engine will recognize this as a "fatigue state." This emotional data is also sent to the server along with behavioral data for analysis.

[1078] 4. Behavior prediction and home appliance operation

[1079] The server predicts the user's next action based on the detected behavioral patterns and emotional data. This prediction result is then sent back to the device. For example, if the user has a habit of watching TV at 10 p.m. and the emotional data for that day indicates that they want to relax, the device will dim the lights and automatically turn on the TV at that time.

[1080] 5. Realizing natural conversation

[1081] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. For example, if the user says, "I'm tired today," the server's AI will respond, "Thank you for your hard work. I'll play some relaxing music." This response is generated taking into account the user's emotional state.

[1082] 6. Security and monitoring functions

[1083] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[1084] Specific examples

[1085] Morning support:

[1086] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[1087] Remote work support:

[1088] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[1089] Through the above-described specific processing and functions, the specialized domestic butler robot of the present invention comprehensively learns the user's behavior and emotions, provides more effective support, and provides a comfortable home environment.

[1090] The processing flow will be explained below.

[1091] Specific flow of program processing

[1092] User behavior pattern learning

[1093] Step 1:

[1094] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[1095] Step 2:

[1096] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[1097] Step 3:

[1098] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[1099] Step 4:

[1100] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[1101] Step 5:

[1102] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[1103] Emotion data collection using an emotion engine

[1104] Step 1:

[1105] The user interacts with the device and performs everyday activities. For example, the user might say, "I'm tired today."

[1106] Step 2:

[1107] The device detects the user's voice and facial expressions, and analyzes this data using an emotion engine. The device recognizes the user's emotional state (e.g., "fatigue," "comfort," etc.) from the voice data and facial image data.

[1108] Step 3:

[1109] The device sends the recognized emotion data to the server, where it is integrated with the behavioral data and analyzed.

[1110] Behavior prediction and home appliance operation

[1111] Step 1:

[1112] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. For example, if the user has a habit of watching TV at 10 p.m., it predicts the user's emotional state for that day as "relaxed."

[1113] Step 2:

[1114] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[1115] Step 3:

[1116] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[1117] Realizing natural conversation

[1118] Step 1:

[1119] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[1120] Step 2:

[1121] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[1122] Step 3:

[1123] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[1124] Step 4:

[1125] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[1126] Security and monitoring functions

[1127] Step 1:

[1128] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[1129] Step 2:

[1130] The device turns off home appliances such as lights, televisions, and air conditioners.

[1131] Step 3:

[1132] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[1133] Step 4:

[1134] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[1135] Step 5:

[1136] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[1137] Specific examples

[1138] Morning support:

[1139] Step 1:

[1140] The system learns that the user wakes up at 7am every morning.

[1141] Step 2:

[1142] The emotion engine recognizes your emotional state from the past morning as "refreshed."

[1143] Step 3:

[1144] The device will turn on the lights at 7:00 and open the curtains, and will also provide a voice greeting such as, "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[1145] Remote work support:

[1146] Step 1:

[1147] The emotion engine recognizes that users are feeling stressed while working from home.

[1148] Step 2:

[1149] The device plays relaxing music before the meeting starts.

[1150] Step 3:

[1151] During the meeting, the device will adjust the air conditioning temperature appropriately and start the projector.

[1152] These steps will enable a comprehensive realization of learning user behavior patterns, collecting and analyzing emotional data, predictive home appliance operation, natural conversation, and crime prevention and monitoring functions.

[1153] Example 2

[1154] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1155] In modern homes, it is difficult to accurately grasp a user's behavioral patterns and emotional state and provide support based on that. Furthermore, while there is a demand for effectively combining security and monitoring functions, it is difficult to build a comprehensive system using existing technology. To address these issues, the present invention provides a system that learns a user's behavioral patterns and emotional state and provides optimal support.

[1156] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting user behavior data in real time using cameras, microphones, and sensors installed in the home; means for transmitting the collected user behavior data to the server and analyzing it to detect behavioral patterns; means for recognizing the user's voice and facial expressions and recording emotional data; means for transmitting the recorded emotional data to the server and using it for analysis; means for predicting the user's next behavior based on the detected behavioral patterns and emotional data; means for operating home appliances based on the prediction; means for recognizing voice input from the user and generating an appropriate voice response using a generative AI model; and means for detecting abnormal behavior in the home and controlling the cameras and sensors to provide security and monitoring functions. This makes it possible to comprehensively learn the user's behavior and emotions, provide effective support, and provide a comfortable and safe home environment.

[1157] A "camera" is a photographic device for capturing visual information.

[1158] A "microphone" is an acoustic transducer for collecting sound.

[1159] A "sensor" is a device that collects physical environmental information and has the ability to detect temperature, movement, light, etc.

[1160] "Behavioral data" refers to data that indicates information about a user's actions, movements, and the equipment they use.

[1161] A "server" is a computer system that analyzes, stores, and communicates data.

[1162] "Behavioral patterns" are data that show regularities and trends extracted from a series of user actions.

[1163] "Emotion data" refers to data that indicates the emotional state of a user that can be inferred from their speech and facial expressions.

[1164] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses or predictions from input data.

[1165] "Security" is a function that aims to protect the safety of the home by detecting intrusions by suspicious individuals and abnormal situations.

[1166] "Monitoring" is a function that checks the safety of children, elderly people, etc. and notifies you if any abnormal behavior occurs.

[1167] "Real-time" means that processing and reaction occur immediately in accordance with actual time.

[1168] "Voice input" refers to data that the system captures and processes from the user's speech.

[1169] "Speech recognition" is the technology that analyzes voice input and understands what it means.

[1170] A "voice response" is a voice message that the system responds to a user's voice input.

[1171] This invention relates to a specialized in-home butler robot system that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot system collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[1172] Collecting user behavior data

[1173] The device collects user behavior data in real time using cameras, microphones, and sensors installed in the home. This includes cameras and sensors installed in each room of the home. This allows the device to record which room the user is in and which home appliances they are using. This behavioral data includes the user's movements, movement history, and usage status of home appliances.

[1174] Analyzing user behavior data and detecting patterns

[1175] The collected behavioral data is sent from the device to a server, which analyzes the data to detect specific behavioral patterns. For example, the server may detect that the user wakes up at 7am every morning and turns on the living room light. This analysis is performed using a data analysis algorithm.

[1176] Emotion recognition by emotion engine

[1177] The device recognizes the user's voice and facial expressions in real time and records them as emotional data. For example, if a user says, "I'm tired today," the device's microphone captures this voice and converts it into text using voice recognition software. The emotion engine analyzes this and recognizes it as "fatigue." This emotional data is also sent to the server for further analysis.

[1178] Behavior prediction and home appliance operation

[1179] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. One example of this prediction is when a user has a habit of watching TV at 10pm, and the server determines from that day's emotional data that they "want to relax." The server then sends the device an instruction to "dim the lights and automatically turn on the TV at 10pm." The device receives this instruction and operates the home appliances.

[1180] Realizing natural conversation

[1181] To achieve natural conversation with the user, the device receives voice input and generates an appropriate response using a generative AI model. For example, if the user says, "I'm tired today," the server's generative AI model generates a response such as, "Thank you for your hard work. I'll play some relaxing music for you," and sends it to the device. The device then conveys this as voice to the user.

[1182] Security and monitoring functions

[1183] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[1184] Specific examples

[1185] Morning support:

[1186] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[1187] Remote work support:

[1188] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[1189] Prompt Sentence Examples

[1190] When a user inputs "I'm tired today" into the generative AI model, the model responds, "Thank you for your hard work. I'll play some relaxing music right now."

[1191] Through these specific processing steps, the specialized domestic butler robot can comprehensively learn the user's behavior and emotions, provide more effective support, and provide a comfortable and safe home environment.

[1192] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1193] Step 1: Collect user behavior data

[1194] The device uses cameras, microphones, and sensors installed in the home to collect user behavior data in real time. Inputs include camera footage, audio data, and sensor data. This input data is processed and output as user location information and behavior data. Specifically, the camera detects the user's movements, the microphone records speech, and the sensors detect temperature and movement.

[1195] Step 2: Analyze user behavior data and detect patterns

[1196] The collected user behavior data is sent from the device to a server. The server analyzes this data and detects behavioral patterns. The input is the collected behavioral data. A data analysis algorithm is used to analyze regularity and frequency, and the detected behavioral patterns are output. Specifically, the server cross-references the user's behavior history and detects a pattern of waking up at 7am every morning and turning on the lights.

[1197] Step 3: Emotion recognition by the emotion engine

[1198] The device recognizes the user's voice and facial expressions and records them as emotional data. The input is the user's voice and facial expression data. Speech recognition software is used to convert the voice into text, and an emotional engine analyzes the emotional state. The analyzed emotional data is sent to the server. For example, if the user says, "I'm tired today," the voice input is output as "fatigue state."

[1199] Step 4: Behavior prediction and home appliance operation

[1200] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. The input is behavioral pattern data and emotional data. The server uses a prediction algorithm to predict the user's next action and sends the prediction result to the device. The device then executes instructions to operate home appliances. For example, if it predicts that the user will watch TV at 10 p.m., it dims the lights and turns on the TV.

[1201] Step 5: Achieving natural conversation

[1202] The device recognizes voice input from the user and uses a generative AI model to generate an appropriate voice response. The input is the user's voice data. This is converted into text data through speech recognition software, and the generative AI model generates a response. The device outputs the generated voice response. For example, if the user says, "I'm tired today," the generative AI will respond, "Thank you for your hard work. I'll play some relaxing music."

[1203] Step 6: Security and monitoring functions

[1204] When the user goes out, the device switches to security mode. The input is the away mode setting. The camera and sensors installed on the device monitor abnormal movements and detect the intrusion of suspicious individuals. Any detected abnormalities are sent to the server, which then sends an alert to the user in real time. For example, if a suspicious individual is detected, an alert notification is output to the user's smartphone.

[1205] (Application example 2)

[1206] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1207] The present invention relates to a system that learns user behavior patterns in the home, recognizes emotions, and provides optimal home appliance operation, natural conversation, security, and monitoring functions. However, conventional technologies have had challenges in detecting customer movements and emotions in real time within stores and economic activity areas, and in using this information to make appropriate product recommendations and improve operational efficiency. While conventional technology systems are suitable for the specific environment of the home, they are not suitable for use in a wide range of economic activity areas, making it difficult to simultaneously improve customer satisfaction and store operational efficiency. Furthermore, recognizing customer emotions in real time and dynamically responding based on this information also present technical challenges.

[1208] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user behavior data, means for analyzing the behavior data and detecting behavior patterns, means for predicting next behavior based on the behavior patterns, means for operating home appliances, means for recognizing voice input and generating voice responses, means for controlling cameras and sensors for security and surveillance, means for detecting customer movements and emotions within an economic activity site in real time, means for providing optimal product recommendations and location guidance to customers based on the analysis results, and means for using a generative AI model including a staffing suggestion function to improve operational efficiency. As a result, customer behavior and emotions can be recognized and analyzed in real time not only at home but also at economic activity sites, and optimal responses can be taken, thereby improving customer satisfaction and streamlining store operations.

[1209] "User behavior patterns" refers to trends in user behavior, movement, and home appliance usage.

[1210] "Behavioral data" is data based on the user's movements, location information, voice, facial expressions, etc.

[1211] "Behavioral pattern analysis" is the process of detecting specific behavioral tendencies and patterns of users based on collected behavioral data.

[1212] "Predicting next behavior" means estimating what behavior a user will take in the future based on analyzed behavioral patterns.

[1213] "Operating home appliances" means controlling household appliances such as air conditioners, lights, and televisions.

[1214] "User voice input" refers to speech uttered by a user that is analyzed by a voice recognition system.

[1215] "Generating a voice response" is the process of generating an appropriate response to a voice input.

[1216] "Crime prevention and surveillance" refers to ensuring safety within homes and stores and monitoring for abnormal situations.

[1217] "Cameras and sensors" is a general term for devices that detect images, sounds, and motion.

[1218] "Points of economic activity" refer to places where goods and services are provided, such as physical stores and commercial facilities.

[1219] "Real-time detection" is the process of collecting data instantly and analyzing it near-simultaneously.

[1220] "Product suggestion" refers to recommending products based on a customer's preferences and purchasing history.

[1221] "Location guidance" refers to providing guidance to the optimal route and location based on the customer's current location and purpose.

[1222] "Operational efficiency improvement" is the process of optimizing store and facility operations and increasing productivity.

[1223] The "staffing suggestion function" refers to a function that suggests appropriate staffing according to location and time of day.

[1224] A "generative AI model" is an artificial intelligence model that automatically generates responses and predictions based on data.

[1225] The present invention relates to a system for detecting and analyzing a user's movements and emotions in real time and providing optimal support. This system operates to effectively support users and customers at home or at points of economic activity. Specific embodiments are described below.

[1226] Hardware and Software

[1227] 1. Hardware Configuration

[1228] Camera: A device installed within an economic activity point to capture the movements and expressions of users and customers. For example, HD cameras and infrared cameras are used.

[1229] Microphone: A device used to collect voice input and recognize user or customer speech.

[1230] Sensors: Devices installed to collect movement data and environmental data. These include temperature sensors, motion sensors, etc.

[1231] Server: A central device for data collection, analysis, and control. It is desirable to have a high-performance processor and a large amount of memory.

[1232] 2. Software Configuration

[1233] Movement and Emotion Detection Software: Detects movements and facial expressions from camera footage using Python and OpenCV.

[1234] Speech recognition system: Analyzes voice data and converts it into text in real time. Google Speech-to-Text API is used.

[1235] Emotion recognition engine: Using a deep learning model with TensorFlow and Keras, emotions are recognized from collected voice and facial expression data.

[1236] Server-side processing: Back-end processing and data management are performed using Node.js and Express.

[1237] Database management system: MongoDB is used to store and query the collected data.

[1238] Processing flow

[1239] 1. Data collection

[1240] The server collects user and customer behavioral and voice data through cameras and microphones.

[1241] Data is also collected from sensors, which transmit location and environmental data of users and customers to a server.

[1242] 2. Data Analysis

[1243] The server uses Python scripts to analyze video data from the cameras and detect customer movements and behavior patterns.

[1244] An emotion recognition engine using TensorFlow and Keras analyzes voice and facial expression data to identify the emotional state of users and customers in real time.

[1245] 3. Creating Actions

[1246] A server-side Node.js application uses the analyzed data to predict the next action and, based on the results, optimally operate home appliances and make product suggestions.

[1247] The server uses a generative AI model to generate appropriate responses to voice input.

[1248] 4. Examples and Prompts

[1249] Example: If a customer spends too long at a particular shelf and their facial expression indicates confusion, the server will notify a nearby staff member and instruct them to explain the product. An audio guide will also ask, "Can I help you with this product?"

[1250] Example prompt sentence:

[1251] "Explain how you can use customer facial expression data to estimate their current emotional state and provide a pleasant in-store experience."

[1252] "Please show us an algorithm that analyzes customer movement data and purchase history and recommends products that are likely to be purchased again."

[1253] This will realize a system that can comprehensively recognize and analyze the behavior and emotions of users and customers at home and at points of economic activity, and respond appropriately. This system is designed to improve customer satisfaction and streamline store operations.

[1254] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1255] Step 1:

[1256] The device collects user movements and voice in real time from cameras and microphones installed in the physical store. As input, it acquires camera video data and audio data. The camera data is broken down into video frames, and the audio data is converted into a frequency spectrum. As output, the acquired frame data and audio spectrum data are sent to the server.

[1257] Step 2:

[1258] The server uses Python and OpenCV to analyze the user's movements from the received frame data. Camera image data is used as input. Data processing involves extracting movement data using background subtraction and object detection algorithms. User movement data is generated as output.

[1259] Step 3:

[1260] The server uses TensorFlow and Keras to analyze the user's emotions from voice data and facial expression data. Voice spectrum data and facial frame data are used as input. Data calculations are performed to classify the emotional state using a voice emotion recognition model and a facial expression recognition model. Emotional state data is generated as output.

[1261] Step 4:

[1262] The server predicts the user's next behavior based on the analyzed behavioral and emotional data. The inputs are movement data and emotional data. The data is calculated using a time series analysis method to build a behavioral pattern model, and the next behavior is estimated using a prediction algorithm. The output is next behavior prediction data.

[1263] Step 5:

[1264] The server uses a generative AI model to generate optimal home appliance operations and product suggestions based on the predicted behavior and emotion data. Behavior prediction data and emotion data are used as input. Data processing involves selecting appropriate responses and actions based on pre-set scenarios and rules. Specific operation commands and suggestions are generated as output.

[1265] Step 6:

[1266] The server sends the generated operation commands and suggestions to terminals in the store to control the appliances and provide product information. The operation commands and suggestions from the generative AI model are used as input. As data calculations, the terminal executes the commands and feeds the results back to the server. As output, data on the status of the appliances and customer reactions is generated.

[1267] Step 7:

[1268] The server updates the generative AI model based on the feedback data, improving accuracy from the next time onwards. Execution result data and customer response data are used as input. Data processing involves retraining the model using a machine learning algorithm. An updated generative AI model is generated as output.

[1269] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1270] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1271] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1272] [Fourth embodiment]

[1273] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1274] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1275] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1276] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1277] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1278] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1279] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1280] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1281] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1282] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1283] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1284] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1285] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[1287] 1. Collecting user behavior data

[1288] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[1289] 2. Analyzing user behavior data and detecting patterns

[1290] The collected user behavior data is sent to a server. The server analyzes this data and detects certain behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This pattern information is used to predict the user's next action.

[1291] 3. Predict your next move and operate your home appliances

[1292] The server predicts the user's next action based on the detected behavioral patterns. This prediction result is then sent back to the device. For example, if the user has a pattern of watching TV every night at 10 p.m., the device will automatically turn on the TV at that time.

[1293] 4. Realizing natural conversation

[1294] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. If the user asks, "What's the weather like tomorrow?", the device analyzes this voice input, and the server's generation AI generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees." The device then conveys this to the user via voice.

[1295] 5. Security and monitoring functions

[1296] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[1297] Specific examples

[1298] Morning support:

[1299] If the device learns that the user wakes up at 7:00 every morning, it will turn on the lights and open the curtains at 7:00. It will also provide weather information via voice, along with a greeting such as "Good morning. It's sunny today."

[1300] Remote work support:

[1301] When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[1302] Through the above-described specific processes and functions, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[1303] The processing flow will be explained below.

[1304] Specific flow of program processing

[1305] User behavior pattern learning

[1306] Step 1:

[1307] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[1308] Step 2:

[1309] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[1310] Step 3:

[1311] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[1312] Step 4:

[1313] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[1314] Step 5:

[1315] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[1316] Behavior prediction and home appliance operation

[1317] Step 1:

[1318] The server predicts the user's next action based on the analyzed behavioral patterns, for example, predicting that the user has a habit of watching TV at 10 p.m.

[1319] Step 2:

[1320] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[1321] Step 3:

[1322] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[1323] Realizing natural conversation

[1324] Step 1:

[1325] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[1326] Step 2:

[1327] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[1328] Step 3:

[1329] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[1330] Step 4:

[1331] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[1332] Security and monitoring functions

[1333] Step 1:

[1334] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[1335] Step 2:

[1336] The device turns off home appliances such as lights, televisions, and air conditioners.

[1337] Step 3:

[1338] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[1339] Step 4:

[1340] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[1341] Step 5:

[1342] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[1343] Through the above steps, learning of user behavior patterns, predictive operation of home appliances, natural conversation, crime prevention and monitoring functions can be effectively realized.

[1344] Example 1

[1345] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1346] In modern homes, systems that learn user behavior patterns and automatically operate and respond to the necessary home appliances are needed to make users' daily lives more efficient and comfortable. Functions such as monitoring elderly people and children and crime prevention are also required. However, existing systems often have separate functions and do not provide comprehensive support. This creates the complexity of having to install multiple systems and operate them individually.

[1347] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1348] In this invention, the server includes means for collecting user behavior data in real time using multiple sensors, cameras, and microphones, means for transmitting the collected behavior data to the server, and means for analyzing the user behavior data and detecting behavior patterns in the server, thereby enabling more efficient support for users' daily lives and improving comfort and safety in the home.

[1349] A "sensor" is a device that detects physical phenomena (temperature, humidity, movement, etc.) and provides that information as data.

[1350] A "camera" is a device used to capture images and videos and to visually record a user's actions and situations.

[1351] A "microphone" is a device that picks up sound and converts it into an electrical signal, and is used to obtain a user's voice input.

[1352] A "server" is a computing device for receiving, storing, analyzing data, and generating a response.

[1353] "Behavioral data" refers to data that includes information related to a user's daily life, such as the user's specific movements, location, and voice.

[1354] "Analysis" is the process of extracting meaning and patterns from collected data.

[1355] "Behavioral patterns" refer to repetition of a user's behavior or tendencies in behavior at a specific time.

[1356] A "machine learning algorithm" is a method of learning specific trends and patterns based on large amounts of data and making predictions and classifications for new data.

[1357] A "generative AI model" is an artificial intelligence model that generates responses or results that are meaningful to humans based on given input data.

[1358] "Speech recognition means" is a technology that converts voice input into text data.

[1359] "Crime prevention" refers to activities aimed at protecting physical spaces from unauthorized intrusion and crime.

[1360] "Monitoring" involves monitoring elderly people, children, and others to ensure their safety, and taking appropriate action if any abnormalities are detected.

[1361] "Encryption" is a technology that converts data using a specific algorithm to make it difficult for third parties to decipher.

[1362] "Real-time" means that processing or response occurs immediately after data is collected, and refers to operations or processes that require immediacy.

[1363] "Home comfort" refers to the physical and psychological satisfaction that users feel when spending time at home.

[1364] A "user interface" is the means or screen layout by which a user directly interacts with a system or device.

[1365] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and supports the user's daily life. This robot collects, analyzes, and predicts the user's behavioral data, and based on the results, automatically operates household appliances in the home. It also has natural conversational capabilities, security functions, and monitoring functions.

[1366] Hardware and Software Configuration

[1367] Terminal: Infrared sensor, environmental sensor, IP camera, microphone

[1368] Server: A computer device for data analysis. Software used includes Python, Pandas, scikit-learn, TensorFlow, etc.

[1369] Network: Use secure protocols such as HTTPS for data transmission.

[1370] Speech Recognition: Speech recognition technologies such as Google Cloud Speech-to-Text.

[1371] Text-to-speech: Text-to-speech technologies such as Google Cloud Text-to-Speech.

[1372] Collecting user behavior data

[1373] The device uses multiple sensors, cameras, and microphones to collect real-time user behavior data, such as which room the user is in and which appliances they are using, allowing the device to understand the user's daily behavioral patterns.

[1374] Example: If a user is listening to music in their living room, the device's camera captures their movements, the microphone captures audio data, and the environmental sensors record the temperature and humidity in the living room.

[1375] Data transmission and analysis

[1376] The collected user behavior data is sent from the device to a server, which receives the data and stores it in a database. It then uses machine learning algorithms to analyze the behavioral patterns and detect specific behavioral patterns.

[1377] Example: Based on data from the past week, the server detects a pattern: "The user wakes up at 7am every morning and has breakfast at 8am."

[1378] Behavior prediction and automatic operation of home appliances

[1379] The server predicts the user's next action based on the detected behavioral patterns, and the prediction results are sent back to the device, allowing the appliance to prepare ahead of the user's actions.

[1380] Example: The server predicts that the user will have breakfast at 8:00 and sends that information to the device. The device then turns on the kitchen lights and starts the coffee maker at 8:00. It also adjusts the air conditioner temperature appropriately.

[1381] Realizing natural conversation

[1382] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response, which is then sent back to the device and conveyed to the user using speech synthesis software.

[1383] Example: When a user asks, "What's the weather going to be like tomorrow?", the device converts the speech to text and sends it to a server. The server generates "It's going to be sunny tomorrow," which the device then replies to the user by voice.

[1384] Security and monitoring functions

[1385] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary.

[1386] Example: If the motion sensor in the room detects abnormal activity, for example, that a suspicious person has entered through the window, the device reports the situation to the server, and the server immediately sends an alert to the user's smartphone.

[1387] Examples of specific examples and prompts

[1388] Wake-up support: If the device learns that the user wakes up at 7:00 a.m. every morning, it will turn on the lights and open the curtains at 7:00 a.m. It will also provide weather information by voice, along with a greeting such as "Good morning. It's sunny today."

[1389] Example prompt: "When the user wakes up at 7:00, turn on the lights and give them the weather report."

[1390] Support for remote work: When a user working from home schedules a meeting, the device will notify them 10 minutes before the meeting starts, and will also adjust the air conditioning temperature appropriately and start the projector during the meeting.

[1391] Example prompt: "Please notify me 10 minutes before a meeting starts and adjust the air conditioning temperature when users are working remotely."

[1392] As described above, the specialized in-home butler robot of the present invention supports the user's life in many ways and provides an efficient and comfortable home environment.

[1393] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1394] Step 1:

[1395] Collecting user behavior data

[1396] The device collects user behavior data in real time using infrared sensors, environmental sensors, IP cameras, and microphones. Specifically, the device's camera captures the user's movements, and the microphone captures audio data. The environmental sensors collect data such as temperature and humidity. This data indicates which room the user is in and which appliances they are using. For example, if a user is listening to music in the living room, the camera records the user's movements and the microphone captures their audio.

[1397] Input: Data from infrared sensors, environmental sensors, IP cameras, microphones

[1398] Output: User behavior data (location, movement, audio, environmental data)

[1399] Step 2:

[1400] Sending behavioral data to the server

[1401] The device encrypts the collected user behavior data and sends it to a server via the Internet. Specifically, the device batches the collected data and sends it to the server using a secure protocol such as HTTPS. For example, data on the user's presence in the living room is sent to the server every hour.

[1402] Input: Collected user behavior data

[1403] Output: Data sent to the server

[1404] Step 3:

[1405] Behavioral data analysis and pattern detection

[1406] The server stores the received user behavior data in a database and analyzes the data using analytical software (e.g., Python, Pandas). The server then analyzes the data using machine learning algorithms (e.g., scikit-learn, TensorFlow) to detect specific behavioral patterns. For example, the server might detect a pattern such as "the user wakes up at 7:00 every morning and eats breakfast at 8:00."

[1407] Input: User behavior data sent to the server

[1408] Output: Detected behavioral patterns

[1409] Step 4:

[1410] Predicting next actions

[1411] The server predicts the user's next action based on the detected behavioral patterns. The server uses a machine learning model to predict the user's next action based on past behavioral data. The prediction result is then sent back to the device. For example, a prediction that "the user will have breakfast at 8 o'clock" is sent to the device.

[1412] Input: Detected behavioral patterns

[1413] Output: Predicted data of next action

[1414] Step 5:

[1415] Automatic operation of home appliances

[1416] The device automatically controls home appliances based on the next action prediction received from the server. Specifically, the device controls home appliances through smart plugs and smart remote controls. For example, the device might turn on the kitchen lights and start the coffee maker at 8:00 a.m., and also adjust the air conditioner temperature appropriately.

[1417] Input: Next action prediction data received from the server

[1418] Output: Operation instructions for home appliances

[1419] Step 6:

[1420] Realizing natural conversation

[1421] The device receives the user's voice input, and the server uses a generative AI model (e.g., GPT-3) to generate an appropriate response. The generated response is then sent back to the device, where it is communicated to the user using speech synthesis software (e.g., Google Cloud Text-to-Speech). For example, if a user asks, "What's the weather like tomorrow?" the device converts the speech into text and sends it to the server. The server generates a response, which the device then returns to the user via voice.

[1422] Input: User voice input

[1423] Output: The generated audio response

[1424] Step 7:

[1425] Security and monitoring functions

[1426] When the user leaves the house, the device automatically switches to security mode. If the device's camera or sensors detect any abnormal activity, the server sends a real-time notification to the user's smartphone. The monitoring function detects abnormal behavior of children or elderly people and notifies them as necessary. For example, if the device's motion sensor detects abnormal activity in a room, it will report the situation to the server, stating that a "suspicious person has entered through a window," and the server will immediately send an alert to the user's smartphone.

[1427] Input: Anomaly detection data from cameras and sensors

[1428] Output: User notification

[1429] (Application example 1)

[1430] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] Conventional home support systems have the ability to learn the user's behavioral patterns and operate home appliances, but they lack the ability to automatically switch security modes or notify users of suspicious activity in real time. This makes it difficult for users to leave their home in the care of the system while they are out. To solve this issue, there is a need for enhanced home security and monitoring functions and for real-time notification when suspicious activity is detected.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1433] In this invention, the server includes means for collecting user behavior data to learn user behavior patterns and provide support to the user, means for analyzing user behavior data and detecting behavior patterns, means for predicting the user's next action based on the detected behavior patterns, means for operating home appliances based on the prediction, means for recognizing user voice input and generating appropriate voice responses to have natural conversations with the user, means for controlling cameras and sensors for home security and surveillance, and means for automatically switching to security mode when the user goes out and for notifying in real time if suspicious activity is detected, allowing the user to leave their home in the care of the appliances while they are out with peace of mind.

[1434] "User behavior data" is data that records a user's daily behavior and lifestyle patterns.

[1435] "Behavioral patterns" refer to specific actions or habits that a user repeatedly performs.

[1436] "User voice input" refers to words or commands that a user speaks to a robot.

[1437] "Cameras and sensors" is a general term for optical and electronic devices used to detect images, sounds, and motion.

[1438] "Security mode" is an operating mode for strengthening security in the home when the user is away.

[1439] "Real-time notification" refers to notifying users immediately when abnormal or suspicious behavior is detected.

[1440] This invention is a specialized domestic butler robot system that learns the user's behavioral patterns and supports the user's daily life. This system is realized by combining various technologies, including the collection, analysis, and prediction of user behavior data, security functions, and natural conversation functions.

[1441] 1. Collecting user behavior data

[1442] The system is equipped with multiple sensors and cameras that monitor and record users' behavior in real time. These data provide a detailed record of users' daily movements, allowing the system to know which room the user is in and which appliances they are using.

[1443] 2. Analyzing user behavior data

[1444] The server analyzes the collected behavioral data to detect user behavior patterns. This process involves using a generative AI model to analyze patterns and predict the user's next action. To do this, the collected data is stored in a database and the AI ​​model learns from it.

[1445] 3. Controlling home appliances

[1446] Based on the user's behavioral patterns, the server predicts the user's next actions and operates the necessary home appliances. For example, the server can set the living room lights and TV to turn off at 10 p.m. every night. This allows the user to automatically operate the home appliances and provide a comfortable living environment.

[1447] 4. Realizing natural conversation

[1448] The device recognizes voice input from the user and generates an appropriate response. For example, if a user asks, "What's the weather like tomorrow?", the server-based generative AI model generates an answer such as, "It will be sunny tomorrow, with a temperature of 25 degrees," and conveys this to the user via voice. This process uses natural language processing technology to analyze the user's intent.

[1449] 5. Security features

[1450] When the user leaves the house, the system automatically switches to security mode and monitors the home using cameras and sensors for any unusual activity. If any suspicious activity is detected, the system sends a real-time notification to the user's smartphone, urging immediate action. If an abnormality is detected, an alarm will sound and the system can also notify security authorities.

[1451] 6. Program Processing

[1452] The server and device collect user behavior data in real time and store it in a database. Generative AI models are used to analyze and predict the data, and the results are used to automatically operate home appliances. Voice recognition software and natural language processing technology are combined to enable natural conversations with users. For security purposes, the system analyzes data detected by cameras and sensors, and notifies users in real time if an abnormality is detected.

[1453] As a concrete example, if a user asks, "What time is a good time to leave the house today?", they can input the following prompt into the generative AI model:

[1454] "The system learns the pattern of when users leave home at 8 o'clock every morning and automatically switches on security mode. This learning is based on simple rules, but in the future, we will be able to predict behavior with greater accuracy by introducing reinforcement learning using generative AI. Specifically, we will input the following prompt into the generative AI: 'Please tell me how I can learn the patterns of when users often leave home and automatically switch on security mode according to that time of day.'"

[1455] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1456] Step 1:

[1457] Collecting user behavior data

[1458] The device uses multiple sensors and cameras installed in the home to collect user behavior data in real time. Specifically, it obtains information on which room the user is in and which home appliances they are using. This behavior data is sent from the device to a server and stored in a database.

[1459] Input: User behavior data from sensors and cameras.

[1460] Output: The collected user behavior data is sent to the server and stored in a database.

[1461] Step 2:

[1462] Analyzing user behavior data

[1463] The server analyzes the user behavior data stored in the database and detects specific behavioral patterns. Using a generative AI model, it learns from this data and discovers user behavioral patterns. For example, it discovers that the user wakes up at 7am every morning and watches TV in the living room.

[1464] Input: User behavior data stored in a database.

[1465] Output: Analyzed by a generative AI model and detected as a behavioral pattern.

[1466] Step 3:

[1467] Predicting next actions based on behavioral patterns

[1468] The server predicts the user's next action based on the detected user behavior patterns. The generative AI model makes predictions based on past data, and the server generates predictions based on the user's lifestyle habits. For example, it predicts that the TV will be turned off at 10 p.m.

[1469] Input: Detected behavioral patterns.

[1470] Output: Predicted result of next action.

[1471] Step 4:

[1472] Automatic operation of home appliances

[1473] The server automatically controls household appliances based on the prediction results. For example, if the prediction is to "turn off the TV at 10 o'clock," the server will issue a command to turn off the TV at that time. This provides a comfortable living environment for the user.

[1474] Input: Predicted outcome of the action.

[1475] Output: Status of automatically operated appliances.

[1476] Step 5:

[1477] Achieving natural conversations with users

[1478] The device uses a voice recognition system to receive voice input from the user, and the server uses a generative AI model to generate appropriate responses to the user's questions and commands, which are then spoken to the user via the device.

[1479] Input: User's voice input.

[1480] Output: Natural-sounding conversational responses generated by the generative AI model.

[1481] Step 6:

[1482] Security features implemented

[1483] When the device detects that the user has left the house, it automatically switches to security mode. Cameras and sensors monitor for suspicious activity, and if the server detects an abnormality, it sends a real-time notification to the user's smartphone.

[1484] Input: External environment data from cameras and sensors.

[1485] Output: Real-time notification of suspicious behavior.

[1486] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1487] This invention relates to a specialized in-home butler robot that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[1488] 1. Collecting user behavior data

[1489] The device uses cameras, microphones, and sensors installed in the home to collect real-time user behavior data. For example, by recording which room the user is in and which home appliances they are using, the device can understand the user's daily behavioral patterns. This data includes the user's movements, movement history, and usage of home appliances.

[1490] 2. Analyzing user behavior data and detecting patterns

[1491] The collected user behavior data is sent to a server, which analyzes it and detects specific behavioral patterns. For example, it can learn that the user wakes up at 7am every morning and turns on the living room light. This provides the basis for predicting the user's next behavior.

[1492] 3. Emotion Recognition by Emotion Engine

[1493] The device recognizes the user's voice and facial expressions and records them as emotional data. For example, if the user looks tired, the emotion engine will recognize this as a "fatigue state." This emotional data is also sent to the server along with behavioral data for analysis.

[1494] 4. Behavior prediction and home appliance operation

[1495] The server predicts the user's next action based on the detected behavioral patterns and emotional data. This prediction result is then sent back to the device. For example, if the user has a habit of watching TV at 10 p.m. and the emotional data for that day indicates that they want to relax, the device will dim the lights and automatically turn on the TV at that time.

[1496] 5. Realizing natural conversation

[1497] The device has the ability to receive voice input and generate appropriate responses to enable natural conversation with the user. For example, if the user says, "I'm tired today," the server's AI will respond, "Thank you for your hard work. I'll play some relaxing music." This response is generated taking into account the user's emotional state.

[1498] 6. Security and monitoring functions

[1499] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[1500] Specific examples

[1501] Morning support:

[1502] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[1503] Remote work support:

[1504] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[1505] Through the above-described specific processing and functions, the specialized domestic butler robot of the present invention comprehensively learns the user's behavior and emotions, provides more effective support, and provides a comfortable home environment.

[1506] The processing flow will be explained below.

[1507] Specific flow of program processing

[1508] User behavior pattern learning

[1509] Step 1:

[1510] The user performs daily activities such as watching TV in the living room, sleeping in the bedroom, and cooking in the kitchen.

[1511] Step 2:

[1512] The device uses cameras, microphones, and sensors to collect real-time data about your behavior, such as recording the time you enter the living room and turn on the TV.

[1513] Step 3:

[1514] The device sends the collected behavioral data to a server, including the user's location, movements, and operation history of home appliances.

[1515] Step 4:

[1516] The server analyzes the received behavioral data and uses a data analysis algorithm to detect user behavior patterns.

[1517] Step 5:

[1518] The server accumulates the analysis results and updates the model to be used to predict the next behavior.

[1519] Emotion data collection using an emotion engine

[1520] Step 1:

[1521] The user interacts with the device and performs everyday activities. For example, the user might say, "I'm tired today."

[1522] Step 2:

[1523] The device detects the user's voice and facial expressions, and analyzes this data using an emotion engine. The device recognizes the user's emotional state (e.g., "fatigue," "comfort," etc.) from the voice data and facial image data.

[1524] Step 3:

[1525] The device sends the recognized emotion data to the server, where it is integrated with the behavioral data and analyzed.

[1526] Behavior prediction and home appliance operation

[1527] Step 1:

[1528] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. For example, if the user has a habit of watching TV at 10 p.m., it predicts the user's emotional state for that day as "relaxed."

[1529] Step 2:

[1530] The server sends the forecast information to the device, including instructions such as "Turn on the TV at 10pm."

[1531] Step 3:

[1532] The device then controls home appliances based on the forecast information it receives, turning on the lights, operating the air conditioner, and turning on the TV at the specified time.

[1533] Realizing natural conversation

[1534] Step 1:

[1535] The user speaks a request to the device, for example, "Tell me what the weather will be tomorrow."

[1536] Step 2:

[1537] The device receives the user's voice input through the microphone, analyzes it with a voice recognition system, and converts it into text data.

[1538] Step 3:

[1539] The server receives the text data and uses generative AI to generate an appropriate response, such as "Tomorrow will be sunny and the temperature will be 25 degrees."

[1540] Step 4:

[1541] The server generates a response and sends it to the device, which then verbally conveys it to the user.

[1542] Security and monitoring functions

[1543] Step 1:

[1544] The user tells the device, "I'm leaving," which initiates the switch to security mode.

[1545] Step 2:

[1546] The device turns off home appliances such as lights, televisions, and air conditioners.

[1547] Step 3:

[1548] The device activates cameras and sensors and begins monitoring the home, for example, monitoring whether doors and windows are open or closed.

[1549] Step 4:

[1550] If the terminal detects an abnormality, for example, if it detects the intrusion of a suspicious person, it sends that information to the server.

[1551] Step 5:

[1552] The server analyzes abnormal information and sends alerts to the user's smartphone in real time.

[1553] Specific examples

[1554] Morning support:

[1555] Step 1:

[1556] The system learns that the user wakes up at 7am every morning.

[1557] Step 2:

[1558] The emotion engine recognizes your emotional state from the past morning as "refreshed."

[1559] Step 3:

[1560] The device will turn on the lights at 7:00 and open the curtains, and will also provide a voice greeting such as, "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[1561] Remote work support:

[1562] Step 1:

[1563] The emotion engine recognizes that users are feeling stressed while working from home.

[1564] Step 2:

[1565] The device plays relaxing music before the meeting starts.

[1566] Step 3:

[1567] During the meeting, the device will adjust the air conditioning temperature appropriately and start the projector.

[1568] These steps will enable a comprehensive realization of learning user behavior patterns, collecting and analyzing emotional data, predictive home appliance operation, natural conversation, and crime prevention and monitoring functions.

[1569] Example 2

[1570] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1571] In modern homes, it is difficult to accurately grasp a user's behavioral patterns and emotional state and provide support based on that. Furthermore, while there is a demand for effectively combining security and monitoring functions, it is difficult to build a comprehensive system using existing technology. To address these issues, the present invention provides a system that learns a user's behavioral patterns and emotional state and provides optimal support.

[1572] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting user behavior data in real time using cameras, microphones, and sensors installed in the home; means for transmitting the collected user behavior data to the server and analyzing it to detect behavioral patterns; means for recognizing the user's voice and facial expressions and recording emotional data; means for transmitting the recorded emotional data to the server and using it for analysis; means for predicting the user's next behavior based on the detected behavioral patterns and emotional data; means for operating home appliances based on the prediction; means for recognizing voice input from the user and generating an appropriate voice response using a generative AI model; and means for detecting abnormal behavior in the home and controlling the cameras and sensors to provide security and monitoring functions. This makes it possible to comprehensively learn the user's behavior and emotions, provide effective support, and provide a comfortable and safe home environment.

[1573] A "camera" is a photographic device for capturing visual information.

[1574] A "microphone" is an acoustic transducer for collecting sound.

[1575] A "sensor" is a device that collects physical environmental information and has the ability to detect temperature, movement, light, etc.

[1576] "Behavioral data" refers to data that indicates information about a user's actions, movements, and the equipment they use.

[1577] A "server" is a computer system that analyzes, stores, and communicates data.

[1578] "Behavioral patterns" are data that show regularities and trends extracted from a series of user actions.

[1579] "Emotion data" refers to data that indicates the emotional state of a user that can be inferred from their speech and facial expressions.

[1580] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses or predictions from input data.

[1581] "Security" is a function that aims to protect the safety of the home by detecting intrusions by suspicious individuals and abnormal situations.

[1582] "Monitoring" is a function that checks the safety of children, elderly people, etc. and notifies you if any abnormal behavior occurs.

[1583] "Real-time" means that processing and reaction occur immediately in accordance with actual time.

[1584] "Voice input" refers to data that the system captures and processes from the user's speech.

[1585] "Speech recognition" is the technology that analyzes voice input and understands what it means.

[1586] A "voice response" is a voice message that the system responds to a user's voice input.

[1587] This invention relates to a specialized in-home butler robot system that learns the user's behavioral patterns and recognizes the user's emotions to provide support. This robot system collects and analyzes the user's behavioral and emotional data, and based on that data, provides optimal home appliance operation, natural conversation, crime prevention, and monitoring functions.

[1588] Collecting user behavior data

[1589] The device collects user behavior data in real time using cameras, microphones, and sensors installed in the home. This includes cameras and sensors installed in each room of the home. This allows the device to record which room the user is in and which home appliances they are using. This behavioral data includes the user's movements, movement history, and usage status of home appliances.

[1590] Analyzing user behavior data and detecting patterns

[1591] The collected behavioral data is sent from the device to a server, which analyzes the data to detect specific behavioral patterns. For example, the server may detect that the user wakes up at 7am every morning and turns on the living room light. This analysis is performed using a data analysis algorithm.

[1592] Emotion recognition by emotion engine

[1593] The device recognizes the user's voice and facial expressions in real time and records them as emotional data. For example, if a user says, "I'm tired today," the device's microphone captures this voice and converts it into text using voice recognition software. The emotion engine analyzes this and recognizes it as "fatigue." This emotional data is also sent to the server for further analysis.

[1594] Behavior prediction and home appliance operation

[1595] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. One example of this prediction is when a user has a habit of watching TV at 10pm, and the server determines from that day's emotional data that they "want to relax." The server then sends the device an instruction to "dim the lights and automatically turn on the TV at 10pm." The device receives this instruction and operates the home appliances.

[1596] Realizing natural conversation

[1597] To achieve natural conversation with the user, the device receives voice input and generates an appropriate response using a generative AI model. For example, if the user says, "I'm tired today," the server's generative AI model generates a response such as, "Thank you for your hard work. I'll play some relaxing music for you," and sends it to the device. The device then conveys this as voice to the user.

[1598] Security and monitoring functions

[1599] When the user leaves the house, the device switches to security mode. The device's built-in cameras and sensors monitor any unusual activity within the home. For example, if a suspicious person is detected, the server sends a real-time alert to the user's smartphone. The device also has a function to monitor children and the elderly, and will notify the user if any unusual behavior is detected.

[1600] Specific examples

[1601] Morning support:

[1602] If the user's pattern of waking up at 7am every morning is learned and the emotion engine recognizes their past morning emotional state as "refreshed," the device will turn on the lights and open the curtains at 7am. It will also provide a voice greeting such as "Good morning. It's a beautiful day today. Is there anything I can help you with?"

[1603] Remote work support:

[1604] If the emotion engine detects that a user working from home is feeling stressed, the device will play relaxing music before a meeting starts, adjust the air conditioning temperature appropriately during the meeting, and start the projector.

[1605] Prompt Sentence Examples

[1606] When a user inputs "I'm tired today" into the generative AI model, the model responds, "Thank you for your hard work. I'll play some relaxing music right now."

[1607] Through these specific processing steps, the specialized domestic butler robot can comprehensively learn the user's behavior and emotions, provide more effective support, and provide a comfortable and safe home environment.

[1608] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1609] Step 1: Collect user behavior data

[1610] The device uses cameras, microphones, and sensors installed in the home to collect user behavior data in real time. Inputs include camera footage, audio data, and sensor data. This input data is processed and output as user location information and behavior data. Specifically, the camera detects the user's movements, the microphone records speech, and the sensors detect temperature and movement.

[1611] Step 2: Analyze user behavior data and detect patterns

[1612] The collected user behavior data is sent from the device to a server. The server analyzes this data and detects behavioral patterns. The input is the collected behavioral data. A data analysis algorithm is used to analyze regularity and frequency, and the detected behavioral patterns are output. Specifically, the server cross-references the user's behavior history and detects a pattern of waking up at 7am every morning and turning on the lights.

[1613] Step 3: Emotion recognition by the emotion engine

[1614] The device recognizes the user's voice and facial expressions and records them as emotional data. The input is the user's voice and facial expression data. Speech recognition software is used to convert the voice into text, and an emotional engine analyzes the emotional state. The analyzed emotional data is sent to the server. For example, if the user says, "I'm tired today," the voice input is output as "fatigue state."

[1615] Step 4: Behavior prediction and home appliance operation

[1616] The server predicts the user's next action based on the analyzed behavioral patterns and emotional data. The input is behavioral pattern data and emotional data. The server uses a prediction algorithm to predict the user's next action and sends the prediction result to the device. The device then executes instructions to operate home appliances. For example, if it predicts that the user will watch TV at 10 p.m., it dims the lights and turns on the TV.

[1617] Step 5: Achieving natural conversation

[1618] The device recognizes voice input from the user and uses a generative AI model to generate an appropriate voice response. The input is the user's voice data. This is converted into text data through speech recognition software, and the generative AI model generates a response. The device outputs the generated voice response. For example, if the user says, "I'm tired today," the generative AI will respond, "Thank you for your hard work. I'll play some relaxing music."

[1619] Step 6: Security and monitoring functions

[1620] When the user goes out, the device switches to security mode. The input is the away mode setting. The camera and sensors installed on the device monitor abnormal movements and detect the intrusion of suspicious individuals. Any detected abnormalities are sent to the server, which then sends an alert to the user in real time. For example, if a suspicious individual is detected, an alert notification is output to the user's smartphone.

[1621] (Application example 2)

[1622] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] The present invention relates to a system that learns user behavior patterns in the home, recognizes emotions, and provides optimal home appliance operation, natural conversation, security, and monitoring functions. However, conventional technologies have had challenges in detecting customer movements and emotions in real time within stores and economic activity areas, and in using this information to make appropriate product recommendations and improve operational efficiency. While conventional technology systems are suitable for the specific environment of the home, they are not suitable for use in a wide range of economic activity areas, making it difficult to simultaneously improve customer satisfaction and store operational efficiency. Furthermore, recognizing customer emotions in real time and dynamically responding based on this information also present technical challenges.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user behavior data, means for analyzing the behavior data and detecting behavior patterns, means for predicting next behavior based on the behavior patterns, means for operating home appliances, means for recognizing voice input and generating voice responses, means for controlling cameras and sensors for security and surveillance, means for detecting customer movements and emotions within an economic activity site in real time, means for providing optimal product recommendations and location guidance to customers based on the analysis results, and means for using a generative AI model including a staffing suggestion function to improve operational efficiency. As a result, customer behavior and emotions can be recognized and analyzed in real time not only at home but also at economic activity sites, and optimal responses can be taken, thereby improving customer satisfaction and streamlining store operations.

[1625] "User behavior patterns" refers to trends in user behavior, movement, and home appliance usage.

[1626] "Behavioral data" is data based on the user's movements, location information, voice, facial expressions, etc.

[1627] "Behavioral pattern analysis" is the process of detecting specific behavioral tendencies and patterns of users based on collected behavioral data.

[1628] "Predicting next behavior" means estimating what behavior a user will take in the future based on analyzed behavioral patterns.

[1629] "Operating home appliances" means controlling household appliances such as air conditioners, lights, and televisions.

[1630] "User voice input" refers to speech uttered by a user that is analyzed by a voice recognition system.

[1631] "Generating a voice response" is the process of generating an appropriate response to a voice input.

[1632] "Crime prevention and surveillance" refers to ensuring safety within homes and stores and monitoring for abnormal situations.

[1633] "Cameras and sensors" is a general term for devices that detect images, sounds, and motion.

[1634] "Points of economic activity" refer to places where goods and services are provided, such as physical stores and commercial facilities.

[1635] "Real-time detection" is the process of collecting data instantly and analyzing it near-simultaneously.

[1636] "Product suggestion" refers to recommending products based on a customer's preferences and purchasing history.

[1637] "Location guidance" refers to providing guidance to the optimal route and location based on the customer's current location and purpose.

[1638] "Operational efficiency improvement" is the process of optimizing store and facility operations and increasing productivity.

[1639] The "staffing suggestion function" refers to a function that suggests appropriate staffing according to location and time of day.

[1640] A "generative AI model" is an artificial intelligence model that automatically generates responses and predictions based on data.

[1641] The present invention relates to a system for detecting and analyzing a user's movements and emotions in real time and providing optimal support. This system operates to effectively support users and customers at home or at points of economic activity. Specific embodiments are described below.

[1642] Hardware and Software

[1643] 1. Hardware Configuration

[1644] Camera: A device installed within an economic activity point to capture the movements and expressions of users and customers. For example, HD cameras and infrared cameras are used.

[1645] Microphone: A device used to collect voice input and recognize user or customer speech.

[1646] Sensors: Devices installed to collect movement data and environmental data. These include temperature sensors, motion sensors, etc.

[1647] Server: A central device for data collection, analysis, and control. It is desirable to have a high-performance processor and a large amount of memory.

[1648] 2. Software Configuration

[1649] Movement and Emotion Detection Software: Detects movements and facial expressions from camera footage using Python and OpenCV.

[1650] Speech recognition system: Analyzes voice data and converts it into text in real time. Google Speech-to-Text API is used.

[1651] Emotion recognition engine: Using a deep learning model with TensorFlow and Keras, emotions are recognized from collected voice and facial expression data.

[1652] Server-side processing: Back-end processing and data management are performed using Node.js and Express.

[1653] Database management system: MongoDB is used to store and query the collected data.

[1654] Processing flow

[1655] 1. Data collection

[1656] The server collects user and customer behavioral and voice data through cameras and microphones.

[1657] Data is also collected from sensors, which transmit location and environmental data of users and customers to a server.

[1658] 2. Data Analysis

[1659] The server uses Python scripts to analyze video data from the cameras and detect customer movements and behavior patterns.

[1660] An emotion recognition engine using TensorFlow and Keras analyzes voice and facial expression data to identify the emotional state of users and customers in real time.

[1661] 3. Creating Actions

[1662] A server-side Node.js application uses the analyzed data to predict the next action and, based on the results, optimally operate home appliances and make product suggestions.

[1663] The server uses a generative AI model to generate appropriate responses to voice input.

[1664] 4. Examples and Prompts

[1665] Example: If a customer spends too long at a particular shelf and their facial expression indicates confusion, the server will notify a nearby staff member and instruct them to explain the product. An audio guide will also ask, "Can I help you with this product?"

[1666] Example prompt sentence:

[1667] "Explain how you can use customer facial expression data to estimate their current emotional state and provide a pleasant in-store experience."

[1668] "Please show us an algorithm that analyzes customer movement data and purchase history and recommends products that are likely to be purchased again."

[1669] This will realize a system that can comprehensively recognize and analyze the behavior and emotions of users and customers at home and at points of economic activity, and respond appropriately. This system is designed to improve customer satisfaction and streamline store operations.

[1670] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1671] Step 1:

[1672] The device collects user movements and voice in real time from cameras and microphones installed in the physical store. As input, it acquires camera video data and audio data. The camera data is broken down into video frames, and the audio data is converted into a frequency spectrum. As output, the acquired frame data and audio spectrum data are sent to the server.

[1673] Step 2:

[1674] The server uses Python and OpenCV to analyze the user's movements from the received frame data. Camera image data is used as input. Data processing involves extracting movement data using background subtraction and object detection algorithms. User movement data is generated as output.

[1675] Step 3:

[1676] The server uses TensorFlow and Keras to analyze the user's emotions from voice data and facial expression data. Voice spectrum data and facial frame data are used as input. Data calculations are performed to classify the emotional state using a voice emotion recognition model and a facial expression recognition model. Emotional state data is generated as output.

[1677] Step 4:

[1678] The server predicts the user's next behavior based on the analyzed behavioral and emotional data. The inputs are movement data and emotional data. The data is calculated using a time series analysis method to build a behavioral pattern model, and the next behavior is estimated using a prediction algorithm. The output is next behavior prediction data.

[1679] Step 5:

[1680] The server uses a generative AI model to generate optimal home appliance operations and product suggestions based on the predicted behavior and emotion data. Behavior prediction data and emotion data are used as input. Data processing involves selecting appropriate responses and actions based on pre-set scenarios and rules. Specific operation commands and suggestions are generated as output.

[1681] Step 6:

[1682] The server sends the generated operation commands and suggestions to terminals in the store to control the appliances and provide product information. The operation commands and suggestions from the generative AI model are used as input. As data calculations, the terminal executes the commands and feeds the results back to the server. As output, data on the status of the appliances and customer reactions is generated.

[1683] Step 7:

[1684] The server updates the generative AI model based on the feedback data, improving accuracy from the next time onwards. Execution result data and customer response data are used as input. Data processing involves retraining the model using a machine learning algorithm. An updated generative AI model is generated as output.

[1685] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1686] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1687] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1688] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1689] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1690] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1691] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1692] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1693] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1694] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1695] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1696] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1697] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1698] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1699] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1700] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1701] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1702] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1703] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1704] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1705] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1706] The following is further disclosed regarding the above embodiment.

[1707] (Claim 1)

[1708] A means for collecting user behavior data to learn user behavior patterns and provide user assistance;

[1709] A means for analyzing user behavior data and detecting behavioral patterns;

[1710] A means for predicting a next action of the user based on the detected behavioral pattern;

[1711] a means for operating the appliance based on the prediction;

[1712] means for recognizing user voice input and generating appropriate voice responses to engage in natural conversation with the user;

[1713] A means for controlling cameras and sensors for home security and surveillance;

[1714] A system including:

[1715] (Claim 2)

[1716] 10. The system of claim 1, further comprising means for tracking a user's location in real time and operating the appliance based on the user's location.

[1717] (Claim 3)

[1718] 10. The system of claim 1, further comprising user interface means that allow a user to set specific conditions.

[1719] "Example 1"

[1720] (Claim 1)

[1721] A means of collecting user behavior data in real time using multiple sensors, cameras, and microphones,

[1722] means for transmitting the collected behavioral data to a server;

[1723] A means for analyzing user behavior data on a server and detecting behavioral patterns;

[1724] a means for predicting a next action of the user based on the detected behavioral pattern;

[1725] A means for automatically operating home appliances based on predictions;

[1726] a means for receiving user voice input and generating an appropriate voice response using a generative AI model on a server;

[1727] A means to control cameras and sensors for home security and surveillance, and notify users if an abnormality is detected.

[1728] A system including:

[1729] (Claim 2)

[1730] 10. The system of claim 1, wherein the analysis of the user behavior data uses a machine learning algorithm.

[1731] (Claim 3)

[1732] 10. The system of claim 1, further comprising a speech recognition means for converting a user's voice input into text for input to the generative AI model.

[1733] "Application Example 1"

[1734] (Claim 1)

[1735] A means for collecting user behavior data to learn user behavior patterns and provide user assistance;

[1736] A means for analyzing user behavior data and detecting behavioral patterns;

[1737] A means for predicting a next action of the user based on the detected behavioral pattern;

[1738] a means for operating the appliance based on the prediction;

[1739] means for recognizing user voice input and generating appropriate voice responses to engage in natural conversation with the user;

[1740] A means for controlling cameras and sensors for home security and surveillance;

[1741] It automatically switches to security mode when the user goes out and notifies the user in real time if any suspicious activity is detected.

[1742] A system including:

[1743] (Claim 2)

[1744] 10. The system of claim 1, further comprising means for tracking a user's location in real time and operating the appliance based on the user's location.

[1745] (Claim 3)

[1746] 10. The system of claim 1, further comprising user interface means that allow a user to set specific conditions.

[1747] "Example 2: Combining Emotion Engines"

[1748] (Claim 1)

[1749] A means of collecting user behavior data in real time using cameras, microphones, and sensors installed in the home;

[1750] a means for transmitting the collected user behavior data to a server and analyzing the data to detect behavioral patterns;

[1751] A means for recognizing the user's voice and facial expressions and recording emotional data;

[1752] a means for transmitting the recorded emotion data to a server for analysis;

[1753] a means for predicting a next action based on the detected behavioral patterns and emotion data;

[1754] a means for operating the appliance based on the prediction;

[1755] a means for recognizing voice input from a user and generating an appropriate voice response using a generative AI model;

[1756] A means for controlling cameras and sensors to detect abnormal activity in the home and provide security and monitoring functions;

[1757] A system including:

[1758] (Claim 2)

[1759] 10. The system of claim 1, further comprising means for tracking a user's location in real time and operating the appliance based on the user's location.

[1760] (Claim 3)

[1761] 10. The system of claim 1, further comprising user interface means that allow a user to set specific conditions.

[1762] "Application example 2 when combining emotion engines"

[1763] New invention details

[1764] Extracting new technical aspects

[1765] The novel parts extracted from the application example, Smart Store Assistant, are as follows:

[1766] A means of detecting customer movements and emotions in real time within physical stores

[1767] A means of providing optimal product suggestions and store guidance to customers based on analysis results

[1768] Staffing suggestion function to improve store management efficiency

[1769] A generative AI model that provides natural voice guidance to customers

[1770] New invention combinations

[1771] (Claim 1)

[1772] A means for collecting user behavior data to learn user behavior patterns and provide user assistance;

[1773] A means for analyzing user behavior data and detecting behavioral patterns;

[1774] A means for predicting a next action of the user based on the detected behavioral pattern;

[1775] a means for operating the appliance based on the prediction;

[1776] means for recognizing user voice input and generating appropriate voice responses to engage in natural conversation with the user;

[1777] A means for controlling cameras and sensors for home security and surveillance;

[1778] A means of detecting customer movements and sentiment within economic activity points in real time;

[1779] A means of providing optimal product suggestions and location guidance to customers based on the analysis results,

[1780] A means using generative AI models, including staffing recommendations to improve operational efficiency;

[1781] A system including:

[1782] (Claim 2)

[1783] 10. The system of claim 1, further comprising means for tracking a user's location in real time and operating the appliance based on the user's location.

[1784] (Claim 3)

[1785] 10. The system of claim 1, further comprising user interface means that allow a user to set specific conditions. [Explanation of symbols]

[1786] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting user behavior data to learn user behavior patterns and provide user assistance; A means for analyzing user behavior data and detecting behavioral patterns; A means for predicting a next action of the user based on the detected behavioral pattern; a means for operating the appliance based on the prediction; means for recognizing user voice input and generating appropriate voice responses to engage in natural conversation with the user; A means for controlling cameras and sensors for home security and surveillance; A system including:

2. The system of claim 1 , further comprising means for tracking a user's location in real time and operating a home appliance based on the user's location.

3. 10. The system of claim 1, further comprising user interface means for enabling a user to set specific conditions.

Images