system

Abstract

We provide the system. [Solution] A means for monitoring the posture of users within a living space and generating and notifying them of an exercise reminder when an abnormal posture is detected, A method for monitoring the operating status of home appliances and automatically turning off the power when not in use, A means for recording the location of an item and notifying the user of the location of the item, A means of generating and notifying advice on tidying up based on the state of the living space, A system that includes means for detecting suspicious persons using cameras and generating an alarm when a suspicious person is detected.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern living spaces, user health management, energy conservation, item location identification, efficient tidying, and crime prevention are important issues. However, when these functions are utilized individually, there are problems such as increased costs and management effort, and a lack of overall efficiency. Conventional technologies lack means to integrally provide these functions, making it difficult to comprehensively support users' lives.

Means for Solving the Problems

[0005] To address these challenges, the present invention provides a means for monitoring a user's posture within a living space and generating exercise reminders when abnormalities are detected. It also includes means for monitoring the operating status of home appliances and reducing energy consumption by automatically turning them off when not in use. Furthermore, it includes means for recording the location of items and notifying the user of their location upon request, as well as means for providing tidying and organization advice based on the state of the living space. Finally, it provides security through the detection of suspicious individuals using cameras and the generation of alarms. In this way, the present invention, as a single integrated system, can provide multifaceted support for the user's life.

[0006] "Living space" refers to the indoor environment in which an individual lives their daily life, and usually includes spaces such as houses and apartments.

[0007] "User" refers to an individual or member of a household who uses this system.

[0008] "Posture" refers to the position and arrangement of the user's body, and in particular, the state of the spine and limbs when standing or sitting.

[0009] An "exercise reminder" refers to an automatically generated notification that encourages the user to move their body.

[0010] "Home appliances" refers to household electrical products, including air conditioners, televisions, and lighting.

[0011] "Operating status" refers to information indicating whether an appliance is powered on and in operation.

[0012] "Energy consumption" refers to the amount of electricity used by home appliances and other electrical devices.

[0013] "Items" refer to objects used within the home, including personal belongings such as keys and remote controls.

[0014] "Location Specific" refers to clarifying where a specific item currently exists.

[0015] "Sorting and Organizing Advice" means providing guidance and suggestions regarding the arrangement and arrangement method of items within the living space.

[0016] "Suspicious Person" refers to an unknown person who has entered or attempted to enter the living space without permission.

[0017] "Alarm" refers to an alert or notification issued when the intrusion of a suspicious person is detected.

Brief Explanation of Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0022] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention is an integrated system that improves the quality of life in living spaces, and is designed to provide multiple functions related to health management, energy saving, item location, organization and tidiness advice, and crime prevention. This system is primarily configured to exchange information between a server, terminals, and users, and to provide multifaceted support for the user's life.

[0040] The server acquires information from cameras and various sensors installed in the living space and analyzes this data in real time. Specifically, the server monitors the user's posture and, if it detects prolonged periods of inappropriate posture, generates a reminder to encourage exercise. This helps users maintain healthy lifestyle habits.

[0041] Regarding energy management, the server constantly monitors the operating status of each appliance and reduces unnecessary power consumption by automatically turning off unused appliances. It also uses cameras to check the room's condition and the location of items, locating lost items as needed and providing this information to the user via a terminal. This feature allows users to find items efficiently.

[0042] Furthermore, the tidying advice function analyzes the current state of the living space via a server and provides specific tidying advice to the user through their device. This allows users to create a more comfortable living environment.

[0043] Furthermore, as a security feature, the server continuously monitors camera footage and immediately sends an alert to the user via their device if a suspicious person is detected. This allows users to respond quickly and ensure their safety.

[0044] For example, if a user is working at their desk for a long time, the server will recognize their posture and display a reminder on their device saying, "Please do 5 minutes of stretching." Also, if a user leaves their bedroom, the server will detect their movement and automatically turn off unused lights and air conditioning. Furthermore, if a user is looking for their keys, the server will notify their device of the last observed location of the keys.

[0045] Thus, the present invention aims to make users' lives more comfortable and safer by providing multiple functions in an integrated manner.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The server begins receiving real-time video and data from cameras and sensors installed in the living space. This includes posture data and the operating status of home appliances.

[0049] Step 2:

[0050] The server analyzes the received data and determines the user's posture. If the posture is inappropriate, the server immediately generates an exercise reminder.

[0051] Step 3:

[0052] The server sends the generated exercise reminder to the device. The device notifies the user visually or audibly.

[0053] Step 4:

[0054] The server monitors the operating status of home appliances and generates an automatic shut-off command if any appliances are not in use.

[0055] Step 5:

[0056] The terminal receives commands from the server and saves energy by turning off the power to unused home appliances.

[0057] Step 6:

[0058] The server receives an item location search request from the user and determines its location based on camera data.

[0059] Step 7:

[0060] The server transmits the location information of the identified item to the terminal, and the terminal informs the user of the item's location.

[0061] Step 8:

[0062] The server evaluates the tidiness of the living space and generates tidying-up advice as needed.

[0063] Step 9:

[0064] The terminal provides the user with advice from the server and encourages them to organize and tidy up.

[0065] Step 10:

[0066] The server constantly monitors camera footage and, if it detects a suspicious person, immediately generates an alarm and notifies the user via their device.

[0067] Step 11:

[0068] Users take appropriate safety measures based on alerts from their devices.

[0069] (Example 1)

[0070] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0071] In modern living environments, users often demand simultaneous improvements in health management, energy conservation, safety, and efficient item management. However, currently, there is no integrated system that can meet these demands, and users rely on devices with individual functions. As a result, users are burdened with managing multiple systems, making it difficult to ensure true comfort and safety. Therefore, there is a need for an integrated system that can solve these problems simultaneously.

[0072] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0073] In this invention, the server includes means for monitoring the physical condition of users in a living space and generating and notifying information to encourage exercise when abnormal physical movements are detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the location of items and notifying the user of the location of said items upon request. This enables users to maintain their health, efficiently conserve energy, and manage items safely and effectively.

[0074] "Living space" refers to a place where life and activities take place, and primarily refers to the interior space of a house, apartment, or similar building.

[0075] "Users" refer to individuals who use this system and seek to improve their health management or quality of life.

[0076] "Physical condition" refers to information about the user's posture and movements, and is considered an important element in health management.

[0077] "Equipment" refers to electrical appliances and home appliances used in living spaces, and is subject to management.

[0078] "Operating status" refers to the state in which the equipment is functioning, including whether it is on or off.

[0079] "Items" refer to objects or possessions that exist within a living space and are subject to management or specific designation.

[0080] "Image device" refers to a device that acquires visual information, such as a camera or sensor.

[0081] An "abnormality" refers to a situation that deviates from the normal state, and is something that the system will detect.

[0082] The present invention will now be described in terms of embodiments. This system aims to improve the quality of life in living spaces by integrating health management, energy saving, item management, and security functions. The server collects and analyzes data from various sensor devices installed in the user's living space. The hardware includes cameras, temperature sensors, motion sensors, etc., which provide information to the server via Wi-Fi or Bluetooth.

[0083] The server uses machine learning algorithms to process data in real time and analyze the user's physical condition and the operating status of the equipment. If appropriate processing is performed, for example, if abnormal physical movement is detected, it will generate a notification prompting exercise. Also, if the equipment is found to be unused, it will automatically shut off the power.

[0084] The device receives information from the server and provides appropriate notifications to the user. This is done using smartphone applications and voice assistants.

[0085] For example, if a user is doing desk work for a long time, the server analyzes the video data from the camera and determines that the user's posture is hunched over. Based on this, it sends a notification to the device prompting the user to stretch for 5 minutes.

[0086] Furthermore, when a user leaves the room, the server uses data from motion sensors to monitor the operating status of devices within the living space and automatically turns off any unnecessary devices. Additionally, if the user is searching for their keys, the server consults camera footage to locate the last observed key position and transmits that information to the user's device.

[0087] An example of a prompt sentence to input into a generating AI model is, "Please tell me how to check the user's posture while they are working at their desk and generate appropriate exercise reminders." This prompt sentence allows the AI ​​model to understand instructions regarding motion acquisition and analysis and generate specific output.

[0088] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0089] Step 1:

[0090] The server acquires data from various sensors and cameras installed in the living space. Inputs include video data, temperature, humidity, motion, and other sensor information. The server receives these inputs and organizes the data to prepare for subsequent analysis. Specifically, the server receives signals from various sensors and begins real-time monitoring.

[0091] Step 2:

[0092] The server analyzes the acquired data. Here, machine learning algorithms are used to analyze the user's posture and the state of the room, detecting anomalies and identified events. The input is the organized data collected in step 1, and the output is information on posture anomalies and the status of unused appliances. At this stage, the server feeds the collected data into an AI model to recognize, for example, that the user has been in the same posture for a long time.

[0093] Step 3:

[0094] The server determines specific actions based on the analysis results. Specifically, if an anomaly is detected, it generates an exercise reminder and shuts off power to unused appliances. The input for this stage is the analysis results from step 2, and the output includes notifications to the user and control signals. Based on this, the server performs conditional checks to determine the next action to be taken.

[0095] Step 4:

[0096] The terminal notifies the user of information received from the server. Specifically, the terminal displays and plays messages such as "Please perform stretches" on the screen or via audio. The input is the notification content generated in step 3, and the output is the actual notification to the user. The terminal follows the received command and prompts the user to take the appropriate action.

[0097] Step 5:

[0098] The server monitors user feedback and behavior to evaluate the system's effectiveness. Inputs include new user actions and environmental changes, while outputs are further data collected for the system's next action. Here, the system checks whether the user takes action in response to a notification, and accumulates that response as training data.

[0099] In this way, the entire process works in coordination, providing continuous support that improves user health, energy efficiency, and safety.

[0100] (Application Example 1)

[0101] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0102] In factories and workplaces, improving workers' posture and reducing fatigue are necessary to ensure efficient work while maintaining worker health. Furthermore, there is a need for systems that reduce unnecessary energy consumption of equipment and efficiently manage the location of objects within the work environment and implement security measures.

[0103] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0104] In this invention, the server includes means for monitoring the physical posture of workers in the workspace and generating and notifying information recommending exercises when an abnormal posture is detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the position of an object and notifying the worker of the position of the object upon request. This enables the maintenance of worker health and improvement of productivity, as well as the efficient use of energy and safe management of the workspace.

[0105] A "workspace" refers to a physical environment such as a factory or production facility, where workers perform their daily tasks.

[0106] A "worker" refers to a person who performs tasks within a workspace, and their health and work efficiency are important factors.

[0107] "Body posture" refers to the arrangement and angle of a worker's body, and is a factor that affects work efficiency and health.

[0108] "Equipment" refers to the machinery and equipment used within factories and facilities, and their operation affects energy consumption.

[0109] "Operating status" refers to information indicating the current state in which equipment or machinery is operating.

[0110] "Objects" refer to all concrete things present in the workspace, including resources and tools used in the work.

[0111] "Position" refers to the point or location that an object occupies within the working space.

[0112] "Abnormal behavior" refers to irregular or unplanned movements or actions in a work environment that may interfere with normal operations or safety.

[0113] "Information recommending exercise" refers to messages that include advice on specific exercises and movements that workers should perform to improve their posture and maintain their health.

[0114] "To shut down power" means to temporarily or permanently interrupt the operation of machinery or equipment in order to avoid wasting energy on the equipment.

[0115] To implement this invention, a system is constructed in which cameras and various sensors are installed at each work station in the workspace, and the obtained data is processed on a central server. The server captures the posture of the worker's body with the camera and analyzes the posture using the image processing library OpenCV. Furthermore, it analyzes the posture data using TENSORFLOW®, and if an abnormal posture is detected, it notifies the terminal via the Pushover API with information prompting the worker to perform exercises.

[0116] The server acquires data from sensors attached to each piece of equipment to monitor its operating status and controls the power supply to automatically shut off when the equipment is not in use. This is done using the MQTT protocol to enable real-time data communication between the sensors and the server.

[0117] Furthermore, if an operator needs location information for an object, the system analyzes the video data of the object captured by the camera to identify its location and notify the operator. For object management, a MySQL® database is used to efficiently record and retrieve location information.

[0118] For example, if a worker continues to work in the same posture for a long period of time, their posture is monitored via camera footage, and a message is sent to their terminal saying, "Please take a break or do some exercises to improve your posture." Furthermore, if idle production equipment is running for an extended period, it is automatically shut down to ensure efficient energy use.

[0119] An example of a prompt for a generated AI model is: "Design an AI assistant that provides advice on optimal posture improvement to improve the health management of factory workers. It will analyze the current working posture and suggest the best posture."

[0120] This system makes it possible to create an efficient work environment and improve safety.

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The server acquires video data from cameras installed within the workspace. The input is a video stream from the cameras, and the output is image data that can be analyzed in real time. This image data is processed using OpenCV to detect the posture of the worker's body.

[0124] Step 2:

[0125] The server uses TensorFlow to analyze posture information obtained from image data. The input is posture data processed with OpenCV, and the server determines whether or not an abnormal posture is present based on the analysis results. If an abnormal posture is detected, information prompting exercises is generated.

[0126] Step 3:

[0127] The terminal receives information prompting exercises from the server via the Pushover API and displays it to the worker as a message. The input is the notification information from the server, and the output is a message visually presented to the worker.

[0128] Step 4:

[0129] The server monitors the operating status of the equipment via sensors attached to it. The input is operational data transmitted from the sensors, and the output is status information indicating whether it is currently running or not. Based on this information, it generates a control signal to automatically shut off the power when not in use.

[0130] Step 5:

[0131] The server retrieves the recorded location of objects from a database and, upon request, identifies the location information of the object desired by the worker and notifies the terminal. Input is the object's name or ID, and output is information indicating the object's current location. Location information is managed using MySQL for efficient retrieval.

[0132] Step 6:

[0133] The server continuously monitors camera footage in the workspace and immediately sends an alarm to the terminal if any abnormal activity is detected. The input is real-time camera footage, and alarm information is generated after an image analysis process detects suspicious movements. The output is an alarm message sent to the worker or manager.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] This invention is an integrated system that, in addition to providing support for user health management, energy conservation management, item location, organization, and intruder detection within a living space, utilizes an emotion engine to provide responses based on the user's emotional state. This system mainly consists of a server, terminals, an emotion engine, and a user, and operates autonomously in response to various situations within the living space.

[0136] The server acquires diverse data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors placed in the living space. In particular, the emotion engine analyzes this data to identify the user's emotional state. Based on this data, if the server determines that the user's emotions are stress-based, it provides suggestions and measures to reduce stress via the terminal. In this process, it can automatically play relaxing music and adjust the lighting.

[0137] Furthermore, the server enhances existing health management functions, taking emotional data into account when monitoring posture and generating exercise reminders. For example, if a user has poor posture and is under stress, it can suggest exercises that prioritize relaxation more than usual.

[0138] Furthermore, the server optimizes the operation of home appliances by combining them with emotional data. When the user is in a relaxed emotional state, comfort can be prioritized within the limits of maintaining energy efficiency.

[0139] As a concrete example, when a user is tired and stressed from long hours of desk work, the server detects their emotions from camera and microphone data. At this time, the server adjusts the lighting to a calming tone and plays relaxing music through the device. It also sends a reminder to encourage exercise, notifying the user that "light stretching would be effective." This allows the user to reduce stress and maintain a healthy state.

[0140] Thus, the present invention is designed as a multi-functional integrated system that provides comprehensive life support, including the emotional aspects of the user.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The server acquires audio and video data from cameras and microphones installed in the living space, capturing the user's posture, facial expressions, and voice tone.

[0144] Step 2:

[0145] The server sends the data acquired in real time to the emotion engine, which analyzes the user's emotional state. The analysis identifies the user's emotional state, such as joy, anger, sadness, or stress.

[0146] Step 3:

[0147] The server determines what needs to be addressed based on the user's emotional state. For example, if it determines that the user is stressed, it will consider appropriate actions to promote relaxation.

[0148] Step 4:

[0149] The server sends the selected action to the device. Based on the received instructions, the device plays relaxing music or adjusts the room lighting to a calming color tone.

[0150] Step 5:

[0151] The server analyzes posture data in combination with emotional data to assess the user's health status. If an inappropriate posture and stressful state persist, it immediately generates an exercise reminder.

[0152] Step 6:

[0153] The device notifies the user of exercise reminders and simultaneously offers advice such as, "Light stretching can help reduce stress."

[0154] Step 7:

[0155] The user performs stretches according to the device's suggestions, aiming to improve their emotional state. During this process, the emotion engine re-evaluates the changed emotional state.

[0156] Step 8:

[0157] The server adjusts the operating status of home appliances based on the results of emotion analysis. For example, when the user is relaxed, it sets the air conditioner to a comfortable temperature while also considering energy efficiency.

[0158] Step 9:

[0159] The server continuously monitors the user's state and adjusts the entire system to ensure user comfort, regardless of any changes in their emotions.

[0160] (Example 2)

[0161] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0162] The challenge lies in appropriately understanding the emotional state of users within their living spaces and providing corresponding lifestyle support to improve health management, energy conservation, and comfort. Furthermore, the aim is to improve the quality of life by automatically adjusting the environment without the user's conscious effort.

[0163] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0164] This invention includes a server that includes means for detecting the user's posture in the living space and generating and notifying exercise reminders that also take emotional data into consideration; means for optimizing the operation of home appliances based on the emotional state, prioritizing comfort while maintaining energy efficiency; and means for analyzing data such as voice tone and facial expressions to identify the user's emotions and automatically implementing measures to reduce stress. This enables detailed lifestyle support tailored to the user's emotional state.

[0165] "Living space" refers to the interior and surrounding areas used by residents for their daily lives, and primarily includes the interior of a home.

[0166] "User" refers to an individual or group of individuals who use the system, including residents within the living space covered by this system.

[0167] "Emotional data" refers to data that indicates a user's emotional state, analyzed based on information such as the user's facial expressions, posture, and voice tone.

[0168] A "exercise reminder" refers to a system function that sends information, including notifications and suggestions, to users to encourage them to exercise appropriately.

[0169] "Measures" refer to specific actions and environmental adjustments provided in accordance with the user's emotional state and health condition, with the aim of reducing stress and improving comfort.

[0170] "Energy efficiency" refers to the balance between the power consumed by home appliances and other electrical equipment and the effectiveness of their use, representing the degree to which unnecessary power consumption is minimized and equipment is operated effectively.

[0171] "Health management" is a general term for providing information and lifestyle support to maintain and improve the physical health of users.

[0172] In embodiments of this invention, a system is provided that primarily aims to maintain comfort and health in living spaces, utilizing a server, terminal, and emotion engine. The server uses cameras and sensors installed in the living space to collect diverse data in real time, such as the user's posture, movements, facial expressions, and voice tone. Hardware such as optical cameras, infrared sensors, and voice recognition devices are used for this purpose.

[0173] The emotion engine analyzes collected data to identify the user's emotional state. Machine learning algorithms and natural language processing techniques are applied to the data analysis. Based on the user's emotional data identified by the emotion engine, the server selects measures such as suggesting relaxation-oriented music or adjusting lighting, and executes these via the user's device. This entire process enables automated environment optimization that takes emotional state into account.

[0174] For example, if the emotion engine determines that a user is experiencing stress due to prolonged desk work, the server will send a command to the terminal to play relaxing music. It can also support the reduction of user stress by instructing the lighting system to adjust to a warmer color tone. In this process, the server inputs a prompt message such as "Based on the user's emotional state, please suggest measures to reduce stress" into the AI ​​model, enabling it to propose the most appropriate measures.

[0175] Furthermore, to support user health management, the server generates exercise reminders based on the user's posture and movements and notifies the user. For example, if an inappropriate posture and stress level are detected simultaneously, the server will provide a reminder via the device saying, "We recommend you do some light stretching," thereby promoting the user's health maintenance.

[0176] Ultimately, this system aims to provide customized lifestyle support tailored to each user's individual needs and feelings, pursuing a better quality of life in their living space.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The server collects data on the user's posture, movements, facial expressions, and voice tone using cameras and sensors placed within the living space. It receives video data, audio data, and sensor data as input, processes them in real time, and outputs raw data that indicates the user's physical state. This raw data includes the user's sitting posture, changes in facial expressions, and voice tone.

[0180] Step 2:

[0181] The server inputs raw data into the emotion engine and analyzes the user's emotional state. Here, machine learning algorithms are applied to perform data analysis. The raw data obtained in the previous step is used as input, and the output includes labels indicating the user's emotional state and a numerical emotion score. Specifically, it determines the user's stress level and relaxation level based on features such as voice pitch and facial expressions.

[0182] Step 3:

[0183] The server determines appropriate measures based on emotional state data obtained from the emotion engine. It takes the user's emotional state score as input and generates specific measure suggestions as output. This step includes specific instructions such as "play relaxing music" or "adjust lighting to warmer colors." The generating AI model is then prompted with a message such as "Please suggest measures to reduce stress based on the user's emotional state" to obtain suggested measures.

[0184] Step 4:

[0185] The device receives instructions from the server and performs actual environmental adjustments based on those instructions. Its input is policy instructions from the server, and its output is changes to the environmental settings within the living space. Specific actions include playing relaxation music through the speaker or changing the color tone of smart lighting.

[0186] Step 5:

[0187] The server evaluates the user's posture data while considering emotional state data as part of health management. It takes raw data from Step 1 and emotional data from Step 2 as input, and generates health status analysis results and appropriate exercise reminders as output. Specifically, if an inappropriate posture persists, a notification such as "We recommend doing some light stretching" is sent from the device to the user.

[0188] Step 6:

[0189] The server optimizes the operation of home appliances based on emotional data. Inputs include emotional scores and the current operating status of the appliances, and the output generates optimal operating instructions that consider energy efficiency. For example, if a user is relaxed, the server will maintain comfort without unnecessarily lowering the air conditioner's temperature setting.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] In modern living spaces, the need for user health management and energy conservation measures is increasing. Furthermore, efficient location tracking and organization of items, as well as detection of suspicious individuals, are also crucial issues. Additionally, there is a demand for immediate responses tailored to users' emotional states and improved customer experiences in physical stores. However, a system that comprehensively addresses all of these needs does not yet exist.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for monitoring the user's posture in the living space and generating and notifying an exercise reminder when an abnormal posture is detected; means for monitoring the operating status of home appliances and automatically turning off the power when not in use; means for recording the location of items and notifying the user of the location of said items upon request; means for generating and notifying tidying-up advice based on the state of the living space; means for detecting suspicious persons using a camera and generating an alarm when detected; means for analyzing the user's emotional state using emotion analysis technology and optimizing the living space environment settings; and means for analyzing the customer's emotional state and providing purchasing support information and services. This integrates diverse functions, improves the user's quality of life, and enables the optimization of the customer experience in physical stores.

[0195] "Living space" refers to the space in which an individual or family conducts their daily life, and includes all rooms and facilities within the home.

[0196] "User" refers to an individual who operates or experiences a living space or system.

[0197] A "posture monitor" is a device or system used to detect and analyze the positioning and movement of a user's body.

[0198] An "exercise reminder" is a feature that generates notifications or alerts prompting users to perform specific exercises.

[0199] "Home appliances" refer to electrical equipment used in the home, including washing machines, refrigerators, and air conditioners.

[0200] "Item location recording" is the process of tracking where a specific item is located and maintaining that information.

[0201] "Organization and tidying advice" means providing recommendations for the efficient arrangement and storage of items.

[0202] A "shooting device" refers to equipment used to capture still images or videos, such as cameras and video cameras.

[0203] "Suspicious person detection" is the process of identifying individuals who are engaging in unusual or suspicious activities within a living space.

[0204] "Emotional analysis technology" is a technology that identifies a user's emotional state from their facial expressions, voice, and behavior.

[0205] "Optimizing environmental settings" refers to adjusting elements such as lighting, sound, and temperature in a living space to suit the user's comfort.

[0206] "Purchase support information" refers to information and suggestions that help customers make purchasing decisions.

[0207] "Service provision" refers to the act of providing value-added services that meet customer needs.

[0208] This invention is a system that monitors the health and emotions of users in living spaces and retail spaces to create a comfortable living environment.

[0209] The server acquires data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors within the living space. Based on this data, the server monitors the user's health status and generates and notifies them of exercise reminders if their posture is inappropriate.

[0210] Furthermore, the server monitors the operating status of home appliances and automatically turns them off when not in use to minimize energy consumption. It also provides features to help with organization, such as recording the location of items and notifying users of their location upon request.

[0211] By using emotion analysis technology, the server identifies the user's emotional state and automatically optimizes the environment settings to consider relaxation and energy efficiency. This allows users to live comfortably without feeling stressed. For example, if a user feels fatigued after working at a desk for a long time, the server automatically adjusts the lighting and plays relaxing music. It also sends reminders to encourage exercise to enhance the relaxation effect.

[0212] Furthermore, in physical stores, it is possible to analyze customers' facial expressions and voice tone to provide optimal purchasing support information and services. This process is based on emotion analysis technology and generative AI models (e.g., OpenAI's GPT-3).

[0213] For example, if a customer is walking around the store and appears tense, the background music will be switched to something calming, and product suggestions will be displayed on their smart device.

[0214] An example of a prompt message is as follows:

[0215] "We will provide the customer's facial expressions and tone of voice in the following format. Based on this data, please tell us what emotions the customer is currently experiencing."

[0216] Facial expression data: No smile, frowning.

[0217] Voice tone: Slightly fast-paced and high-pitched.

[0218] Based on this, please propose appropriate purchasing support services.

[0219] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0220] Step 1:

[0221] The server acquires data from cameras and sensors installed in the living space, including the user's posture, movement, facial expressions, and voice tone. Input data includes real-time video streams and audio input. This data is stored in temporary storage and used in the next processing step.

[0222] Step 2:

[0223] The server uses the acquired data to execute emotion analysis techniques and motion analysis algorithms. The input includes posture, facial expressions, and voice tone collected in step 1, which are then analyzed to determine the user's current emotional and postural state. A generative AI model is used here, utilizing prompt sentences to aid in the analysis and identify the user's emotional state. The output is an analysis result including emotional and postural states.

[0224] Step 3:

[0225] The server generates and notifies users of exercise reminders via the user interface based on the results of emotion and posture analysis. It also suggests environmental adjustments to optimize energy efficiency and improve the customer experience. Specific adjustments include lighting and music settings. Users can adjust the color temperature and brightness of the lighting and play relaxing music via their device. Outputs include exercise reminders and environmental adjustment suggestions.

[0226] Step 4:

[0227] The device, based on instructions from the server, notifies the user of generated reminders and adjusts environmental settings. For example, reminders are displayed as notifications on the smartphone, and music and lighting adjustments are performed directly through smart home appliances. The output is the reminder notification and the adjusted environment.

[0228] Step 5:

[0229] The user receives notifications and performs recommended exercise or relaxation activities. Based on the information obtained from the device, the user adjusts their behavior to maintain a comfortable state. The output is the specific action the user takes.

[0230] Step 6:

[0231] The server continuously runs the same process, dynamically adjusting in response to changes in the user's state and environment. This helps users maintain a comfortable environment for extended periods. The output consists of continuously updated analysis results and recommended actions.

[0232] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0233] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0234] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0235] [Second Embodiment]

[0236] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0237] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0238] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0239] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0240] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0241] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0242] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0243] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0244] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0245] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0246] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0247] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0248] This invention is an integrated system that improves the quality of life in living spaces, and is designed to provide multiple functions related to health management, energy saving, item location, organization and tidiness advice, and crime prevention. This system is primarily configured to exchange information between a server, terminals, and users, and to provide multifaceted support for the user's life.

[0249] The server acquires information from cameras and various sensors installed in the living space and analyzes this data in real time. Specifically, the server monitors the user's posture and, if it detects prolonged periods of inappropriate posture, generates a reminder to encourage exercise. This helps users maintain healthy lifestyle habits.

[0250] Regarding energy management, the server constantly monitors the operating status of each appliance and reduces unnecessary power consumption by automatically turning off unused appliances. It also uses cameras to check the room's condition and the location of items, locating lost items as needed and providing this information to the user via a terminal. This feature allows users to find items efficiently.

[0251] Furthermore, the tidying advice function analyzes the current state of the living space via a server and provides specific tidying advice to the user through their device. This allows users to create a more comfortable living environment.

[0252] Furthermore, as a security feature, the server continuously monitors camera footage and immediately sends an alert to the user via their device if a suspicious person is detected. This allows users to respond quickly and ensure their safety.

[0253] For example, if a user is working at their desk for a long time, the server will recognize their posture and display a reminder on their device saying, "Please do 5 minutes of stretching." Also, if a user leaves their bedroom, the server will detect their movement and automatically turn off unused lights and air conditioning. Furthermore, if a user is looking for their keys, the server will notify their device of the last observed location of the keys.

[0254] Thus, the present invention aims to make users' lives more comfortable and safer by providing multiple functions in an integrated manner.

[0255] The following describes the processing flow.

[0256] Step 1:

[0257] The server begins receiving real-time video and data from cameras and sensors installed in the living space. This includes posture data and the operating status of home appliances.

[0258] Step 2:

[0259] The server analyzes the received data and determines the user's posture. If the posture is inappropriate, the server immediately generates an exercise reminder.

[0260] Step 3:

[0261] The server sends the generated exercise reminder to the device. The device notifies the user visually or audibly.

[0262] Step 4:

[0263] The server monitors the operating status of home appliances and generates an automatic shut-off command if any appliances are not in use.

[0264] Step 5:

[0265] The terminal receives commands from the server and saves energy by turning off the power to unused home appliances.

[0266] Step 6:

[0267] The server receives an item location search request from the user and determines its location based on camera data.

[0268] Step 7:

[0269] The server transmits the location information of the identified item to the terminal, and the terminal informs the user of the item's location.

[0270] Step 8:

[0271] The server evaluates the tidiness of the living space and generates tidying-up advice as needed.

[0272] Step 9:

[0273] The terminal provides the user with advice from the server and encourages them to organize and tidy up.

[0274] Step 10:

[0275] The server constantly monitors camera footage and, if it detects a suspicious person, immediately generates an alarm and notifies the user via their device.

[0276] Step 11:

[0277] Users take appropriate safety measures based on alerts from their devices.

[0278] (Example 1)

[0279] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0280] In modern living environments, users often demand simultaneous improvements in health management, energy conservation, safety, and efficient item management. However, currently, there is no integrated system that can meet these demands, and users rely on devices with individual functions. As a result, users are burdened with managing multiple systems, making it difficult to ensure true comfort and safety. Therefore, there is a need for an integrated system that can solve these problems simultaneously.

[0281] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0282] In this invention, the server includes means for monitoring the physical condition of a user in a living space, generating and notifying information for promoting exercise when detecting abnormal physical movements, means for monitoring the operating status of devices and automatically cutting off power when not in use, and means for recording the location of articles and notifying the location of the articles in response to a request from the user. As a result, the user can efficiently save energy while maintaining health and manage articles safely and effectively.

[0283] The "living space" is a place where life and activities are carried out, mainly referring to the internal space of a house, apartment, etc.

[0284] The "user" is a person who uses this system, referring to those who seek health management and improvement of the quality of life.

[0285] The "physical condition" refers to information regarding the posture and movements of the user, which is an important factor in health management.

[0286] The "devices" refer to electrical appliances and household appliances used in the living space and are the objects of management.

[0287] The "operating status" refers to the operating state of the device and includes the on and off states.

[0288] The "articles" refer to objects and possessions existing in the living space and are the objects of management and identification.

[0289] The "image device" refers to a device that acquires visual information such as a camera or sensor.

[0290] <​​​​The present invention will now be described in terms of embodiments. This system aims to improve the quality of life in living spaces by integrating health management, energy saving, item management, and security functions. The server collects and analyzes data from various sensor devices installed in the user's living space. The hardware includes cameras, temperature sensors, motion sensors, etc., which provide information to the server via Wi-Fi or Bluetooth.

[0292] The server uses machine learning algorithms to process data in real time and analyze the user's physical condition and the operating status of the equipment. If appropriate processing is performed, for example, if abnormal physical movement is detected, it will generate a notification prompting exercise. Also, if the equipment is found to be unused, it will automatically shut off the power.

[0293] The device receives information from the server and provides appropriate notifications to the user. This is done using smartphone applications and voice assistants.

[0294] For example, if a user is doing desk work for a long time, the server analyzes the video data from the camera and determines that the user's posture is hunched over. Based on this, it sends a notification to the device prompting the user to stretch for 5 minutes.

[0295] Furthermore, when a user leaves the room, the server uses data from motion sensors to monitor the operating status of devices within the living space and automatically turns off any unnecessary devices. Additionally, if the user is searching for their keys, the server consults camera footage to locate the last observed key position and transmits that information to the user's device.

[0296] An example of a prompt sentence to input into a generating AI model is, "Please tell me how to check the user's posture while they are working at their desk and generate appropriate exercise reminders." This prompt sentence allows the AI ​​model to understand instructions regarding motion acquisition and analysis and generate specific output.

[0297] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0298] Step 1:

[0299] The server acquires data from various sensors and cameras installed in the living space. Inputs include video data, temperature, humidity, motion, and other sensor information. The server receives these inputs and organizes the data to prepare for subsequent analysis. Specifically, the server receives signals from various sensors and begins real-time monitoring.

[0300] Step 2:

[0301] The server analyzes the acquired data. Here, machine learning algorithms are used to analyze the user's posture and the state of the room, detecting anomalies and identified events. The input is the organized data collected in step 1, and the output is information on posture anomalies and the status of unused appliances. At this stage, the server feeds the collected data into an AI model to recognize, for example, that the user has been in the same posture for a long time.

[0302] Step 3:

[0303] The server determines specific actions based on the analysis results. Specifically, if an anomaly is detected, it generates an exercise reminder and shuts off power to unused appliances. The input for this stage is the analysis results from step 2, and the output includes notifications to the user and control signals. Based on this, the server performs conditional checks to determine the next action to be taken.

[0304] Step 4:

[0305] The terminal notifies the user of the information received from the server. As a specific operation, the terminal displays and plays messages such as "Please stretch" on the screen or by voice. The input is the notification content generated in Step 3, and the output is the actual notification to the user. The terminal prompts the user to take appropriate actions according to the received instructions.

[0306] Step 5:

[0307] The server monitors the feedback and actions from the user and evaluates the effectiveness of the system. The inputs include the user's new actions and environmental changes, and the output is the result of further data collection for the next action of the system. Here, it is checked whether the user takes an action in response to the notification, and the reaction is accumulated as learning data.

[0308] In this way, through the cooperation of the overall processing, the system provides continuous support to improve the user's health, energy conservation, and safety.

[0309] (Application Example 1)

[0310] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0311] In factories and workspaces, in order to work efficiently while maintaining the health of workers, it is necessary to improve the posture and reduce fatigue of workers. In addition, there is a need for a mechanism to reduce unnecessary energy consumption of equipment and efficiently manage the positions of objects and implement security measures within the business environment.

[0312] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0313] In this invention, the server includes means for monitoring the physical posture of workers in the workspace and generating and notifying information recommending exercises when an abnormal posture is detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the position of an object and notifying the worker of the position of the object upon request. This enables the maintenance of worker health and improvement of productivity, as well as the efficient use of energy and safe management of the workspace.

[0314] A "workspace" refers to a physical environment such as a factory or production facility, where workers perform their daily tasks.

[0315] A "worker" refers to a person who performs tasks within a workspace, and their health and work efficiency are important factors.

[0316] "Body posture" refers to the arrangement and angle of a worker's body, and is a factor that affects work efficiency and health.

[0317] "Equipment" refers to the machinery and equipment used within factories and facilities, and their operation affects energy consumption.

[0318] "Operating status" refers to information indicating the current state in which equipment or machinery is operating.

[0319] "Objects" refer to all concrete things present in the workspace, including resources and tools used in the work.

[0320] "Position" refers to the point or location that an object occupies within the working space.

[0321] "Abnormal behavior" refers to irregular or unplanned movements or actions in a work environment that may interfere with normal operations or safety.

[0322] "Information recommending exercise" refers to messages that include advice on specific exercises and movements that workers should perform to improve their posture and maintain their health.

[0323] "To shut down power" means to temporarily or permanently interrupt the operation of machinery or equipment in order to avoid wasting energy on the equipment.

[0324] To implement this invention, a system is constructed in which cameras and various sensors are installed at each work station in the workspace, and the acquired data is processed on a central server. The server captures the posture of the worker's body with the camera and analyzes the posture using the image processing library OpenCV. Furthermore, it analyzes the posture data using TensorFlow, and if an abnormal posture is detected, it notifies the terminal via the Pushover API with information prompting the worker to perform exercises.

[0325] The server acquires data from sensors attached to each piece of equipment to monitor its operating status and controls the power supply to automatically shut off when the equipment is not in use. This is done using the MQTT protocol to enable real-time data communication between the sensors and the server.

[0326] Furthermore, if an operator needs location information for an object, the system analyzes the video data of the object captured by the camera to identify its location and notify the operator. A MySQL database is used to efficiently record and retrieve location information for object management.

[0327] For example, if a worker continues to work in the same posture for a long period of time, their posture is monitored via camera footage, and a message is sent to their terminal saying, "Please take a break or do some exercises to improve your posture." Furthermore, if idle production equipment is running for an extended period, it is automatically shut down to ensure efficient energy use.

[0328] An example of a prompt for a generated AI model is: "Design an AI assistant that provides advice on optimal posture improvement to improve the health management of factory workers. It will analyze the current working posture and suggest the best posture."

[0329] This system makes it possible to create an efficient work environment and improve safety.

[0330] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0331] Step 1:

[0332] The server acquires video data from cameras installed within the workspace. The input is a video stream from the cameras, and the output is image data that can be analyzed in real time. This image data is processed using OpenCV to detect the posture of the worker's body.

[0333] Step 2:

[0334] The server uses TensorFlow to analyze posture information obtained from image data. The input is posture data processed with OpenCV, and the server determines whether or not an abnormal posture is present based on the analysis results. If an abnormal posture is detected, information prompting exercises is generated.

[0335] Step 3:

[0336] The terminal receives information prompting exercises from the server via the Pushover API and displays it to the worker as a message. The input is the notification information from the server, and the output is a message visually presented to the worker.

[0337] Step 4:

[0338] The server monitors the operating status of the equipment via sensors attached to it. The input is operational data transmitted from the sensors, and the output is status information indicating whether it is currently running or not. Based on this information, it generates a control signal to automatically shut off the power when not in use.

[0339] Step 5:

[0340] The server retrieves the recorded location of objects from a database and, upon request, identifies the location information of the object desired by the worker and notifies the terminal. Input is the object's name or ID, and output is information indicating the object's current location. Location information is managed using MySQL for efficient retrieval.

[0341] Step 6:

[0342] The server continuously monitors camera footage in the workspace and immediately sends an alarm to the terminal if any abnormal activity is detected. The input is real-time camera footage, and alarm information is generated after an image analysis process detects suspicious movements. The output is an alarm message sent to the worker or manager.

[0343] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0344] This invention is an integrated system that, in addition to providing support for user health management, energy conservation management, item location, organization, and intruder detection within a living space, utilizes an emotion engine to provide responses based on the user's emotional state. This system mainly consists of a server, terminals, an emotion engine, and a user, and operates autonomously in response to various situations within the living space.

[0345] The server acquires diverse data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors placed in the living space. In particular, the emotion engine analyzes this data to identify the user's emotional state. Based on this data, if the server determines that the user's emotions are stress-based, it provides suggestions and measures to reduce stress via the terminal. In this process, it can automatically play relaxing music and adjust the lighting.

[0346] Furthermore, the server enhances existing health management functions, taking emotional data into account when monitoring posture and generating exercise reminders. For example, if a user has poor posture and is under stress, it can suggest exercises that prioritize relaxation more than usual.

[0347] Furthermore, the server optimizes the operation of home appliances by combining them with emotional data. When the user is in a relaxed emotional state, comfort can be prioritized within the limits of maintaining energy efficiency.

[0348] As a concrete example, when a user is tired and stressed from long hours of desk work, the server detects their emotions from camera and microphone data. At this time, the server adjusts the lighting to a calming tone and plays relaxing music through the device. It also sends a reminder to encourage exercise, notifying the user that "light stretching would be effective." This allows the user to reduce stress and maintain a healthy state.

[0349] Thus, the present invention is designed as a multi-functional integrated system that provides comprehensive life support, including the emotional aspects of the user.

[0350] The following describes the processing flow.

[0351] Step 1:

[0352] The server acquires audio and video data from cameras and microphones installed in the living space, capturing the user's posture, facial expressions, and voice tone.

[0353] Step 2:

[0354] The server sends the data acquired in real time to the emotion engine, which analyzes the user's emotional state. The analysis identifies the user's emotional state, such as joy, anger, sadness, or stress.

[0355] Step 3:

[0356] The server determines what needs to be addressed based on the user's emotional state. For example, if it determines that the user is stressed, it will consider appropriate actions to promote relaxation.

[0357] Step 4:

[0358] The server sends the selected action to the device. Based on the received instructions, the device plays relaxing music or adjusts the room lighting to a calming color tone.

[0359] Step 5:

[0360] The server analyzes posture data in combination with emotional data to assess the user's health status. If an inappropriate posture and stressful state persist, it immediately generates an exercise reminder.

[0361] Step 6:

[0362] The device notifies the user of exercise reminders and simultaneously offers advice such as, "Light stretching can help reduce stress."

[0363] Step 7:

[0364] The user performs stretches according to the device's suggestions, aiming to improve their emotional state. During this process, the emotion engine re-evaluates the changed emotional state.

[0365] Step 8:

[0366] The server adjusts the operating status of home appliances based on the results of emotion analysis. For example, when the user is relaxed, it sets the air conditioner to a comfortable temperature while also considering energy efficiency.

[0367] Step 9:

[0368] The server continuously monitors the user's state and adjusts the entire system to ensure user comfort, regardless of any changes in their emotions.

[0369] (Example 2)

[0370] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0371] The challenge lies in appropriately understanding the emotional state of users within their living spaces and providing corresponding lifestyle support to improve health management, energy conservation, and comfort. Furthermore, the aim is to improve the quality of life by automatically adjusting the environment without the user's conscious effort.

[0372] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0373] This invention includes a server that includes means for detecting the user's posture in the living space and generating and notifying exercise reminders that also take emotional data into consideration; means for optimizing the operation of home appliances based on the emotional state, prioritizing comfort while maintaining energy efficiency; and means for analyzing data such as voice tone and facial expressions to identify the user's emotions and automatically implementing measures to reduce stress. This enables detailed lifestyle support tailored to the user's emotional state.

[0374] "Living space" refers to the interior and surrounding areas used by residents for their daily lives, and primarily includes the interior of a home.

[0375] "User" refers to an individual or group of individuals who use the system, including residents within the living space covered by this system.

[0376] "Emotional data" refers to data that indicates a user's emotional state, analyzed based on information such as the user's facial expressions, posture, and voice tone.

[0377] A "exercise reminder" refers to a system function that sends information, including notifications and suggestions, to users to encourage them to exercise appropriately.

[0378] "Measures" refer to specific actions and environmental adjustments provided in accordance with the user's emotional state and health condition, with the aim of reducing stress and improving comfort.

[0379] "Energy efficiency" refers to the balance between the power consumed by home appliances and other electrical equipment and the effectiveness of their use, representing the degree to which unnecessary power consumption is minimized and equipment is operated effectively.

[0380] "Health management" is a general term for providing information and lifestyle support to maintain and improve the physical health of users.

[0381] In embodiments of this invention, a system is provided that primarily aims to maintain comfort and health in living spaces, utilizing a server, terminal, and emotion engine. The server uses cameras and sensors installed in the living space to collect diverse data in real time, such as the user's posture, movements, facial expressions, and voice tone. Hardware such as optical cameras, infrared sensors, and voice recognition devices are used for this purpose.

[0382] The emotion engine analyzes collected data to identify the user's emotional state. Machine learning algorithms and natural language processing techniques are applied to the data analysis. Based on the user's emotional data identified by the emotion engine, the server selects measures such as suggesting relaxation-oriented music or adjusting lighting, and executes these via the user's device. This entire process enables automated environment optimization that takes emotional state into account.

[0383] For example, if the emotion engine determines that a user is experiencing stress due to prolonged desk work, the server will send a command to the terminal to play relaxing music. It can also support the reduction of user stress by instructing the lighting system to adjust to a warmer color tone. In this process, the server inputs a prompt message such as "Based on the user's emotional state, please suggest measures to reduce stress" into the AI ​​model, enabling it to propose the most appropriate measures.

[0384] Furthermore, to support user health management, the server generates exercise reminders based on the user's posture and movements and notifies the user. For example, if an inappropriate posture and stress level are detected simultaneously, the server will provide a reminder via the device saying, "We recommend you do some light stretching," thereby promoting the user's health maintenance.

[0385] Ultimately, this system aims to provide customized lifestyle support tailored to each user's individual needs and feelings, pursuing a better quality of life in their living space.

[0386] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0387] Step 1:

[0388] The server collects data on the user's posture, movements, facial expressions, and voice tone using cameras and sensors placed within the living space. It receives video data, audio data, and sensor data as input, processes them in real time, and outputs raw data that indicates the user's physical state. This raw data includes the user's sitting posture, changes in facial expressions, and voice tone.

[0389] Step 2:

[0390] The server inputs raw data into the emotion engine and analyzes the user's emotional state. Here, machine learning algorithms are applied to perform data analysis. The raw data obtained in the previous step is used as input, and the output includes labels indicating the user's emotional state and a numerical emotion score. Specifically, it determines the user's stress level and relaxation level based on features such as voice pitch and facial expressions.

[0391] Step 3:

[0392] The server determines appropriate measures based on emotional state data obtained from the emotion engine. It takes the user's emotional state score as input and generates specific measure suggestions as output. This step includes specific instructions such as "play relaxing music" or "adjust lighting to warmer colors." The generating AI model is then prompted with a message such as "Please suggest measures to reduce stress based on the user's emotional state" to obtain suggested measures.

[0393] Step 4:

[0394] The device receives instructions from the server and performs actual environmental adjustments based on those instructions. Its input is policy instructions from the server, and its output is changes to the environmental settings within the living space. Specific actions include playing relaxation music through the speaker or changing the color tone of smart lighting.

[0395] Step 5:

[0396] The server evaluates the user's posture data while considering emotional state data as part of health management. It takes raw data from Step 1 and emotional data from Step 2 as input, and generates health status analysis results and appropriate exercise reminders as output. Specifically, if an inappropriate posture persists, a notification such as "We recommend doing some light stretching" is sent from the device to the user.

[0397] Step 6:

[0398] The server optimizes the operation of home appliances based on emotional data. Inputs include emotional scores and the current operating status of the appliances, and the output generates optimal operating instructions that consider energy efficiency. For example, if a user is relaxed, the server will maintain comfort without unnecessarily lowering the air conditioner's temperature setting.

[0399] (Application Example 2)

[0400] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0401] In modern living spaces, the need for user health management and energy conservation measures is increasing. Furthermore, efficient location tracking and organization of items, as well as detection of suspicious individuals, are also crucial issues. Additionally, there is a demand for immediate responses tailored to users' emotional states and improved customer experiences in physical stores. However, a system that comprehensively addresses all of these needs does not yet exist.

[0402] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0403] In this invention, the server includes means for monitoring the user's posture in the living space and generating and notifying an exercise reminder when an abnormal posture is detected; means for monitoring the operating status of home appliances and automatically turning off the power when not in use; means for recording the location of items and notifying the user of the location of said items upon request; means for generating and notifying tidying-up advice based on the state of the living space; means for detecting suspicious persons using a camera and generating an alarm when detected; means for analyzing the user's emotional state using emotion analysis technology and optimizing the living space environment settings; and means for analyzing the customer's emotional state and providing purchasing support information and services. This integrates diverse functions, improves the user's quality of life, and enables the optimization of the customer experience in physical stores.

[0404] "Living space" refers to the space in which an individual or family conducts their daily life, and includes all rooms and facilities within the home.

[0405] "User" refers to an individual who operates or experiences a living space or system.

[0406] A "posture monitor" is a device or system used to detect and analyze the positioning and movement of a user's body.

[0407] An "exercise reminder" is a feature that generates notifications or alerts prompting users to perform specific exercises.

[0408] "Home appliances" refer to electrical equipment used in the home, including washing machines, refrigerators, and air conditioners.

[0409] "Item location recording" is the process of tracking where a specific item is located and maintaining that information.

[0410] "Organization and tidying advice" means providing recommendations for the efficient arrangement and storage of items.

[0411] A "shooting device" refers to equipment used to capture still images or videos, such as cameras and video cameras.

[0412] "Suspicious person detection" is the process of identifying individuals who are engaging in unusual or suspicious activities within a living space.

[0413] "Emotional analysis technology" is a technology that identifies a user's emotional state from their facial expressions, voice, and behavior.

[0414] "Optimizing environmental settings" refers to adjusting elements such as lighting, sound, and temperature in a living space to suit the user's comfort.

[0415] "Purchase support information" refers to information and suggestions that help customers make purchasing decisions.

[0416] "Service provision" refers to the act of providing value-added services that meet customer needs.

[0417] This invention is a system that monitors the health and emotions of users in living spaces and retail spaces to create a comfortable living environment.

[0418] The server acquires data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors within the living space. Based on this data, the server monitors the user's health status and generates and notifies them of exercise reminders if their posture is inappropriate.

[0419] Furthermore, the server monitors the operating status of home appliances and automatically turns them off when not in use to minimize energy consumption. It also provides features to help with organization, such as recording the location of items and notifying users of their location upon request.

[0420] By using emotion analysis technology, the server identifies the user's emotional state and automatically optimizes the environment settings to consider relaxation and energy efficiency. This allows users to live comfortably without feeling stressed. For example, if a user feels fatigued after working at a desk for a long time, the server automatically adjusts the lighting and plays relaxing music. It also sends reminders to encourage exercise to enhance the relaxation effect.

[0421] Furthermore, in physical stores, it is possible to analyze customers' facial expressions and voice tone to provide optimal purchasing support information and services. This process is based on emotion analysis technology and generative AI models (e.g., OpenAI's GPT-3).

[0422] For example, if a customer is walking around the store and appears tense, the background music will be switched to something calming, and product suggestions will be displayed on their smart device.

[0423] An example of a prompt message is as follows:

[0424] "We will provide the customer's facial expressions and tone of voice in the following format. Based on this data, please tell us what emotions the customer is currently experiencing."

[0425] Facial expression data: No smile, frowning.

[0426] Voice tone: Slightly fast-paced and high-pitched.

[0427] Based on this, please propose appropriate purchasing support services.

[0428] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0429] Step 1:

[0430] The server acquires data from cameras and sensors installed in the living space, including the user's posture, movement, facial expressions, and voice tone. Input data includes real-time video streams and audio input. This data is stored in temporary storage and used in the next processing step.

[0431] Step 2:

[0432] The server uses the acquired data to execute emotion analysis techniques and motion analysis algorithms. The input includes posture, facial expressions, and voice tone collected in step 1, which are then analyzed to determine the user's current emotional and postural state. A generative AI model is used here, utilizing prompt sentences to aid in the analysis and identify the user's emotional state. The output is an analysis result including emotional and postural states.

[0433] Step 3:

[0434] The server generates and notifies users of exercise reminders via the user interface based on the results of emotion and posture analysis. It also suggests environmental adjustments to optimize energy efficiency and improve the customer experience. Specific adjustments include lighting and music settings. Users can adjust the color temperature and brightness of the lighting and play relaxing music via their device. Outputs include exercise reminders and environmental adjustment suggestions.

[0435] Step 4:

[0436] The device, based on instructions from the server, notifies the user of generated reminders and adjusts environmental settings. For example, reminders are displayed as notifications on the smartphone, and music and lighting adjustments are performed directly through smart home appliances. The output is the reminder notification and the adjusted environment.

[0437] Step 5:

[0438] The user receives notifications and performs recommended exercise or relaxation activities. Based on the information obtained from the device, the user adjusts their behavior to maintain a comfortable state. The output is the specific action the user takes.

[0439] Step 6:

[0440] The server continuously runs the same process, dynamically adjusting in response to changes in the user's state and environment. This helps users maintain a comfortable environment for extended periods. The output consists of continuously updated analysis results and recommended actions.

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0442] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0444] [Third Embodiment]

[0445] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0446] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0447] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0449] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0451] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0452] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0453] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0456] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0457] This invention is an integrated system that improves the quality of life in living spaces, and is designed to provide multiple functions related to health management, energy saving, item location, organization and tidiness advice, and crime prevention. This system is primarily configured to exchange information between a server, terminals, and users, and to provide multifaceted support for the user's life.

[0458] The server acquires information from cameras and various sensors installed in the living space and analyzes this data in real time. Specifically, the server monitors the user's posture and, if it detects prolonged periods of inappropriate posture, generates a reminder to encourage exercise. This helps users maintain healthy lifestyle habits.

[0459] Regarding energy management, the server constantly monitors the operating status of each appliance and reduces unnecessary power consumption by automatically turning off unused appliances. It also uses cameras to check the room's condition and the location of items, locating lost items as needed and providing this information to the user via a terminal. This feature allows users to find items efficiently.

[0460] Furthermore, the tidying advice function analyzes the current state of the living space via a server and provides specific tidying advice to the user through their device. This allows users to create a more comfortable living environment.

[0461] Furthermore, as a security feature, the server continuously monitors camera footage and immediately sends an alert to the user via their device if a suspicious person is detected. This allows users to respond quickly and ensure their safety.

[0462] For example, if a user is working at their desk for a long time, the server will recognize their posture and display a reminder on their device saying, "Please do 5 minutes of stretching." Also, if a user leaves their bedroom, the server will detect their movement and automatically turn off unused lights and air conditioning. Furthermore, if a user is looking for their keys, the server will notify their device of the last observed location of the keys.

[0463] Thus, the present invention aims to make users' lives more comfortable and safer by providing multiple functions in an integrated manner.

[0464] The following describes the processing flow.

[0465] Step 1:

[0466] The server begins receiving real-time video and data from cameras and sensors installed in the living space. This includes posture data and the operating status of home appliances.

[0467] Step 2:

[0468] The server analyzes the received data and determines the user's posture. If the posture is inappropriate, the server immediately generates an exercise reminder.

[0469] Step 3:

[0470] The server sends the generated exercise reminder to the device. The device notifies the user visually or audibly.

[0471] Step 4:

[0472] The server monitors the operating status of home appliances and generates an automatic shut-off command if any appliances are not in use.

[0473] Step 5:

[0474] The terminal receives commands from the server and saves energy by turning off the power to unused home appliances.

[0475] Step 6:

[0476] The server receives an item location search request from the user and determines its location based on camera data.

[0477] Step 7:

[0478] The server transmits the location information of the identified item to the terminal, and the terminal informs the user of the item's location.

[0479] Step 8:

[0480] The server evaluates the tidiness of the living space and generates tidying-up advice as needed.

[0481] Step 9:

[0482] The terminal provides the user with advice from the server and encourages them to organize and tidy up.

[0483] Step 10:

[0484] The server constantly monitors camera footage and, if it detects a suspicious person, immediately generates an alarm and notifies the user via their device.

[0485] Step 11:

[0486] Users take appropriate safety measures based on alerts from their devices.

[0487] (Example 1)

[0488] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0489] In modern living environments, users often demand simultaneous improvements in health management, energy conservation, safety, and efficient item management. However, currently, there is no integrated system that can meet these demands, and users rely on devices with individual functions. As a result, users are burdened with managing multiple systems, making it difficult to ensure true comfort and safety. Therefore, there is a need for an integrated system that can solve these problems simultaneously.

[0490] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0491] In this invention, the server includes means for monitoring the physical condition of users in a living space and generating and notifying information to encourage exercise when abnormal physical movements are detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the location of items and notifying the user of the location of said items upon request. This enables users to maintain their health, efficiently conserve energy, and manage items safely and effectively.

[0492] "Living space" refers to a place where life and activities take place, and primarily refers to the interior space of a house, apartment, or similar building.

[0493] "Users" refer to individuals who use this system and seek to improve their health management or quality of life.

[0494] "Physical condition" refers to information about the user's posture and movements, and is considered an important element in health management.

[0495] "Equipment" refers to electrical appliances and home appliances used in living spaces, and is subject to management.

[0496] "Operating status" refers to the state in which the equipment is functioning, including whether it is on or off.

[0497] "Items" refer to objects or possessions that exist within a living space and are subject to management or specific designation.

[0498] "Image device" refers to a device that acquires visual information, such as a camera or sensor.

[0499] An "abnormality" refers to a situation that deviates from the normal state, and is something that the system will detect.

[0500] The present invention will now be described in terms of embodiments. This system aims to improve the quality of life in living spaces by integrating health management, energy saving, item management, and security functions. The server collects and analyzes data from various sensor devices installed in the user's living space. The hardware includes cameras, temperature sensors, motion sensors, etc., which provide information to the server via Wi-Fi or Bluetooth.

[0501] The server uses machine learning algorithms to process data in real time and analyze the user's physical condition and the operating status of the equipment. If appropriate processing is performed, for example, if abnormal physical movement is detected, it will generate a notification prompting exercise. Also, if the equipment is found to be unused, it will automatically shut off the power.

[0502] The device receives information from the server and provides appropriate notifications to the user. This is done using smartphone applications and voice assistants.

[0503] For example, if a user is doing desk work for a long time, the server analyzes the video data from the camera and determines that the user's posture is hunched over. Based on this, it sends a notification to the device prompting the user to stretch for 5 minutes.

[0504] Furthermore, when a user leaves the room, the server uses data from motion sensors to monitor the operating status of devices within the living space and automatically turns off any unnecessary devices. Additionally, if the user is searching for their keys, the server consults camera footage to locate the last observed key position and transmits that information to the user's device.

[0505] An example of a prompt sentence to input into a generating AI model is, "Please tell me how to check the user's posture while they are working at their desk and generate appropriate exercise reminders." This prompt sentence allows the AI ​​model to understand instructions regarding motion acquisition and analysis and generate specific output.

[0506] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0507] Step 1:

[0508] The server acquires data from various sensors and cameras installed in the living space. Inputs include video data, temperature, humidity, motion, and other sensor information. The server receives these inputs and organizes the data to prepare for subsequent analysis. Specifically, the server receives signals from various sensors and begins real-time monitoring.

[0509] Step 2:

[0510] The server analyzes the acquired data. Here, machine learning algorithms are used to analyze the user's posture and the state of the room, detecting anomalies and identified events. The input is the organized data collected in step 1, and the output is information on posture anomalies and the status of unused appliances. At this stage, the server feeds the collected data into an AI model to recognize, for example, that the user has been in the same posture for a long time.

[0511] Step 3:

[0512] The server determines specific actions based on the analysis results. Specifically, if an anomaly is detected, it generates an exercise reminder and shuts off power to unused appliances. The input for this stage is the analysis results from step 2, and the output includes notifications to the user and control signals. Based on this, the server performs conditional checks to determine the next action to be taken.

[0513] Step 4:

[0514] The terminal notifies the user of information received from the server. Specifically, the terminal displays and plays messages such as "Please perform stretches" on the screen or via audio. The input is the notification content generated in step 3, and the output is the actual notification to the user. The terminal follows the received command and prompts the user to take the appropriate action.

[0515] Step 5:

[0516] The server monitors user feedback and behavior to evaluate the system's effectiveness. Inputs include new user actions and environmental changes, while outputs are further data collected for the system's next action. Here, the system checks whether the user takes action in response to a notification, and accumulates that response as training data.

[0517] In this way, the entire process works in coordination, providing continuous support that improves user health, energy efficiency, and safety.

[0518] (Application Example 1)

[0519] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0520] In factories and workplaces, improving workers' posture and reducing fatigue are necessary to ensure efficient work while maintaining worker health. Furthermore, there is a need for systems that reduce unnecessary energy consumption of equipment and efficiently manage the location of objects within the work environment and implement security measures.

[0521] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0522] In this invention, the server includes means for monitoring the physical posture of workers in the workspace and generating and notifying information recommending exercises when an abnormal posture is detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the position of an object and notifying the worker of the position of the object upon request. This enables the maintenance of worker health and improvement of productivity, as well as the efficient use of energy and safe management of the workspace.

[0523] A "workspace" refers to a physical environment such as a factory or production facility, where workers perform their daily tasks.

[0524] A "worker" refers to a person who performs tasks within a workspace, and their health and work efficiency are important factors.

[0525] "Body posture" refers to the arrangement and angle of a worker's body, and is a factor that affects work efficiency and health.

[0526] "Equipment" refers to the machinery and equipment used within factories and facilities, and their operation affects energy consumption.

[0527] "Operating status" refers to information indicating the current state in which equipment or machinery is operating.

[0528] "Objects" refer to all concrete things present in the workspace, including resources and tools used in the work.

[0529] "Position" refers to the point or location that an object occupies within the working space.

[0530] "Abnormal behavior" refers to irregular or unplanned movements or actions in a work environment that may interfere with normal operations or safety.

[0531] "Information recommending exercise" refers to messages that include advice on specific exercises and movements that workers should perform to improve their posture and maintain their health.

[0532] "To shut down power" means to temporarily or permanently interrupt the operation of machinery or equipment in order to avoid wasting energy on the equipment.

[0533] To implement this invention, a system is constructed in which cameras and various sensors are installed at each work station in the workspace, and the acquired data is processed on a central server. The server captures the posture of the worker's body with the camera and analyzes the posture using the image processing library OpenCV. Furthermore, it analyzes the posture data using TensorFlow, and if an abnormal posture is detected, it notifies the terminal via the Pushover API with information prompting the worker to perform exercises.

[0534] The server acquires data from sensors attached to each piece of equipment to monitor its operating status and controls the power supply to automatically shut off when the equipment is not in use. This is done using the MQTT protocol to enable real-time data communication between the sensors and the server.

[0535] Furthermore, if an operator needs location information for an object, the system analyzes the video data of the object captured by the camera to identify its location and notify the operator. A MySQL database is used to efficiently record and retrieve location information for object management.

[0536] For example, if a worker continues to work in the same posture for a long period of time, their posture is monitored via camera footage, and a message is sent to their terminal saying, "Please take a break or do some exercises to improve your posture." Furthermore, if idle production equipment is running for an extended period, it is automatically shut down to ensure efficient energy use.

[0537] An example of a prompt for a generated AI model is: "Design an AI assistant that provides advice on optimal posture improvement to improve the health management of factory workers. It will analyze the current working posture and suggest the best posture."

[0538] This system makes it possible to create an efficient work environment and improve safety.

[0539] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0540] Step 1:

[0541] The server acquires video data from cameras installed within the workspace. The input is a video stream from the cameras, and the output is image data that can be analyzed in real time. This image data is processed using OpenCV to detect the posture of the worker's body.

[0542] Step 2:

[0543] The server uses TensorFlow to analyze posture information obtained from image data. The input is posture data processed with OpenCV, and the server determines whether or not an abnormal posture is present based on the analysis results. If an abnormal posture is detected, information prompting exercises is generated.

[0544] Step 3:

[0545] The terminal receives information prompting exercises from the server via the Pushover API and displays it to the worker as a message. The input is the notification information from the server, and the output is a message visually presented to the worker.

[0546] Step 4:

[0547] The server monitors the operating status of the equipment via sensors attached to it. The input is operational data transmitted from the sensors, and the output is status information indicating whether it is currently running or not. Based on this information, it generates a control signal to automatically shut off the power when not in use.

[0548] Step 5:

[0549] The server retrieves the recorded location of objects from a database and, upon request, identifies the location information of the object desired by the worker and notifies the terminal. Input is the object's name or ID, and output is information indicating the object's current location. Location information is managed using MySQL for efficient retrieval.

[0550] Step 6:

[0551] The server continuously monitors camera footage in the workspace and immediately sends an alarm to the terminal if any abnormal activity is detected. The input is real-time camera footage, and alarm information is generated after an image analysis process detects suspicious movements. The output is an alarm message sent to the worker or manager.

[0552] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0553] This invention is an integrated system that, in addition to providing support for user health management, energy conservation management, item location, organization, and intruder detection within a living space, utilizes an emotion engine to provide responses based on the user's emotional state. This system mainly consists of a server, terminals, an emotion engine, and a user, and operates autonomously in response to various situations within the living space.

[0554] The server acquires diverse data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors placed in the living space. In particular, the emotion engine analyzes this data to identify the user's emotional state. Based on this data, if the server determines that the user's emotions are stress-based, it provides suggestions and measures to reduce stress via the terminal. In this process, it can automatically play relaxing music and adjust the lighting.

[0555] Furthermore, the server enhances existing health management functions, taking emotional data into account when monitoring posture and generating exercise reminders. For example, if a user has poor posture and is under stress, it can suggest exercises that prioritize relaxation more than usual.

[0556] Furthermore, the server optimizes the operation of home appliances by combining them with emotional data. When the user is in a relaxed emotional state, comfort can be prioritized within the limits of maintaining energy efficiency.

[0557] As a concrete example, when a user is tired and stressed from long hours of desk work, the server detects their emotions from camera and microphone data. At this time, the server adjusts the lighting to a calming tone and plays relaxing music through the device. It also sends a reminder to encourage exercise, notifying the user that "light stretching would be effective." This allows the user to reduce stress and maintain a healthy state.

[0558] Thus, the present invention is designed as a multi-functional integrated system that provides comprehensive life support, including the emotional aspects of the user.

[0559] The following describes the processing flow.

[0560] Step 1:

[0561] The server acquires audio and video data from cameras and microphones installed in the living space, capturing the user's posture, facial expressions, and voice tone.

[0562] Step 2:

[0563] The server sends the data acquired in real time to the emotion engine, which analyzes the user's emotional state. The analysis identifies the user's emotional state, such as joy, anger, sadness, or stress.

[0564] Step 3:

[0565] The server determines what needs to be addressed based on the user's emotional state. For example, if it determines that the user is stressed, it will consider appropriate actions to promote relaxation.

[0566] Step 4:

[0567] The server sends the selected action to the device. Based on the received instructions, the device plays relaxing music or adjusts the room lighting to a calming color tone.

[0568] Step 5:

[0569] The server analyzes posture data in combination with emotional data to assess the user's health status. If an inappropriate posture and stressful state persist, it immediately generates an exercise reminder.

[0570] Step 6:

[0571] The device notifies the user of exercise reminders and simultaneously offers advice such as, "Light stretching can help reduce stress."

[0572] Step 7:

[0573] The user performs stretches according to the device's suggestions, aiming to improve their emotional state. During this process, the emotion engine re-evaluates the changed emotional state.

[0574] Step 8:

[0575] The server adjusts the operating status of home appliances based on the results of emotion analysis. For example, when the user is relaxed, it sets the air conditioner to a comfortable temperature while also considering energy efficiency.

[0576] Step 9:

[0577] The server continuously monitors the user's state and adjusts the entire system to ensure user comfort, regardless of any changes in their emotions.

[0578] (Example 2)

[0579] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0580] The challenge lies in appropriately understanding the emotional state of users within their living spaces and providing corresponding lifestyle support to improve health management, energy conservation, and comfort. Furthermore, the aim is to improve the quality of life by automatically adjusting the environment without the user's conscious effort.

[0581] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0582] This invention includes a server that includes means for detecting the user's posture in the living space and generating and notifying exercise reminders that also take emotional data into consideration; means for optimizing the operation of home appliances based on the emotional state, prioritizing comfort while maintaining energy efficiency; and means for analyzing data such as voice tone and facial expressions to identify the user's emotions and automatically implementing measures to reduce stress. This enables detailed lifestyle support tailored to the user's emotional state.

[0583] "Living space" refers to the interior and surrounding areas used by residents for their daily lives, and primarily includes the interior of a home.

[0584] "User" refers to an individual or group of individuals who use the system, including residents within the living space covered by this system.

[0585] "Emotional data" refers to data that indicates a user's emotional state, analyzed based on information such as the user's facial expressions, posture, and voice tone.

[0586] A "exercise reminder" refers to a system function that sends information, including notifications and suggestions, to users to encourage them to exercise appropriately.

[0587] "Measures" refer to specific actions and environmental adjustments provided in accordance with the user's emotional state and health condition, with the aim of reducing stress and improving comfort.

[0588] "Energy efficiency" refers to the balance between the power consumed by home appliances and other electrical equipment and the effectiveness of their use, representing the degree to which unnecessary power consumption is minimized and equipment is operated effectively.

[0589] "Health management" is a general term for providing information and lifestyle support to maintain and improve the physical health of users.

[0590] In embodiments of this invention, a system is provided that primarily aims to maintain comfort and health in living spaces, utilizing a server, terminal, and emotion engine. The server uses cameras and sensors installed in the living space to collect diverse data in real time, such as the user's posture, movements, facial expressions, and voice tone. Hardware such as optical cameras, infrared sensors, and voice recognition devices are used for this purpose.

[0591] The emotion engine analyzes collected data to identify the user's emotional state. Machine learning algorithms and natural language processing techniques are applied to the data analysis. Based on the user's emotional data identified by the emotion engine, the server selects measures such as suggesting relaxation-oriented music or adjusting lighting, and executes these via the user's device. This entire process enables automated environment optimization that takes emotional state into account.

[0592] For example, if the emotion engine determines that a user is experiencing stress due to prolonged desk work, the server will send a command to the terminal to play relaxing music. It can also support the reduction of user stress by instructing the lighting system to adjust to a warmer color tone. In this process, the server inputs a prompt message such as "Based on the user's emotional state, please suggest measures to reduce stress" into the AI ​​model, enabling it to propose the most appropriate measures.

[0593] Furthermore, to support user health management, the server generates exercise reminders based on the user's posture and movements and notifies the user. For example, if an inappropriate posture and stress level are detected simultaneously, the server will provide a reminder via the device saying, "We recommend you do some light stretching," thereby promoting the user's health maintenance.

[0594] Ultimately, this system aims to provide customized lifestyle support tailored to each user's individual needs and feelings, pursuing a better quality of life in their living space.

[0595] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0596] Step 1:

[0597] The server collects data on the user's posture, movements, facial expressions, and voice tone using cameras and sensors placed within the living space. It receives video data, audio data, and sensor data as input, processes them in real time, and outputs raw data that indicates the user's physical state. This raw data includes the user's sitting posture, changes in facial expressions, and voice tone.

[0598] Step 2:

[0599] The server inputs raw data into the emotion engine and analyzes the user's emotional state. Here, machine learning algorithms are applied to perform data analysis. The raw data obtained in the previous step is used as input, and the output includes labels indicating the user's emotional state and a numerical emotion score. Specifically, it determines the user's stress level and relaxation level based on features such as voice pitch and facial expressions.

[0600] Step 3:

[0601] The server determines appropriate measures based on emotional state data obtained from the emotion engine. It takes the user's emotional state score as input and generates specific measure suggestions as output. This step includes specific instructions such as "play relaxing music" or "adjust lighting to warmer colors." The generating AI model is then prompted with a message such as "Please suggest measures to reduce stress based on the user's emotional state" to obtain suggested measures.

[0602] Step 4:

[0603] The device receives instructions from the server and performs actual environmental adjustments based on those instructions. Its input is policy instructions from the server, and its output is changes to the environmental settings within the living space. Specific actions include playing relaxation music through the speaker or changing the color tone of smart lighting.

[0604] Step 5:

[0605] The server evaluates the user's posture data while considering emotional state data as part of health management. It takes raw data from Step 1 and emotional data from Step 2 as input, and generates health status analysis results and appropriate exercise reminders as output. Specifically, if an inappropriate posture persists, a notification such as "We recommend doing some light stretching" is sent from the device to the user.

[0606] Step 6:

[0607] The server optimizes the operation of home appliances based on emotional data. Inputs include emotional scores and the current operating status of the appliances, and the output generates optimal operating instructions that consider energy efficiency. For example, if a user is relaxed, the server will maintain comfort without unnecessarily lowering the air conditioner's temperature setting.

[0608] (Application Example 2)

[0609] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0610] In modern living spaces, the need for user health management and energy conservation measures is increasing. Furthermore, efficient location tracking and organization of items, as well as detection of suspicious individuals, are also crucial issues. Additionally, there is a demand for immediate responses tailored to users' emotional states and improved customer experiences in physical stores. However, a system that comprehensively addresses all of these needs does not yet exist.

[0611] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0612] In this invention, the server includes means for monitoring the user's posture in the living space and generating and notifying an exercise reminder when an abnormal posture is detected; means for monitoring the operating status of home appliances and automatically turning off the power when not in use; means for recording the location of items and notifying the user of the location of said items upon request; means for generating and notifying tidying-up advice based on the state of the living space; means for detecting suspicious persons using a camera and generating an alarm when detected; means for analyzing the user's emotional state using emotion analysis technology and optimizing the living space environment settings; and means for analyzing the customer's emotional state and providing purchasing support information and services. This integrates diverse functions, improves the user's quality of life, and enables the optimization of the customer experience in physical stores.

[0613] "Living space" refers to the space in which an individual or family conducts their daily life, and includes all rooms and facilities within the home.

[0614] "User" refers to an individual who operates or experiences a living space or system.

[0615] A "posture monitor" is a device or system used to detect and analyze the positioning and movement of a user's body.

[0616] An "exercise reminder" is a feature that generates notifications or alerts prompting users to perform specific exercises.

[0617] "Home appliances" refer to electrical equipment used in the home, including washing machines, refrigerators, and air conditioners.

[0618] "Item location recording" is the process of tracking where a specific item is located and maintaining that information.

[0619] "Organization and tidying advice" means providing recommendations for the efficient arrangement and storage of items.

[0620] A "shooting device" refers to equipment used to capture still images or videos, such as cameras and video cameras.

[0621] "Suspicious person detection" is the process of identifying individuals who are engaging in unusual or suspicious activities within a living space.

[0622] "Emotional analysis technology" is a technology that identifies a user's emotional state from their facial expressions, voice, and behavior.

[0623] "Optimizing environmental settings" refers to adjusting elements such as lighting, sound, and temperature in a living space to suit the user's comfort.

[0624] "Purchase support information" refers to information and suggestions that help customers make purchasing decisions.

[0625] "Service provision" refers to the act of providing value-added services that meet customer needs.

[0626] This invention is a system that monitors the health and emotions of users in living spaces and retail spaces to create a comfortable living environment.

[0627] The server acquires data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors within the living space. Based on this data, the server monitors the user's health status and generates and notifies them of exercise reminders if their posture is inappropriate.

[0628] Furthermore, the server monitors the operating status of home appliances and automatically turns them off when not in use to minimize energy consumption. It also provides features to help with organization, such as recording the location of items and notifying users of their location upon request.

[0629] By using emotion analysis technology, the server identifies the user's emotional state and automatically optimizes the environment settings to consider relaxation and energy efficiency. This allows users to live comfortably without feeling stressed. For example, if a user feels fatigued after working at a desk for a long time, the server automatically adjusts the lighting and plays relaxing music. It also sends reminders to encourage exercise to enhance the relaxation effect.

[0630] Furthermore, in physical stores, it is possible to analyze customers' facial expressions and voice tone to provide optimal purchasing support information and services. This process is based on emotion analysis technology and generative AI models (e.g., OpenAI's GPT-3).

[0631] For example, if a customer is walking around the store and appears tense, the background music will be switched to something calming, and product suggestions will be displayed on their smart device.

[0632] An example of a prompt message is as follows:

[0633] "We will provide the customer's facial expressions and tone of voice in the following format. Based on this data, please tell us what emotions the customer is currently experiencing."

[0634] Facial expression data: No smile, frowning.

[0635] Voice tone: Slightly fast-paced and high-pitched.

[0636] Based on this, please propose appropriate purchasing support services.

[0637] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0638] Step 1:

[0639] The server acquires data from cameras and sensors installed in the living space, including the user's posture, movement, facial expressions, and voice tone. Input data includes real-time video streams and audio input. This data is stored in temporary storage and used in the next processing step.

[0640] Step 2:

[0641] The server uses the acquired data to execute emotion analysis techniques and motion analysis algorithms. The input includes posture, facial expressions, and voice tone collected in step 1, which are then analyzed to determine the user's current emotional and postural state. A generative AI model is used here, utilizing prompt sentences to aid in the analysis and identify the user's emotional state. The output is an analysis result including emotional and postural states.

[0642] Step 3:

[0643] The server generates and notifies users of exercise reminders via the user interface based on the results of emotion and posture analysis. It also suggests environmental adjustments to optimize energy efficiency and improve the customer experience. Specific adjustments include lighting and music settings. Users can adjust the color temperature and brightness of the lighting and play relaxing music via their device. Outputs include exercise reminders and environmental adjustment suggestions.

[0644] Step 4:

[0645] The device, based on instructions from the server, notifies the user of generated reminders and adjusts environmental settings. For example, reminders are displayed as notifications on the smartphone, and music and lighting adjustments are performed directly through smart home appliances. The output is the reminder notification and the adjusted environment.

[0646] Step 5:

[0647] The user receives notifications and performs recommended exercise or relaxation activities. Based on the information obtained from the device, the user adjusts their behavior to maintain a comfortable state. The output is the specific action the user takes.

[0648] Step 6:

[0649] The server continuously runs the same process, dynamically adjusting in response to changes in the user's state and environment. This helps users maintain a comfortable environment for extended periods. The output consists of continuously updated analysis results and recommended actions.

[0650] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0651] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0652] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0653] [Fourth Embodiment]

[0654] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0655] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0656] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0657] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0658] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0659] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0660] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0661] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0662] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0663] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0664] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0665] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0666] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0667] This invention is an integrated system that improves the quality of life in living spaces, and is designed to provide multiple functions related to health management, energy saving, item location, organization and tidiness advice, and crime prevention. This system is primarily configured to exchange information between a server, terminals, and users, and to provide multifaceted support for the user's life.

[0668] The server acquires information from cameras and various sensors installed in the living space and analyzes this data in real time. Specifically, the server monitors the user's posture and, if it detects prolonged periods of inappropriate posture, generates a reminder to encourage exercise. This helps users maintain healthy lifestyle habits.

[0669] Regarding energy management, the server constantly monitors the operating status of each appliance and reduces unnecessary power consumption by automatically turning off unused appliances. It also uses cameras to check the room's condition and the location of items, locating lost items as needed and providing this information to the user via a terminal. This feature allows users to find items efficiently.

[0670] Furthermore, the tidying advice function analyzes the current state of the living space via a server and provides specific tidying advice to the user through their device. This allows users to create a more comfortable living environment.

[0671] Furthermore, as a security feature, the server continuously monitors camera footage and immediately sends an alert to the user via their device if a suspicious person is detected. This allows users to respond quickly and ensure their safety.

[0672] For example, if a user is working at their desk for a long time, the server will recognize their posture and display a reminder on their device saying, "Please do 5 minutes of stretching." Also, if a user leaves their bedroom, the server will detect their movement and automatically turn off unused lights and air conditioning. Furthermore, if a user is looking for their keys, the server will notify their device of the last observed location of the keys.

[0673] Thus, the present invention aims to make users' lives more comfortable and safer by providing multiple functions in an integrated manner.

[0674] The following describes the processing flow.

[0675] Step 1:

[0676] The server begins receiving real-time video and data from cameras and sensors installed in the living space. This includes posture data and the operating status of home appliances.

[0677] Step 2:

[0678] The server analyzes the received data and determines the user's posture. If the posture is inappropriate, the server immediately generates an exercise reminder.

[0679] Step 3:

[0680] The server sends the generated exercise reminder to the device. The device notifies the user visually or audibly.

[0681] Step 4:

[0682] The server monitors the operating status of home appliances and generates an automatic shut-off command if any appliances are not in use.

[0683] Step 5:

[0684] The terminal receives commands from the server and saves energy by turning off the power to unused home appliances.

[0685] Step 6:

[0686] The server receives an item location search request from the user and determines its location based on camera data.

[0687] Step 7:

[0688] The server transmits the location information of the identified item to the terminal, and the terminal informs the user of the item's location.

[0689] Step 8:

[0690] The server evaluates the tidiness of the living space and generates tidying-up advice as needed.

[0691] Step 9:

[0692] The terminal provides the user with advice from the server and encourages them to organize and tidy up.

[0693] Step 10:

[0694] The server constantly monitors camera footage and, if it detects a suspicious person, immediately generates an alarm and notifies the user via their device.

[0695] Step 11:

[0696] Users take appropriate safety measures based on alerts from their devices.

[0697] (Example 1)

[0698] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0699] In modern living environments, users often demand simultaneous improvements in health management, energy conservation, safety, and efficient item management. However, currently, there is no integrated system that can meet these demands, and users rely on devices with individual functions. As a result, users are burdened with managing multiple systems, making it difficult to ensure true comfort and safety. Therefore, there is a need for an integrated system that can solve these problems simultaneously.

[0700] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0701] In this invention, the server includes means for monitoring the physical condition of users in a living space and generating and notifying information to encourage exercise when abnormal physical movements are detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the location of items and notifying the user of the location of said items upon request. This enables users to maintain their health, efficiently conserve energy, and manage items safely and effectively.

[0702] "Living space" refers to a place where life and activities take place, and primarily refers to the interior space of a house, apartment, or similar building.

[0703] "Users" refer to individuals who use this system and seek to improve their health management or quality of life.

[0704] "Physical condition" refers to information about the user's posture and movements, and is considered an important element in health management.

[0705] "Equipment" refers to electrical appliances and home appliances used in living spaces, and is subject to management.

[0706] "Operating status" refers to the state in which the equipment is functioning, including whether it is on or off.

[0707] "Items" refer to objects or possessions that exist within a living space and are subject to management or specific designation.

[0708] "Image device" refers to a device that acquires visual information, such as a camera or sensor.

[0709] An "abnormality" refers to a situation that deviates from the normal state, and is something that the system will detect.

[0710] The present invention will now be described in terms of embodiments. This system aims to improve the quality of life in living spaces by integrating health management, energy saving, item management, and security functions. The server collects and analyzes data from various sensor devices installed in the user's living space. The hardware includes cameras, temperature sensors, motion sensors, etc., which provide information to the server via Wi-Fi or Bluetooth.

[0711] The server uses machine learning algorithms to process data in real time and analyze the user's physical condition and the operating status of the equipment. If appropriate processing is performed, for example, if abnormal physical movement is detected, it will generate a notification prompting exercise. Also, if the equipment is found to be unused, it will automatically shut off the power.

[0712] The device receives information from the server and provides appropriate notifications to the user. This is done using smartphone applications and voice assistants.

[0713] For example, if a user is doing desk work for a long time, the server analyzes the video data from the camera and determines that the user's posture is hunched over. Based on this, it sends a notification to the device prompting the user to stretch for 5 minutes.

[0714] Furthermore, when a user leaves the room, the server uses data from motion sensors to monitor the operating status of devices within the living space and automatically turns off any unnecessary devices. Additionally, if the user is searching for their keys, the server consults camera footage to locate the last observed key position and transmits that information to the user's device.

[0715] An example of a prompt sentence to input into a generating AI model is, "Please tell me how to check the user's posture while they are working at their desk and generate appropriate exercise reminders." This prompt sentence allows the AI ​​model to understand instructions regarding motion acquisition and analysis and generate specific output.

[0716] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0717] Step 1:

[0718] The server acquires data from various sensors and cameras installed in the living space. Inputs include video data, temperature, humidity, motion, and other sensor information. The server receives these inputs and organizes the data to prepare for subsequent analysis. Specifically, the server receives signals from various sensors and begins real-time monitoring.

[0719] Step 2:

[0720] The server analyzes the acquired data. Here, machine learning algorithms are used to analyze the user's posture and the state of the room, detecting anomalies and identified events. The input is the organized data collected in step 1, and the output is information on posture anomalies and the status of unused appliances. At this stage, the server feeds the collected data into an AI model to recognize, for example, that the user has been in the same posture for a long time.

[0721] Step 3:

[0722] The server determines specific actions based on the analysis results. Specifically, if an anomaly is detected, it generates an exercise reminder and shuts off power to unused appliances. The input for this stage is the analysis results from step 2, and the output includes notifications to the user and control signals. Based on this, the server performs conditional checks to determine the next action to be taken.

[0723] Step 4:

[0724] The terminal notifies the user of information received from the server. Specifically, the terminal displays and plays messages such as "Please perform stretches" on the screen or via audio. The input is the notification content generated in step 3, and the output is the actual notification to the user. The terminal follows the received command and prompts the user to take the appropriate action.

[0725] Step 5:

[0726] The server monitors user feedback and behavior to evaluate the system's effectiveness. Inputs include new user actions and environmental changes, while outputs are further data collected for the system's next action. Here, the system checks whether the user takes action in response to a notification, and accumulates that response as training data.

[0727] In this way, the entire process works in coordination, providing continuous support that improves user health, energy efficiency, and safety.

[0728] (Application Example 1)

[0729] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0730] In factories and workplaces, improving workers' posture and reducing fatigue are necessary to ensure efficient work while maintaining worker health. Furthermore, there is a need for systems that reduce unnecessary energy consumption of equipment and efficiently manage the location of objects within the work environment and implement security measures.

[0731] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0732] In this invention, the server includes means for monitoring the physical posture of workers in the workspace and generating and notifying information recommending exercises when an abnormal posture is detected; means for monitoring the operating status of equipment and automatically shutting off power when not in use; and means for recording the position of an object and notifying the worker of the position of the object upon request. This enables the maintenance of worker health and improvement of productivity, as well as the efficient use of energy and safe management of the workspace.

[0733] A "workspace" refers to a physical environment such as a factory or production facility, where workers perform their daily tasks.

[0734] A "worker" refers to a person who performs tasks within a workspace, and their health and work efficiency are important factors.

[0735] "Body posture" refers to the arrangement and angle of a worker's body, and is a factor that affects work efficiency and health.

[0736] "Equipment" refers to the machinery and equipment used within factories and facilities, and their operation affects energy consumption.

[0737] "Operating status" refers to information indicating the current state in which equipment or machinery is operating.

[0738] "Objects" refer to all concrete things present in the workspace, including resources and tools used in the work.

[0739] "Position" refers to the point or location that an object occupies within the working space.

[0740] "Abnormal behavior" refers to irregular or unplanned movements or actions in a work environment that may interfere with normal operations or safety.

[0741] "Information recommending exercise" refers to messages that include advice on specific exercises and movements that workers should perform to improve their posture and maintain their health.

[0742] "To shut down power" means to temporarily or permanently interrupt the operation of machinery or equipment in order to avoid wasting energy on the equipment.

[0743] To implement this invention, a system is constructed in which cameras and various sensors are installed at each work station in the workspace, and the acquired data is processed on a central server. The server captures the posture of the worker's body with the camera and analyzes the posture using the image processing library OpenCV. Furthermore, it analyzes the posture data using TensorFlow, and if an abnormal posture is detected, it notifies the terminal via the Pushover API with information prompting the worker to perform exercises.

[0744] The server acquires data from sensors attached to each piece of equipment to monitor its operating status and controls the power supply to automatically shut off when the equipment is not in use. This is done using the MQTT protocol to enable real-time data communication between the sensors and the server.

[0745] Furthermore, if an operator needs location information for an object, the system analyzes the video data of the object captured by the camera to identify its location and notify the operator. A MySQL database is used to efficiently record and retrieve location information for object management.

[0746] For example, if a worker continues to work in the same posture for a long period of time, their posture is monitored via camera footage, and a message is sent to their terminal saying, "Please take a break or do some exercises to improve your posture." Furthermore, if idle production equipment is running for an extended period, it is automatically shut down to ensure efficient energy use.

[0747] An example of a prompt for a generated AI model is: "Design an AI assistant that provides advice on optimal posture improvement to improve the health management of factory workers. It will analyze the current working posture and suggest the best posture."

[0748] This system makes it possible to create an efficient work environment and improve safety.

[0749] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0750] Step 1:

[0751] The server acquires video data from cameras installed within the workspace. The input is a video stream from the cameras, and the output is image data that can be analyzed in real time. This image data is processed using OpenCV to detect the posture of the worker's body.

[0752] Step 2:

[0753] The server uses TensorFlow to analyze posture information obtained from image data. The input is posture data processed with OpenCV, and the server determines whether or not an abnormal posture is present based on the analysis results. If an abnormal posture is detected, information prompting exercises is generated.

[0754] Step 3:

[0755] The terminal receives information prompting exercises from the server via the Pushover API and displays it to the worker as a message. The input is the notification information from the server, and the output is a message visually presented to the worker.

[0756] Step 4:

[0757] The server monitors the operating status of the equipment via sensors attached to it. The input is operational data transmitted from the sensors, and the output is status information indicating whether it is currently running or not. Based on this information, it generates a control signal to automatically shut off the power when not in use.

[0758] Step 5:

[0759] The server retrieves the recorded location of objects from a database and, upon request, identifies the location information of the object desired by the worker and notifies the terminal. Input is the object's name or ID, and output is information indicating the object's current location. Location information is managed using MySQL for efficient retrieval.

[0760] Step 6:

[0761] The server continuously monitors camera footage in the workspace and immediately sends an alarm to the terminal if any abnormal activity is detected. The input is real-time camera footage, and alarm information is generated after an image analysis process detects suspicious movements. The output is an alarm message sent to the worker or manager.

[0762] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0763] This invention is an integrated system that, in addition to providing support for user health management, energy conservation management, item location, organization, and intruder detection within a living space, utilizes an emotion engine to provide responses based on the user's emotional state. This system mainly consists of a server, terminals, an emotion engine, and a user, and operates autonomously in response to various situations within the living space.

[0764] The server acquires diverse data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors placed in the living space. In particular, the emotion engine analyzes this data to identify the user's emotional state. Based on this data, if the server determines that the user's emotions are stress-based, it provides suggestions and measures to reduce stress via the terminal. In this process, it can automatically play relaxing music and adjust the lighting.

[0765] Furthermore, the server enhances existing health management functions, taking emotional data into account when monitoring posture and generating exercise reminders. For example, if a user has poor posture and is under stress, it can suggest exercises that prioritize relaxation more than usual.

[0766] Furthermore, the server optimizes the operation of home appliances by combining them with emotional data. When the user is in a relaxed emotional state, comfort can be prioritized within the limits of maintaining energy efficiency.

[0767] As a concrete example, when a user is tired and stressed from long hours of desk work, the server detects their emotions from camera and microphone data. At this time, the server adjusts the lighting to a calming tone and plays relaxing music through the device. It also sends a reminder to encourage exercise, notifying the user that "light stretching would be effective." This allows the user to reduce stress and maintain a healthy state.

[0768] Thus, the present invention is designed as a multi-functional integrated system that provides comprehensive life support, including the emotional aspects of the user.

[0769] The following describes the processing flow.

[0770] Step 1:

[0771] The server acquires audio and video data from cameras and microphones installed in the living space, capturing the user's posture, facial expressions, and voice tone.

[0772] Step 2:

[0773] The server sends the data acquired in real time to the emotion engine, which analyzes the user's emotional state. The analysis identifies the user's emotional state, such as joy, anger, sadness, or stress.

[0774] Step 3:

[0775] The server determines what needs to be addressed based on the user's emotional state. For example, if it determines that the user is stressed, it will consider appropriate actions to promote relaxation.

[0776] Step 4:

[0777] The server sends the selected action to the device. Based on the received instructions, the device plays relaxing music or adjusts the room lighting to a calming color tone.

[0778] Step 5:

[0779] The server analyzes posture data in combination with emotional data to assess the user's health status. If an inappropriate posture and stressful state persist, it immediately generates an exercise reminder.

[0780] Step 6:

[0781] The device notifies the user of exercise reminders and simultaneously offers advice such as, "Light stretching can help reduce stress."

[0782] Step 7:

[0783] The user performs stretches according to the device's suggestions, aiming to improve their emotional state. During this process, the emotion engine re-evaluates the changed emotional state.

[0784] Step 8:

[0785] The server adjusts the operating status of home appliances based on the results of emotion analysis. For example, when the user is relaxed, it sets the air conditioner to a comfortable temperature while also considering energy efficiency.

[0786] Step 9:

[0787] The server continuously monitors the user's state and adjusts the entire system to ensure user comfort, regardless of any changes in their emotions.

[0788] (Example 2)

[0789] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0790] The challenge lies in appropriately understanding the emotional state of users within their living spaces and providing corresponding lifestyle support to improve health management, energy conservation, and comfort. Furthermore, the aim is to improve the quality of life by automatically adjusting the environment without the user's conscious effort.

[0791] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0792] This invention includes a server that includes means for detecting the user's posture in the living space and generating and notifying exercise reminders that also take emotional data into consideration; means for optimizing the operation of home appliances based on the emotional state, prioritizing comfort while maintaining energy efficiency; and means for analyzing data such as voice tone and facial expressions to identify the user's emotions and automatically implementing measures to reduce stress. This enables detailed lifestyle support tailored to the user's emotional state.

[0793] "Living space" refers to the interior and surrounding areas used by residents for their daily lives, and primarily includes the interior of a home.

[0794] "User" refers to an individual or group of individuals who use the system, including residents within the living space covered by this system.

[0795] "Emotional data" refers to data that indicates a user's emotional state, analyzed based on information such as the user's facial expressions, posture, and voice tone.

[0796] A "exercise reminder" refers to a system function that sends information, including notifications and suggestions, to users to encourage them to exercise appropriately.

[0797] "Measures" refer to specific actions and environmental adjustments provided in accordance with the user's emotional state and health condition, with the aim of reducing stress and improving comfort.

[0798] "Energy efficiency" refers to the balance between the power consumed by home appliances and other electrical equipment and the effectiveness of their use, representing the degree to which unnecessary power consumption is minimized and equipment is operated effectively.

[0799] "Health management" is a general term for providing information and lifestyle support to maintain and improve the physical health of users.

[0800] In embodiments of this invention, a system is provided that primarily aims to maintain comfort and health in living spaces, utilizing a server, terminal, and emotion engine. The server uses cameras and sensors installed in the living space to collect diverse data in real time, such as the user's posture, movements, facial expressions, and voice tone. Hardware such as optical cameras, infrared sensors, and voice recognition devices are used for this purpose.

[0801] The emotion engine analyzes collected data to identify the user's emotional state. Machine learning algorithms and natural language processing techniques are applied to the data analysis. Based on the user's emotional data identified by the emotion engine, the server selects measures such as suggesting relaxation-oriented music or adjusting lighting, and executes these via the user's device. This entire process enables automated environment optimization that takes emotional state into account.

[0802] For example, if the emotion engine determines that a user is experiencing stress due to prolonged desk work, the server will send a command to the terminal to play relaxing music. It can also support the reduction of user stress by instructing the lighting system to adjust to a warmer color tone. In this process, the server inputs a prompt message such as "Based on the user's emotional state, please suggest measures to reduce stress" into the AI ​​model, enabling it to propose the most appropriate measures.

[0803] Furthermore, to support user health management, the server generates exercise reminders based on the user's posture and movements and notifies the user. For example, if an inappropriate posture and stress level are detected simultaneously, the server will provide a reminder via the device saying, "We recommend you do some light stretching," thereby promoting the user's health maintenance.

[0804] Ultimately, this system aims to provide customized lifestyle support tailored to each user's individual needs and feelings, pursuing a better quality of life in their living space.

[0805] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0806] Step 1:

[0807] The server collects data on the user's posture, movements, facial expressions, and voice tone using cameras and sensors placed within the living space. It receives video data, audio data, and sensor data as input, processes them in real time, and outputs raw data that indicates the user's physical state. This raw data includes the user's sitting posture, changes in facial expressions, and voice tone.

[0808] Step 2:

[0809] The server inputs raw data into the emotion engine and analyzes the user's emotional state. Here, machine learning algorithms are applied to perform data analysis. The raw data obtained in the previous step is used as input, and the output includes labels indicating the user's emotional state and a numerical emotion score. Specifically, it determines the user's stress level and relaxation level based on features such as voice pitch and facial expressions.

[0810] Step 3:

[0811] The server determines appropriate measures based on emotional state data obtained from the emotion engine. It takes the user's emotional state score as input and generates specific measure suggestions as output. This step includes specific instructions such as "play relaxing music" or "adjust lighting to warmer colors." The generating AI model is then prompted with a message such as "Please suggest measures to reduce stress based on the user's emotional state" to obtain suggested measures.

[0812] Step 4:

[0813] The device receives instructions from the server and performs actual environmental adjustments based on those instructions. Its input is policy instructions from the server, and its output is changes to the environmental settings within the living space. Specific actions include playing relaxation music through the speaker or changing the color tone of smart lighting.

[0814] Step 5:

[0815] The server evaluates the user's posture data while considering emotional state data as part of health management. It takes raw data from Step 1 and emotional data from Step 2 as input, and generates health status analysis results and appropriate exercise reminders as output. Specifically, if an inappropriate posture persists, a notification such as "We recommend doing some light stretching" is sent from the device to the user.

[0816] Step 6:

[0817] The server optimizes the operation of home appliances based on emotional data. Inputs include emotional scores and the current operating status of the appliances, and the output generates optimal operating instructions that consider energy efficiency. For example, if a user is relaxed, the server will maintain comfort without unnecessarily lowering the air conditioner's temperature setting.

[0818] (Application Example 2)

[0819] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0820] In modern living spaces, the need for user health management and energy conservation measures is increasing. Furthermore, efficient location tracking and organization of items, as well as detection of suspicious individuals, are also crucial issues. Additionally, there is a demand for immediate responses tailored to users' emotional states and improved customer experiences in physical stores. However, a system that comprehensively addresses all of these needs does not yet exist.

[0821] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0822] In this invention, the server includes means for monitoring the user's posture in the living space and generating and notifying an exercise reminder when an abnormal posture is detected; means for monitoring the operating status of home appliances and automatically turning off the power when not in use; means for recording the location of items and notifying the user of the location of said items upon request; means for generating and notifying tidying-up advice based on the state of the living space; means for detecting suspicious persons using a camera and generating an alarm when detected; means for analyzing the user's emotional state using emotion analysis technology and optimizing the living space environment settings; and means for analyzing the customer's emotional state and providing purchasing support information and services. This integrates diverse functions, improves the user's quality of life, and enables the optimization of the customer experience in physical stores.

[0823] "Living space" refers to the space in which an individual or family conducts their daily life, and includes all rooms and facilities within the home.

[0824] "User" refers to an individual who operates or experiences a living space or system.

[0825] A "posture monitor" is a device or system used to detect and analyze the positioning and movement of a user's body.

[0826] An "exercise reminder" is a feature that generates notifications or alerts prompting users to perform specific exercises.

[0827] "Home appliances" refer to electrical equipment used in the home, including washing machines, refrigerators, and air conditioners.

[0828] "Item location recording" is the process of tracking where a specific item is located and maintaining that information.

[0829] "Organization and tidying advice" means providing recommendations for the efficient arrangement and storage of items.

[0830] A "shooting device" refers to equipment used to capture still images or videos, such as cameras and video cameras.

[0831] "Suspicious person detection" is the process of identifying individuals who are engaging in unusual or suspicious activities within a living space.

[0832] "Emotional analysis technology" is a technology that identifies a user's emotional state from their facial expressions, voice, and behavior.

[0833] "Optimizing environmental settings" refers to adjusting elements such as lighting, sound, and temperature in a living space to suit the user's comfort.

[0834] "Purchase support information" refers to information and suggestions that help customers make purchasing decisions.

[0835] "Service provision" refers to the act of providing value-added services that meet customer needs.

[0836] This invention is a system that monitors the health and emotions of users in living spaces and retail spaces to create a comfortable living environment.

[0837] The server acquires data, including the user's posture, movements, facial expressions, and voice tone, through cameras and sensors within the living space. Based on this data, the server monitors the user's health status and generates and notifies them of exercise reminders if their posture is inappropriate.

[0838] Furthermore, the server monitors the operating status of home appliances and automatically turns them off when not in use to minimize energy consumption. It also provides features to help with organization, such as recording the location of items and notifying users of their location upon request.

[0839] By using emotion analysis technology, the server identifies the user's emotional state and automatically optimizes the environment settings to consider relaxation and energy efficiency. This allows users to live comfortably without feeling stressed. For example, if a user feels fatigued after working at a desk for a long time, the server automatically adjusts the lighting and plays relaxing music. It also sends reminders to encourage exercise to enhance the relaxation effect.

[0840] Furthermore, in physical stores, it is possible to analyze customers' facial expressions and voice tone to provide optimal purchasing support information and services. This process is based on emotion analysis technology and generative AI models (e.g., OpenAI's GPT-3).

[0841] For example, if a customer is walking around the store and appears tense, the background music will be switched to something calming, and product suggestions will be displayed on their smart device.

[0842] An example of a prompt message is as follows:

[0843] "We will provide the customer's facial expressions and tone of voice in the following format. Based on this data, please tell us what emotions the customer is currently experiencing."

[0844] Facial expression data: No smile, frowning.

[0845] Voice tone: Slightly fast-paced and high-pitched.

[0846] Based on this, please propose appropriate purchasing support services.

[0847] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0848] Step 1:

[0849] The server acquires data from cameras and sensors installed in the living space, including the user's posture, movement, facial expressions, and voice tone. Input data includes real-time video streams and audio input. This data is stored in temporary storage and used in the next processing step.

[0850] Step 2:

[0851] The server uses the acquired data to execute emotion analysis techniques and motion analysis algorithms. The input includes posture, facial expressions, and voice tone collected in step 1, which are then analyzed to determine the user's current emotional and postural state. A generative AI model is used here, utilizing prompt sentences to aid in the analysis and identify the user's emotional state. The output is an analysis result including emotional and postural states.

[0852] Step 3:

[0853] The server generates and notifies users of exercise reminders via the user interface based on the results of emotion and posture analysis. It also suggests environmental adjustments to optimize energy efficiency and improve the customer experience. Specific adjustments include lighting and music settings. Users can adjust the color temperature and brightness of the lighting and play relaxing music via their device. Outputs include exercise reminders and environmental adjustment suggestions.

[0854] Step 4:

[0855] The device, based on instructions from the server, notifies the user of generated reminders and adjusts environmental settings. For example, reminders are displayed as notifications on the smartphone, and music and lighting adjustments are performed directly through smart home appliances. The output is the reminder notification and the adjusted environment.

[0856] Step 5:

[0857] The user receives notifications and performs recommended exercise or relaxation activities. Based on the information obtained from the device, the user adjusts their behavior to maintain a comfortable state. The output is the specific action the user takes.

[0858] Step 6:

[0859] The server continuously runs the same process, dynamically adjusting in response to changes in the user's state and environment. This helps users maintain a comfortable environment for extended periods. The output consists of continuously updated analysis results and recommended actions.

[0860] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0861] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0862] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0863] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0864] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0865] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0866] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0867] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0868] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0869] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0870] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0871] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0872] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0873] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0874] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0875] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0876] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0877] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0878] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0879] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0880] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0881] The following is further disclosed regarding the embodiments described above.

[0882] (Claim 1)

[0883] A means for monitoring the posture of users within a living space and generating and notifying them of an exercise reminder when an abnormal posture is detected,

[0884] A method for monitoring the operating status of home appliances and automatically turning off the power when not in use,

[0885] A means for recording the location of an item and notifying the user of the location of the item,

[0886] A means of generating and notifying advice on tidying up based on the state of the living space,

[0887] A system that includes means for detecting suspicious persons using cameras and generating an alarm when a suspicious person is detected.

[0888] (Claim 2)

[0889] The system according to claim 1, which monitors the user's health status and recommends appropriate exercise.

[0890] (Claim 3)

[0891] The system according to claim 1, which manages power supply based on operating status in order to minimize energy consumption of home appliances.

[0892] "Example 1"

[0893] (Claim 1)

[0894] A means for monitoring the physical condition of users within a living space, and for generating and notifying information to encourage exercise when abnormal physical movements are detected,

[0895] A means to monitor the operating status of the equipment and automatically shut off the power when it is not in use,

[0896] A means for recording the location of an item and notifying the user of the location of the item upon request,

[0897] A means for generating and notifying tidying advice based on the condition of the living space,

[0898] A system that includes means for detecting anomalies using an imaging device and generating an alarm when an anomaly is detected.

[0899] (Claim 2)

[0900] The system according to claim 1, which monitors the user's health management and suggests appropriate exercise.

[0901] (Claim 3)

[0902] The system according to claim 1, which performs power management based on operating conditions in order to optimize the energy use of the equipment.

[0903] "Application Example 1"

[0904] (Claim 1)

[0905] A means for monitoring the physical posture of workers in the workspace and generating and notifying them of information recommending exercises when an abnormal posture is detected,

[0906] A means to monitor the operating status of the equipment and automatically shut off the power when not in use,

[0907] A means for recording the position of an object and notifying the worker of the position of the object upon request,

[0908] A means for generating and notifying tidying advice based on the state of the workspace,

[0909] A means for detecting abnormal behavior using a video device and generating a warning upon detection,

[0910] A means of optimizing energy consumption by analyzing the usage status of equipment in the work environment and automatically stopping unused equipment,

[0911] A system that includes this.

[0912] (Claim 2)

[0913] The system according to claim 1, which monitors the physical health of workers and encourages them to perform appropriate exercises.

[0914] (Claim 3)

[0915] The system according to claim 1, which performs power control based on the operating status in order to minimize the energy consumption of the equipment.

[0916] "Example 2 of combining an emotion engine"

[0917] (Claim 1)

[0918] A means for detecting the user's posture within a living space and generating and notifying them of exercise reminders that also take emotional data into consideration,

[0919] A means of optimizing the operation of home appliances based on emotional state, prioritizing comfort while maintaining energy efficiency,

[0920] A method to analyze data such as voice tone and facial expressions to identify the user's emotions and automatically implement measures to reduce stress,

[0921] A means of analyzing data collected from cameras and sensors using an emotion engine to determine a response based on the user's emotional state,

[0922] A system including means for notifying the user of the location of items in response to their request, and means for generating and notifying tidying-up advice based on the state of the living space.

[0923] (Claim 2)

[0924] The system according to claim 1, which monitors the user's health status while taking into account the user's emotional state and recommends appropriate exercise.

[0925] (Claim 3)

[0926] The system according to claim 1, which performs power management to minimize the energy consumption of home appliances based on the user's emotional state.

[0927] "Application example 2 when combining with an emotional engine"

[0928] (Claim 1)

[0929] A device that monitors the posture of users in their living space and generates and notifies them of an exercise reminder when an abnormal posture is detected,

[0930] A device that monitors the operating status of home appliances and automatically turns off the power when not in use,

[0931] A device that records the location of an item and notifies the user of the location of the item upon request,

[0932] A device that generates and notifies users of tidying and organizing advice based on the state of their living space,

[0933] A device that uses a camera to detect suspicious persons and generates an alarm when a suspicious person is detected,

[0934] A device that analyzes the user's emotional state using emotion analysis technology and optimizes the environmental settings of the living space,

[0935] A system that includes devices that analyze the emotional state of customers and provide purchasing support information and services.

[0936] (Claim 2)

[0937] The system according to claim 1, which monitors the user's health status, recommends appropriate exercise, and proposes relaxation based on emotional analysis.

[0938] (Claim 3)

[0939] The system according to claim 1, which manages power supply based on the operating status and the emotional state of the user in order to minimize the energy consumption of home appliances. [Explanation of Symbols]

[0940] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for monitoring the posture of users within a living space and generating and notifying them of an exercise reminder when an abnormal posture is detected, A method for monitoring the operating status of home appliances and automatically turning off the power when not in use, A means for recording the location of an item and notifying the user of the location of the item, A means of generating and notifying advice on tidying up based on the state of the living space, A system that includes means for detecting suspicious persons using cameras and generating an alarm when a suspicious person is detected.

2. The system according to claim 1, which monitors the user's health status and recommends appropriate exercise.

3. The system according to claim 1, which manages power supply based on operating status in order to minimize energy consumption of home appliances.

Images