system

Abstract

We provide the system. [Solution] A recognition means for analyzing facial data acquired by a video acquisition device and confirming the survival of the recognized individual, A cleaning method that automatically performs environmental cleaning using specialized machinery and equipment, An analysis means for analyzing environmental information acquired by multiple sensor devices and detecting anomalies, An aggregation means for integrating information obtained from the aforementioned recognition means, cleaning means, and analysis means to create a report, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In condominium apartments where the aging process is advancing, it is required to reduce the burden of operation while ensuring the safety and comfort of the residents' lives. However, the conventional management method requires a large amount of human resources, increasing the burden on the management association. In particular, since regular operations such as survival confirmation, cleaning, and environmental monitoring are not automated, it hinders efficient operation. Also, it is difficult to respond promptly when an abnormality occurs, and the construction of an appropriate system for ensuring the safety of residents is required.

Means for Solving the Problems

[0005] This invention provides a system comprising: recognition means for confirming the survival of residents using facial data analysis from an image acquisition device; cleaning means for performing automatic cleaning using specialized mechanical devices; analysis means for detecting abnormalities by analyzing environmental information from multiple sensor devices; and aggregation means for integrating this information to create a report. This will improve the efficiency of operational tasks, enable rapid response to abnormalities, and create an environment in which residents can live with peace of mind in condominium complexes with aging populations.

[0006] An "image acquisition device" is a device that captures images or videos of an object and collects that data.

[0007] "Facial data" refers to characteristic information acquired to identify human faces, and is digital data that enables individual recognition.

[0008] "Recognition means" refers to a technical method or device for analyzing acquired data and identifying or confirming a specific object.

[0009] A "specialized machine" is an automated device designed specifically for a particular purpose or task.

[0010] "Cleaning means" refers to techniques or devices that remove dirt and impurities in order to maintain or beautify the environment.

[0011] A "sensor device" is a device that detects physical or chemical changes and converts those changes into signals.

[0012] "Environmental information" refers to data that indicates conditions such as temperature, humidity, light intensity, sound, and movement within a specific range.

[0013] "Analysis means" refers to a method or apparatus for analyzing collected data and identifying specific patterns or anomalies.

[0014] "Aggregation means" refers to a method or apparatus for analyzing multiple collected data and integrating the results.

[0015] A "report" is a document that summarizes data on specific events or situations, and is a document for explaining or recording such situations.

Brief Description of the Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The present invention aims to facilitate the smooth operation of condominium complexes by using an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means.

[0038] First, a terminal (a video acquisition device installed in the entrance) captures the resident's face and sends it to the server. The server uses recognition technology to compare the received face data with a registered database and confirms the resident's well-being. If the result is abnormal, the server notifies the administrator, and the terminal displays an alert.

[0039] In addition, during cleaning operations, the server instructs specialized machinery on the cleaning schedule. The cleaning robots, acting as terminals, clean the designated areas and report their status to the server upon completion. The server records this report in a database and manages it as a cleaning history.

[0040] Furthermore, during inspection and patrol work, terminals (sensor devices) acquire environmental information in real time and transmit the data to the server. The server analyzes this data using analytical tools, and if an anomaly is detected, it immediately notifies the administrator and prompts them to take action.

[0041] For reporting, the server aggregates data obtained from each function using aggregation tools and generates daily or monthly reports. These reports are accessible to users through a web portal, facilitating comparisons with historical data and trend analysis.

[0042] As a concrete example, there is a system in place where a cleaning robot automatically starts cleaning the entrance every morning, and its progress is reported to a server. Users can check the cleaning status and the history of detected anomalies at any time using their smartphones or tablets. This reduces the workload of the management association and allows residents to live with greater peace of mind.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The terminal (video acquisition device) captures the resident's face and generates facial image data. It then prepares to send this data to the server.

[0046] Step 2:

[0047] The server queries the received facial image data against a database and compares it with registered facial data. A recognition system is then used to confirm the person's survival.

[0048] Step 3:

[0049] The server evaluates the facial recognition results and determines whether the survival check was successful. If an anomaly is detected, it generates and sends a notification to the administrator.

[0050] Step 4:

[0051] The terminal receives the results of the liveness check from the server, as well as any necessary anomaly notifications, and displays the results on a local display.

[0052] Step 5:

[0053] The server sends a cleaning start command to the cleaning robot based on the cleaning schedule.

[0054] Step 6:

[0055] The terminal (cleaning robot) begins cleaning the designated area and performs the cleaning task. The progress of the work is reported to the server in real time.

[0056] Step 7:

[0057] After cleaning is complete, the cleaning robot sends a status report to the server, which then records it in its database.

[0058] Step 8:

[0059] The terminal (sensor device) acquires environmental information (temperature, humidity, etc.) in real time and transmits it to the server.

[0060] Step 9:

[0061] The server analyzes the sensor data and uses analytical tools to evaluate whether there are any abnormalities. If an abnormality is detected, the administrator is immediately notified.

[0062] Step 10:

[0063] The server aggregates all this data and generates reports periodically. Users can access these reports and view the information through a web portal.

[0064] (Example 1)

[0065] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0066] Modern apartment buildings require improved operational efficiency while ensuring the safety and comfort of residents. However, traditional methods rely heavily on human resources, making efficient management difficult and potentially hindering rapid response in the event of an emergency. Furthermore, centrally managing records of cleaning and periodic inspections is challenging. A system is needed to address these issues and enable more efficient and safer management.

[0067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0068] In this invention, the server includes recognition means, cleaning means, and analysis means. This enables rapid survival confirmation through automatic recognition of residents' facial information, improved operational efficiency through automation of regular cleaning, and anomaly detection and prediction through real-time analysis of sensor information.

[0069] "Image acquisition means" refers to a device or system installed in a specific location for collecting facial information of a subject.

[0070] "Facial information" refers to distinctive data that enables the identification of a target individual, and includes image data obtained from cameras, etc.

[0071] A "recognition means" is a system that has the function of identifying individuals and confirming their survival using acquired facial information.

[0072] A "cleaning means" is a system that has the function of performing cleaning work in a designated area using automated, specialized machinery and equipment.

[0073] "Specialized machinery and equipment" refers to machinery and equipment designed to suit specific tasks or conditions, and includes those with autonomous cleaning functions.

[0074] "Analysis means" refers to algorithms and systems for analyzing environmental information collected from multiple detection devices and detecting anomalies.

[0075] A "detection device" is a device or system that includes multiple sensors installed to detect various parameters in the environment.

[0076] A "data aggregation system" is a system that integrates information obtained from multiple sources and has the function of supporting analysis and report creation.

[0077] A "verification method" is a system that verifies anomalies based on authentication results using facial information and has the function of notifying administrators or issuing warnings to display devices.

[0078] A "prediction means" is a system that includes calculation methods and algorithms for predicting future anomalies from data collected using analysis means.

[0079] This invention is an integrated management system aimed at the efficient management of apartment buildings. The system includes the following elements:

[0080] First, a video acquisition device installed as a terminal collects facial information of the residents. This device is a high-resolution camera used to accurately capture facial information. The captured data is immediately sent to the server.

[0081] Next, the server receives this facial data and analyzes it using an AI-based recognition system. This system uses a generative AI model trained by a machine learning algorithm, and checks for the person's survival by comparing it with existing data in the database.

[0082] Furthermore, the server manages specialized machinery, specifically autonomous cleaning robots. These robots automatically clean designated areas based on a programmed cleaning schedule. Once cleaning is complete, the robots send a completion notification to the server, which is then recorded as history in the database.

[0083] Furthermore, multiple sensor devices placed on the terminal continuously collect environmental information within the building (temperature, humidity, CO2 concentration, etc.). This information is transmitted to a server, where it is analyzed to check for any abnormalities. If an abnormality is detected, the administrator is immediately notified and prompted to take action.

[0084] The server aggregates the data obtained from these processes and generates periodic reports. This is designed to improve the transparency and efficiency of operations. Users can view the reports via a web portal and perform comparative analysis with historical data.

[0085] As a concrete example, every morning, a cleaning robot automatically starts cleaning the entrance and reports its status to a server upon completion. This report, along with past history, is easily accessible to users via smartphones and tablets.

[0086] An example of a prompt message is, "Please tell me how to collect data for resident facial recognition and cleaning robot status checks, and analyze trends based on recent cleaning history." The goal of operating such a system is to significantly improve the management efficiency of apartment buildings.

[0087] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0088] Step 1:

[0089] The terminal uses a camera installed in the entrance to capture residents' facial information. The input is video data acquired by the camera in real time. As output, the facial information is saved in file format and sent to the server. Specifically, when a resident passes in front of the camera, a trigger is activated and an image for facial recognition is captured.

[0090] Step 2:

[0091] The server analyzes the received facial data using a generating AI model. The input is the facial information sent in step 1. As part of the data processing, the AI ​​model extracts facial feature points and compares them with a database to recognize individuals and verify their survival. The output is the recognition result. Specifically, the AI ​​model identifies patterns in facial features and performs authentication by comparing them with specific individuals.

[0092] Step 3:

[0093] The server checks for anomalies based on the facial recognition results and sends a notification to the administrator if an anomaly is detected. The input is the recognition result obtained in step 2. As part of the data processing, an anomaly detection algorithm detects discrepancies with the normal recognition pattern. The output is a flag indicating the presence or absence of an anomaly and a notification message. Specifically, even if only a slight anomaly is detected, it is immediately reported to the administrator via email or push notification.

[0094] Step 4:

[0095] The server issues cleaning instructions to the specialized cleaning robot based on a pre-set schedule. The input is the cleaning schedule set within the system. During the data calculation process, the schedule is compared with the actual operating status to generate appropriate instructions. The output is a cleaning instruction command. In terms of specific operation, the cleaning robot begins cleaning the designated area according to the schedule.

[0096] Step 5:

[0097] The cleaning robot, acting as the terminal, reports its status to the server after completing the cleaning task. The input is operational data collected by the robot during cleaning. The output is a cleaning status report, which is then sent to the server. Specifically, upon completion of cleaning, the status is automatically updated and recorded as a report.

[0098] Step 6:

[0099] The sensor device installed at the terminal continuously collects environmental information within the building and transmits it to the server. Inputs include sensor data such as temperature, humidity, and CO2 concentration. Outputs are environmental information data. Specifically, the sensor collects various data at set intervals and transmits it in real time.

[0100] Step 7:

[0101] The server analyzes sensor information in real time and detects anomalies. The input is the environmental information data transmitted in step 6. During data processing, an analysis algorithm compares the data to standard values ​​and detects anomalies. The output is the anomaly detection result. Specifically, when an anomaly occurs, a procedure is initiated that prompts immediate corrective action.

[0102] Step 8:

[0103] The server aggregates all data and generates daily or monthly reports. Inputs include historical facial recognition, cleaning data, and sensor data. Output is a detailed report accessible to the user. Its specific operation involves retrieving historical information from the database and outputting it as a visually easy-to-understand report.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] In condominium complexes and public facilities, ensuring security and streamlining operations are crucial challenges. In particular, verifying residents and visitors, managing environmental cleanliness, and promptly detecting and responding to anomalies are required. However, performing these tasks manually is labor-intensive and hinders quick responses. Therefore, efficient and automated systems are needed.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes identification means for analyzing facial data acquired by a video acquisition device to confirm the recognized individual, maintenance means for automatically performing environmental maintenance using specialized mechanical devices, and processing means for processing environmental information acquired by multiple sensor devices to detect anomalies. This enables improved security levels and more efficient environmental management.

[0109] A "video acquisition device" is a device used to acquire the appearance of a target location or individual as digital data.

[0110] "Identification means" refers to methods or devices that analyze acquired data and have the function of identifying and confirming individuals based on specific characteristics.

[0111] A "specialized machine" is a machine specifically designed to efficiently perform a particular task or operation.

[0112] "Maintenance means" refers to mechanisms and methods for automatically performing environmental maintenance and cleaning using specialized machinery and equipment.

[0113] A "sensor device" is a device used to measure information about the surrounding environment and to electronically acquire that data.

[0114] "Processing means" refers to a method or system used to analyze acquired information and identify anomalies.

[0115] An "aggregation method" is a system or method for integrating and centrally managing information obtained from different sources.

[0116] A "communication means" is a system that has the function of communication and information provision to notify relevant parties of detected anomalies.

[0117] This invention provides a system that automates security and environmental management in condominium complexes using a server, video acquisition device, specialized mechanical device, and sensor device. The server first collects facial data using the video acquisition device and verifies individuals using identification means. This process uses facial recognition software such as Amazon Rekognition or Microsoft® Face API for analysis.

[0118] Next, specialized machinery and equipment are used to automatically prepare the environment through maintenance methods, and a server monitors the status of the maintenance work. This maintenance is achieved by using cleaning robots such as Roomba to clean designated areas.

[0119] Furthermore, environmental information collected by sensor devices is analyzed in real time by processing tools, and if an anomaly is detected, the server immediately sends a warning to the administrator. AWS® IoT Core and Google® Cloud IoT are used for processing, and AWS Kinesis Data Analytics and other real-time analysis tools are used for rapid data processing.

[0120] Users can receive this information via a smartphone app, whether at home or on the go, and check the status of security and environmental improvements. For example, it is possible to set up a system that immediately sends a push notification if a stranger enters the property.

[0121] As a concrete example, an example of a prompt statement is "Generate a prompt regarding a system for detecting anomalies in common areas of a condominium complex and notifying administrators." In this way, the objective of this invention is to provide an environment in which residents and administrators can live with peace of mind on a sustainable basis.

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The server receives facial data from the video acquisition device. The acquired facial data is compared against the registration database using facial recognition software such as Amazon Rekognition or Microsoft Face API. The input for checking for anomalies is the facial data, and the output is the authentication result of the identified individual. Based on this, the server checks whether the authenticated individual has been registered without any problems.

[0125] Step 2:

[0126] The server sends maintenance work commands to cleaning robots, which are specialized mechanical devices. These commands are scheduled and perform cleaning to maintain the environment of the designated area. Inputs are the cleaning schedule and area information, and output is the completion status of the cleaning work. The server monitors the progress of the maintenance work in real time and records it in the database upon completion.

[0127] Step 3:

[0128] The sensor device, acting as a terminal, collects environmental sensor data and transmits it to the server. The server ingests this data using AWS IoT Core and performs analysis using AWS Kinesis Data Analytics, etc. The input is environmental sensor data, and the output is the analysis results regarding the presence or absence of anomalies. Based on these analysis results, the server detects anomalies in real time.

[0129] Step 4:

[0130] When an anomaly is detected, the server sends an alert to administrators and relevant parties using a communication method. Firebase Cloud Messaging (FCM) is used to immediately send push notifications to smartphones or compatible devices. The input is the analysis result of the anomaly detection, and the output is the alert notification to administrators. This enables a rapid response.

[0131] Step 5:

[0132] Users can check the security status and maintenance status of their environment in real time through a smartphone app. Users can receive notifications and take necessary actions accordingly. The input to this process is notifications from the server, and the output is information provided to the user. This allows users to understand the situation and go about their daily lives with peace of mind.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] The system of the present invention comprises an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means, as well as an emotion engine that recognizes the user's emotions. This emotion engine can receive the user's voice and facial expression data and analyze their emotional state. The system uses this data to dynamically adjust its operation in order to improve the comfort and safety of the living environment.

[0135] First, a terminal (video acquisition device) captures the resident's face and sends it to the server. The recognition system on the server uses this data to confirm the resident's well-being. In parallel, an emotion engine recognizes the user's voice and facial expressions and analyzes their emotional state. Based on this analysis, the server adjusts the system's operation and response, taking necessary actions.

[0136] For example, if negative emotions such as anxiety or stress are detected when a user passes through the entrance, the server can use that information to send a notification to the administrator via the terminal. Furthermore, operational tasks can be carried out using emotional information, such as adjusting the cleaning schedule to perform cleaning at a time when users feel most comfortable.

[0137] The sensor device continues to collect environmental information and transmit it to the server. The server analyzes this information in conjunction with the data from the emotion engine and immediately notifies the administrator if an anomaly is detected. The system also has a function to issue an alert if the emotion information acquired by the emotion engine is abnormal.

[0138] Through this series of processes, the system of the present invention not only improves operational efficiency but also enables thoughtful responses that respond to the feelings of residents, making it possible to provide a safe and comfortable living environment even in housing where the population is aging.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] The terminal (video acquisition device) captures the face and voice data of users passing through the entrance and prepares to send it to the server.

[0142] Step 2:

[0143] Based on the facial data received by the server, recognition tools are used to compare it with information in the database. Along with confirming the user's well-being, the emotion engine analyzes the user's emotional state in conjunction with the voice data.

[0144] Step 3:

[0145] The server evaluates the matching result, and if authentication is successful, it proceeds to the next process. If the emotion engine analysis determines that the user is experiencing stress, the server creates and sends a notification to the administrator.

[0146] Step 4:

[0147] The server checks the cleaning schedule and sends instructions to adjust the time and area of ​​the cleaning method (cleaning robot) according to the user's emotional state.

[0148] Step 5:

[0149] Based on the instructions received by the terminal (cleaning robot), it begins cleaning work at the specified time and area, and reports its progress to the server.

[0150] Step 6:

[0151] The terminal (sensor device) collects environmental information (temperature, humidity, light intensity, etc.) and transmits the data to the server in real time.

[0152] Step 7:

[0153] The server comprehensively analyzes sensor data and emotional state analysis data, and if an anomaly is detected, it sends a notification to the administrator and displays an alert on the display device.

[0154] Step 8:

[0155] The server aggregates all the data and generates reports periodically. Users can access these reports to check information about their living environment.

[0156] (Example 2)

[0157] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0158] In an aging society, there is a need to achieve efficient environmental management while ensuring the safety and comfort of residents. In particular, the lack of dynamic responses that take into account the emotional state of residents makes it difficult to adjust the environment to meet individual needs. Therefore, technology is needed to analyze emotional states and individually adjust the operation of systems.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0160] In this invention, the server includes a decision-making means, an operating means, and an analysis means. This enables the use of emotional information to flexibly adjust the environment according to the residents and improve safety.

[0161] The "determination means" refers to a function that analyzes facial information of an individual obtained through an image acquisition device to confirm the existence of that individual.

[0162] "Operating means" refers to a function that automatically performs environmental maintenance using specific-purpose equipment.

[0163] "Analysis means" refers to a function that analyzes environmental information acquired from multiple detection devices to detect anomalies.

[0164] The "aggregation means" is a function that integrates information obtained from the judgment means, the operation means, and the analysis means to create a list.

[0165] "Analysis tools" refer to functions for analyzing an individual's emotional state and providing data related to environmental adjustments.

[0166] This invention is an environmental control system for improving safety and comfort within living spaces. The system's main purpose is to analyze the emotional state of the residents and dynamically adjust the environment and system operation based on that analysis.

[0167] This system includes multiple elements, the main of which are a video acquisition device, a specific-purpose device, multiple detection devices, and an emotion analysis function. Specifically, the server analyzes facial information obtained from the video acquisition device using a judgment means to confirm the presence of residents. The analysis means continuously monitors environmental information obtained from the detection devices and responds quickly if an anomaly is detected. The specific-purpose device, acting as an operating means, automatically adjusts the environment to support residents in living comfortably. Furthermore, an emotion engine operates as an analysis means, analyzing the resident's voice and facial expressions to determine their emotional state.

[0168] As a concrete example, if an anxious feeling is detected when a user enters the entrance, the server will adjust the lighting to induce a sense of security. Furthermore, if necessary, it will immediately notify the administrator to support the resolution of the problem.

[0169] For example, by inputting a prompt message such as "What is the most effective interface adjustment when a user feels stressed?" into the AI ​​model, it can find effective countermeasures.

[0170] In this way, the system aims to provide advanced life support based on emotional information by combining natural language processing and environmental control.

[0171] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0172] Step 1:

[0173] The terminal uses a video acquisition device to capture facial information of residents in the living space in real time. In this step, facial images of the user as they move become input data. The terminal sends this facial information to a server, which analyzes it using a decision-making tool to confirm the presence of the resident. This analysis results in an output confirming the resident's survival.

[0174] Step 2:

[0175] The server uses an emotion engine to receive the user's voice and facial expression data and analyze their emotional state. The input consists of voice and facial expression data obtained from the user. The server processes this data to determine the user's emotional state (e.g., reassured, stressed, anxious). The output of this step is the analyzed emotional data.

[0176] Step 3:

[0177] The sensor device continuously collects environmental information (temperature, humidity, brightness, etc.) and transmits it to the server. The input is this collected environmental data. The server's analysis system processes the data to determine the current state of the environment and whether or not there are any abnormalities. This process yields an output indicating whether the state is normal or abnormal.

[0178] Step 4:

[0179] The server adjusts the operation of the entire system based on the acquired emotional and environmental data. Specifically, the server controls the operating mechanisms to adjust the brightness of the lighting or operate the air conditioning. The input is the emotional and environmental state data obtained in the previous step. The output is the optimized environmental conditions and the results of the operation.

[0180] Step 5:

[0181] If the server detects an anomaly or a significant emotional state, it sends a notification to the administrator via the terminal. This step uses the anomaly detection and emotion analysis results as input. The output is a notification sent to the administrator, allowing for necessary action.

[0182] Step 6:

[0183] This step uses a generative AI model to obtain suggestions for system adjustments in a specific situation. The input for this step is the prompt "What is the most effective interface adjustment when a user experiences stress?". The AI ​​model analyzes this and proposes the optimal system adjustment method. This proposal is the output.

[0184] Through these steps, the system can flexibly respond to the user's emotions and living environment, providing a comfortable and safe living space.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] Current security systems primarily focus on detecting physical anomalies, making it difficult to respond flexibly based on the emotional state of residents and users. As a result, they fail to provide maximum effectiveness in terms of security and comfort. Furthermore, the lack of appropriate support that can immediately respond to changes in users' emotions during emergencies prevents them from providing a sense of security.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes means for analyzing facial and voice data acquired by a video acquisition device to confirm the survival of individuals and analyze their emotional state, means for automatically performing environmental cleaning using a specialized mechanical device, and means for analyzing environmental information acquired by multiple sensor devices to detect abnormalities and emotional states. This enables security measures and the maintenance of a comfortable environment that take into account the emotional state of residents and users.

[0190] A "video acquisition device" is a device that captures the visual information of an object and acquires it as digital data.

[0191] "Facial data" refers to digital data that represents the characteristics of an individual's face and is used for identification and recognition.

[0192] "Audio data" refers to data that can record audio and be stored or analyzed in digital format.

[0193] "Recognition means" refers to means equipped with the function of identifying a specific individual or analyzing its state based on acquired data.

[0194] A "specialized machine" is a machine with a specialized structure designed to perform a specific purpose or function.

[0195] "Cleaning means" refers to devices or functions that automatically clean the environment.

[0196] A "sensor device" is a device that detects the physical or chemical state of the environment and acquires that information as digital data.

[0197] "Analysis means" refers to methods for processing acquired data and extracting meaningful information.

[0198] "Aggregation means" refers to a device or function used to integrate multiple pieces of information and create reports or perform data analysis.

[0199] "Emotional state" is an indicator that represents an individual's psychological state and can be evaluated based on voice and facial expressions.

[0200] An "emotion engine" is software equipped with algorithms and functions to analyze a user's voice and facial expression data and identify their emotional state.

[0201] A "security agency" is an organization or company that is responsible for safety and security-related operations.

[0202] The system implementing this invention is configured to combine a video acquisition device, an audio acquisition device, a recognition means, a cleaning means, a sensor device, an analysis means, an aggregation means, and an emotion engine. A terminal installed in the user's living environment acquires the user's facial and voice data in real time using the video acquisition device and the audio acquisition device. This data is transmitted to a server, where the recognition means confirms the individual's survival and analyzes their emotional state.

[0203] The server controls cleaning methods that perform environmental cleaning using specialized mechanical equipment, and analysis methods detect anomalies based on various environmental information acquired from sensor devices. The diverse data obtained from these methods are integrated by aggregation methods to create reports, and at the same time, dynamic system responses that take into account the user's emotional state are realized.

[0204] Specifically, when a user is relaxing at home, the system recognizes their emotions and adjusts lighting, music, and other elements to provide a comfortable environment. Furthermore, if the user is experiencing anxiety or stress, the server immediately notifies security agencies for a swift response.

[0205] The hardware used includes smart devices with cameras and microphones with voice input capabilities. Data analysis is performed using software such as Rekognition and Polly from Amazon Web Services (AWS). AWS SNS is used for alarm notifications, enabling real-time notifications to users and administrators.

[0206] A concrete example of a prompt using a generative AI model is: "I want to design an app that identifies emotions from a user's face and voice, and sends a notification to a security company if negative emotions are detected. Therefore, please tell me how to perform emotion analysis using AWS Rekognition and Polly."

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The device acquires the user's face and voice. It captures face and voice data in real time using the camera and microphone, and sends this data to the server as input.

[0210] Step 2:

[0211] The server processes the received facial data using recognition technology. This process analyzes facial features and performs individual identification. The recognition result is obtained as output and passed to the next processing step.

[0212] Step 3:

[0213] The emotion engine on the server analyzes the audio data and evaluates the user's emotional state. It extracts emotions from the tone and rhythm of the voice and outputs the results.

[0214] Step 4:

[0215] The server integrates the recognition methods and the output of the emotion engine to comprehensively evaluate the user's state. It determines the presence or absence of abnormalities or negative emotions and uses the results in the next step.

[0216] Step 5:

[0217] The server analyzes environmental information from sensor devices and detects anomalies. The output is data indicating whether or not there are environmental anomalies. Based on these results, the server determines the necessary actions.

[0218] Step 6:

[0219] The server integrates the results from the previous step using aggregation tools and creates a report. Simultaneously, it sends a notification to the security agency if any anomalies are detected. This step also includes setting up alerts via voice notifications and display devices.

[0220] Step 7:

[0221] The user receives feedback from the server. For example, if the server determines that the user is relaxed, actions are taken to improve the user experience, such as automatically playing soothing music.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] The present invention aims to facilitate the smooth operation of condominium complexes by using an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means.

[0239] First, a terminal (a video acquisition device installed in the entrance) captures the resident's face and sends it to the server. The server uses recognition technology to compare the received face data with a registered database and confirms the resident's well-being. If the result is abnormal, the server notifies the administrator, and the terminal displays an alert.

[0240] In addition, during cleaning operations, the server instructs specialized machinery on the cleaning schedule. The cleaning robots, acting as terminals, clean the designated areas and report their status to the server upon completion. The server records this report in a database and manages it as a cleaning history.

[0241] Furthermore, during inspection and patrol work, terminals (sensor devices) acquire environmental information in real time and transmit the data to the server. The server analyzes this data using analytical tools, and if an anomaly is detected, it immediately notifies the administrator and prompts them to take action.

[0242] For reporting, the server aggregates data obtained from each function using aggregation tools and generates daily or monthly reports. These reports are accessible to users through a web portal, facilitating comparisons with historical data and trend analysis.

[0243] As a concrete example, there is a system in place where a cleaning robot automatically starts cleaning the entrance every morning, and its progress is reported to a server. Users can check the cleaning status and the history of detected anomalies at any time using their smartphones or tablets. This reduces the workload of the management association and allows residents to live with greater peace of mind.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The terminal (video acquisition device) captures the resident's face and generates facial image data. It then prepares to send this data to the server.

[0247] Step 2:

[0248] The server queries the received facial image data against a database and compares it with registered facial data. A recognition system is then used to confirm the person's survival.

[0249] Step 3:

[0250] The server evaluates the facial recognition results and determines whether the survival check was successful. If an anomaly is detected, it generates and sends a notification to the administrator.

[0251] Step 4:

[0252] The terminal receives the results of the liveness check from the server, as well as any necessary anomaly notifications, and displays the results on a local display.

[0253] Step 5:

[0254] The server sends a cleaning start command to the cleaning robot based on the cleaning schedule.

[0255] Step 6:

[0256] The terminal (cleaning robot) begins cleaning the designated area and performs the cleaning task. The progress of the work is reported to the server in real time.

[0257] Step 7:

[0258] After cleaning is complete, the cleaning robot sends a status report to the server, which then records it in its database.

[0259] Step 8:

[0260] The terminal (sensor device) acquires environmental information (temperature, humidity, etc.) in real time and transmits it to the server.

[0261] Step 9:

[0262] The server analyzes the sensor data and uses analytical tools to evaluate whether there are any abnormalities. If an abnormality is detected, the administrator is immediately notified.

[0263] Step 10:

[0264] The server aggregates all this data and generates reports periodically. Users can access these reports and view the information through a web portal.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0267] Modern apartment buildings require improved operational efficiency while ensuring the safety and comfort of residents. However, traditional methods rely heavily on human resources, making efficient management difficult and potentially hindering rapid response in the event of an emergency. Furthermore, centrally managing records of cleaning and periodic inspections is challenging. A system is needed to address these issues and enable more efficient and safer management.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes recognition means, cleaning means, and analysis means. This enables rapid survival confirmation through automatic recognition of residents' facial information, improved operational efficiency through automation of regular cleaning, and anomaly detection and prediction through real-time analysis of sensor information.

[0270] "Image acquisition means" refers to a device or system installed in a specific location for collecting facial information of a subject.

[0271] "Facial information" refers to distinctive data that enables the identification of a target individual, and includes image data obtained from cameras, etc.

[0272] A "recognition means" is a system that has the function of identifying individuals and confirming their survival using acquired facial information.

[0273] A "cleaning means" is a system that has the function of performing cleaning work in a designated area using automated, specialized machinery and equipment.

[0274] "Specialized machinery and equipment" refers to machinery and equipment designed to suit specific tasks or conditions, and includes those with autonomous cleaning functions.

[0275] "Analysis means" refers to algorithms and systems for analyzing environmental information collected from multiple detection devices and detecting anomalies.

[0276] A "detection device" is a device or system that includes multiple sensors installed to detect various parameters in the environment.

[0277] A "data aggregation system" is a system that integrates information obtained from multiple sources and has the function of supporting analysis and report creation.

[0278] A "verification method" is a system that verifies anomalies based on authentication results using facial information and has the function of notifying administrators or issuing warnings to display devices.

[0279] A "prediction means" is a system that includes calculation methods and algorithms for predicting future anomalies from data collected using analysis means.

[0280] This invention is an integrated management system aimed at the efficient management of apartment buildings. The system includes the following elements:

[0281] First, a video acquisition device installed as a terminal collects facial information of the residents. This device is a high-resolution camera used to accurately capture facial information. The captured data is immediately sent to the server.

[0282] Next, the server receives these face data and performs analysis using AI-based recognition means. This means uses a generative AI model trained by a machine learning algorithm and verifies the survival by comparing with the existing data in the database.

[0283] Furthermore, the server manages a specialized mechanical device, specifically an autonomous cleaning robot. This robot automatically cleans the designated area based on the programmed cleaning schedule. When the cleaning is completed, the robot sends a completion notice to the server and is recorded as a history in the database.

[0284] Also, a plurality of sensor devices arranged on the terminal continuously collect environmental information (temperature, humidity, CO2 concentration, etc.) inside the building. This information is sent to the server and verified for abnormalities using analysis means. If an abnormality is detected, it is immediately notified to the administrator to prompt a response.

[0285] The server aggregates the data obtained in these processes and generates a periodic report. It is designed to improve the transparency and efficiency of the operation. Users can check the report via the web portal and perform comparative analysis with past data.

[0286] As a specific example, every morning, the cleaning robot automatically starts cleaning the entrance and reports the status to the server after completion. This report is made easily accessible to users through smartphones or tablets in combination with past history.

[0287] As an example of the prompt sentence, "Please teach me a method of aggregating data for resident face recognition and status confirmation of the cleaning robot and analyzing trends based on the recent cleaning history." is given. The operation of such a system aims to significantly improve the management efficiency of apartment buildings.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The terminal uses a camera installed in the entrance to capture residents' facial information. The input is video data acquired by the camera in real time. As output, the facial information is saved in file format and sent to the server. Specifically, when a resident passes in front of the camera, a trigger is activated and an image for facial recognition is captured.

[0291] Step 2:

[0292] The server analyzes the received facial data using a generating AI model. The input is the facial information sent in step 1. As part of the data processing, the AI ​​model extracts facial feature points and compares them with a database to recognize individuals and verify their survival. The output is the recognition result. Specifically, the AI ​​model identifies patterns in facial features and performs authentication by comparing them with specific individuals.

[0293] Step 3:

[0294] The server checks for anomalies based on the facial recognition results and sends a notification to the administrator if an anomaly is detected. The input is the recognition result obtained in step 2. As part of the data processing, an anomaly detection algorithm detects discrepancies with the normal recognition pattern. The output is a flag indicating the presence or absence of an anomaly and a notification message. Specifically, even if only a slight anomaly is detected, it is immediately reported to the administrator via email or push notification.

[0295] Step 4:

[0296] The server issues cleaning instructions to the specialized cleaning robot based on a pre-set schedule. The input is the cleaning schedule set within the system. During the data calculation process, the schedule is compared with the actual operating status to generate appropriate instructions. The output is a cleaning instruction command. In terms of specific operation, the cleaning robot begins cleaning the designated area according to the schedule.

[0297] Step 5:

[0298] The cleaning robot, acting as the terminal, reports its status to the server after completing the cleaning task. The input is operational data collected by the robot during cleaning. The output is a cleaning status report, which is then sent to the server. Specifically, upon completion of cleaning, the status is automatically updated and recorded as a report.

[0299] Step 6:

[0300] The sensor device installed at the terminal continuously collects environmental information within the building and transmits it to the server. Inputs include sensor data such as temperature, humidity, and CO2 concentration. Outputs are environmental information data. Specifically, the sensor collects various data at set intervals and transmits it in real time.

[0301] Step 7:

[0302] The server analyzes sensor information in real time and detects anomalies. The input is the environmental information data transmitted in step 6. During data processing, an analysis algorithm compares the data to standard values ​​and detects anomalies. The output is the anomaly detection result. Specifically, when an anomaly occurs, a procedure is initiated that prompts immediate corrective action.

[0303] Step 8:

[0304] The server aggregates all data and generates daily or monthly reports. Inputs include historical facial recognition, cleaning data, and sensor data. Output is a detailed report accessible to the user. Its specific operation involves retrieving historical information from the database and outputting it as a visually easy-to-understand report.

[0305] (Application Example 1)

[0306] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0307] In condominiums and public facilities, ensuring security and improving the efficiency of operation are important issues. In particular, confirmation of residents and visitors, management of the cleaning status of the environment, early detection of abnormalities, and response are required, but it is labor-intensive to perform these manually, and it is difficult to respond quickly. Therefore, an efficient and automated system is required.

[0308] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0309] In this invention, the server includes an identification means for confirming an individual recognized by analyzing face data acquired by an image acquisition device, a maintenance means for automatically performing environmental maintenance by a specialized mechanical device, and a processing means for processing environmental information acquired by a plurality of sensor devices and detecting abnormalities. Thereby, it becomes possible to improve the security level and the efficiency of environmental management.

[0310] The "image acquisition device" is a device for acquiring the appearance of a target location or individual as digital data.

[0311] The "identification means" is a method or device having a function of analyzing acquired data and identifying and confirming an individual based on specific characteristics.

[0312] The "specialized mechanical device" is a dedicated machine designed to efficiently perform specific operations or tasks.

[0313] The "maintenance means" is a mechanism or method for automatically performing environmental maintenance and cleaning using a specialized mechanical device.

[0314] The "sensor device" is a device for measuring ambient environmental information and electronically acquiring the data.

[0315] "Processing means" refers to a method or system used to analyze acquired information and identify anomalies.

[0316] An "aggregation method" is a system or method for integrating and centrally managing information obtained from different sources.

[0317] A "communication means" is a system that has the function of communication and information provision to notify relevant parties of detected anomalies.

[0318] In this invention, the server provides a system that automates security and environmental management in condominium complexes using a video acquisition device, specialized mechanical device, and sensor device. The server first collects facial data using the video acquisition device and verifies individuals using identification means. This process uses facial recognition software such as Amazon Rekognition or Microsoft Face API for analysis.

[0319] Next, specialized machinery and equipment are used to automatically prepare the environment through maintenance methods, and a server monitors the status of the maintenance work. This maintenance is achieved by using cleaning robots such as Roomba to clean designated areas.

[0320] Furthermore, environmental information collected by sensor devices is analyzed in real time by processing devices, and if an anomaly is detected, the server immediately sends a warning to the administrator. AWS IoT Core and Google Cloud IoT are used for processing, and AWS Kinesis Data Analytics and other real-time analysis tools are used for rapid data processing.

[0321] Users can receive this information via a smartphone app, whether at home or on the go, and check the status of security and environmental improvements. For example, it is possible to set up a system that immediately sends a push notification if a stranger enters the property.

[0322] As a concrete example, an example of a prompt statement is "Generate a prompt regarding a system for detecting anomalies in common areas of a condominium complex and notifying administrators." In this way, the objective of this invention is to provide an environment in which residents and administrators can live with peace of mind on a sustainable basis.

[0323] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0324] Step 1:

[0325] The server receives facial data from the video acquisition device. The acquired facial data is compared against the registration database using facial recognition software such as Amazon Rekognition or Microsoft Face API. The input for checking for anomalies is the facial data, and the output is the authentication result of the identified individual. Based on this, the server checks whether the authenticated individual has been registered without any problems.

[0326] Step 2:

[0327] The server sends maintenance work commands to cleaning robots, which are specialized mechanical devices. These commands are scheduled and perform cleaning to maintain the environment of the designated area. Inputs are the cleaning schedule and area information, and output is the completion status of the cleaning work. The server monitors the progress of the maintenance work in real time and records it in the database upon completion.

[0328] Step 3:

[0329] The sensor device, acting as a terminal, collects environmental sensor data and transmits it to the server. The server ingests this data using AWS IoT Core and performs analysis using AWS Kinesis Data Analytics, etc. The input is environmental sensor data, and the output is the analysis results regarding the presence or absence of anomalies. Based on these analysis results, the server detects anomalies in real time.

[0330] Step 4:

[0331] When an anomaly is detected, the server sends an alert to administrators and relevant parties using a communication method. Firebase Cloud Messaging (FCM) is used to immediately send push notifications to smartphones or compatible devices. The input is the analysis result of the anomaly detection, and the output is the alert notification to administrators. This enables a rapid response.

[0332] Step 5:

[0333] Users can check the security status and maintenance status of their environment in real time through a smartphone app. Users can receive notifications and take necessary actions accordingly. The input to this process is notifications from the server, and the output is information provided to the user. This allows users to understand the situation and go about their daily lives with peace of mind.

[0334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0335] The system of the present invention comprises an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means, as well as an emotion engine that recognizes the user's emotions. This emotion engine can receive the user's voice and facial expression data and analyze their emotional state. The system uses this data to dynamically adjust its operation in order to improve the comfort and safety of the living environment.

[0336] First, a terminal (video acquisition device) captures the resident's face and sends it to the server. The recognition system on the server uses this data to confirm the resident's well-being. In parallel, an emotion engine recognizes the user's voice and facial expressions and analyzes their emotional state. Based on this analysis, the server adjusts the system's operation and response, taking necessary actions.

[0337] For example, if negative emotions such as anxiety or stress are detected when a user passes through the entrance, the server can use that information to send a notification to the administrator via the terminal. Furthermore, operational tasks can be carried out using emotional information, such as adjusting the cleaning schedule to perform cleaning at a time when users feel most comfortable.

[0338] The sensor device continues to collect environmental information and transmit it to the server. The server analyzes this information in conjunction with the data from the emotion engine and immediately notifies the administrator if an anomaly is detected. The system also has a function to issue an alert if the emotion information acquired by the emotion engine is abnormal.

[0339] Through this series of processes, the system of the present invention not only improves operational efficiency but also enables thoughtful responses that respond to the feelings of residents, making it possible to provide a safe and comfortable living environment even in housing where the population is aging.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The terminal (video acquisition device) captures the face and voice data of users passing through the entrance and prepares to send it to the server.

[0343] Step 2:

[0344] Based on the facial data received by the server, recognition tools are used to compare it with information in the database. Along with confirming the user's well-being, the emotion engine analyzes the user's emotional state in conjunction with the voice data.

[0345] Step 3:

[0346] The server evaluates the matching result, and if authentication is successful, it proceeds to the next process. If the emotion engine analysis determines that the user is experiencing stress, the server creates and sends a notification to the administrator.

[0347] Step 4:

[0348] The server checks the cleaning schedule and sends instructions to adjust the time and area of ​​the cleaning method (cleaning robot) according to the user's emotional state.

[0349] Step 5:

[0350] Based on the instructions received by the terminal (cleaning robot), it begins cleaning work at the specified time and area, and reports its progress to the server.

[0351] Step 6:

[0352] The terminal (sensor device) collects environmental information (temperature, humidity, light intensity, etc.) and transmits the data to the server in real time.

[0353] Step 7:

[0354] The server comprehensively analyzes sensor data and emotional state analysis data, and if an anomaly is detected, it sends a notification to the administrator and displays an alert on the display device.

[0355] Step 8:

[0356] The server aggregates all the data and generates reports periodically. Users can access these reports to check information about their living environment.

[0357] (Example 2)

[0358] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0359] In an aging society, there is a need to achieve efficient environmental management while ensuring the safety and comfort of residents. In particular, the lack of dynamic responses that take into account the emotional state of residents makes it difficult to adjust the environment to meet individual needs. Therefore, technology is needed to analyze emotional states and individually adjust the operation of systems.

[0360] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0361] In this invention, the server includes a decision-making means, an operating means, and an analysis means. This enables the use of emotional information to flexibly adjust the environment according to the residents and improve safety.

[0362] The "determination means" refers to a function that analyzes facial information of an individual obtained through an image acquisition device to confirm the existence of that individual.

[0363] "Operating means" refers to a function that automatically performs environmental maintenance using specific-purpose equipment.

[0364] "Analysis means" refers to a function that analyzes environmental information acquired from multiple detection devices to detect anomalies.

[0365] The "aggregation means" is a function that integrates information obtained from the judgment means, the operation means, and the analysis means to create a list.

[0366] "Analysis tools" refer to functions for analyzing an individual's emotional state and providing data related to environmental adjustments.

[0367] This invention is an environmental control system for improving safety and comfort within living spaces. The system's main purpose is to analyze the emotional state of the residents and dynamically adjust the environment and system operation based on that analysis.

[0368] This system includes multiple elements, the main of which are a video acquisition device, a specific-purpose device, multiple detection devices, and an emotion analysis function. Specifically, the server analyzes facial information obtained from the video acquisition device using a judgment means to confirm the presence of residents. The analysis means continuously monitors environmental information obtained from the detection devices and responds quickly if an anomaly is detected. The specific-purpose device, acting as an operating means, automatically adjusts the environment to support residents in living comfortably. Furthermore, an emotion engine operates as an analysis means, analyzing the resident's voice and facial expressions to determine their emotional state.

[0369] As a concrete example, if an anxious feeling is detected when a user enters the entrance, the server will adjust the lighting to induce a sense of security. Furthermore, if necessary, it will immediately notify the administrator to support the resolution of the problem.

[0370] For example, by inputting a prompt message such as "What is the most effective interface adjustment when a user feels stressed?" into the AI ​​model, it can find effective countermeasures.

[0371] In this way, the system aims to provide advanced life support based on emotional information by combining natural language processing and environmental control.

[0372] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0373] Step 1:

[0374] The terminal uses a video acquisition device to capture facial information of residents in the living space in real time. In this step, facial images of the user as they move become input data. The terminal sends this facial information to a server, which analyzes it using a decision-making tool to confirm the presence of the resident. This analysis results in an output confirming the resident's survival.

[0375] Step 2:

[0376] The server uses an emotion engine to receive the user's voice and facial expression data and analyze their emotional state. The input consists of voice and facial expression data obtained from the user. The server processes this data to determine the user's emotional state (e.g., reassured, stressed, anxious). The output of this step is the analyzed emotional data.

[0377] Step 3:

[0378] The sensor device continuously collects environmental information (temperature, humidity, brightness, etc.) and transmits it to the server. The input is this collected environmental data. The server's analysis system processes the data to determine the current state of the environment and whether or not there are any abnormalities. This process yields an output indicating whether the state is normal or abnormal.

[0379] Step 4:

[0380] The server adjusts the operation of the entire system based on the acquired emotional and environmental data. Specifically, the server controls the operating mechanisms to adjust the brightness of the lighting or operate the air conditioning. The input is the emotional and environmental state data obtained in the previous step. The output is the optimized environmental conditions and the results of the operation.

[0381] Step 5:

[0382] If the server detects an anomaly or a significant emotional state, it sends a notification to the administrator via the terminal. This step uses the anomaly detection and emotion analysis results as input. The output is a notification sent to the administrator, allowing for necessary action.

[0383] Step 6:

[0384] This step uses a generative AI model to obtain suggestions for system adjustments in a specific situation. The input for this step is the prompt "What is the most effective interface adjustment when a user experiences stress?". The AI ​​model analyzes this and proposes the optimal system adjustment method. This proposal is the output.

[0385] Through these steps, the system can flexibly respond to the user's emotions and living environment, providing a comfortable and safe living space.

[0386] (Application Example 2)

[0387] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0388] Current security systems primarily focus on detecting physical anomalies, making it difficult to respond flexibly based on the emotional state of residents and users. As a result, they fail to provide maximum effectiveness in terms of security and comfort. Furthermore, the lack of appropriate support that can immediately respond to changes in users' emotions during emergencies prevents them from providing a sense of security.

[0389] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0390] In this invention, the server includes means for analyzing facial and voice data acquired by a video acquisition device to confirm the survival of individuals and analyze their emotional state, means for automatically performing environmental cleaning using a specialized mechanical device, and means for analyzing environmental information acquired by multiple sensor devices to detect abnormalities and emotional states. This enables security measures and the maintenance of a comfortable environment that take into account the emotional state of residents and users.

[0391] A "video acquisition device" is a device that captures the visual information of an object and acquires it as digital data.

[0392] "Facial data" refers to digital data that represents the characteristics of an individual's face and is used for identification and recognition.

[0393] "Audio data" refers to data that can record audio and be stored or analyzed in digital format.

[0394] "Recognition means" refers to means equipped with the function of identifying a specific individual or analyzing its state based on acquired data.

[0395] A "specialized machine" is a machine with a specialized structure designed to perform a specific purpose or function.

[0396] "Cleaning means" refers to devices or functions that automatically clean the environment.

[0397] A "sensor device" is a device that detects the physical or chemical state of the environment and acquires that information as digital data.

[0398] "Analysis means" refers to methods for processing acquired data and extracting meaningful information.

[0399] "Aggregation means" refers to a device or function used to integrate multiple pieces of information and create reports or perform data analysis.

[0400] "Emotional state" is an indicator that represents an individual's psychological state and can be evaluated based on voice and facial expressions.

[0401] An "emotion engine" is software equipped with algorithms and functions to analyze a user's voice and facial expression data and identify their emotional state.

[0402] A "security agency" is an organization or company that is responsible for safety and security-related operations.

[0403] The system implementing this invention is configured to combine a video acquisition device, an audio acquisition device, a recognition means, a cleaning means, a sensor device, an analysis means, an aggregation means, and an emotion engine. A terminal installed in the user's living environment acquires the user's facial and voice data in real time using the video acquisition device and the audio acquisition device. This data is transmitted to a server, where the recognition means confirms the individual's survival and analyzes their emotional state.

[0404] The server controls cleaning methods that perform environmental cleaning using specialized mechanical equipment, and analysis methods detect anomalies based on various environmental information acquired from sensor devices. The diverse data obtained from these methods are integrated by aggregation methods to create reports, and at the same time, dynamic system responses that take into account the user's emotional state are realized.

[0405] Specifically, when a user is relaxing at home, the system recognizes their emotions and adjusts lighting, music, and other elements to provide a comfortable environment. Furthermore, if the user is experiencing anxiety or stress, the server immediately notifies security agencies for a swift response.

[0406] The hardware used includes smart devices with cameras and microphones with voice input capabilities. Data analysis is performed using software such as Rekognition and Polly from Amazon Web Services (AWS). AWS SNS is used for alarm notifications, enabling real-time notifications to users and administrators.

[0407] A concrete example of a prompt using a generative AI model is: "I want to design an app that identifies emotions from a user's face and voice, and sends a notification to a security company if negative emotions are detected. Therefore, please tell me how to perform emotion analysis using AWS Rekognition and Polly."

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The device acquires the user's face and voice. It captures face and voice data in real time using the camera and microphone, and sends this data to the server as input.

[0411] Step 2:

[0412] The server processes the received facial data using recognition technology. This process analyzes facial features and performs individual identification. The recognition result is obtained as output and passed to the next processing step.

[0413] Step 3:

[0414] The emotion engine on the server analyzes the audio data and evaluates the user's emotional state. It extracts emotions from the tone and rhythm of the voice and outputs the results.

[0415] Step 4:

[0416] The server integrates the recognition methods and the output of the emotion engine to comprehensively evaluate the user's state. It determines the presence or absence of abnormalities or negative emotions and uses the results in the next step.

[0417] Step 5:

[0418] The server analyzes environmental information from sensor devices and detects anomalies. The output is data indicating whether or not there are environmental anomalies. Based on these results, the server determines the necessary actions.

[0419] Step 6:

[0420] The server integrates the results from the previous step using aggregation tools and creates a report. Simultaneously, it sends a notification to the security agency if any anomalies are detected. This step also includes setting up alerts via voice notifications and display devices.

[0421] Step 7:

[0422] The user receives feedback from the server. For example, if the server determines that the user is relaxed, actions are taken to improve the user experience, such as automatically playing soothing music.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] The present invention aims to facilitate the smooth operation of condominium complexes by using an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means.

[0440] First, a terminal (a video acquisition device installed in the entrance) captures the resident's face and sends it to the server. The server uses recognition technology to compare the received face data with a registered database and confirms the resident's well-being. If the result is abnormal, the server notifies the administrator, and the terminal displays an alert.

[0441] In addition, during cleaning operations, the server instructs specialized machinery on the cleaning schedule. The cleaning robots, acting as terminals, clean the designated areas and report their status to the server upon completion. The server records this report in a database and manages it as a cleaning history.

[0442] Furthermore, during inspection and patrol work, terminals (sensor devices) acquire environmental information in real time and transmit the data to the server. The server analyzes this data using analytical tools, and if an anomaly is detected, it immediately notifies the administrator and prompts them to take action.

[0443] For reporting, the server aggregates data obtained from each function using aggregation tools and generates daily or monthly reports. These reports are accessible to users through a web portal, facilitating comparisons with historical data and trend analysis.

[0444] As a concrete example, there is a system in place where a cleaning robot automatically starts cleaning the entrance every morning, and its progress is reported to a server. Users can check the cleaning status and the history of detected anomalies at any time using their smartphones or tablets. This reduces the workload of the management association and allows residents to live with greater peace of mind.

[0445] The following describes the processing flow.

[0446] Step 1:

[0447] The terminal (video acquisition device) captures the resident's face and generates facial image data. It then prepares to send this data to the server.

[0448] Step 2:

[0449] The server queries the received facial image data against a database and compares it with registered facial data. A recognition system is then used to confirm the person's survival.

[0450] Step 3:

[0451] The server evaluates the facial recognition results and determines whether the survival check was successful. If an anomaly is detected, it generates and sends a notification to the administrator.

[0452] Step 4:

[0453] The terminal receives the results of the liveness check from the server, as well as any necessary anomaly notifications, and displays the results on a local display.

[0454] Step 5:

[0455] The server sends a cleaning start command to the cleaning robot based on the cleaning schedule.

[0456] Step 6:

[0457] The terminal (cleaning robot) begins cleaning the designated area and performs the cleaning task. The progress of the work is reported to the server in real time.

[0458] Step 7:

[0459] After cleaning is complete, the cleaning robot sends a status report to the server, which then records it in its database.

[0460] Step 8:

[0461] The terminal (sensor device) acquires environmental information (temperature, humidity, etc.) in real time and transmits it to the server.

[0462] Step 9:

[0463] The server analyzes the sensor data and uses analytical tools to evaluate whether there are any abnormalities. If an abnormality is detected, the administrator is immediately notified.

[0464] Step 10:

[0465] The server aggregates all this data and generates reports periodically. Users can access these reports and view the information through a web portal.

[0466] (Example 1)

[0467] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0468] Modern apartment buildings require improved operational efficiency while ensuring the safety and comfort of residents. However, traditional methods rely heavily on human resources, making efficient management difficult and potentially hindering rapid response in the event of an emergency. Furthermore, centrally managing records of cleaning and periodic inspections is challenging. A system is needed to address these issues and enable more efficient and safer management.

[0469] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0470] In this invention, the server includes recognition means, cleaning means, and analysis means. This enables rapid survival confirmation through automatic recognition of residents' facial information, improved operational efficiency through automation of regular cleaning, and anomaly detection and prediction through real-time analysis of sensor information.

[0471] "Image acquisition means" refers to a device or system installed in a specific location for collecting facial information of a subject.

[0472] "Facial information" refers to distinctive data that enables the identification of a target individual, and includes image data obtained from cameras, etc.

[0473] A "recognition means" is a system that has the function of identifying individuals and confirming their survival using acquired facial information.

[0474] A "cleaning means" is a system that has the function of performing cleaning work in a designated area using automated, specialized machinery and equipment.

[0475] "Specialized machinery and equipment" refers to machinery and equipment designed to suit specific tasks or conditions, and includes those with autonomous cleaning functions.

[0476] "Analysis means" refers to algorithms and systems for analyzing environmental information collected from multiple detection devices and detecting anomalies.

[0477] A "detection device" is a device or system that includes multiple sensors installed to detect various parameters in the environment.

[0478] A "data aggregation system" is a system that integrates information obtained from multiple sources and has the function of supporting analysis and report creation.

[0479] A "verification method" is a system that verifies anomalies based on authentication results using facial information and has the function of notifying administrators or issuing warnings to display devices.

[0480] A "prediction means" is a system that includes calculation methods and algorithms for predicting future anomalies from data collected using analysis means.

[0481] This invention is an integrated management system aimed at the efficient management of apartment buildings. The system includes the following elements:

[0482] First, a video acquisition device installed as a terminal collects facial information of the residents. This device is a high-resolution camera used to accurately capture facial information. The captured data is immediately sent to the server.

[0483] Next, the server receives this facial data and analyzes it using an AI-based recognition system. This system uses a generative AI model trained by a machine learning algorithm, and checks for the person's survival by comparing it with existing data in the database.

[0484] Furthermore, the server manages specialized machinery, specifically autonomous cleaning robots. These robots automatically clean designated areas based on a programmed cleaning schedule. Once cleaning is complete, the robots send a completion notification to the server, which is then recorded as history in the database.

[0485] Furthermore, multiple sensor devices placed on the terminal continuously collect environmental information within the building (temperature, humidity, CO2 concentration, etc.). This information is transmitted to a server, where it is analyzed to check for any abnormalities. If an abnormality is detected, the administrator is immediately notified and prompted to take action.

[0486] The server aggregates the data obtained from these processes and generates periodic reports. This is designed to improve the transparency and efficiency of operations. Users can view the reports via a web portal and perform comparative analysis with historical data.

[0487] As a concrete example, every morning, a cleaning robot automatically starts cleaning the entrance and reports its status to a server upon completion. This report, along with past history, is easily accessible to users via smartphones and tablets.

[0488] An example of a prompt message is, "Please tell me how to collect data for resident facial recognition and cleaning robot status checks, and analyze trends based on recent cleaning history." The goal of operating such a system is to significantly improve the management efficiency of apartment buildings.

[0489] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0490] Step 1:

[0491] The terminal uses a camera installed in the entrance to capture residents' facial information. The input is video data acquired by the camera in real time. As output, the facial information is saved in file format and sent to the server. Specifically, when a resident passes in front of the camera, a trigger is activated and an image for facial recognition is captured.

[0492] Step 2:

[0493] The server analyzes the received facial data using a generating AI model. The input is the facial information sent in step 1. As part of the data processing, the AI ​​model extracts facial feature points and compares them with a database to recognize individuals and verify their survival. The output is the recognition result. Specifically, the AI ​​model identifies patterns in facial features and performs authentication by comparing them with specific individuals.

[0494] Step 3:

[0495] The server checks for anomalies based on the facial recognition results and sends a notification to the administrator if an anomaly is detected. The input is the recognition result obtained in step 2. As part of the data processing, an anomaly detection algorithm detects discrepancies with the normal recognition pattern. The output is a flag indicating the presence or absence of an anomaly and a notification message. Specifically, even if only a slight anomaly is detected, it is immediately reported to the administrator via email or push notification.

[0496] Step 4:

[0497] The server issues cleaning instructions to the specialized cleaning robot based on a pre-set schedule. The input is the cleaning schedule set within the system. During the data calculation process, the schedule is compared with the actual operating status to generate appropriate instructions. The output is a cleaning instruction command. In terms of specific operation, the cleaning robot begins cleaning the designated area according to the schedule.

[0498] Step 5:

[0499] The cleaning robot, acting as the terminal, reports its status to the server after completing the cleaning task. The input is operational data collected by the robot during cleaning. The output is a cleaning status report, which is then sent to the server. Specifically, upon completion of cleaning, the status is automatically updated and recorded as a report.

[0500] Step 6:

[0501] The sensor device installed at the terminal continuously collects environmental information within the building and transmits it to the server. Inputs include sensor data such as temperature, humidity, and CO2 concentration. Outputs are environmental information data. Specifically, the sensor collects various data at set intervals and transmits it in real time.

[0502] Step 7:

[0503] The server analyzes sensor information in real time and detects anomalies. The input is the environmental information data transmitted in step 6. During data processing, an analysis algorithm compares the data to standard values ​​and detects anomalies. The output is the anomaly detection result. Specifically, when an anomaly occurs, a procedure is initiated that prompts immediate corrective action.

[0504] Step 8:

[0505] The server aggregates all data and generates daily or monthly reports. Inputs include historical facial recognition, cleaning data, and sensor data. Output is a detailed report accessible to the user. Its specific operation involves retrieving historical information from the database and outputting it as a visually easy-to-understand report.

[0506] (Application Example 1)

[0507] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0508] In condominium complexes and public facilities, ensuring security and streamlining operations are crucial challenges. In particular, verifying residents and visitors, managing environmental cleanliness, and promptly detecting and responding to anomalies are required. However, performing these tasks manually is labor-intensive and hinders quick responses. Therefore, efficient and automated systems are needed.

[0509] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0510] In this invention, the server includes identification means for analyzing facial data acquired by a video acquisition device to confirm the recognized individual, maintenance means for automatically performing environmental maintenance using specialized mechanical devices, and processing means for processing environmental information acquired by multiple sensor devices to detect anomalies. This enables improved security levels and more efficient environmental management.

[0511] A "video acquisition device" is a device used to acquire the appearance of a target location or individual as digital data.

[0512] "Identification means" refers to methods or devices that analyze acquired data and have the function of identifying and confirming individuals based on specific characteristics.

[0513] A "specialized machine" is a machine specifically designed to efficiently perform a particular task or operation.

[0514] "Maintenance means" refers to mechanisms and methods for automatically performing environmental maintenance and cleaning using specialized machinery and equipment.

[0515] A "sensor device" is a device used to measure information about the surrounding environment and to electronically acquire that data.

[0516] "Processing means" refers to a method or system used to analyze acquired information and identify anomalies.

[0517] An "aggregation method" is a system or method for integrating and centrally managing information obtained from different sources.

[0518] A "communication means" is a system that has the function of communication and information provision to notify relevant parties of detected anomalies.

[0519] In this invention, the server provides a system that automates security and environmental management in condominium complexes using a video acquisition device, specialized mechanical device, and sensor device. The server first collects facial data using the video acquisition device and verifies individuals using identification means. This process uses facial recognition software such as Amazon Rekognition or Microsoft Face API for analysis.

[0520] Next, specialized machinery and equipment are used to automatically prepare the environment through maintenance methods, and a server monitors the status of the maintenance work. This maintenance is achieved by using cleaning robots such as Roomba to clean designated areas.

[0521] Furthermore, environmental information collected by sensor devices is analyzed in real time by processing devices, and if an anomaly is detected, the server immediately sends a warning to the administrator. AWS IoT Core and Google Cloud IoT are used for processing, and AWS Kinesis Data Analytics and other real-time analysis tools are used for rapid data processing.

[0522] Users can receive this information via a smartphone app, whether at home or on the go, and check the status of security and environmental improvements. For example, it is possible to set up a system that immediately sends a push notification if a stranger enters the property.

[0523] As a concrete example, an example of a prompt statement is "Generate a prompt regarding a system for detecting anomalies in common areas of a condominium complex and notifying administrators." In this way, the objective of this invention is to provide an environment in which residents and administrators can live with peace of mind on a sustainable basis.

[0524] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0525] Step 1:

[0526] The server receives facial data from the video acquisition device. The acquired facial data is compared against the registration database using facial recognition software such as Amazon Rekognition or Microsoft Face API. The input for checking for anomalies is the facial data, and the output is the authentication result of the identified individual. Based on this, the server checks whether the authenticated individual has been registered without any problems.

[0527] Step 2:

[0528] The server sends maintenance work commands to cleaning robots, which are specialized mechanical devices. These commands are scheduled and perform cleaning to maintain the environment of the designated area. Inputs are the cleaning schedule and area information, and output is the completion status of the cleaning work. The server monitors the progress of the maintenance work in real time and records it in the database upon completion.

[0529] Step 3:

[0530] The sensor device, acting as a terminal, collects environmental sensor data and transmits it to the server. The server ingests this data using AWS IoT Core and performs analysis using AWS Kinesis Data Analytics, etc. The input is environmental sensor data, and the output is the analysis results regarding the presence or absence of anomalies. Based on these analysis results, the server detects anomalies in real time.

[0531] Step 4:

[0532] When an anomaly is detected, the server sends an alert to administrators and relevant parties using a communication method. Firebase Cloud Messaging (FCM) is used to immediately send push notifications to smartphones or compatible devices. The input is the analysis result of the anomaly detection, and the output is the alert notification to administrators. This enables a rapid response.

[0533] Step 5:

[0534] Users can check the security status and maintenance status of their environment in real time through a smartphone app. Users can receive notifications and take necessary actions accordingly. The input to this process is notifications from the server, and the output is information provided to the user. This allows users to understand the situation and go about their daily lives with peace of mind.

[0535] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0536] The system of the present invention comprises an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means, as well as an emotion engine that recognizes the user's emotions. This emotion engine can receive the user's voice and facial expression data and analyze their emotional state. The system uses this data to dynamically adjust its operation in order to improve the comfort and safety of the living environment.

[0537] First, a terminal (video acquisition device) captures the resident's face and sends it to the server. The recognition system on the server uses this data to confirm the resident's well-being. In parallel, an emotion engine recognizes the user's voice and facial expressions and analyzes their emotional state. Based on this analysis, the server adjusts the system's operation and response, taking necessary actions.

[0538] For example, if negative emotions such as anxiety or stress are detected when a user passes through the entrance, the server can use that information to send a notification to the administrator via the terminal. Furthermore, operational tasks can be carried out using emotional information, such as adjusting the cleaning schedule to perform cleaning at a time when users feel most comfortable.

[0539] The sensor device continues to collect environmental information and transmit it to the server. The server analyzes this information in conjunction with the data from the emotion engine and immediately notifies the administrator if an anomaly is detected. The system also has a function to issue an alert if the emotion information acquired by the emotion engine is abnormal.

[0540] Through this series of processes, the system of the present invention not only improves operational efficiency but also enables thoughtful responses that respond to the feelings of residents, making it possible to provide a safe and comfortable living environment even in housing where the population is aging.

[0541] The following describes the processing flow.

[0542] Step 1:

[0543] The terminal (video acquisition device) captures the face and voice data of users passing through the entrance and prepares to send it to the server.

[0544] Step 2:

[0545] Based on the facial data received by the server, recognition tools are used to compare it with information in the database. Along with confirming the user's well-being, the emotion engine analyzes the user's emotional state in conjunction with the voice data.

[0546] Step 3:

[0547] The server evaluates the matching result, and if authentication is successful, it proceeds to the next process. If the emotion engine analysis determines that the user is experiencing stress, the server creates and sends a notification to the administrator.

[0548] Step 4:

[0549] The server checks the cleaning schedule and sends instructions to adjust the time and area of ​​the cleaning method (cleaning robot) according to the user's emotional state.

[0550] Step 5:

[0551] Based on the instructions received by the terminal (cleaning robot), it begins cleaning work at the specified time and area, and reports its progress to the server.

[0552] Step 6:

[0553] The terminal (sensor device) collects environmental information (temperature, humidity, light intensity, etc.) and transmits the data to the server in real time.

[0554] Step 7:

[0555] The server comprehensively analyzes sensor data and emotional state analysis data, and if an anomaly is detected, it sends a notification to the administrator and displays an alert on the display device.

[0556] Step 8:

[0557] The server aggregates all the data and generates reports periodically. Users can access these reports to check information about their living environment.

[0558] (Example 2)

[0559] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] In an aging society, there is a need to achieve efficient environmental management while ensuring the safety and comfort of residents. In particular, the lack of dynamic responses that take into account the emotional state of residents makes it difficult to adjust the environment to meet individual needs. Therefore, technology is needed to analyze emotional states and individually adjust the operation of systems.

[0561] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0562] In this invention, the server includes a decision-making means, an operating means, and an analysis means. This enables the use of emotional information to flexibly adjust the environment according to the residents and improve safety.

[0563] The "determination means" refers to a function that analyzes facial information of an individual obtained through an image acquisition device to confirm the existence of that individual.

[0564] "Operating means" refers to a function that automatically performs environmental maintenance using specific-purpose equipment.

[0565] "Analysis means" refers to a function that analyzes environmental information acquired from multiple detection devices to detect anomalies.

[0566] The "aggregation means" is a function that integrates information obtained from the judgment means, the operation means, and the analysis means to create a list.

[0567] "Analysis tools" refer to functions for analyzing an individual's emotional state and providing data related to environmental adjustments.

[0568] This invention is an environmental control system for improving safety and comfort within living spaces. The system's main purpose is to analyze the emotional state of the residents and dynamically adjust the environment and system operation based on that analysis.

[0569] This system includes multiple elements, the main of which are a video acquisition device, a specific-purpose device, multiple detection devices, and an emotion analysis function. Specifically, the server analyzes facial information obtained from the video acquisition device using a judgment means to confirm the presence of residents. The analysis means continuously monitors environmental information obtained from the detection devices and responds quickly if an anomaly is detected. The specific-purpose device, acting as an operating means, automatically adjusts the environment to support residents in living comfortably. Furthermore, an emotion engine operates as an analysis means, analyzing the resident's voice and facial expressions to determine their emotional state.

[0570] As a concrete example, if an anxious feeling is detected when a user enters the entrance, the server will adjust the lighting to induce a sense of security. Furthermore, if necessary, it will immediately notify the administrator to support the resolution of the problem.

[0571] For example, by inputting a prompt message such as "What is the most effective interface adjustment when a user feels stressed?" into the AI ​​model, it can find effective countermeasures.

[0572] In this way, the system aims to provide advanced life support based on emotional information by combining natural language processing and environmental control.

[0573] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0574] Step 1:

[0575] The terminal uses a video acquisition device to capture facial information of residents in the living space in real time. In this step, facial images of the user as they move become input data. The terminal sends this facial information to a server, which analyzes it using a decision-making tool to confirm the presence of the resident. This analysis results in an output confirming the resident's survival.

[0576] Step 2:

[0577] The server uses an emotion engine to receive the user's voice and facial expression data and analyze their emotional state. The input consists of voice and facial expression data obtained from the user. The server processes this data to determine the user's emotional state (e.g., reassured, stressed, anxious). The output of this step is the analyzed emotional data.

[0578] Step 3:

[0579] The sensor device continuously collects environmental information (temperature, humidity, brightness, etc.) and transmits it to the server. The input is this collected environmental data. The server's analysis system processes the data to determine the current state of the environment and whether or not there are any abnormalities. This process yields an output indicating whether the state is normal or abnormal.

[0580] Step 4:

[0581] The server adjusts the operation of the entire system based on the acquired emotional and environmental data. Specifically, the server controls the operating mechanisms to adjust the brightness of the lighting or operate the air conditioning. The input is the emotional and environmental state data obtained in the previous step. The output is the optimized environmental conditions and the results of the operation.

[0582] Step 5:

[0583] If the server detects an anomaly or a significant emotional state, it sends a notification to the administrator via the terminal. This step uses the anomaly detection and emotion analysis results as input. The output is a notification sent to the administrator, allowing for necessary action.

[0584] Step 6:

[0585] This step uses a generative AI model to obtain suggestions for system adjustments in a specific situation. The input for this step is the prompt "What is the most effective interface adjustment when a user experiences stress?". The AI ​​model analyzes this and proposes the optimal system adjustment method. This proposal is the output.

[0586] Through these steps, the system can flexibly respond to the user's emotions and living environment, providing a comfortable and safe living space.

[0587] (Application Example 2)

[0588] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] Current security systems primarily focus on detecting physical anomalies, making it difficult to respond flexibly based on the emotional state of residents and users. As a result, they fail to provide maximum effectiveness in terms of security and comfort. Furthermore, the lack of appropriate support that can immediately respond to changes in users' emotions during emergencies prevents them from providing a sense of security.

[0590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0591] In this invention, the server includes means for analyzing facial and voice data acquired by a video acquisition device to confirm the survival of individuals and analyze their emotional state, means for automatically performing environmental cleaning using a specialized mechanical device, and means for analyzing environmental information acquired by multiple sensor devices to detect abnormalities and emotional states. This enables security measures and the maintenance of a comfortable environment that take into account the emotional state of residents and users.

[0592] A "video acquisition device" is a device that captures the visual information of an object and acquires it as digital data.

[0593] "Facial data" refers to digital data that represents the characteristics of an individual's face and is used for identification and recognition.

[0594] "Audio data" refers to data that can record audio and be stored or analyzed in digital format.

[0595] "Recognition means" refers to means equipped with the function of identifying a specific individual or analyzing its state based on acquired data.

[0596] A "specialized machine" is a machine with a specialized structure designed to perform a specific purpose or function.

[0597] "Cleaning means" refers to devices or functions that automatically clean the environment.

[0598] A "sensor device" is a device that detects the physical or chemical state of the environment and acquires that information as digital data.

[0599] "Analysis means" refers to methods for processing acquired data and extracting meaningful information.

[0600] "Aggregation means" refers to a device or function used to integrate multiple pieces of information and create reports or perform data analysis.

[0601] "Emotional state" is an indicator that represents an individual's psychological state and can be evaluated based on voice and facial expressions.

[0602] An "emotion engine" is software equipped with algorithms and functions to analyze a user's voice and facial expression data and identify their emotional state.

[0603] A "security agency" is an organization or company that is responsible for safety and security-related operations.

[0604] The system implementing this invention is configured to combine a video acquisition device, an audio acquisition device, a recognition means, a cleaning means, a sensor device, an analysis means, an aggregation means, and an emotion engine. A terminal installed in the user's living environment acquires the user's facial and voice data in real time using the video acquisition device and the audio acquisition device. This data is transmitted to a server, where the recognition means confirms the individual's survival and analyzes their emotional state.

[0605] The server controls cleaning methods that perform environmental cleaning using specialized mechanical equipment, and analysis methods detect anomalies based on various environmental information acquired from sensor devices. The diverse data obtained from these methods are integrated by aggregation methods to create reports, and at the same time, dynamic system responses that take into account the user's emotional state are realized.

[0606] Specifically, when a user is relaxing at home, the system recognizes their emotions and adjusts lighting, music, and other elements to provide a comfortable environment. Furthermore, if the user is experiencing anxiety or stress, the server immediately notifies security agencies for a swift response.

[0607] The hardware used includes smart devices with cameras and microphones with voice input capabilities. Data analysis is performed using software such as Rekognition and Polly from Amazon Web Services (AWS). AWS SNS is used for alarm notifications, enabling real-time notifications to users and administrators.

[0608] A concrete example of a prompt using a generative AI model is: "I want to design an app that identifies emotions from a user's face and voice, and sends a notification to a security company if negative emotions are detected. Therefore, please tell me how to perform emotion analysis using AWS Rekognition and Polly."

[0609] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0610] Step 1:

[0611] The device acquires the user's face and voice. It captures face and voice data in real time using the camera and microphone, and sends this data to the server as input.

[0612] Step 2:

[0613] The server processes the received facial data using recognition technology. This process analyzes facial features and performs individual identification. The recognition result is obtained as output and passed to the next processing step.

[0614] Step 3:

[0615] The emotion engine on the server analyzes the audio data and evaluates the user's emotional state. It extracts emotions from the tone and rhythm of the voice and outputs the results.

[0616] Step 4:

[0617] The server integrates the recognition methods and the output of the emotion engine to comprehensively evaluate the user's state. It determines the presence or absence of abnormalities or negative emotions and uses the results in the next step.

[0618] Step 5:

[0619] The server analyzes environmental information from sensor devices and detects anomalies. The output is data indicating whether or not there are environmental anomalies. Based on these results, the server determines the necessary actions.

[0620] Step 6:

[0621] The server integrates the results from the previous step using aggregation tools and creates a report. Simultaneously, it sends a notification to the security agency if any anomalies are detected. This step also includes setting up alerts via voice notifications and display devices.

[0622] Step 7:

[0623] The user receives feedback from the server. For example, if the server determines that the user is relaxed, actions are taken to improve the user experience, such as automatically playing soothing music.

[0624] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0625] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0627] [Fourth Embodiment]

[0628] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0629] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0630] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0632] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0634] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0635] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0636] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0637] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0640] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] The present invention aims to facilitate the smooth operation of condominium complexes by using an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means.

[0642] First, a terminal (a video acquisition device installed in the entrance) captures the resident's face and sends it to the server. The server uses recognition technology to compare the received face data with a registered database and confirms the resident's well-being. If the result is abnormal, the server notifies the administrator, and the terminal displays an alert.

[0643] In addition, during cleaning operations, the server instructs specialized machinery on the cleaning schedule. The cleaning robots, acting as terminals, clean the designated areas and report their status to the server upon completion. The server records this report in a database and manages it as a cleaning history.

[0644] Furthermore, during inspection and patrol work, terminals (sensor devices) acquire environmental information in real time and transmit the data to the server. The server analyzes this data using analytical tools, and if an anomaly is detected, it immediately notifies the administrator and prompts them to take action.

[0645] For reporting, the server aggregates data obtained from each function using aggregation tools and generates daily or monthly reports. These reports are accessible to users through a web portal, facilitating comparisons with historical data and trend analysis.

[0646] As a concrete example, there is a system in place where a cleaning robot automatically starts cleaning the entrance every morning, and its progress is reported to a server. Users can check the cleaning status and the history of detected anomalies at any time using their smartphones or tablets. This reduces the workload of the management association and allows residents to live with greater peace of mind.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The terminal (video acquisition device) captures the resident's face and generates facial image data. It then prepares to send this data to the server.

[0650] Step 2:

[0651] The server queries the received facial image data against a database and compares it with registered facial data. A recognition system is then used to confirm the person's survival.

[0652] Step 3:

[0653] The server evaluates the facial recognition results and determines whether the survival check was successful. If an anomaly is detected, it generates and sends a notification to the administrator.

[0654] Step 4:

[0655] The terminal receives the results of the liveness check from the server, as well as any necessary anomaly notifications, and displays the results on a local display.

[0656] Step 5:

[0657] The server sends a cleaning start command to the cleaning robot based on the cleaning schedule.

[0658] Step 6:

[0659] The terminal (cleaning robot) begins cleaning the designated area and performs the cleaning task. The progress of the work is reported to the server in real time.

[0660] Step 7:

[0661] After cleaning is complete, the cleaning robot sends a status report to the server, which then records it in its database.

[0662] Step 8:

[0663] The terminal (sensor device) acquires environmental information (temperature, humidity, etc.) in real time and transmits it to the server.

[0664] Step 9:

[0665] The server analyzes the sensor data and uses analytical tools to evaluate whether there are any abnormalities. If an abnormality is detected, the administrator is immediately notified.

[0666] Step 10:

[0667] The server aggregates all this data and generates reports periodically. Users can access these reports and view the information through a web portal.

[0668] (Example 1)

[0669] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] Modern apartment buildings require improved operational efficiency while ensuring the safety and comfort of residents. However, traditional methods rely heavily on human resources, making efficient management difficult and potentially hindering rapid response in the event of an emergency. Furthermore, centrally managing records of cleaning and periodic inspections is challenging. A system is needed to address these issues and enable more efficient and safer management.

[0671] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0672] In this invention, the server includes recognition means, cleaning means, and analysis means. This enables rapid survival confirmation through automatic recognition of residents' facial information, improved operational efficiency through automation of regular cleaning, and anomaly detection and prediction through real-time analysis of sensor information.

[0673] "Image acquisition means" refers to a device or system installed in a specific location for collecting facial information of a subject.

[0674] "Facial information" refers to distinctive data that enables the identification of a target individual, and includes image data obtained from cameras, etc.

[0675] A "recognition means" is a system that has the function of identifying individuals and confirming their survival using acquired facial information.

[0676] A "cleaning means" is a system that has the function of performing cleaning work in a designated area using automated, specialized machinery and equipment.

[0677] "Specialized machinery and equipment" refers to machinery and equipment designed to suit specific tasks or conditions, and includes those with autonomous cleaning functions.

[0678] "Analysis means" refers to algorithms and systems for analyzing environmental information collected from multiple detection devices and detecting anomalies.

[0679] A "detection device" is a device or system that includes multiple sensors installed to detect various parameters in the environment.

[0680] A "data aggregation system" is a system that integrates information obtained from multiple sources and has the function of supporting analysis and report creation.

[0681] A "verification method" is a system that verifies anomalies based on authentication results using facial information and has the function of notifying administrators or issuing warnings to display devices.

[0682] A "prediction means" is a system that includes calculation methods and algorithms for predicting future anomalies from data collected using analysis means.

[0683] This invention is an integrated management system aimed at the efficient management of apartment buildings. The system includes the following elements:

[0684] First, a video acquisition device installed as a terminal collects facial information of the residents. This device is a high-resolution camera used to accurately capture facial information. The captured data is immediately sent to the server.

[0685] Next, the server receives this facial data and analyzes it using an AI-based recognition system. This system uses a generative AI model trained by a machine learning algorithm, and checks for the person's survival by comparing it with existing data in the database.

[0686] Furthermore, the server manages specialized machinery, specifically autonomous cleaning robots. These robots automatically clean designated areas based on a programmed cleaning schedule. Once cleaning is complete, the robots send a completion notification to the server, which is then recorded as history in the database.

[0687] Furthermore, multiple sensor devices placed on the terminal continuously collect environmental information within the building (temperature, humidity, CO2 concentration, etc.). This information is transmitted to a server, where it is analyzed to check for any abnormalities. If an abnormality is detected, the administrator is immediately notified and prompted to take action.

[0688] The server aggregates the data obtained from these processes and generates periodic reports. This is designed to improve the transparency and efficiency of operations. Users can view the reports via a web portal and perform comparative analysis with historical data.

[0689] As a concrete example, every morning, a cleaning robot automatically starts cleaning the entrance and reports its status to a server upon completion. This report, along with past history, is easily accessible to users via smartphones and tablets.

[0690] An example of a prompt message is, "Please tell me how to collect data for resident facial recognition and cleaning robot status checks, and analyze trends based on recent cleaning history." The goal of operating such a system is to significantly improve the management efficiency of apartment buildings.

[0691] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0692] Step 1:

[0693] The terminal uses a camera installed in the entrance to capture residents' facial information. The input is video data acquired by the camera in real time. As output, the facial information is saved in file format and sent to the server. Specifically, when a resident passes in front of the camera, a trigger is activated and an image for facial recognition is captured.

[0694] Step 2:

[0695] The server analyzes the received facial data using a generating AI model. The input is the facial information sent in step 1. As part of the data processing, the AI ​​model extracts facial feature points and compares them with a database to recognize individuals and verify their survival. The output is the recognition result. Specifically, the AI ​​model identifies patterns in facial features and performs authentication by comparing them with specific individuals.

[0696] Step 3:

[0697] The server checks for anomalies based on the facial recognition results and sends a notification to the administrator if an anomaly is detected. The input is the recognition result obtained in step 2. As part of the data processing, an anomaly detection algorithm detects discrepancies with the normal recognition pattern. The output is a flag indicating the presence or absence of an anomaly and a notification message. Specifically, even if only a slight anomaly is detected, it is immediately reported to the administrator via email or push notification.

[0698] Step 4:

[0699] The server issues cleaning instructions to the specialized cleaning robot based on a pre-set schedule. The input is the cleaning schedule set within the system. During the data calculation process, the schedule is compared with the actual operating status to generate appropriate instructions. The output is a cleaning instruction command. In terms of specific operation, the cleaning robot begins cleaning the designated area according to the schedule.

[0700] Step 5:

[0701] The cleaning robot, acting as the terminal, reports its status to the server after completing the cleaning task. The input is operational data collected by the robot during cleaning. The output is a cleaning status report, which is then sent to the server. Specifically, upon completion of cleaning, the status is automatically updated and recorded as a report.

[0702] Step 6:

[0703] The sensor device installed at the terminal continuously collects environmental information within the building and transmits it to the server. Inputs include sensor data such as temperature, humidity, and CO2 concentration. Outputs are environmental information data. Specifically, the sensor collects various data at set intervals and transmits it in real time.

[0704] Step 7:

[0705] The server analyzes sensor information in real time and detects anomalies. The input is the environmental information data transmitted in step 6. During data processing, an analysis algorithm compares the data to standard values ​​and detects anomalies. The output is the anomaly detection result. Specifically, when an anomaly occurs, a procedure is initiated that prompts immediate corrective action.

[0706] Step 8:

[0707] The server aggregates all data and generates daily or monthly reports. Inputs include historical facial recognition, cleaning data, and sensor data. Output is a detailed report accessible to the user. Its specific operation involves retrieving historical information from the database and outputting it as a visually easy-to-understand report.

[0708] (Application Example 1)

[0709] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0710] In condominium complexes and public facilities, ensuring security and streamlining operations are crucial challenges. In particular, verifying residents and visitors, managing environmental cleanliness, and promptly detecting and responding to anomalies are required. However, performing these tasks manually is labor-intensive and hinders quick responses. Therefore, efficient and automated systems are needed.

[0711] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0712] In this invention, the server includes identification means for analyzing facial data acquired by a video acquisition device to confirm the recognized individual, maintenance means for automatically performing environmental maintenance using specialized mechanical devices, and processing means for processing environmental information acquired by multiple sensor devices to detect anomalies. This enables improved security levels and more efficient environmental management.

[0713] A "video acquisition device" is a device used to acquire the appearance of a target location or individual as digital data.

[0714] "Identification means" refers to methods or devices that analyze acquired data and have the function of identifying and confirming individuals based on specific characteristics.

[0715] A "specialized machine" is a machine specifically designed to efficiently perform a particular task or operation.

[0716] "Maintenance means" refers to mechanisms and methods for automatically performing environmental maintenance and cleaning using specialized machinery and equipment.

[0717] A "sensor device" is a device used to measure information about the surrounding environment and to electronically acquire that data.

[0718] "Processing means" refers to a method or system used to analyze acquired information and identify anomalies.

[0719] An "aggregation method" is a system or method for integrating and centrally managing information obtained from different sources.

[0720] A "communication means" is a system that has the function of communication and information provision to notify relevant parties of detected anomalies.

[0721] In this invention, the server provides a system that automates security and environmental management in condominium complexes using a video acquisition device, specialized mechanical device, and sensor device. The server first collects facial data using the video acquisition device and verifies individuals using identification means. This process uses facial recognition software such as Amazon Rekognition or Microsoft Face API for analysis.

[0722] Next, specialized machinery and equipment are used to automatically prepare the environment through maintenance methods, and a server monitors the status of the maintenance work. This maintenance is achieved by using cleaning robots such as Roomba to clean designated areas.

[0723] Furthermore, environmental information collected by sensor devices is analyzed in real time by processing devices, and if an anomaly is detected, the server immediately sends a warning to the administrator. AWS IoT Core and Google Cloud IoT are used for processing, and AWS Kinesis Data Analytics and other real-time analysis tools are used for rapid data processing.

[0724] Users can receive this information via a smartphone app, whether at home or on the go, and check the status of security and environmental improvements. For example, it is possible to set up a system that immediately sends a push notification if a stranger enters the property.

[0725] As a concrete example, an example of a prompt statement is "Generate a prompt regarding a system for detecting anomalies in common areas of a condominium complex and notifying administrators." In this way, the objective of this invention is to provide an environment in which residents and administrators can live with peace of mind on a sustainable basis.

[0726] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0727] Step 1:

[0728] The server receives facial data from the video acquisition device. The acquired facial data is compared against the registration database using facial recognition software such as Amazon Rekognition or Microsoft Face API. The input for checking for anomalies is the facial data, and the output is the authentication result of the identified individual. Based on this, the server checks whether the authenticated individual has been registered without any problems.

[0729] Step 2:

[0730] The server sends maintenance work commands to cleaning robots, which are specialized mechanical devices. These commands are scheduled and perform cleaning to maintain the environment of the designated area. Inputs are the cleaning schedule and area information, and output is the completion status of the cleaning work. The server monitors the progress of the maintenance work in real time and records it in the database upon completion.

[0731] Step 3:

[0732] The sensor device, acting as a terminal, collects environmental sensor data and transmits it to the server. The server ingests this data using AWS IoT Core and performs analysis using AWS Kinesis Data Analytics, etc. The input is environmental sensor data, and the output is the analysis results regarding the presence or absence of anomalies. Based on these analysis results, the server detects anomalies in real time.

[0733] Step 4:

[0734] When an anomaly is detected, the server sends an alert to administrators and relevant parties using a communication method. Firebase Cloud Messaging (FCM) is used to immediately send push notifications to smartphones or compatible devices. The input is the analysis result of the anomaly detection, and the output is the alert notification to administrators. This enables a rapid response.

[0735] Step 5:

[0736] Users can check the security status and maintenance status of their environment in real time through a smartphone app. Users can receive notifications and take necessary actions accordingly. The input to this process is notifications from the server, and the output is information provided to the user. This allows users to understand the situation and go about their daily lives with peace of mind.

[0737] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0738] The system of the present invention comprises an image acquisition device, recognition means, specialized mechanical device, cleaning means, sensor device, analysis means, and aggregation means, as well as an emotion engine that recognizes the user's emotions. This emotion engine can receive the user's voice and facial expression data and analyze their emotional state. The system uses this data to dynamically adjust its operation in order to improve the comfort and safety of the living environment.

[0739] First, a terminal (video acquisition device) captures the resident's face and sends it to the server. The recognition system on the server uses this data to confirm the resident's well-being. In parallel, an emotion engine recognizes the user's voice and facial expressions and analyzes their emotional state. Based on this analysis, the server adjusts the system's operation and response, taking necessary actions.

[0740] For example, if negative emotions such as anxiety or stress are detected when a user passes through the entrance, the server can use that information to send a notification to the administrator via the terminal. Furthermore, operational tasks can be carried out using emotional information, such as adjusting the cleaning schedule to perform cleaning at a time when users feel most comfortable.

[0741] The sensor device continues to collect environmental information and transmit it to the server. The server analyzes this information in conjunction with the data from the emotion engine and immediately notifies the administrator if an anomaly is detected. The system also has a function to issue an alert if the emotion information acquired by the emotion engine is abnormal.

[0742] Through this series of processes, the system of the present invention not only improves operational efficiency but also enables thoughtful responses that respond to the feelings of residents, making it possible to provide a safe and comfortable living environment even in housing where the population is aging.

[0743] The following describes the processing flow.

[0744] Step 1:

[0745] The terminal (video acquisition device) captures the face and voice data of users passing through the entrance and prepares to send it to the server.

[0746] Step 2:

[0747] Based on the facial data received by the server, recognition tools are used to compare it with information in the database. Along with confirming the user's well-being, the emotion engine analyzes the user's emotional state in conjunction with the voice data.

[0748] Step 3:

[0749] The server evaluates the matching result, and if authentication is successful, it proceeds to the next process. If the emotion engine analysis determines that the user is experiencing stress, the server creates and sends a notification to the administrator.

[0750] Step 4:

[0751] The server checks the cleaning schedule and sends instructions to adjust the time and area of ​​the cleaning method (cleaning robot) according to the user's emotional state.

[0752] Step 5:

[0753] Based on the instructions received by the terminal (cleaning robot), it begins cleaning work at the specified time and area, and reports its progress to the server.

[0754] Step 6:

[0755] The terminal (sensor device) collects environmental information (temperature, humidity, light intensity, etc.) and transmits the data to the server in real time.

[0756] Step 7:

[0757] The server comprehensively analyzes sensor data and emotional state analysis data, and if an anomaly is detected, it sends a notification to the administrator and displays an alert on the display device.

[0758] Step 8:

[0759] The server aggregates all the data and generates reports periodically. Users can access these reports to check information about their living environment.

[0760] (Example 2)

[0761] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] In an aging society, there is a need to achieve efficient environmental management while ensuring the safety and comfort of residents. In particular, the lack of dynamic responses that take into account the emotional state of residents makes it difficult to adjust the environment to meet individual needs. Therefore, technology is needed to analyze emotional states and individually adjust the operation of systems.

[0763] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0764] In this invention, the server includes a decision-making means, an operating means, and an analysis means. This enables the use of emotional information to flexibly adjust the environment according to the residents and improve safety.

[0765] The "determination means" refers to a function that analyzes facial information of an individual obtained through an image acquisition device to confirm the existence of that individual.

[0766] "Operating means" refers to a function that automatically performs environmental maintenance using specific-purpose equipment.

[0767] "Analysis means" refers to a function that analyzes environmental information acquired from multiple detection devices to detect anomalies.

[0768] The "aggregation means" is a function that integrates information obtained from the judgment means, the operation means, and the analysis means to create a list.

[0769] "Analysis tools" refer to functions for analyzing an individual's emotional state and providing data related to environmental adjustments.

[0770] This invention is an environmental control system for improving safety and comfort within living spaces. The system's main purpose is to analyze the emotional state of the residents and dynamically adjust the environment and system operation based on that analysis.

[0771] This system includes multiple elements, the main of which are a video acquisition device, a specific-purpose device, multiple detection devices, and an emotion analysis function. Specifically, the server analyzes facial information obtained from the video acquisition device using a judgment means to confirm the presence of residents. The analysis means continuously monitors environmental information obtained from the detection devices and responds quickly if an anomaly is detected. The specific-purpose device, acting as an operating means, automatically adjusts the environment to support residents in living comfortably. Furthermore, an emotion engine operates as an analysis means, analyzing the resident's voice and facial expressions to determine their emotional state.

[0772] As a concrete example, if an anxious feeling is detected when a user enters the entrance, the server will adjust the lighting to induce a sense of security. Furthermore, if necessary, it will immediately notify the administrator to support the resolution of the problem.

[0773] For example, by inputting a prompt message such as "What is the most effective interface adjustment when a user feels stressed?" into the AI ​​model, it can find effective countermeasures.

[0774] In this way, the system aims to provide advanced life support based on emotional information by combining natural language processing and environmental control.

[0775] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0776] Step 1:

[0777] The terminal uses a video acquisition device to capture facial information of residents in the living space in real time. In this step, facial images of the user as they move become input data. The terminal sends this facial information to a server, which analyzes it using a decision-making tool to confirm the presence of the resident. This analysis results in an output confirming the resident's survival.

[0778] Step 2:

[0779] The server uses an emotion engine to receive the user's voice and facial expression data and analyze their emotional state. The input consists of voice and facial expression data obtained from the user. The server processes this data to determine the user's emotional state (e.g., reassured, stressed, anxious). The output of this step is the analyzed emotional data.

[0780] Step 3:

[0781] The sensor device continuously collects environmental information (temperature, humidity, brightness, etc.) and transmits it to the server. The input is this collected environmental data. The server's analysis system processes the data to determine the current state of the environment and whether or not there are any abnormalities. This process yields an output indicating whether the state is normal or abnormal.

[0782] Step 4:

[0783] The server adjusts the operation of the entire system based on the acquired emotional and environmental data. Specifically, the server controls the operating mechanisms to adjust the brightness of the lighting or operate the air conditioning. The input is the emotional and environmental state data obtained in the previous step. The output is the optimized environmental conditions and the results of the operation.

[0784] Step 5:

[0785] If the server detects an anomaly or a significant emotional state, it sends a notification to the administrator via the terminal. This step uses the anomaly detection and emotion analysis results as input. The output is a notification sent to the administrator, allowing for necessary action.

[0786] Step 6:

[0787] This step uses a generative AI model to obtain suggestions for system adjustments in a specific situation. The input for this step is the prompt "What is the most effective interface adjustment when a user experiences stress?". The AI ​​model analyzes this and proposes the optimal system adjustment method. This proposal is the output.

[0788] Through these steps, the system can flexibly respond to the user's emotions and living environment, providing a comfortable and safe living space.

[0789] (Application Example 2)

[0790] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0791] Current security systems primarily focus on detecting physical anomalies, making it difficult to respond flexibly based on the emotional state of residents and users. As a result, they fail to provide maximum effectiveness in terms of security and comfort. Furthermore, the lack of appropriate support that can immediately respond to changes in users' emotions during emergencies prevents them from providing a sense of security.

[0792] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0793] In this invention, the server includes means for analyzing facial and voice data acquired by a video acquisition device to confirm the survival of individuals and analyze their emotional state, means for automatically performing environmental cleaning using a specialized mechanical device, and means for analyzing environmental information acquired by multiple sensor devices to detect abnormalities and emotional states. This enables security measures and the maintenance of a comfortable environment that take into account the emotional state of residents and users.

[0794] A "video acquisition device" is a device that captures the visual information of an object and acquires it as digital data.

[0795] "Facial data" refers to digital data that represents the characteristics of an individual's face and is used for identification and recognition.

[0796] "Audio data" refers to data that can record audio and be stored or analyzed in digital format.

[0797] "Recognition means" refers to means equipped with the function of identifying a specific individual or analyzing its state based on acquired data.

[0798] A "specialized machine" is a machine with a specialized structure designed to perform a specific purpose or function.

[0799] "Cleaning means" refers to devices or functions that automatically clean the environment.

[0800] A "sensor device" is a device that detects the physical or chemical state of the environment and acquires that information as digital data.

[0801] "Analysis means" refers to methods for processing acquired data and extracting meaningful information.

[0802] "Aggregation means" refers to a device or function used to integrate multiple pieces of information and create reports or perform data analysis.

[0803] "Emotional state" is an indicator that represents an individual's psychological state and can be evaluated based on voice and facial expressions.

[0804] An "emotion engine" is software equipped with algorithms and functions to analyze a user's voice and facial expression data and identify their emotional state.

[0805] A "security agency" is an organization or company that is responsible for safety and security-related operations.

[0806] The system implementing this invention is configured to combine a video acquisition device, an audio acquisition device, a recognition means, a cleaning means, a sensor device, an analysis means, an aggregation means, and an emotion engine. A terminal installed in the user's living environment acquires the user's facial and voice data in real time using the video acquisition device and the audio acquisition device. This data is transmitted to a server, where the recognition means confirms the individual's survival and analyzes their emotional state.

[0807] The server controls cleaning methods that perform environmental cleaning using specialized mechanical equipment, and analysis methods detect anomalies based on various environmental information acquired from sensor devices. The diverse data obtained from these methods are integrated by aggregation methods to create reports, and at the same time, dynamic system responses that take into account the user's emotional state are realized.

[0808] Specifically, when a user is relaxing at home, the system recognizes their emotions and adjusts lighting, music, and other elements to provide a comfortable environment. Furthermore, if the user is experiencing anxiety or stress, the server immediately notifies security agencies for a swift response.

[0809] The hardware used includes smart devices with cameras and microphones with voice input capabilities. Data analysis is performed using software such as Rekognition and Polly from Amazon Web Services (AWS). AWS SNS is used for alarm notifications, enabling real-time notifications to users and administrators.

[0810] A concrete example of a prompt using a generative AI model is: "I want to design an app that identifies emotions from a user's face and voice, and sends a notification to a security company if negative emotions are detected. Therefore, please tell me how to perform emotion analysis using AWS Rekognition and Polly."

[0811] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0812] Step 1:

[0813] The device acquires the user's face and voice. It captures face and voice data in real time using the camera and microphone, and sends this data to the server as input.

[0814] Step 2:

[0815] The server processes the received facial data using recognition technology. This process analyzes facial features and performs individual identification. The recognition result is obtained as output and passed to the next processing step.

[0816] Step 3:

[0817] The emotion engine on the server analyzes the audio data and evaluates the user's emotional state. It extracts emotions from the tone and rhythm of the voice and outputs the results.

[0818] Step 4:

[0819] The server integrates the recognition methods and the output of the emotion engine to comprehensively evaluate the user's state. It determines the presence or absence of abnormalities or negative emotions and uses the results in the next step.

[0820] Step 5:

[0821] The server analyzes environmental information from sensor devices and detects anomalies. The output is data indicating whether or not there are environmental anomalies. Based on these results, the server determines the necessary actions.

[0822] Step 6:

[0823] The server integrates the results from the previous step using aggregation tools and creates a report. Simultaneously, it sends a notification to the security agency if any anomalies are detected. This step also includes setting up alerts via voice notifications and display devices.

[0824] Step 7:

[0825] The user receives feedback from the server. For example, if the server determines that the user is relaxed, actions are taken to improve the user experience, such as automatically playing soothing music.

[0826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0827] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0829] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0830] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0831] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0832] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0833] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0834] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0836] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0837] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0838] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0840] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0841] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0842] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0843] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0844] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0845] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0846] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0847] The following is further disclosed regarding the embodiments described above.

[0848] (Claim 1)

[0849] A recognition means for analyzing facial data acquired by a video acquisition device and confirming the survival of the recognized individual,

[0850] A cleaning method that automatically performs environmental cleaning using specialized machinery and equipment,

[0851] An analysis means for analyzing environmental information acquired by multiple sensor devices and detecting anomalies,

[0852] An aggregation means for integrating information obtained from the aforementioned recognition means, cleaning means, and analysis means to create a report,

[0853] A system that includes this.

[0854] (Claim 2)

[0855] The system according to claim 1, which includes a matching means that, if the recognition means detects an abnormality based on the authentication result, sends a notification to an administrator and displays an alert on a display device.

[0856] (Claim 3)

[0857] The system according to claim 1, wherein the analysis means includes a prediction means that processes data from a sensor device in real time and uses an algorithm to predict anomalies in advance.

[0858] "Example 1"

[0859] (Claim 1)

[0860] A recognition means for processing facial information acquired by a video acquisition means and confirming the state of the recognized object,

[0861] A cleaning method that automatically performs environmental cleaning using specialized machinery and equipment,

[0862] An analysis means for analyzing environmental information acquired by multiple detection devices and detecting anomalies,

[0863] An aggregation means for integrating information obtained from the aforementioned recognition means, cleaning means, and analysis means to generate a periodic report,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, which includes a confirmation means that, if the recognition means detects an abnormality based on the authentication result, contacts the controller and instructs the display device to pay attention.

[0867] (Claim 3)

[0868] The system according to claim 1, wherein the analysis means includes a prediction means that processes data from a detection device in real time and uses a calculation method to predict anomalies in advance.

[0869] "Application Example 1"

[0870] (Claim 1)

[0871] An identification means for analyzing facial data acquired by a video acquisition device and confirming the recognized individual,

[0872] Maintenance methods that automatically perform environmental maintenance using specialized machinery and equipment,

[0873] Processing means for processing environmental information acquired by multiple sensor devices and detecting anomalies,

[0874] A means for aggregating information obtained from the aforementioned identification means, maintenance means, and processing means, and for creating a document,

[0875] A means for transmitting a warning via a communication device when an abnormality occurs,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, wherein the identification means includes a verification means that, if an abnormality is found based on the authentication result, sends a notification to the controller and displays a warning on the display device.

[0879] (Claim 3)

[0880] The system according to claim 1, wherein the processing means includes a prediction device that immediately handles data from a sensor device and uses an algorithm to predict abnormalities in advance.

[0881] "Example 2 of combining an emotion engine"

[0882] (Claim 1)

[0883] A means for determining the existence of a recognized individual by analyzing facial information of an individual acquired by a video acquisition device,

[0884] An operating means that automatically performs environmental maintenance using specific-purpose equipment,

[0885] An analysis means for analyzing environmental information acquired by multiple detection devices and detecting anomalies,

[0886] Aggregation means for integrating information obtained from the aforementioned determination means, operating means, and analysis means to create a list,

[0887] Analytical tools for analyzing emotional states,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, which includes a comparison means that, if the determination means has an abnormality based on the authentication result, sends a notification to the administrator and displays an alarm on a display device.

[0891] (Claim 3)

[0892] The system according to claim 1, wherein the analysis means includes a prediction means that processes data from a detection device in real time and uses a calculation method to predict anomalies in advance.

[0893] "Application example 2 when combining with an emotional engine"

[0894] (Claim 1)

[0895] A recognition means for analyzing facial and audio data acquired by a video acquisition device to confirm the survival of an individual and to analyze their emotional state,

[0896] A cleaning method that automatically performs environmental cleaning using specialized machinery and equipment,

[0897] An analysis means for analyzing environmental information acquired by multiple sensor devices to detect abnormalities and emotional states,

[0898] An aggregation means for integrating information obtained from the aforementioned recognition means, cleaning means, and analysis means, creating a report, and generating a response that takes into account the sentiment analysis results,

[0899] A system that includes this.

[0900] (Claim 2)

[0901] The system according to claim 1, further comprising a matching means that, if the recognition means detects an abnormality based on authentication and sentiment analysis results, sends a notification to an administrator or security agency and displays an alert on a display device.

[0902] (Claim 3)

[0903] The system according to claim 1, wherein the analysis means includes a prediction means that processes data from a sensor device and an emotion engine in real time and uses an algorithm to predict abnormalities and emotional changes in advance. [Explanation of Symbols]

[0904] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A recognition means for analyzing facial data acquired by a video acquisition device and confirming the survival of the recognized individual, A cleaning method that automatically performs environmental cleaning using specialized machinery and equipment, An analysis means for analyzing environmental information acquired by multiple sensor devices and detecting anomalies, An aggregation means for integrating information obtained from the aforementioned recognition means, cleaning means, and analysis means to create a report, A system that includes this.

2. The system according to claim 1, which includes a verification means that, if the recognition means detects an abnormality based on the authentication result, sends a notification to the administrator and displays an alert on a display device.

3. The system according to claim 1, wherein the analysis means includes a prediction means that processes data from a sensor device in real time and uses an algorithm to predict anomalies in advance.

Images