System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving and sending authentication information from a user to a server; means for the server to match the received authentication information against a database; means for generating and sending an authentication token to a terminal based on a matching result; and means for the terminal to display an authentication result to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, population decline due to a declining birthrate and aging population, and labor shortages due to technological advances, are serious issues, calling for improved safety, security, and convenience in daily life. In particular, improving support systems for the elderly, people with disabilities, and children, early detection and prevention of problems, and crime prevention measures are important issues. There is also a need for integrated management of various IoT devices in homes and offices. However, solving these issues requires systems that are multifunctional and capable of advanced information processing, and conventional technologies are inadequate. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. The system includes a terminal having means for receiving authentication information from a user and transmitting it to a server, means for the server to compare the received authentication information with a database, generate an authentication token based on the comparison result, and transmit the token to the terminal, and means for the terminal to display the authentication result to the user. The system also includes means for collecting environmental data from sensors and transmitting it to the terminal, means for the terminal to transmit the collected environmental data to the server, the server to analyze the environmental data and detect anomalies, means for generating recommended measures when an anomaly is detected and transmitting the measures to the terminal, and means for the terminal to display the recommended measures to the user. The system also provides means for linking with various sensors to collect trouble-related data and transmitting it to the server, means for the server to analyze the trouble-related data and generate a real-time report, and means for transmitting the generated real-time report to the terminal and notifying the user. This improves safety, security, and convenience in daily life, and increases customer satisfaction.

[0006] "User authentication" is the process by which a system user enters information such as a username and password, and the server then authenticates them based on that information, in order to verify that they are the real person.

[0007] A "terminal" is a device used by a user, such as a smartphone, tablet, or PC, that provides an interface for accessing the system.

[0008] A "server" is a computer system that performs central information processing within the system, receiving requests from users, checking them against a database, and carrying out the necessary processing.

[0009] An "authentication token" is information that indicates that a user has been properly authenticated, and is an identifier that simplifies subsequent accesses.

[0010] A "database" is an information storage device for centrally managing data used by the system, and stores authentication information, environmental data, etc.

[0011] A "sensor" is a device that collects physical environmental data (temperature, humidity, smell, sound, etc.) and transmits that data to a terminal or server.

[0012] "Environmental data" is information such as temperature, humidity, odor, and sound collected by sensors that the system uses to determine the state of the environment.

[0013] An "emotion analysis engine" is software or hardware that analyzes a user's voice data and identifies their emotions.

[0014] "Trouble-related data" refers to data that indicates abnormal or dangerous situations within the system, and is used for early detection and prevention of problems.

[0015] "Real-time reports" are documents or notifications that instantly report the current situation based on collected trouble-related data.

[0016] "IoT devices" are physical devices (e.g., smart locks, smart lights, etc.) that can communicate with each other via the Internet, and can be automated and remotely controlled by linking them with systems. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[0039] User Authentication

[0040] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a specific example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[0041] Emotion understanding and environmental sensor data analysis

[0042] The device collects temperature, humidity, smell, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a specific example, if the indoor temperature suddenly rises, the data is analyzed by a cloud server and an immediate adjustment to the air conditioner is recommended.

[0043] Real-time reporting and notifications

[0044] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a specific example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[0045] Integration with IoT devices

[0046] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[0047] Assistance Features

[0048] A user inputs a question or request into a terminal. The terminal sends this to a server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[0049] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[0050] The processing flow will be explained below.

[0051] User Authentication

[0052] Step 1:

[0053] The user enters a username and password into the terminal.

[0054] Step 2:

[0055] The terminal transmits the entered authentication information to the server.

[0056] Step 3:

[0057] The server compares the received authentication information with the database and generates an authentication result.

[0058] Step 4:

[0059] The server generates an authentication token based on the matching result and sends it to the terminal.

[0060] Step 5:

[0061] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[0062] Emotion understanding and environmental sensor data analysis

[0063] Step 1:

[0064] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[0065] Step 2:

[0066] The terminal transmits the collected environmental data to the server.

[0067] Step 3:

[0068] The server analyzes the received environmental data and detects abnormalities.

[0069] Step 4:

[0070] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[0071] Step 5:

[0072] The device displays recommended measures to the user.

[0073] Step 6:

[0074] The terminal collects the user's voice using a microphone and transmits it to the server.

[0075] Step 7:

[0076] The server analyzes the voice data and identifies the user's emotions.

[0077] Step 8:

[0078] The server sends the results of the emotion analysis to the device.

[0079] Step 9:

[0080] The device displays the results of the emotion analysis to the user.

[0081] Real-time reporting and notifications

[0082] Step 1:

[0083] The terminal collects trouble-related data from sensors.

[0084] Step 2:

[0085] The trouble-related data collected by the terminal is sent to the server.

[0086] Step 3:

[0087] The server analyzes the trouble-related data and generates real-time reports.

[0088] Step 4:

[0089] The server sends the generated real-time report to the terminal.

[0090] Step 5:

[0091] The terminal notifies the user of real-time reports.

[0092] Step 6:

[0093] The device will display the report contents to the user and also notify them of recommended measures.

[0094] Integration with IoT devices

[0095] Step 1:

[0096] The terminal collects status data from various IoT devices.

[0097] Step 2:

[0098] The terminal transmits the collected status data to the server.

[0099] Step 3:

[0100] The server consolidates and analyzes data from IoT devices.

[0101] Step 4:

[0102] The server determines abnormal values ​​and necessary actions and generates control signals.

[0103] Step 5:

[0104] The server sends a control signal to the IoT device via the terminal.

[0105] Step 6:

[0106] The terminal displays the status of IoT devices and actions taken to the user.

[0107] Assistance Features

[0108] Step 1:

[0109] The user types a question or request into the terminal.

[0110] Step 2:

[0111] The device sends the user's questions or requests to the server.

[0112] Step 3:

[0113] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[0114] Step 4:

[0115] The server generates a response and sends it to the terminal.

[0116] Step 5:

[0117] The terminal displays the answer to the user.

[0118] As a result, the present invention enables detailed user authentication, emotion understanding, environmental sensor data analysis, real-time reporting, collaboration with IoT devices, and assistance functions, providing a safe and convenient lifestyle.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] In conventional systems, user authentication, environmental data analysis, trouble detection, and integration with various devices were all performed separately, resulting in a lack of overall integration and efficiency. Using multiple systems that are not centrally managed is inconvenient for users, and system management and operation require significant resources. Furthermore, real-time data analysis and trouble notification may not be possible, potentially compromising user safety and convenience.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes a means for receiving authentication information from a user and transmitting it to the server, a means for the server to compare the received authentication information with a database, and a means for generating an authentication token based on the comparison result and transmitting it to the terminal, thereby enabling efficient user authentication in an integrated system.

[0124] The system also includes a means for the terminal to collect data from the environmental sensors and send it to the server, a means for the server to analyze the received environmental data, detect abnormalities, generate recommended countermeasures and send them to the terminal, and a means for the user to input questions or requests to the terminal, which the server receives and provides corresponding information. This enables real-time analysis of environmental data, prompt presentation of countermeasures to the user, and interactive information provision.

[0125] "User authentication" is the process in which the server receives the authentication information (user name, password, etc.) entered by the user into the terminal and compares it with a database to perform authentication.

[0126] "Authentication information" is information that identifies a user, such as a username and password.

[0127] A "terminal" is a device (smartphone, computer, etc.) operated by a user.

[0128] A "server" is a system that receives authentication information and environmental data, analyzes them, and returns the results to the terminal.

[0129] A "database" is a collection of information that stores user authentication information, environmental data, and the like.

[0130] An "authentication token" is a unique digital key issued to identify a user once authenticated.

[0131] An "environmental sensor" is a device that collects environmental data such as temperature, humidity, odor, and sound.

[0132] An "anomaly" is data or an event that deviates from the normal range.

[0133] "Recommended measures" are countermeasures that the system generates when an abnormality is detected.

[0134] "Data analysis" is the process of detecting anomalies and generating countermeasures based on collected data.

[0135] "Trouble-related data" is data related to troubles such as the detection of abnormal sounds and unauthorized intrusions.

[0136] A "real-time report" is a report that is generated based on analyzed data and provided to the user immediately.

[0137] "IoT devices" are various home appliances and devices connected to the Internet.

[0138] A "control signal" is an operation command generated by a server and sent to an IoT device.

[0139] An "assistance function" is a function in which a server provides answers to user questions or requests by referencing databases and Internet resources.

[0140] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[0141] First, let's explain the user authentication process. The user enters their username and password into their device. The device encrypts this authentication information and sends it to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a concrete example, when a user logs in to a smartphone app, the authentication information is compared with the cloud authentication service and the result is returned instantly.

[0142] Next, we will explain the processes of emotion understanding and environmental sensor data analysis. The device collects temperature, humidity, smell, sound, etc. from the environmental sensors. The device encrypts the collected data and sends it to the server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a concrete example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[0143] Next, we will explain the process of real-time reports and notifications. The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a concrete example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[0144] We will also explain the process of linking with IoT devices. The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. As a concrete example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[0145] Finally, we will explain the assistance function. The user inputs a question or request into the terminal. The terminal sends this to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. As a concrete example, if the user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[0146] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[0147] Prompt Sentence Examples

[0148] "Please explain how user authentication works in a smartphone app."

[0149] "Describe a system that uses environmental sensors and audio data to analyze situations and emotions."

[0150] "Please explain the procedure for using sensors to detect problems in real time and notify users."

[0151] "Please explain a system that improves home safety by linking with IoT devices."

[0152] "Please explain how the assistance feature works to provide information in response to a user's question."

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] (user authentication)

[0155] Step 1:

[0156] The user enters a username and password into the terminal.

[0157] Input: Username, Password.

[0158] Output: The authentication information sent to the device.

[0159] Specific operation: The user enters their username and password into the login screen on their smartphone.

[0160] Step 2:

[0161] The device encrypts the authentication information and sends it to the server.

[0162] Input: The credentials entered by the user.

[0163] Output: The encrypted authentication information sent to the server.

[0164] Specific operation: Authentication information is encrypted inside the terminal using the SSL / TLS protocol and sent to the server.

[0165] Step 3:

[0166] The server checks the received authentication information against a database.

[0167] Input: Encrypted credentials.

[0168] Output: Matching result (authentication success or failure).

[0169] What happens: The server decrypts the received authentication information, checks it against a database, and verifies the integrity of the password using a hash function.

[0170] Step 4:

[0171] The server generates an authentication token or an error message based on the result of the match and sends it to the terminal.

[0172] Input: Matching result.

[0173] Output: An authentication token or an error message.

[0174] Specific operation: If authentication is successful, the server generates a unique authentication token; if it fails, it generates an error message. The generated result is encrypted and sent to the terminal.

[0175] Step 5:

[0176] The terminal displays the authentication result to the user.

[0177] Input: An authentication token or an error message.

[0178] Output: The authentication result that is displayed to the user.

[0179] Specific operation: The device decrypts the received authentication token, and if authentication is successful, it transitions to the app's home screen, or if it fails, it displays an error message.

[0180] (Emotion understanding and environmental sensor data analysis)

[0181] Step 1:

[0182] The device collects data from environmental sensors.

[0183] Input: Measurement data from environmental sensors (temperature, humidity, odor, sound, etc.).

[0184] Output: Collected environmental data.

[0185] Specific operation: The device periodically collects data from connected thermometers, hygrometers, air quality sensors, etc.

[0186] Step 2:

[0187] The data collected by the device is encrypted and sent to the server.

[0188] Input: Collected environmental data.

[0189] Output: Encrypted environment data.

[0190] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[0191] Step 3:

[0192] The server analyzes the received environmental data.

[0193] Input: Encrypted environment data.

[0194] Output: Analysis result (normal or abnormal).

[0195] Specific operation: The server decrypts the received environmental data and detects outliers using statistical methods and machine learning models.

[0196] Step 4:

[0197] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[0198] Input: Analysis results (if anomaly is detected).

[0199] Output: Recommended action.

[0200] Specific operation: If an abnormality is detected, such as a sudden rise in temperature, the server generates recommended measures such as "adjusting the air conditioner," encrypts them, and sends them to the device.

[0201] Step 5:

[0202] The device displays recommended measures to the user.

[0203] Input: Recommended measures.

[0204] Output: Recommended action displayed to the user.

[0205] Specific operation: The device will decode the recommended measures received and display a pop-up message or notification area saying "The temperature is too high. Please adjust the air conditioner."

[0206] (Real-time reports and notifications)

[0207] Step 1:

[0208] The terminal collects trouble-related data from sensors.

[0209] Input: Sensor measurement data (abnormal sounds, suspicious movements, etc.).

[0210] Output: Collected trouble-related data.

[0211] How it works: The device collects data in real time from sensors that detect abnormal sounds and movements.

[0212] Step 2:

[0213] The device encrypts the collected data and sends it to the server.

[0214] Input: Collected trouble-related data.

[0215] Output: Encrypted trouble-related data.

[0216] Specific operation: The device immediately encrypts the collected data using the SSL / TLS protocol and sends it to the server.

[0217] Step 3:

[0218] The server analyzes the data and generates real-time reports.

[0219] Input: Encrypted trouble-related data.

[0220] Output: Real-time reports.

[0221] How it works: The server decrypts the data, analyzes it using machine learning models, and generates a real-time report detailing any anomalies detected.

[0222] Step 4:

[0223] The server encrypts the generated real-time report and sends it to the terminal.

[0224] Enter: Real-time reports.

[0225] Output: Encrypted real-time reports.

[0226] Specific operation: The generated report is encrypted and sent to the device.

[0227] Step 5:

[0228] The device notifies the user of real-time reports received and recommended measures.

[0229] Input: Encrypted real-time reports.

[0230] Output: Real-time reports and recommended actions notified to the user.

[0231] Specific behavior: The device will decode the report and notify the user via push notification or alert message, for example, "Suspicious activity was detected at 8:30 PM. Please check for details."

[0232] (Linkage with IoT devices)

[0233] Step 1:

[0234] The terminal collects data from IoT devices.

[0235] Input: IoT device state data.

[0236] Output: Collected IoT device data.

[0237] Specific operation: The device periodically collects status data from the home's smart locks, smart lights, etc.

[0238] Step 2:

[0239] The device encrypts the collected data and sends it to the server.

[0240] Input: Collected IoT device data.

[0241] Output: Encrypted IoT device data.

[0242] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[0243] Step 3:

[0244] The server analyzes and integrates the data.

[0245] Input: Encrypted IoT device data.

[0246] Output: Analysis results.

[0247] How it works: The server decrypts the data, aggregates and analyzes the data from multiple IoT devices, for example, to check whether a door is locked.

[0248] Step 4:

[0249] The server generates the necessary control signals and sends them to the terminal.

[0250] Input: Analysis results.

[0251] Output: Control signal.

[0252] Specific operation: A control signal is generated based on the analysis result. For example, if the door is open, a lock signal is generated.

[0253] Step 5:

[0254] The terminal sends a control signal to the IoT device.

[0255] Input: Control signal.

[0256] Output: The control signal sent to the IoT device.

[0257] Specific operation: The terminal decodes the received control signal and sends it to the target IoT device, for example, sending a locking signal to a smart lock and confirming its execution.

[0258] (Application example 1)

[0259] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0260] With conventional systems, it was difficult to centrally manage data collected from multiple sensors and devices, detect anomalies in real time, and take countermeasures. Furthermore, delays in notifying users and providing countermeasures could result in insufficient security measures being implemented. Furthermore, it was difficult to understand users' emotions and situations and provide appropriate actions accordingly.

[0261] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0262] In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the authentication information received with a database; means for generating an authentication token based on the comparison result and transmitting it to the terminal; means for the terminal to display the authentication result to the user; means for collecting environmental data from a sensor and transmitting it to the terminal; means for the terminal to transmit the collected environmental data to the server; means for the server to analyze the environmental data and detect anomalies; means for generating recommended measures when an abnormality is detected and transmitting them to the terminal; means for the terminal to display the recommended measures to the user; means for linking with various sensors and collecting trouble-related data and transmitting it to the server; means for the server to analyze the trouble-related data and generate a real-time report; means for transmitting the generated real-time report to the terminal and notifying the user; means for linking with physical devices such as smart home appliances and security devices; means for automatically controlling physical devices in the event of an abnormality; means for providing optimal information based on a user's question or request; means for generating an answer to a user's question using a generative AI model; and means for displaying the generated answer to the user. This makes it possible to detect anomalies in real time, implement countermeasures, and provide appropriate actions by understanding user emotions.

[0263] "User authentication" is the process of verifying a user's identity using authentication information such as a username and password when the user accesses a system.

[0264] A "terminal" is a device operated by a user, such as a PC, smartphone, or tablet.

[0265] A "server" is a central computing device that processes and stores data and communicates with terminals via a network.

[0266] A "database" is a system for organizing, storing, and managing large amounts of data.

[0267] An "authentication token" is an identifier issued by the server when user authentication is successful, and is used to simplify subsequent authentications.

[0268] An "environmental sensor" is a device that detects environmental data such as temperature, humidity, sound, and smell.

[0269] "Real-time reporting" refers to information that instantly processes data collected from sensors and devices and reports current conditions and abnormalities.

[0270] "Recommended measures" are specific countermeasures that are presented to the user when an abnormality or problem occurs.

[0271] An "IoT device" is a physical device that can connect to the Internet, collect data, and be controlled.

[0272] "Smart home appliances" are household electrical appliances that have internet connectivity and can be remotely controlled or automated.

[0273] A "physical device" is a hardware device that actually operates, such as a smart lock or security camera.

[0274] A "generative AI model" is an algorithmic model that learns from large amounts of data and generates predictions and answers for new data.

[0275] "Emotion understanding" is the process of detecting a user's emotional state from their voice, facial expressions, etc., and understanding that state.

[0276] "Notification" is the act of conveying information to a user in real time via a terminal.

[0277] The embodiments for carrying out the present invention will be described in detail below.

[0278] User Authentication

[0279] The server receives the authentication information entered from the user's device and compares it with the database. If the comparison is successful, it generates an authentication token and sends it to the device. The device receives this token and displays the authentication result to the user. This achieves highly secure access control.

[0280] Emotion understanding and environmental sensor data analysis

[0281] The device sends environmental data, such as temperature, humidity, and sound, collected from environmental sensors to a server. The server analyzes this data and determines whether there are any abnormalities. If an abnormality is detected, the server generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. The device also collects the user's voice and sends it to the server. The server uses a generative AI model to analyze this voice data and identify the user's emotions. The server then sends the results to the device and provides feedback appropriate to the user's situation.

[0282] Real-time reporting and notifications

[0283] The device collects trouble-related data from sensors and sends it to the server, which analyzes the data and generates a real-time report of specific troubles or abnormalities. This report is immediately sent to the device and notified to the user.

[0284] Integration with IoT devices

[0285] The server connects to various IoT devices, collects and analyzes data from these devices, and generates control signals as needed and sends them to the IoT devices via the terminals. This allows smart home appliances and security devices to automatically take appropriate action.

[0286] Assistance Features

[0287] When a user inputs a question or request into the device, the device sends it to the server. The server uses a generative AI model to generate the optimal answer to the user's question. This generated answer is displayed to the user via the device. For example, if a user asks, "Which route should I take to get home today?" the server will provide the optimal route taking into account the traffic conditions and weather for that day, and the device will display it.

[0288] Hardware and software used

[0289] The servers are operated using cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP).

[0290] For database management, relational databases such as MySQL and PostgreSQL are used.

[0291] The front-end devices used are iOS and Android smartphones and tablets.

[0292] Environmental sensors and IoT devices use microcontrollers such as Arduino and Raspberry Pi.

[0293] The generative AI model uses AI technologies such as OpenAI's GPT-4.

[0294] Example: Prompt sentence

[0295] For example, if a user requests, "Check if the door is open," the server analyzes data from the connected smart lock to determine the door's status and immediately notifies the user of the results.

[0296] In this way, the system of the present invention integrates multiple functions to provide users with safe and efficient information and control.

[0297] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0298] Step 1:

[0299] The user enters authentication information (user name and password) into the terminal.

[0300] Input: Username and Password.

[0301] Output: Sending authentication information to the server.

[0302] Step 2:

[0303] The server checks the received authentication information against a database.

[0304] Input: The authentication information sent from the device.

[0305] Output: Authentication result (success or failure).

[0306] Step 3:

[0307] If the authentication is successful, the server generates an authentication token and sends it to the terminal.

[0308] Input: Authentication result (success).

[0309] Output: An authentication token.

[0310] Step 4:

[0311] The terminal stores the received authentication token and displays the authentication result to the user.

[0312] Input: Authentication token.

[0313] Output: Display of authentication result.

[0314] Step 5:

[0315] The device collects data from environmental sensors (temperature, humidity, sound, etc.) and sends it to a server.

[0316] Input: Data from environmental sensors.

[0317] Output: Sending environmental data to the server.

[0318] Step 6:

[0319] The server analyzes the collected environmental data and detects abnormalities.

[0320] Input: Environmental data sent from the device.

[0321] Output: Anomaly detection results.

[0322] Step 7:

[0323] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[0324] Input: Anomaly detection results.

[0325] Output: Recommended action.

[0326] Step 8:

[0327] The device displays the received recommended measures to the user.

[0328] Input: Recommended measures.

[0329] Output: Display of recommended actions.

[0330] Step 9:

[0331] The terminal collects trouble-related data (such as abnormal sounds) from sensors and sends it to the server.

[0332] Input: Trouble related data from sensors.

[0333] Output: Sending trouble-related data to the server.

[0334] Step 10:

[0335] The server analyzes the trouble-related data and generates real-time reports.

[0336] Input: Trouble-related data sent from the device.

[0337] Output: Real-time reports.

[0338] Step 11:

[0339] The server transmits the generated real-time report to the terminal and notifies the user.

[0340] Enter: Real-time reports.

[0341] Output: Real-time reports sent to terminal and user notifications.

[0342] Step 12:

[0343] The server collects and analyzes data from smart home appliances and security devices.

[0344] Input: Data from smart home appliances and security devices.

[0345] Output: Analysis results.

[0346] Step 13:

[0347] When necessary, the server generates control signals and sends them to the physical device via the terminal.

[0348] Input: Analysis results.

[0349] Output: Control signals to physical devices.

[0350] Step 14:

[0351] When a user enters a question or request into the terminal, the terminal sends it to the server.

[0352] Input: The user's question or request.

[0353] Output: Sending questions or requests to the server.

[0354] Step 15:

[0355] The server uses a generative AI model to generate the best answer to the user's question.

[0356] Input: The user's question or request.

[0357] Output: The generated answer.

[0358] Step 16:

[0359] The server sends the generated answer to the terminal, which displays it to the user.

[0360] Input: The generated answer.

[0361] Output: Display of answer to user.

[0362] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0363] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of a system that combines emotion engines, and specific examples.

[0364] User Authentication

[0365] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[0366] Emotion understanding and environmental sensor data analysis

[0367] The device collects temperature, humidity, odor, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data with an emotion engine to identify the user's emotion and sends the result to the device. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[0368] Real-time reporting and notifications

[0369] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. For example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[0370] Integration with IoT devices

[0371] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends the data to the server, which then consolidates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if an abnormality is detected.

[0372] Assistance Features

[0373] A user inputs a question or request into a device. The device sends it to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends the answer to the device, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[0374] Emotion Engine

[0375] In the specific operation of the emotion engine, the device collects the user's voice and sends it to the server. The server passes the voice data to the emotion engine and analyzes the user's emotions. Once the analysis results are obtained, the server sends the emotion data to the device. The device displays the received emotion data to the user and may also provide an appropriate assistance plan. For example, if the user is angry, the emotion engine will recognize that emotion and suggest music to help them relax.

[0376] Furthermore, the server can generate control signals based on the emotion data to dynamically adjust the data from the environmental sensors. These control signals are used to change environmental conditions such as temperature and lighting. For example, if a user is feeling stressed, changing the room lighting to warmer colors can promote relaxation.

[0377] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[0378] The processing flow will be explained below.

[0379] User Authentication

[0380] Step 1:

[0381] The user enters a username and password into the terminal.

[0382] Step 2:

[0383] The terminal transmits the entered authentication information to the server.

[0384] Step 3:

[0385] The server compares the received authentication information with the database and generates an authentication result.

[0386] Step 4:

[0387] The server generates an authentication token based on the matching result and sends it to the terminal.

[0388] Step 5:

[0389] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[0390] Emotion understanding and environmental sensor data analysis

[0391] Step 1:

[0392] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[0393] Step 2:

[0394] The terminal transmits the collected environmental data to the server.

[0395] Step 3:

[0396] The server analyzes the received environmental data and detects abnormalities.

[0397] Step 4:

[0398] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[0399] Step 5:

[0400] The device displays recommended measures to the user.

[0401] Step 6:

[0402] The terminal collects the user's voice using a microphone and transmits it to the server.

[0403] Step 7:

[0404] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[0405] Step 8:

[0406] The server sends the results of the emotion analysis to the device.

[0407] Step 9:

[0408] The device displays the results of the emotion analysis to the user.

[0409] Real-time reporting and notifications

[0410] Step 1:

[0411] The terminal collects trouble-related data from sensors.

[0412] Step 2:

[0413] The trouble-related data collected by the terminal is sent to the server.

[0414] Step 3:

[0415] The server analyzes the trouble-related data and generates real-time reports.

[0416] Step 4:

[0417] The server sends the generated real-time report to the terminal.

[0418] Step 5:

[0419] The terminal notifies the user of real-time reports.

[0420] Step 6:

[0421] The device will display the report contents to the user and also notify them of recommended measures.

[0422] Integration with IoT devices

[0423] Step 1:

[0424] The terminal collects status data from various IoT devices.

[0425] Step 2:

[0426] The terminal transmits the collected status data to the server.

[0427] Step 3:

[0428] The server consolidates and analyzes data from IoT devices.

[0429] Step 4:

[0430] The server determines abnormal values ​​and necessary actions and generates control signals.

[0431] Step 5:

[0432] The server sends a control signal to the IoT device via the terminal.

[0433] Step 6:

[0434] The terminal displays the status of IoT devices and actions taken to the user.

[0435] Assistance Features

[0436] Step 1:

[0437] The user types a question or request into the terminal.

[0438] Step 2:

[0439] The device sends the user's questions or requests to the server.

[0440] Step 3:

[0441] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[0442] Step 4:

[0443] The server generates a response and sends it to the terminal.

[0444] Step 5:

[0445] The terminal displays the answer to the user.

[0446] Emotion Engine

[0447] Step 1:

[0448] The terminal collects the user's voice using a microphone.

[0449] Step 2:

[0450] The device sends the collected voice data to the server.

[0451] Step 3:

[0452] The server passes the voice data to the emotion engine.

[0453] Step 4:

[0454] The server performs emotion analysis using an emotion engine.

[0455] Step 5:

[0456] The server generates emotion data based on the analysis results.

[0457] Step 6:

[0458] The server transmits the emotion data to the terminal.

[0459] Step 7:

[0460] The device displays the results of the emotion analysis to the user.

[0461] Step 8:

[0462] The server generates an appropriate assistance plan based on the emotion data.

[0463] Step 9:

[0464] The server transmits the generated assist plan to the terminal.

[0465] Step 10:

[0466] The terminal displays the assistance plan to the user.

[0467] Step 11:

[0468] A server generates control signals for dynamic adjustment of the environment based on the emotion data.

[0469] Step 12:

[0470] The server sends control signals to various sensors and devices.

[0471] Step 13:

[0472] The device notifies the user of changes in the environment in real time.

[0473] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[0474] Example 2

[0475] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0476] Conventional authentication systems and systems for analyzing environmental data from sensors are often specialized in individual functions, and do not adequately consider user emotion analysis or the integration of multiple IoT devices. Furthermore, they lack the ability to respond in real time when a problem occurs or the ability to provide assistance in response to user requests. This forces users to use multiple systems and applications, which can be difficult to use.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0478] In this invention, the server includes means for receiving authentication information from a user and transmitting it to the server, means for the server to compare the authentication information received with a database, means for the server to generate an authentication token based on the comparison result and transmit it to the terminal, means for the terminal to display the authentication result to the user, means for the server to collect the user's voice and transmit it to the server, means for the server to analyze the voice data and identify emotions, and means for the server to transmit the identification result to the terminal and display it to the user. This makes it possible to provide an environment in which users can use authentication, emotion analysis, environmental analysis, trouble notification, IoT device collaboration, and assistance functions in a unified manner.

[0479] "Authentication information" refers to information such as a username and password that a user uses to access a system.

[0480] A "server" is a computer system for processing and managing data.

[0481] A "database" is a system that stores an organized collection of data and allows it to be quickly searched and manipulated when needed.

[0482] An "authentication token" is a digital certificate that is generated by the server when a user is successfully authenticated, and that indicates that the user has been authenticated.

[0483] A "terminal" is a device (e.g., a smartphone, tablet, or personal computer) that a user directly operates to access the system.

[0484] "Environmental data" refers to information such as temperature, humidity, smell, and sound collected through sensors.

[0485] An "emotion engine" is an algorithm or software that analyzes voice data and facial expression data to identify a user's emotions.

[0486] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as a digital or analog signal.

[0487] "Abnormal" refers to a condition that is outside of expected ranges or outside of specified standards.

[0488] "Recommended measures" indicate appropriate methods and actions to be taken in response to detected abnormalities.

[0489] "Trouble-related data" is data that includes detailed information when the system detects an abnormality or problem.

[0490] A "real-time report" is a report that is automatically created and notified to the user immediately after data is generated.

[0491] An "IoT device" is a physical device that has the ability to communicate with other devices and systems over the Internet.

[0492] A "control signal" is a command signal that a system sends to an IoT device to cause it to perform a specific action.

[0493] An "assist plan" is appropriate advice or action plan provided to the user based on the user's emotions and environmental data.

[0494] This invention is an integrated system that includes functions for user authentication, environmental data analysis, emotion analysis, trouble notification, collaboration with IoT devices, and assistance in response to user requests. Specific implementation methods for each function are described below.

[0495] User Authentication

[0496] The user enters their username and password into their device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[0497] Environmental sensor data analysis

[0498] The device collects environmental data such as temperature, humidity, smell, and sound from sensors. The device then sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[0499] emotional understanding

[0500] The device collects the user's voice and sends it to the server. The server analyzes the voice data using an emotion engine to identify the user's emotion. The server sends the identification results to the device, which then displays the emotion result to the user. For example, if the user says, "I'm tired today," the server analyzes it using its emotion engine and identifies the emotion as "fatigue." It then suggests, "Would you like to play some music to relax?"

[0501] Trouble Notification

[0502] The device collects trouble-related data (for example, abnormal sound detection) from the sensor and sends it to the server. The server analyzes the data and generates a real-time report. The generated real-time report is sent to the device and notified to the user. For example, if an abnormal sound is detected by the sensor at night, that information is immediately sent to the user's smartphone.

[0503] Integration with IoT devices

[0504] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. The server generates control signals as needed and sends them to the IoT devices via the terminal. The terminal displays the current status of the IoT devices and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[0505] Assistance Features

[0506] A user inputs a question or request into a device. The device sends it to the server, which then references databases and Internet resources based on the request and generates an answer. The server then sends the answer to the device, which displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[0507] Examples of concrete examples and prompts

[0508] Example 1: User authentication

[0509] Example) Processing when a user logs in to a smartphone app:

[0510] Prompt: "Please explain what happens when a user logs in to your app. In particular, describe how authentication information is sent and received and the results are displayed."

[0511] Example 2: Environmental sensor data analysis

[0512] Example) What to do if the indoor temperature suddenly rises:

[0513] Prompt: "Describe the actions you would take if the room temperature suddenly spiked. Include collecting temperature data, analyzing it on a server, and generating and displaying recommended actions."

[0514] Example 3: Emotional Understanding

[0515] Example: What happens if the user says "I'm tired today"?

[0516] Prompt: "What is the process flow when the emotion engine recognizes that the user is angry? Describe how emotion data is collected, analyzed, and an appropriate assistance plan is presented."

[0517] Example 4: Trouble notification

[0518] Example) Detecting and notifying suspicious activity at night:

[0519] Prompt: "What happens when a sensor detects suspicious activity at night? From data collection to notification."

[0520] Example 5: Integration with IoT devices

[0521] Example) Checking the status of the smart lock and automatically locking it in case of an abnormality:

[0522] Prompt: "Please explain the process for checking whether the smart lock is locked properly. Please also mention how to generate and send the automatic locking signal in the event of an abnormality."

[0523] Example 6: Assistance features

[0524] Example) When a user gets the best route information:

[0525] Prompt: "Describe the flow of the assistance function when a user asks which way to go home. Describe receiving the request, database lookup, generating and displaying the answer."

[0526] The above is a specific embodiment of the present invention. This allows users to link with various sensors and IoT devices to realize safe and convenient daily life. In addition, by providing real-time abnormality notifications and assistance functions that respond to the user's emotions, it is expected that customer satisfaction will be improved.

[0527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0528] User authentication process flow

[0529] Step 1:

[0530] The user enters a username and password into the terminal.

[0531] Input: Username, Password

[0532] Specific operation: The user enters their username and password into the login screen of the smartphone app and presses the "Login" button.

[0533] Output: Authentication information (username, password)

[0534] Step 2:

[0535] The terminal transmits the entered authentication information to the server.

[0536] Input: Credentials

[0537] Specific operation: The device encrypts the entered username and password and sends them to the server using HTTPS.

[0538] Output: Received authentication information (server side)

[0539] Step 3:

[0540] The server checks the received authentication information against a database.

[0541] Input: Received authentication information, user data in the database

[0542] What happens: The server compares the received username and password with the user data in its database.

[0543] Output: Match result (success or failure)

[0544] Step 4:

[0545] Based on the result of the match, the server generates an authentication token and sends it to the terminal.

[0546] Input: Matching result

[0547] Specific operation: If authentication is successful, the server generates a JWT (JSON Web Token), and if authentication fails, it generates an error message. Both are sent to the terminal.

[0548] Output: Authentication token or error message (terminal side)

[0549] Step 5:

[0550] The terminal displays the authentication result to the user.

[0551] Input: Authentication token or error message

[0552] Specific operation: The device saves the received authentication token and displays the home screen. If authentication fails, an error message is displayed to the user.

[0553] Output: Display of authentication result

[0554] Process flow for emotion understanding and environmental sensor data analysis

[0555] Step 1:

[0556] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[0557] Input: Environmental data (from sensors)

[0558] Specific operation: The device collects data in real time from sensors such as thermometers, hygrometers, odor sensors, and microphones.

[0559] Output: Collected environmental data

[0560] Step 2:

[0561] The device sends the collected sensor data to the server.

[0562] Input: Collected environmental data

[0563] Specific operation: The device organizes the collected data and sends it to the server using HTTPS.

[0564] Output: Received environment data (server side)

[0565] Step 3:

[0566] The server analyzes environmental data and detects abnormalities.

[0567] Input: Received environment data

[0568] How it works: The server uses Python's Pandas library to analyze the data and check for deviations from the reference value. For example, if the temperature exceeds 30°C, it is considered abnormal.

[0569] Output: Anomaly detection results

[0570] Step 4:

[0571] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[0572] Input: Anomaly detection results

[0573] Specific operation: When an abnormality is detected, the server generates a pre-defined countermeasure (e.g., turn on the air conditioner) and sends it to the terminal.

[0574] Output: Recommended Action

[0575] Step 5:

[0576] The device displays recommended measures to the user.

[0577] Input: Recommended Action

[0578] Specific behavior: The device notifies the user of the recommended action message received, for example, "The temperature is too high. Please turn on the air conditioner."

[0579] Output: Display of recommended actions

[0580] Step 6:

[0581] The terminal collects the user's voice and transmits it to the server.

[0582] Input: User voice

[0583] Specific operation: The device records the user's voice with a microphone and sends the data to the server.

[0584] Output: Received audio data (server side)

[0585] Step 7:

[0586] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[0587] Input: Audio data

[0588] Specific operation: The server analyzes the voice data using an emotion engine (e.g., IBM Watson Tone Analyzer) to identify emotions (e.g., anger, joy, sadness).

[0589] Output: Emotion identification result

[0590] Step 8:

[0591] The server transmits the identification result to the terminal, and the terminal displays the emotion result to the user.

[0592] Input: Emotion identification result

[0593] Specific operation: The server sends the emotion identification result to the device, and the device displays the emotion result (e.g., "You seem relaxed") to the user. If necessary, it suggests playing relaxing music.

[0594] Output: Display of emotion results

[0595] Real-time Reporting and Notifications Process

[0596] Step 1:

[0597] The device collects trouble-related data from the sensors and sends it to the server.

[0598] Input: Trouble-related data (from sensors)

[0599] Specific operation: The device collects trouble-related data such as abnormal sounds and movements from sensors and sends it to the server.

[0600] Output: Received trouble-related data (server side)

[0601] Step 2:

[0602] The server analyzes the trouble-related data and generates real-time reports.

[0603] Input: Trouble-related data

[0604] How it works: The server analyzes the received data using machine learning algorithms and generates reports in real time.

[0605] Output: Real-time reports

[0606] Step 3:

[0607] The server transmits the generated real-time report to the terminal, and the terminal notifies the user.

[0608] Input: Real-time Report

[0609] Specific operation: The server sends the generated report to the device, and the device notifies the user (e.g., "Suspicious activity was detected during the night").

[0610] Output: Real-time notification

[0611] Process flow for linking with IoT devices

[0612] Step 1:

[0613] The terminal collects status data from various IoT devices and sends it to the server.

[0614] Input: IoT device state data

[0615] Specific operation: The terminal collects status data from IoT devices such as smart locks and smart lights and sends it to the server.

[0616] Output: Received status data (server side)

[0617] Step 2:

[0618] The server integrates and analyzes the received data.

[0619] Input: State data

[0620] Specific operation: The server executes SQL queries to aggregate and analyze device status data.

[0621] Output: Analysis results

[0622] Step 3:

[0623] If necessary, the server generates a control signal and sends it to the IoT device via the terminal.

[0624] Input: Analysis results

[0625] Specific operation: If an abnormality is detected, the server generates an appropriate control signal (e.g., a locking signal) and sends it to the device via the terminal.

[0626] Output: Control signal

[0627] Step 4:

[0628] The terminal displays the current status of IoT devices and actions taken to the user.

[0629] Input: Result of the control signal

[0630] Specific operation: The device displays the result of the received control signal to the user (e.g., "The smart lock has been successfully locked").

[0631] Output: Shows the status and actions

[0632] Assistance function processing flow

[0633] Step 1:

[0634] The user enters a question or request into the terminal and sends it to the server.

[0635] Input: Question or request

[0636] Specific operation: The user enters a question (e.g., "Which way should I take to get home today?") into the input field of the device and presses the send button. The device then sends this to the server.

[0637] Output: Receiving the request (server side)

[0638] Step 2:

[0639] The server generates an answer based on the request by referencing databases and Internet resources.

[0640] Input: Request

[0641] Specific operation: The server refers to a database or API (e.g., Google Maps) and generates answers such as the best route.

[0642] Output: The generated answer

[0643] Step 3:

[0644] The server sends the generated answer to the terminal, which displays it to the user.

[0645] Input: Generated answer

[0646] Specific operation: The server sends the generated answer to the device, and the device displays it to the user (e.g., "The best route is Route A").

[0647] Output: Display the answer

[0648] The above are the specific processing steps of this system's program. This will enable safe and convenient daily life, and is expected to improve customer satisfaction.

[0649] (Application example 2)

[0650] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0651] In traditional brick-and-mortar stores, it is difficult to quickly respond to changes in customer emotions and the environment, which can delay appropriate responses to maintain customer satisfaction. In addition, because automation using environmental sensors and IoT devices is not fully implemented, adjustments to maintain customer comfort are time-consuming. As a result, the customer experience does not improve and efficiency in store operations is reduced.

[0652] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the received authentication information with a database; means for the server to generate an authentication token based on the comparison result and transmit it to the terminal; means for the terminal to display the authentication result to the user; means for the terminal to collect environmental data such as temperature and humidity from an environmental sensor and transmit it to the server; means for the server to analyze the environmental data, detect abnormalities, generate recommended measures, and transmit them to the terminal; means for the terminal to display the recommended measures to the user; means for collecting voice data, transmitting it to the server, analyzing it with an emotion engine, and transmitting the results to the terminal; and means for generating a control signal to automatically adjust environmental settings according to the customer's emotions and transmitting it to an IoT device. This enables rapid response to changes in customer emotions and the environment, significantly improving customer satisfaction. It also improves store operation efficiency and enables prompt appropriate responses.

[0653] "User Credentials" means information provided to identify a user when accessing a system.

[0654] A "server" is a computer system that stores, processes, and manages data over a network.

[0655] A "database" is a system for organizing, storing, and managing data according to a specific structure.

[0656] An "authentication token" is a temporary digital certificate that indicates that a user is authenticated.

[0657] A "terminal" is an electronic device that can be directly operated by a user, and includes, for example, a smartphone or a tablet.

[0658] An "environmental sensor" is a device for measuring physical environmental data such as temperature, humidity, light level, and sound volume.

[0659] "Environmental Data" is information about the physical environment collected by environmental sensors.

[0660] An "emotion engine" is software that analyzes voice data and other user data to identify a user's emotional state.

[0661] An "IoT device" is a physical device that is connected to the Internet and can communicate with other devices.

[0662] A "real-time report" is a report that instantly analyzes collected data and provides results immediately.

[0663] A "control signal" is an electronic command sent to perform a particular action.

[0664] A "recommended action" is a suggestion of the best course of action for a particular situation.

[0665] A "user notification" is a message sent from the system to present information to the user.

[0666] An "API" is an interface that allows different software applications to communicate with each other.

[0667] This paper describes a system and a program for implementing the present invention. Specifically, the system is designed to respond to changes in customer emotions and the environment in a physical store, thereby improving customer satisfaction and streamlining store operations.

[0668] Overall system configuration

[0669] To realize the invention, a device (smartphone) used by the user, a server for analyzing data, various environmental sensors, and IoT devices are required. The server runs in a cloud environment and analyzes data sent from the device and sensors. The emotion engine runs on the server and analyzes voice data to identify the user's emotions.

[0670] User Authentication

[0671] Users authenticate by scanning an identifier such as a QR code using their smartphone. The authentication information (user ID and password) is sent to the server, which compares it with a database to generate an authentication token and sends it to the device. The device then displays the authentication result to the user, completing the identification process.

[0672] Environmental data collection and analysis

[0673] Various environmental sensors (temperature, humidity, volume, etc.) are connected to the device. Data acquired from these sensors is sent to the server via the device. The server analyzes the received environmental data, and if an abnormality is detected, generates recommended measures and sends them to the device. The device then displays the recommended measures to the user.

[0674] Emotional understanding and regulation

[0675] The device collects voice data through a microphone and sends it to a server. The server then uses an emotion engine to analyze the voice data and identify the user's emotional state. Based on the results of the analysis, it generates control signals to automatically change environmental settings (such as changing music or adjusting lighting) if necessary, and sends them to the IoT device.

[0676] Real-time notifications

[0677] The server integrates the analysis results of the environmental sensors and the emotion engine, and generates a report if it detects a problem in real time. This report is sent to the device and notifies the user, allowing for a prompt response.

[0678] Examples of concrete examples and prompts

[0679] For example, when a customer enters a store, a device collects the customer's voice and analyzes their emotions. If the customer is dissatisfied, that information is sent to the staff's device in real time, prompting them to take appropriate action. Also, if the temperature in the store rises and it detects that the customer is feeling stressed, the device will automatically adjust the air conditioning and change the music to something more relaxing.

[0680] Example prompt sentence:

[0681] "Use the emotion engine API to analyze customer emotions from collected voice data, and if the emotion is stressful, generate code to change the music in the store to something relaxing using the IoT device API."

[0682] This makes it possible to respond quickly to changes in customer emotions and the environment, improving customer satisfaction and streamlining store operations.

[0683] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0684] Step 1:

[0685] The user enters authentication information using a terminal and sends it to the server. The entered authentication information is the user ID and password. The server compares the received authentication information with the database and generates an authentication token. If the comparison is positive, the authentication token is sent to the terminal, and the terminal displays the authentication result to the user. The input is the authentication information, and the output is the authentication result and authentication token.

[0686] Step 2:

[0687] The device collects environmental data from environmental sensors (temperature, humidity, volume) and sends it to the server. The server analyzes the received environmental data and generates recommended measures if an abnormality is detected. The generated recommended measures are sent to the device and displayed to the user. The input is environmental data and the output is recommended measures.

[0688] Step 3:

[0689] The terminal collects voice data and sends it to the server for analysis by the emotion engine. The server analyzes the voice data and identifies the user's emotional state. If the identification result indicates that environmental adjustment is necessary, the server generates a control signal and sends it to the IoT device. The input is the voice data and emotional state, and the output is a control signal for environmental adjustment.

[0690] Step 4:

[0691] The server generates real-time reports and immediately notifies the terminal if an abnormal situation is detected. The terminal displays the received real-time reports to the user and generates notifications to support a quick response. The input is trouble-related data, and the output is the real-time report and user notification.

[0692] Step 5:

[0693] If necessary, the server combines environmental sensor data with sentiment analysis results to perform detailed analysis to identify the root cause. If further action is required, the analysis results are sent to the device as recommended countermeasures. The input is environmental data and sentiment analysis results, and the output is detailed analysis results and recommended countermeasures.

[0694] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0695] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0696] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0697] [Second embodiment]

[0698] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0699] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0700] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0701] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0702] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0703] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0704] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0705] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0706] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0707] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0708] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0709] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0710] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[0711] User Authentication

[0712] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a specific example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[0713] Emotion understanding and environmental sensor data analysis

[0714] The device collects temperature, humidity, smell, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a specific example, if the indoor temperature suddenly rises, the data is analyzed by a cloud server and an immediate adjustment to the air conditioner is recommended.

[0715] Real-time reporting and notifications

[0716] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a specific example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[0717] Integration with IoT devices

[0718] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[0719] Assistance Features

[0720] A user inputs a question or request into a terminal. The terminal sends this to a server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[0721] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[0722] The processing flow will be explained below.

[0723] User Authentication

[0724] Step 1:

[0725] The user enters a username and password into the terminal.

[0726] Step 2:

[0727] The terminal transmits the entered authentication information to the server.

[0728] Step 3:

[0729] The server compares the received authentication information with the database and generates an authentication result.

[0730] Step 4:

[0731] The server generates an authentication token based on the matching result and sends it to the terminal.

[0732] Step 5:

[0733] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[0734] Emotion understanding and environmental sensor data analysis

[0735] Step 1:

[0736] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[0737] Step 2:

[0738] The terminal transmits the collected environmental data to the server.

[0739] Step 3:

[0740] The server analyzes the received environmental data and detects abnormalities.

[0741] Step 4:

[0742] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[0743] Step 5:

[0744] The device displays recommended measures to the user.

[0745] Step 6:

[0746] The terminal collects the user's voice using a microphone and transmits it to the server.

[0747] Step 7:

[0748] The server analyzes the voice data and identifies the user's emotions.

[0749] Step 8:

[0750] The server sends the results of the emotion analysis to the device.

[0751] Step 9:

[0752] The device displays the results of the emotion analysis to the user.

[0753] Real-time reporting and notifications

[0754] Step 1:

[0755] The terminal collects trouble-related data from sensors.

[0756] Step 2:

[0757] The trouble-related data collected by the terminal is sent to the server.

[0758] Step 3:

[0759] The server analyzes the trouble-related data and generates real-time reports.

[0760] Step 4:

[0761] The server sends the generated real-time report to the terminal.

[0762] Step 5:

[0763] The terminal notifies the user of real-time reports.

[0764] Step 6:

[0765] The device will display the report contents to the user and also notify them of recommended measures.

[0766] Integration with IoT devices

[0767] Step 1:

[0768] The terminal collects status data from various IoT devices.

[0769] Step 2:

[0770] The terminal transmits the collected status data to the server.

[0771] Step 3:

[0772] The server consolidates and analyzes data from IoT devices.

[0773] Step 4:

[0774] The server determines abnormal values ​​and necessary actions and generates control signals.

[0775] Step 5:

[0776] The server sends a control signal to the IoT device via the terminal.

[0777] Step 6:

[0778] The terminal displays the status of IoT devices and actions taken to the user.

[0779] Assistance Features

[0780] Step 1:

[0781] The user types a question or request into the terminal.

[0782] Step 2:

[0783] The device sends the user's questions or requests to the server.

[0784] Step 3:

[0785] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[0786] Step 4:

[0787] The server generates a response and sends it to the terminal.

[0788] Step 5:

[0789] The terminal displays the answer to the user.

[0790] As a result, the present invention enables detailed user authentication, emotion understanding, environmental sensor data analysis, real-time reporting, collaboration with IoT devices, and assistance functions, providing a safe and convenient lifestyle.

[0791] Example 1

[0792] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0793] In conventional systems, user authentication, environmental data analysis, trouble detection, and integration with various devices were all performed separately, resulting in a lack of overall integration and efficiency. Using multiple systems that are not centrally managed is inconvenient for users, and system management and operation require significant resources. Furthermore, real-time data analysis and trouble notification may not be possible, potentially compromising user safety and convenience.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0795] In this invention, the server includes a means for receiving authentication information from a user and transmitting it to the server, a means for the server to compare the received authentication information with a database, and a means for generating an authentication token based on the comparison result and transmitting it to the terminal, thereby enabling efficient user authentication in an integrated system.

[0796] The system also includes a means for the terminal to collect data from the environmental sensors and send it to the server, a means for the server to analyze the received environmental data, detect abnormalities, generate recommended countermeasures and send them to the terminal, and a means for the user to input questions or requests to the terminal, which the server receives and provides corresponding information. This enables real-time analysis of environmental data, prompt presentation of countermeasures to the user, and interactive information provision.

[0797] "User authentication" is the process in which the server receives the authentication information (user name, password, etc.) entered by the user into the terminal and compares it with a database to perform authentication.

[0798] "Authentication information" is information that identifies a user, such as a username and password.

[0799] A "terminal" is a device (smartphone, computer, etc.) operated by a user.

[0800] A "server" is a system that receives authentication information and environmental data, analyzes them, and returns the results to the terminal.

[0801] A "database" is a collection of information that stores user authentication information, environmental data, and the like.

[0802] An "authentication token" is a unique digital key issued to identify a user once authenticated.

[0803] An "environmental sensor" is a device that collects environmental data such as temperature, humidity, odor, and sound.

[0804] An "anomaly" is data or an event that deviates from the normal range.

[0805] "Recommended measures" are countermeasures that the system generates when an abnormality is detected.

[0806] "Data analysis" is the process of detecting anomalies and generating countermeasures based on collected data.

[0807] "Trouble-related data" is data related to troubles such as the detection of abnormal sounds and unauthorized intrusions.

[0808] A "real-time report" is a report that is generated based on analyzed data and provided to the user immediately.

[0809] "IoT devices" are various home appliances and devices connected to the Internet.

[0810] A "control signal" is an operation command generated by a server and sent to an IoT device.

[0811] An "assistance function" is a function in which a server provides answers to user questions or requests by referencing databases and Internet resources.

[0812] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[0813] First, let's explain the user authentication process. The user enters their username and password into their device. The device encrypts this authentication information and sends it to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a concrete example, when a user logs in to a smartphone app, the authentication information is compared with the cloud authentication service and the result is returned instantly.

[0814] Next, we will explain the processes of emotion understanding and environmental sensor data analysis. The device collects temperature, humidity, smell, sound, etc. from the environmental sensors. The device encrypts the collected data and sends it to the server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a concrete example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[0815] Next, we will explain the process of real-time reports and notifications. The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a concrete example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[0816] We will also explain the process of linking with IoT devices. The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. As a concrete example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[0817] Finally, we will explain the assistance function. The user inputs a question or request into the terminal. The terminal sends this to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. As a concrete example, if the user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[0818] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[0819] Prompt Sentence Examples

[0820] "Please explain how user authentication works in a smartphone app."

[0821] "Describe a system that uses environmental sensors and audio data to analyze situations and emotions."

[0822] "Please explain the procedure for using sensors to detect problems in real time and notify users."

[0823] "Please explain a system that improves home safety by linking with IoT devices."

[0824] "Please explain how the assistance feature works to provide information in response to a user's question."

[0825] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0826] (user authentication)

[0827] Step 1:

[0828] The user enters a username and password into the terminal.

[0829] Input: Username, Password.

[0830] Output: The authentication information sent to the device.

[0831] Specific operation: The user enters their username and password into the login screen on their smartphone.

[0832] Step 2:

[0833] The device encrypts the authentication information and sends it to the server.

[0834] Input: The credentials entered by the user.

[0835] Output: The encrypted authentication information sent to the server.

[0836] Specific operation: Authentication information is encrypted inside the terminal using the SSL / TLS protocol and sent to the server.

[0837] Step 3:

[0838] The server checks the received authentication information against a database.

[0839] Input: Encrypted credentials.

[0840] Output: Matching result (authentication success or failure).

[0841] What happens: The server decrypts the received authentication information, checks it against a database, and verifies the integrity of the password using a hash function.

[0842] Step 4:

[0843] The server generates an authentication token or an error message based on the result of the match and sends it to the terminal.

[0844] Input: Matching result.

[0845] Output: An authentication token or an error message.

[0846] Specific operation: If authentication is successful, the server generates a unique authentication token; if it fails, it generates an error message. The generated result is encrypted and sent to the terminal.

[0847] Step 5:

[0848] The terminal displays the authentication result to the user.

[0849] Input: An authentication token or an error message.

[0850] Output: The authentication result that is displayed to the user.

[0851] Specific operation: The device decrypts the received authentication token, and if authentication is successful, it transitions to the app's home screen, or if it fails, it displays an error message.

[0852] (Emotion understanding and environmental sensor data analysis)

[0853] Step 1:

[0854] The device collects data from environmental sensors.

[0855] Input: Measurement data from environmental sensors (temperature, humidity, odor, sound, etc.).

[0856] Output: Collected environmental data.

[0857] Specific operation: The device periodically collects data from connected thermometers, hygrometers, air quality sensors, etc.

[0858] Step 2:

[0859] The data collected by the device is encrypted and sent to the server.

[0860] Input: Collected environmental data.

[0861] Output: Encrypted environment data.

[0862] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[0863] Step 3:

[0864] The server analyzes the received environmental data.

[0865] Input: Encrypted environment data.

[0866] Output: Analysis result (normal or abnormal).

[0867] Specific operation: The server decrypts the received environmental data and detects outliers using statistical methods and machine learning models.

[0868] Step 4:

[0869] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[0870] Input: Analysis results (if anomaly is detected).

[0871] Output: Recommended action.

[0872] Specific operation: If an abnormality is detected, such as a sudden rise in temperature, the server generates recommended measures such as "adjusting the air conditioner," encrypts them, and sends them to the device.

[0873] Step 5:

[0874] The device displays recommended measures to the user.

[0875] Input: Recommended measures.

[0876] Output: Recommended action displayed to the user.

[0877] Specific operation: The device will decode the recommended measures received and display a pop-up message or notification area saying "The temperature is too high. Please adjust the air conditioner."

[0878] (Real-time reports and notifications)

[0879] Step 1:

[0880] The terminal collects trouble-related data from sensors.

[0881] Input: Sensor measurement data (abnormal sounds, suspicious movements, etc.).

[0882] Output: Collected trouble-related data.

[0883] How it works: The device collects data in real time from sensors that detect abnormal sounds and movements.

[0884] Step 2:

[0885] The device encrypts the collected data and sends it to the server.

[0886] Input: Collected trouble-related data.

[0887] Output: Encrypted trouble-related data.

[0888] Specific operation: The device immediately encrypts the collected data using the SSL / TLS protocol and sends it to the server.

[0889] Step 3:

[0890] The server analyzes the data and generates real-time reports.

[0891] Input: Encrypted trouble-related data.

[0892] Output: Real-time reports.

[0893] How it works: The server decrypts the data, analyzes it using machine learning models, and generates a real-time report detailing any anomalies detected.

[0894] Step 4:

[0895] The server encrypts the generated real-time report and sends it to the terminal.

[0896] Enter: Real-time reports.

[0897] Output: Encrypted real-time reports.

[0898] Specific operation: The generated report is encrypted and sent to the device.

[0899] Step 5:

[0900] The device notifies the user of real-time reports received and recommended measures.

[0901] Input: Encrypted real-time reports.

[0902] Output: Real-time reports and recommended actions notified to the user.

[0903] Specific behavior: The device will decode the report and notify the user via push notification or alert message, for example, "Suspicious activity was detected at 8:30 PM. Please check for details."

[0904] (Linkage with IoT devices)

[0905] Step 1:

[0906] The terminal collects data from IoT devices.

[0907] Input: IoT device state data.

[0908] Output: Collected IoT device data.

[0909] Specific operation: The device periodically collects status data from the home's smart locks, smart lights, etc.

[0910] Step 2:

[0911] The device encrypts the collected data and sends it to the server.

[0912] Input: Collected IoT device data.

[0913] Output: Encrypted IoT device data.

[0914] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[0915] Step 3:

[0916] The server analyzes and integrates the data.

[0917] Input: Encrypted IoT device data.

[0918] Output: Analysis results.

[0919] How it works: The server decrypts the data, aggregates and analyzes the data from multiple IoT devices, for example, to check whether a door is locked.

[0920] Step 4:

[0921] The server generates the necessary control signals and sends them to the terminal.

[0922] Input: Analysis results.

[0923] Output: Control signal.

[0924] Specific operation: A control signal is generated based on the analysis result. For example, if the door is open, a lock signal is generated.

[0925] Step 5:

[0926] The terminal sends a control signal to the IoT device.

[0927] Input: Control signal.

[0928] Output: The control signal sent to the IoT device.

[0929] Specific operation: The terminal decodes the received control signal and sends it to the target IoT device, for example, sending a locking signal to a smart lock and confirming its execution.

[0930] (Application example 1)

[0931] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0932] With conventional systems, it was difficult to centrally manage data collected from multiple sensors and devices, detect anomalies in real time, and take countermeasures. Furthermore, delays in notifying users and providing countermeasures could result in insufficient security measures being implemented. Furthermore, it was difficult to understand users' emotions and situations and provide appropriate actions accordingly.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0934] In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the authentication information received with a database; means for generating an authentication token based on the comparison result and transmitting it to the terminal; means for the terminal to display the authentication result to the user; means for collecting environmental data from a sensor and transmitting it to the terminal; means for the terminal to transmit the collected environmental data to the server; means for the server to analyze the environmental data and detect anomalies; means for generating recommended measures when an abnormality is detected and transmitting them to the terminal; means for the terminal to display the recommended measures to the user; means for linking with various sensors and collecting trouble-related data and transmitting it to the server; means for the server to analyze the trouble-related data and generate a real-time report; means for transmitting the generated real-time report to the terminal and notifying the user; means for linking with physical devices such as smart home appliances and security devices; means for automatically controlling physical devices in the event of an abnormality; means for providing optimal information based on a user's question or request; means for generating an answer to a user's question using a generative AI model; and means for displaying the generated answer to the user. This makes it possible to detect anomalies in real time, implement countermeasures, and provide appropriate actions by understanding user emotions.

[0935] "User authentication" is the process of verifying a user's identity using authentication information such as a username and password when the user accesses a system.

[0936] A "terminal" is a device operated by a user, such as a PC, smartphone, or tablet.

[0937] A "server" is a central computing device that processes and stores data and communicates with terminals via a network.

[0938] A "database" is a system for organizing, storing, and managing large amounts of data.

[0939] An "authentication token" is an identifier issued by the server when user authentication is successful, and is used to simplify subsequent authentications.

[0940] An "environmental sensor" is a device that detects environmental data such as temperature, humidity, sound, and smell.

[0941] "Real-time reporting" refers to information that instantly processes data collected from sensors and devices and reports current conditions and abnormalities.

[0942] "Recommended measures" are specific countermeasures that are presented to the user when an abnormality or problem occurs.

[0943] An "IoT device" is a physical device that can connect to the Internet, collect data, and be controlled.

[0944] "Smart home appliances" are household electrical appliances that have internet connectivity and can be remotely controlled or automated.

[0945] A "physical device" is a hardware device that actually operates, such as a smart lock or security camera.

[0946] A "generative AI model" is an algorithmic model that learns from large amounts of data and generates predictions and answers for new data.

[0947] "Emotion understanding" is the process of detecting a user's emotional state from their voice, facial expressions, etc., and understanding that state.

[0948] "Notification" is the act of conveying information to a user in real time via a terminal.

[0949] The embodiments for carrying out the present invention will be described in detail below.

[0950] User Authentication

[0951] The server receives the authentication information entered from the user's device and compares it with the database. If the comparison is successful, it generates an authentication token and sends it to the device. The device receives this token and displays the authentication result to the user. This achieves highly secure access control.

[0952] Emotion understanding and environmental sensor data analysis

[0953] The device sends environmental data, such as temperature, humidity, and sound, collected from environmental sensors to a server. The server analyzes this data and determines whether there are any abnormalities. If an abnormality is detected, the server generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. The device also collects the user's voice and sends it to the server. The server uses a generative AI model to analyze this voice data and identify the user's emotions. The server then sends the results to the device and provides feedback appropriate to the user's situation.

[0954] Real-time reporting and notifications

[0955] The device collects trouble-related data from sensors and sends it to the server, which analyzes the data and generates a real-time report of specific troubles or abnormalities. This report is immediately sent to the device and notified to the user.

[0956] Integration with IoT devices

[0957] The server connects to various IoT devices, collects and analyzes data from these devices, and generates control signals as needed and sends them to the IoT devices via the terminals. This allows smart home appliances and security devices to automatically take appropriate action.

[0958] Assistance Features

[0959] When a user inputs a question or request into the device, the device sends it to the server. The server uses a generative AI model to generate the optimal answer to the user's question. This generated answer is displayed to the user via the device. For example, if a user asks, "Which route should I take to get home today?" the server will provide the optimal route taking into account the traffic conditions and weather for that day, and the device will display it.

[0960] Hardware and software used

[0961] The servers are operated using cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP).

[0962] For database management, relational databases such as MySQL and PostgreSQL are used.

[0963] The front-end devices used are iOS and Android smartphones and tablets.

[0964] Environmental sensors and IoT devices use microcontrollers such as Arduino and Raspberry Pi.

[0965] The generative AI model uses AI technologies such as OpenAI's GPT-4.

[0966] Example: Prompt sentence

[0967] For example, if a user requests, "Check if the door is open," the server analyzes data from the connected smart lock to determine the door's status and immediately notifies the user of the results.

[0968] In this way, the system of the present invention integrates multiple functions to provide users with safe and efficient information and control.

[0969] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0970] Step 1:

[0971] The user enters authentication information (user name and password) into the terminal.

[0972] Input: Username and Password.

[0973] Output: Sending authentication information to the server.

[0974] Step 2:

[0975] The server checks the received authentication information against a database.

[0976] Input: The authentication information sent from the device.

[0977] Output: Authentication result (success or failure).

[0978] Step 3:

[0979] If the authentication is successful, the server generates an authentication token and sends it to the terminal.

[0980] Input: Authentication result (success).

[0981] Output: An authentication token.

[0982] Step 4:

[0983] The terminal stores the received authentication token and displays the authentication result to the user.

[0984] Input: Authentication token.

[0985] Output: Display of authentication result.

[0986] Step 5:

[0987] The device collects data from environmental sensors (temperature, humidity, sound, etc.) and sends it to a server.

[0988] Input: Data from environmental sensors.

[0989] Output: Sending environmental data to the server.

[0990] Step 6:

[0991] The server analyzes the collected environmental data and detects abnormalities.

[0992] Input: Environmental data sent from the device.

[0993] Output: Anomaly detection results.

[0994] Step 7:

[0995] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[0996] Input: Anomaly detection results.

[0997] Output: Recommended action.

[0998] Step 8:

[0999] The device displays the received recommended measures to the user.

[1000] Input: Recommended measures.

[1001] Output: Display of recommended actions.

[1002] Step 9:

[1003] The terminal collects trouble-related data (such as abnormal sounds) from sensors and sends it to the server.

[1004] Input: Trouble related data from sensors.

[1005] Output: Sending trouble-related data to the server.

[1006] Step 10:

[1007] The server analyzes the trouble-related data and generates real-time reports.

[1008] Input: Trouble-related data sent from the device.

[1009] Output: Real-time reports.

[1010] Step 11:

[1011] The server transmits the generated real-time report to the terminal and notifies the user.

[1012] Enter: Real-time reports.

[1013] Output: Real-time reports sent to terminal and user notifications.

[1014] Step 12:

[1015] The server collects and analyzes data from smart home appliances and security devices.

[1016] Input: Data from smart home appliances and security devices.

[1017] Output: Analysis results.

[1018] Step 13:

[1019] When necessary, the server generates control signals and sends them to the physical device via the terminal.

[1020] Input: Analysis results.

[1021] Output: Control signals to physical devices.

[1022] Step 14:

[1023] When a user enters a question or request into the terminal, the terminal sends it to the server.

[1024] Input: The user's question or request.

[1025] Output: Sending questions or requests to the server.

[1026] Step 15:

[1027] The server uses a generative AI model to generate the best answer to the user's question.

[1028] Input: The user's question or request.

[1029] Output: The generated answer.

[1030] Step 16:

[1031] The server sends the generated answer to the terminal, which displays it to the user.

[1032] Input: The generated answer.

[1033] Output: Display of answer to user.

[1034] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1035] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of a system that combines emotion engines, and specific examples.

[1036] User Authentication

[1037] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[1038] Emotion understanding and environmental sensor data analysis

[1039] The device collects temperature, humidity, odor, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data with an emotion engine to identify the user's emotion and sends the result to the device. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[1040] Real-time reporting and notifications

[1041] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. For example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[1042] Integration with IoT devices

[1043] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends the data to the server, which then consolidates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if an abnormality is detected.

[1044] Assistance Features

[1045] A user inputs a question or request into a device. The device sends it to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends the answer to the device, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[1046] Emotion Engine

[1047] In the specific operation of the emotion engine, the device collects the user's voice and sends it to the server. The server passes the voice data to the emotion engine and analyzes the user's emotions. Once the analysis results are obtained, the server sends the emotion data to the device. The device displays the received emotion data to the user and may also provide an appropriate assistance plan. For example, if the user is angry, the emotion engine will recognize that emotion and suggest music to help them relax.

[1048] Furthermore, the server can generate control signals based on the emotion data to dynamically adjust the data from the environmental sensors. These control signals are used to change environmental conditions such as temperature and lighting. For example, if a user is feeling stressed, changing the room lighting to warmer colors can promote relaxation.

[1049] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[1050] The processing flow will be explained below.

[1051] User Authentication

[1052] Step 1:

[1053] The user enters a username and password into the terminal.

[1054] Step 2:

[1055] The terminal transmits the entered authentication information to the server.

[1056] Step 3:

[1057] The server compares the received authentication information with the database and generates an authentication result.

[1058] Step 4:

[1059] The server generates an authentication token based on the matching result and sends it to the terminal.

[1060] Step 5:

[1061] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[1062] Emotion understanding and environmental sensor data analysis

[1063] Step 1:

[1064] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[1065] Step 2:

[1066] The terminal transmits the collected environmental data to the server.

[1067] Step 3:

[1068] The server analyzes the received environmental data and detects abnormalities.

[1069] Step 4:

[1070] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[1071] Step 5:

[1072] The device displays recommended measures to the user.

[1073] Step 6:

[1074] The terminal collects the user's voice using a microphone and transmits it to the server.

[1075] Step 7:

[1076] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[1077] Step 8:

[1078] The server sends the results of the emotion analysis to the device.

[1079] Step 9:

[1080] The device displays the results of the emotion analysis to the user.

[1081] Real-time reporting and notifications

[1082] Step 1:

[1083] The terminal collects trouble-related data from sensors.

[1084] Step 2:

[1085] The trouble-related data collected by the terminal is sent to the server.

[1086] Step 3:

[1087] The server analyzes the trouble-related data and generates real-time reports.

[1088] Step 4:

[1089] The server sends the generated real-time report to the terminal.

[1090] Step 5:

[1091] The terminal notifies the user of real-time reports.

[1092] Step 6:

[1093] The device will display the report contents to the user and also notify them of recommended measures.

[1094] Integration with IoT devices

[1095] Step 1:

[1096] The terminal collects status data from various IoT devices.

[1097] Step 2:

[1098] The terminal transmits the collected status data to the server.

[1099] Step 3:

[1100] The server consolidates and analyzes data from IoT devices.

[1101] Step 4:

[1102] The server determines abnormal values ​​and necessary actions and generates control signals.

[1103] Step 5:

[1104] The server sends a control signal to the IoT device via the terminal.

[1105] Step 6:

[1106] The terminal displays the status of IoT devices and actions taken to the user.

[1107] Assistance Features

[1108] Step 1:

[1109] The user types a question or request into the terminal.

[1110] Step 2:

[1111] The device sends the user's questions or requests to the server.

[1112] Step 3:

[1113] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[1114] Step 4:

[1115] The server generates a response and sends it to the terminal.

[1116] Step 5:

[1117] The terminal displays the answer to the user.

[1118] Emotion Engine

[1119] Step 1:

[1120] The terminal collects the user's voice using a microphone.

[1121] Step 2:

[1122] The device sends the collected voice data to the server.

[1123] Step 3:

[1124] The server passes the voice data to the emotion engine.

[1125] Step 4:

[1126] The server performs emotion analysis using an emotion engine.

[1127] Step 5:

[1128] The server generates emotion data based on the analysis results.

[1129] Step 6:

[1130] The server transmits the emotion data to the terminal.

[1131] Step 7:

[1132] The device displays the results of the emotion analysis to the user.

[1133] Step 8:

[1134] The server generates an appropriate assistance plan based on the emotion data.

[1135] Step 9:

[1136] The server transmits the generated assist plan to the terminal.

[1137] Step 10:

[1138] The terminal displays the assistance plan to the user.

[1139] Step 11:

[1140] A server generates control signals for dynamic adjustment of the environment based on the emotion data.

[1141] Step 12:

[1142] The server sends control signals to various sensors and devices.

[1143] Step 13:

[1144] The device notifies the user of changes in the environment in real time.

[1145] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[1146] Example 2

[1147] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1148] Conventional authentication systems and systems for analyzing environmental data from sensors are often specialized in individual functions, and do not adequately consider user emotion analysis or the integration of multiple IoT devices. Furthermore, they lack the ability to respond in real time when a problem occurs or the ability to provide assistance in response to user requests. This forces users to use multiple systems and applications, which can be difficult to use.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1150] In this invention, the server includes means for receiving authentication information from a user and transmitting it to the server, means for the server to compare the authentication information received with a database, means for the server to generate an authentication token based on the comparison result and transmit it to the terminal, means for the terminal to display the authentication result to the user, means for the server to collect the user's voice and transmit it to the server, means for the server to analyze the voice data and identify emotions, and means for the server to transmit the identification result to the terminal and display it to the user. This makes it possible to provide an environment in which users can use authentication, emotion analysis, environmental analysis, trouble notification, IoT device collaboration, and assistance functions in a unified manner.

[1151] "Authentication information" refers to information such as a username and password that a user uses to access a system.

[1152] A "server" is a computer system for processing and managing data.

[1153] A "database" is a system that stores an organized collection of data and allows it to be quickly searched and manipulated when needed.

[1154] An "authentication token" is a digital certificate that is generated by the server when a user is successfully authenticated, and that indicates that the user has been authenticated.

[1155] A "terminal" is a device (e.g., a smartphone, tablet, or personal computer) that a user directly operates to access the system.

[1156] "Environmental data" refers to information such as temperature, humidity, smell, and sound collected through sensors.

[1157] An "emotion engine" is an algorithm or software that analyzes voice data and facial expression data to identify a user's emotions.

[1158] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as a digital or analog signal.

[1159] "Abnormal" refers to a condition that is outside of expected ranges or outside of specified standards.

[1160] "Recommended measures" indicate appropriate methods and actions to be taken in response to detected abnormalities.

[1161] "Trouble-related data" is data that includes detailed information when the system detects an abnormality or problem.

[1162] A "real-time report" is a report that is automatically created and notified to the user immediately after data is generated.

[1163] An "IoT device" is a physical device that has the ability to communicate with other devices and systems over the Internet.

[1164] A "control signal" is a command signal that a system sends to an IoT device to cause it to perform a specific action.

[1165] An "assist plan" is appropriate advice or action plan provided to the user based on the user's emotions and environmental data.

[1166] This invention is an integrated system that includes functions for user authentication, environmental data analysis, emotion analysis, trouble notification, collaboration with IoT devices, and assistance in response to user requests. Specific implementation methods for each function are described below.

[1167] User Authentication

[1168] The user enters their username and password into their device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[1169] Environmental sensor data analysis

[1170] The device collects environmental data such as temperature, humidity, smell, and sound from sensors. The device then sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[1171] emotional understanding

[1172] The device collects the user's voice and sends it to the server. The server analyzes the voice data using an emotion engine to identify the user's emotion. The server sends the identification results to the device, which then displays the emotion result to the user. For example, if the user says, "I'm tired today," the server analyzes it using its emotion engine and identifies the emotion as "fatigue." It then suggests, "Would you like to play some music to relax?"

[1173] Trouble Notification

[1174] The device collects trouble-related data (for example, abnormal sound detection) from the sensor and sends it to the server. The server analyzes the data and generates a real-time report. The generated real-time report is sent to the device and notified to the user. For example, if an abnormal sound is detected by the sensor at night, that information is immediately sent to the user's smartphone.

[1175] Integration with IoT devices

[1176] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. The server generates control signals as needed and sends them to the IoT devices via the terminal. The terminal displays the current status of the IoT devices and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[1177] Assistance Features

[1178] A user inputs a question or request into a device. The device sends it to the server, which then references databases and Internet resources based on the request and generates an answer. The server then sends the answer to the device, which displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[1179] Examples of concrete examples and prompts

[1180] Example 1: User authentication

[1181] Example) Processing when a user logs in to a smartphone app:

[1182] Prompt: "Please explain what happens when a user logs in to your app. In particular, describe how authentication information is sent and received and the results are displayed."

[1183] Example 2: Environmental sensor data analysis

[1184] Example) What to do if the indoor temperature suddenly rises:

[1185] Prompt: "Describe the actions you would take if the room temperature suddenly spiked. Include collecting temperature data, analyzing it on a server, and generating and displaying recommended actions."

[1186] Example 3: Emotional Understanding

[1187] Example: What happens if the user says "I'm tired today"?

[1188] Prompt: "What is the process flow when the emotion engine recognizes that the user is angry? Describe how emotion data is collected, analyzed, and an appropriate assistance plan is presented."

[1189] Example 4: Trouble notification

[1190] Example) Detecting and notifying suspicious activity at night:

[1191] Prompt: "What happens when a sensor detects suspicious activity at night? From data collection to notification."

[1192] Example 5: Integration with IoT devices

[1193] Example) Checking the status of the smart lock and automatically locking it in case of an abnormality:

[1194] Prompt: "Please explain the process for checking whether the smart lock is locked properly. Please also mention how to generate and send the automatic locking signal in the event of an abnormality."

[1195] Example 6: Assistance features

[1196] Example) When a user gets the best route information:

[1197] Prompt: "Describe the flow of the assistance function when a user asks which way to go home. Describe receiving the request, database lookup, generating and displaying the answer."

[1198] The above is a specific embodiment of the present invention. This allows users to link with various sensors and IoT devices to realize safe and convenient daily life. In addition, by providing real-time abnormality notifications and assistance functions that respond to the user's emotions, it is expected that customer satisfaction will be improved.

[1199] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1200] User authentication process flow

[1201] Step 1:

[1202] The user enters a username and password into the terminal.

[1203] Input: Username, Password

[1204] Specific operation: The user enters their username and password into the login screen of the smartphone app and presses the "Login" button.

[1205] Output: Authentication information (username, password)

[1206] Step 2:

[1207] The terminal transmits the entered authentication information to the server.

[1208] Input: Credentials

[1209] Specific operation: The device encrypts the entered username and password and sends them to the server using HTTPS.

[1210] Output: Received authentication information (server side)

[1211] Step 3:

[1212] The server checks the received authentication information against a database.

[1213] Input: Received authentication information, user data in the database

[1214] What happens: The server compares the received username and password with the user data in its database.

[1215] Output: Match result (success or failure)

[1216] Step 4:

[1217] Based on the result of the match, the server generates an authentication token and sends it to the terminal.

[1218] Input: Matching result

[1219] Specific operation: If authentication is successful, the server generates a JWT (JSON Web Token), and if authentication fails, it generates an error message. Both are sent to the terminal.

[1220] Output: Authentication token or error message (terminal side)

[1221] Step 5:

[1222] The terminal displays the authentication result to the user.

[1223] Input: Authentication token or error message

[1224] Specific operation: The device saves the received authentication token and displays the home screen. If authentication fails, an error message is displayed to the user.

[1225] Output: Display of authentication result

[1226] Process flow for emotion understanding and environmental sensor data analysis

[1227] Step 1:

[1228] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[1229] Input: Environmental data (from sensors)

[1230] Specific operation: The device collects data in real time from sensors such as thermometers, hygrometers, odor sensors, and microphones.

[1231] Output: Collected environmental data

[1232] Step 2:

[1233] The device sends the collected sensor data to the server.

[1234] Input: Collected environmental data

[1235] Specific operation: The device organizes the collected data and sends it to the server using HTTPS.

[1236] Output: Received environment data (server side)

[1237] Step 3:

[1238] The server analyzes environmental data and detects abnormalities.

[1239] Input: Received environment data

[1240] How it works: The server uses Python's Pandas library to analyze the data and check for deviations from the reference value. For example, if the temperature exceeds 30°C, it is considered abnormal.

[1241] Output: Anomaly detection results

[1242] Step 4:

[1243] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[1244] Input: Anomaly detection results

[1245] Specific operation: When an abnormality is detected, the server generates a pre-defined countermeasure (e.g., turn on the air conditioner) and sends it to the terminal.

[1246] Output: Recommended Action

[1247] Step 5:

[1248] The device displays recommended measures to the user.

[1249] Input: Recommended Action

[1250] Specific behavior: The device notifies the user of the recommended action message received, for example, "The temperature is too high. Please turn on the air conditioner."

[1251] Output: Display of recommended actions

[1252] Step 6:

[1253] The terminal collects the user's voice and transmits it to the server.

[1254] Input: User voice

[1255] Specific operation: The device records the user's voice with a microphone and sends the data to the server.

[1256] Output: Received audio data (server side)

[1257] Step 7:

[1258] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[1259] Input: Audio data

[1260] Specific operation: The server analyzes the voice data using an emotion engine (e.g., IBM Watson Tone Analyzer) to identify emotions (e.g., anger, joy, sadness).

[1261] Output: Emotion identification result

[1262] Step 8:

[1263] The server transmits the identification result to the terminal, and the terminal displays the emotion result to the user.

[1264] Input: Emotion identification result

[1265] Specific operation: The server sends the emotion identification result to the device, and the device displays the emotion result (e.g., "You seem relaxed") to the user. If necessary, it suggests playing relaxing music.

[1266] Output: Display of emotion results

[1267] Real-time Reporting and Notifications Process

[1268] Step 1:

[1269] The device collects trouble-related data from the sensors and sends it to the server.

[1270] Input: Trouble-related data (from sensors)

[1271] Specific operation: The device collects trouble-related data such as abnormal sounds and movements from sensors and sends it to the server.

[1272] Output: Received trouble-related data (server side)

[1273] Step 2:

[1274] The server analyzes the trouble-related data and generates real-time reports.

[1275] Input: Trouble-related data

[1276] How it works: The server analyzes the received data using machine learning algorithms and generates reports in real time.

[1277] Output: Real-time reports

[1278] Step 3:

[1279] The server transmits the generated real-time report to the terminal, and the terminal notifies the user.

[1280] Input: Real-time Report

[1281] Specific operation: The server sends the generated report to the device, and the device notifies the user (e.g., "Suspicious activity was detected during the night").

[1282] Output: Real-time notification

[1283] Process flow for linking with IoT devices

[1284] Step 1:

[1285] The terminal collects status data from various IoT devices and sends it to the server.

[1286] Input: IoT device state data

[1287] Specific operation: The terminal collects status data from IoT devices such as smart locks and smart lights and sends it to the server.

[1288] Output: Received status data (server side)

[1289] Step 2:

[1290] The server integrates and analyzes the received data.

[1291] Input: State data

[1292] Specific operation: The server executes SQL queries to aggregate and analyze device status data.

[1293] Output: Analysis results

[1294] Step 3:

[1295] If necessary, the server generates a control signal and sends it to the IoT device via the terminal.

[1296] Input: Analysis results

[1297] Specific operation: If an abnormality is detected, the server generates an appropriate control signal (e.g., a locking signal) and sends it to the device via the terminal.

[1298] Output: Control signal

[1299] Step 4:

[1300] The terminal displays the current status of IoT devices and actions taken to the user.

[1301] Input: Result of the control signal

[1302] Specific operation: The device displays the result of the received control signal to the user (e.g., "The smart lock has been successfully locked").

[1303] Output: Shows the status and actions

[1304] Assistance function processing flow

[1305] Step 1:

[1306] The user enters a question or request into the terminal and sends it to the server.

[1307] Input: Question or request

[1308] Specific operation: The user enters a question (e.g., "Which way should I take to get home today?") into the input field of the device and presses the send button. The device then sends this to the server.

[1309] Output: Receiving the request (server side)

[1310] Step 2:

[1311] The server generates an answer based on the request by referencing databases and Internet resources.

[1312] Input: Request

[1313] Specific operation: The server refers to a database or API (e.g., Google Maps) and generates answers such as the best route.

[1314] Output: The generated answer

[1315] Step 3:

[1316] The server sends the generated answer to the terminal, which displays it to the user.

[1317] Input: Generated answer

[1318] Specific operation: The server sends the generated answer to the device, and the device displays it to the user (e.g., "The best route is Route A").

[1319] Output: Display the answer

[1320] The above are the specific processing steps of this system's program. This will enable safe and convenient daily life, and is expected to improve customer satisfaction.

[1321] (Application example 2)

[1322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1323] In traditional brick-and-mortar stores, it is difficult to quickly respond to changes in customer emotions and the environment, which can delay appropriate responses to maintain customer satisfaction. In addition, because automation using environmental sensors and IoT devices is not fully implemented, adjustments to maintain customer comfort are time-consuming. As a result, the customer experience does not improve and efficiency in store operations is reduced.

[1324] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the received authentication information with a database; means for the server to generate an authentication token based on the comparison result and transmit it to the terminal; means for the terminal to display the authentication result to the user; means for the terminal to collect environmental data such as temperature and humidity from an environmental sensor and transmit it to the server; means for the server to analyze the environmental data, detect abnormalities, generate recommended measures, and transmit them to the terminal; means for the terminal to display the recommended measures to the user; means for collecting voice data, transmitting it to the server, analyzing it with an emotion engine, and transmitting the results to the terminal; and means for generating a control signal to automatically adjust environmental settings according to the customer's emotions and transmitting it to an IoT device. This enables rapid response to changes in customer emotions and the environment, significantly improving customer satisfaction. It also improves store operation efficiency and enables prompt appropriate responses.

[1325] "User Credentials" means information provided to identify a user when accessing a system.

[1326] A "server" is a computer system that stores, processes, and manages data over a network.

[1327] A "database" is a system for organizing, storing, and managing data according to a specific structure.

[1328] An "authentication token" is a temporary digital certificate that indicates that a user is authenticated.

[1329] A "terminal" is an electronic device that can be directly operated by a user, and includes, for example, a smartphone or a tablet.

[1330] An "environmental sensor" is a device for measuring physical environmental data such as temperature, humidity, light level, and sound volume.

[1331] "Environmental Data" is information about the physical environment collected by environmental sensors.

[1332] An "emotion engine" is software that analyzes voice data and other user data to identify a user's emotional state.

[1333] An "IoT device" is a physical device that is connected to the Internet and can communicate with other devices.

[1334] A "real-time report" is a report that instantly analyzes collected data and provides results immediately.

[1335] A "control signal" is an electronic command sent to perform a particular action.

[1336] A "recommended action" is a suggestion of the best course of action for a particular situation.

[1337] A "user notification" is a message sent from the system to present information to the user.

[1338] An "API" is an interface that allows different software applications to communicate with each other.

[1339] This paper describes a system and a program for implementing the present invention. Specifically, the system is designed to respond to changes in customer emotions and the environment in a physical store, thereby improving customer satisfaction and streamlining store operations.

[1340] Overall system configuration

[1341] To realize the invention, a device (smartphone) used by the user, a server for analyzing data, various environmental sensors, and IoT devices are required. The server runs in a cloud environment and analyzes data sent from the device and sensors. The emotion engine runs on the server and analyzes voice data to identify the user's emotions.

[1342] User Authentication

[1343] Users authenticate by scanning an identifier such as a QR code using their smartphone. The authentication information (user ID and password) is sent to the server, which compares it with a database to generate an authentication token and sends it to the device. The device then displays the authentication result to the user, completing the identification process.

[1344] Environmental data collection and analysis

[1345] Various environmental sensors (temperature, humidity, volume, etc.) are connected to the device. Data acquired from these sensors is sent to the server via the device. The server analyzes the received environmental data, and if an abnormality is detected, generates recommended measures and sends them to the device. The device then displays the recommended measures to the user.

[1346] Emotional understanding and regulation

[1347] The device collects voice data through a microphone and sends it to a server. The server then uses an emotion engine to analyze the voice data and identify the user's emotional state. Based on the results of the analysis, it generates control signals to automatically change environmental settings (such as changing music or adjusting lighting) if necessary, and sends them to the IoT device.

[1348] Real-time notifications

[1349] The server integrates the analysis results of the environmental sensors and the emotion engine, and generates a report if it detects a problem in real time. This report is sent to the device and notifies the user, allowing for a prompt response.

[1350] Examples of concrete examples and prompts

[1351] For example, when a customer enters a store, a device collects the customer's voice and analyzes their emotions. If the customer is dissatisfied, that information is sent to the staff's device in real time, prompting them to take appropriate action. Also, if the temperature in the store rises and it detects that the customer is feeling stressed, the device will automatically adjust the air conditioning and change the music to something more relaxing.

[1352] Example prompt sentence:

[1353] "Use the emotion engine API to analyze customer emotions from collected voice data, and if the emotion is stressful, generate code to change the music in the store to something relaxing using the IoT device API."

[1354] This makes it possible to respond quickly to changes in customer emotions and the environment, improving customer satisfaction and streamlining store operations.

[1355] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1356] Step 1:

[1357] The user enters authentication information using a terminal and sends it to the server. The entered authentication information is the user ID and password. The server compares the received authentication information with the database and generates an authentication token. If the comparison is positive, the authentication token is sent to the terminal, and the terminal displays the authentication result to the user. The input is the authentication information, and the output is the authentication result and authentication token.

[1358] Step 2:

[1359] The device collects environmental data from environmental sensors (temperature, humidity, volume) and sends it to the server. The server analyzes the received environmental data and generates recommended measures if an abnormality is detected. The generated recommended measures are sent to the device and displayed to the user. The input is environmental data and the output is recommended measures.

[1360] Step 3:

[1361] The terminal collects voice data and sends it to the server for analysis by the emotion engine. The server analyzes the voice data and identifies the user's emotional state. If the identification result indicates that environmental adjustment is necessary, the server generates a control signal and sends it to the IoT device. The input is the voice data and emotional state, and the output is a control signal for environmental adjustment.

[1362] Step 4:

[1363] The server generates real-time reports and immediately notifies the terminal if an abnormal situation is detected. The terminal displays the received real-time reports to the user and generates notifications to support a quick response. The input is trouble-related data, and the output is the real-time report and user notification.

[1364] Step 5:

[1365] If necessary, the server combines environmental sensor data with sentiment analysis results to perform detailed analysis to identify the root cause. If further action is required, the analysis results are sent to the device as recommended countermeasures. The input is environmental data and sentiment analysis results, and the output is detailed analysis results and recommended countermeasures.

[1366] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1367] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1368] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1369] [Third embodiment]

[1370] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1371] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1372] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1373] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1374] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1375] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1376] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1377] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1378] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1379] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1380] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1381] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1382] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[1383] User Authentication

[1384] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a specific example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[1385] Emotion understanding and environmental sensor data analysis

[1386] The device collects temperature, humidity, smell, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a specific example, if the indoor temperature suddenly rises, the data is analyzed by a cloud server and an immediate adjustment to the air conditioner is recommended.

[1387] Real-time reporting and notifications

[1388] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a specific example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[1389] Integration with IoT devices

[1390] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[1391] Assistance Features

[1392] A user inputs a question or request into a terminal. The terminal sends this to a server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[1393] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[1394] The processing flow will be explained below.

[1395] User Authentication

[1396] Step 1:

[1397] The user enters a username and password into the terminal.

[1398] Step 2:

[1399] The terminal transmits the entered authentication information to the server.

[1400] Step 3:

[1401] The server compares the received authentication information with the database and generates an authentication result.

[1402] Step 4:

[1403] The server generates an authentication token based on the matching result and sends it to the terminal.

[1404] Step 5:

[1405] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[1406] Emotion understanding and environmental sensor data analysis

[1407] Step 1:

[1408] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[1409] Step 2:

[1410] The terminal transmits the collected environmental data to the server.

[1411] Step 3:

[1412] The server analyzes the received environmental data and detects abnormalities.

[1413] Step 4:

[1414] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[1415] Step 5:

[1416] The device displays recommended measures to the user.

[1417] Step 6:

[1418] The terminal collects the user's voice using a microphone and transmits it to the server.

[1419] Step 7:

[1420] The server analyzes the voice data and identifies the user's emotions.

[1421] Step 8:

[1422] The server sends the results of the emotion analysis to the device.

[1423] Step 9:

[1424] The device displays the results of the emotion analysis to the user.

[1425] Real-time reporting and notifications

[1426] Step 1:

[1427] The terminal collects trouble-related data from sensors.

[1428] Step 2:

[1429] The trouble-related data collected by the terminal is sent to the server.

[1430] Step 3:

[1431] The server analyzes the trouble-related data and generates real-time reports.

[1432] Step 4:

[1433] The server sends the generated real-time report to the terminal.

[1434] Step 5:

[1435] The terminal notifies the user of real-time reports.

[1436] Step 6:

[1437] The device will display the report contents to the user and also notify them of recommended measures.

[1438] Integration with IoT devices

[1439] Step 1:

[1440] The terminal collects status data from various IoT devices.

[1441] Step 2:

[1442] The terminal transmits the collected status data to the server.

[1443] Step 3:

[1444] The server consolidates and analyzes data from IoT devices.

[1445] Step 4:

[1446] The server determines abnormal values ​​and necessary actions and generates control signals.

[1447] Step 5:

[1448] The server sends a control signal to the IoT device via the terminal.

[1449] Step 6:

[1450] The terminal displays the status of IoT devices and actions taken to the user.

[1451] Assistance Features

[1452] Step 1:

[1453] The user types a question or request into the terminal.

[1454] Step 2:

[1455] The device sends the user's questions or requests to the server.

[1456] Step 3:

[1457] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[1458] Step 4:

[1459] The server generates a response and sends it to the terminal.

[1460] Step 5:

[1461] The terminal displays the answer to the user.

[1462] As a result, the present invention enables detailed user authentication, emotion understanding, environmental sensor data analysis, real-time reporting, collaboration with IoT devices, and assistance functions, providing a safe and convenient lifestyle.

[1463] Example 1

[1464] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1465] In conventional systems, user authentication, environmental data analysis, trouble detection, and integration with various devices were all performed separately, resulting in a lack of overall integration and efficiency. Using multiple systems that are not centrally managed is inconvenient for users, and system management and operation require significant resources. Furthermore, real-time data analysis and trouble notification may not be possible, potentially compromising user safety and convenience.

[1466] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1467] In this invention, the server includes a means for receiving authentication information from a user and transmitting it to the server, a means for the server to compare the received authentication information with a database, and a means for generating an authentication token based on the comparison result and transmitting it to the terminal, thereby enabling efficient user authentication in an integrated system.

[1468] The system also includes a means for the terminal to collect data from the environmental sensors and send it to the server, a means for the server to analyze the received environmental data, detect abnormalities, generate recommended countermeasures and send them to the terminal, and a means for the user to input questions or requests to the terminal, which the server receives and provides corresponding information. This enables real-time analysis of environmental data, prompt presentation of countermeasures to the user, and interactive information provision.

[1469] "User authentication" is the process in which the server receives the authentication information (user name, password, etc.) entered by the user into the terminal and compares it with a database to perform authentication.

[1470] "Authentication information" is information that identifies a user, such as a username and password.

[1471] A "terminal" is a device (smartphone, computer, etc.) operated by a user.

[1472] A "server" is a system that receives authentication information and environmental data, analyzes them, and returns the results to the terminal.

[1473] A "database" is a collection of information that stores user authentication information, environmental data, and the like.

[1474] An "authentication token" is a unique digital key issued to identify a user once authenticated.

[1475] An "environmental sensor" is a device that collects environmental data such as temperature, humidity, odor, and sound.

[1476] An "anomaly" is data or an event that deviates from the normal range.

[1477] "Recommended measures" are countermeasures that the system generates when an abnormality is detected.

[1478] "Data analysis" is the process of detecting anomalies and generating countermeasures based on collected data.

[1479] "Trouble-related data" is data related to troubles such as the detection of abnormal sounds and unauthorized intrusions.

[1480] A "real-time report" is a report that is generated based on analyzed data and provided to the user immediately.

[1481] "IoT devices" are various home appliances and devices connected to the Internet.

[1482] A "control signal" is an operation command generated by a server and sent to an IoT device.

[1483] An "assistance function" is a function in which a server provides answers to user questions or requests by referencing databases and Internet resources.

[1484] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[1485] First, let's explain the user authentication process. The user enters their username and password into their device. The device encrypts this authentication information and sends it to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a concrete example, when a user logs in to a smartphone app, the authentication information is compared with the cloud authentication service and the result is returned instantly.

[1486] Next, we will explain the processes of emotion understanding and environmental sensor data analysis. The device collects temperature, humidity, smell, sound, etc. from the environmental sensors. The device encrypts the collected data and sends it to the server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a concrete example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[1487] Next, we will explain the process of real-time reports and notifications. The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a concrete example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[1488] We will also explain the process of linking with IoT devices. The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. As a concrete example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[1489] Finally, we will explain the assistance function. The user inputs a question or request into the terminal. The terminal sends this to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. As a concrete example, if the user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[1490] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[1491] Prompt Sentence Examples

[1492] "Please explain how user authentication works in a smartphone app."

[1493] "Describe a system that uses environmental sensors and audio data to analyze situations and emotions."

[1494] "Please explain the procedure for using sensors to detect problems in real time and notify users."

[1495] "Please explain a system that improves home safety by linking with IoT devices."

[1496] "Please explain how the assistance feature works to provide information in response to a user's question."

[1497] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1498] (user authentication)

[1499] Step 1:

[1500] The user enters a username and password into the terminal.

[1501] Input: Username, Password.

[1502] Output: The authentication information sent to the device.

[1503] Specific operation: The user enters their username and password into the login screen on their smartphone.

[1504] Step 2:

[1505] The device encrypts the authentication information and sends it to the server.

[1506] Input: The credentials entered by the user.

[1507] Output: The encrypted authentication information sent to the server.

[1508] Specific operation: Authentication information is encrypted inside the terminal using the SSL / TLS protocol and sent to the server.

[1509] Step 3:

[1510] The server checks the received authentication information against a database.

[1511] Input: Encrypted credentials.

[1512] Output: Matching result (authentication success or failure).

[1513] What happens: The server decrypts the received authentication information, checks it against a database, and verifies the integrity of the password using a hash function.

[1514] Step 4:

[1515] The server generates an authentication token or an error message based on the result of the match and sends it to the terminal.

[1516] Input: Matching result.

[1517] Output: An authentication token or an error message.

[1518] Specific operation: If authentication is successful, the server generates a unique authentication token; if it fails, it generates an error message. The generated result is encrypted and sent to the terminal.

[1519] Step 5:

[1520] The terminal displays the authentication result to the user.

[1521] Input: An authentication token or an error message.

[1522] Output: The authentication result that is displayed to the user.

[1523] Specific operation: The device decrypts the received authentication token, and if authentication is successful, it transitions to the app's home screen, or if it fails, it displays an error message.

[1524] (Emotion understanding and environmental sensor data analysis)

[1525] Step 1:

[1526] The device collects data from environmental sensors.

[1527] Input: Measurement data from environmental sensors (temperature, humidity, odor, sound, etc.).

[1528] Output: Collected environmental data.

[1529] Specific operation: The device periodically collects data from connected thermometers, hygrometers, air quality sensors, etc.

[1530] Step 2:

[1531] The data collected by the device is encrypted and sent to the server.

[1532] Input: Collected environmental data.

[1533] Output: Encrypted environment data.

[1534] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[1535] Step 3:

[1536] The server analyzes the received environmental data.

[1537] Input: Encrypted environment data.

[1538] Output: Analysis result (normal or abnormal).

[1539] Specific operation: The server decrypts the received environmental data and detects outliers using statistical methods and machine learning models.

[1540] Step 4:

[1541] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[1542] Input: Analysis results (if anomaly is detected).

[1543] Output: Recommended action.

[1544] Specific operation: If an abnormality is detected, such as a sudden rise in temperature, the server generates recommended measures such as "adjusting the air conditioner," encrypts them, and sends them to the device.

[1545] Step 5:

[1546] The device displays recommended measures to the user.

[1547] Input: Recommended measures.

[1548] Output: Recommended action displayed to the user.

[1549] Specific operation: The device will decode the recommended measures received and display a pop-up message or notification area saying "The temperature is too high. Please adjust the air conditioner."

[1550] (Real-time reports and notifications)

[1551] Step 1:

[1552] The terminal collects trouble-related data from sensors.

[1553] Input: Sensor measurement data (abnormal sounds, suspicious movements, etc.).

[1554] Output: Collected trouble-related data.

[1555] How it works: The device collects data in real time from sensors that detect abnormal sounds and movements.

[1556] Step 2:

[1557] The device encrypts the collected data and sends it to the server.

[1558] Input: Collected trouble-related data.

[1559] Output: Encrypted trouble-related data.

[1560] Specific operation: The device immediately encrypts the collected data using the SSL / TLS protocol and sends it to the server.

[1561] Step 3:

[1562] The server analyzes the data and generates real-time reports.

[1563] Input: Encrypted trouble-related data.

[1564] Output: Real-time reports.

[1565] How it works: The server decrypts the data, analyzes it using machine learning models, and generates a real-time report detailing any anomalies detected.

[1566] Step 4:

[1567] The server encrypts the generated real-time report and sends it to the terminal.

[1568] Enter: Real-time reports.

[1569] Output: Encrypted real-time reports.

[1570] Specific operation: The generated report is encrypted and sent to the device.

[1571] Step 5:

[1572] The device notifies the user of real-time reports received and recommended measures.

[1573] Input: Encrypted real-time reports.

[1574] Output: Real-time reports and recommended actions notified to the user.

[1575] Specific behavior: The device will decode the report and notify the user via push notification or alert message, for example, "Suspicious activity was detected at 8:30 PM. Please check for details."

[1576] (Linkage with IoT devices)

[1577] Step 1:

[1578] The terminal collects data from IoT devices.

[1579] Input: IoT device state data.

[1580] Output: Collected IoT device data.

[1581] Specific operation: The device periodically collects status data from the home's smart locks, smart lights, etc.

[1582] Step 2:

[1583] The device encrypts the collected data and sends it to the server.

[1584] Input: Collected IoT device data.

[1585] Output: Encrypted IoT device data.

[1586] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[1587] Step 3:

[1588] The server analyzes and integrates the data.

[1589] Input: Encrypted IoT device data.

[1590] Output: Analysis results.

[1591] How it works: The server decrypts the data, aggregates and analyzes the data from multiple IoT devices, for example, to check whether a door is locked.

[1592] Step 4:

[1593] The server generates the necessary control signals and sends them to the terminal.

[1594] Input: Analysis results.

[1595] Output: Control signal.

[1596] Specific operation: A control signal is generated based on the analysis result. For example, if the door is open, a lock signal is generated.

[1597] Step 5:

[1598] The terminal sends a control signal to the IoT device.

[1599] Input: Control signal.

[1600] Output: The control signal sent to the IoT device.

[1601] Specific operation: The terminal decodes the received control signal and sends it to the target IoT device, for example, sending a locking signal to a smart lock and confirming its execution.

[1602] (Application example 1)

[1603] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1604] With conventional systems, it was difficult to centrally manage data collected from multiple sensors and devices, detect anomalies in real time, and take countermeasures. Furthermore, delays in notifying users and providing countermeasures could result in insufficient security measures being implemented. Furthermore, it was difficult to understand users' emotions and situations and provide appropriate actions accordingly.

[1605] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1606] In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the authentication information received with a database; means for generating an authentication token based on the comparison result and transmitting it to the terminal; means for the terminal to display the authentication result to the user; means for collecting environmental data from a sensor and transmitting it to the terminal; means for the terminal to transmit the collected environmental data to the server; means for the server to analyze the environmental data and detect anomalies; means for generating recommended measures when an abnormality is detected and transmitting them to the terminal; means for the terminal to display the recommended measures to the user; means for linking with various sensors and collecting trouble-related data and transmitting it to the server; means for the server to analyze the trouble-related data and generate a real-time report; means for transmitting the generated real-time report to the terminal and notifying the user; means for linking with physical devices such as smart home appliances and security devices; means for automatically controlling physical devices in the event of an abnormality; means for providing optimal information based on a user's question or request; means for generating an answer to a user's question using a generative AI model; and means for displaying the generated answer to the user. This makes it possible to detect anomalies in real time, implement countermeasures, and provide appropriate actions by understanding user emotions.

[1607] "User authentication" is the process of verifying a user's identity using authentication information such as a username and password when the user accesses a system.

[1608] A "terminal" is a device operated by a user, such as a PC, smartphone, or tablet.

[1609] A "server" is a central computing device that processes and stores data and communicates with terminals via a network.

[1610] A "database" is a system for organizing, storing, and managing large amounts of data.

[1611] An "authentication token" is an identifier issued by the server when user authentication is successful, and is used to simplify subsequent authentications.

[1612] An "environmental sensor" is a device that detects environmental data such as temperature, humidity, sound, and smell.

[1613] "Real-time reporting" refers to information that instantly processes data collected from sensors and devices and reports current conditions and abnormalities.

[1614] "Recommended measures" are specific countermeasures that are presented to the user when an abnormality or problem occurs.

[1615] An "IoT device" is a physical device that can connect to the Internet, collect data, and be controlled.

[1616] "Smart home appliances" are household electrical appliances that have internet connectivity and can be remotely controlled or automated.

[1617] A "physical device" is a hardware device that actually operates, such as a smart lock or security camera.

[1618] A "generative AI model" is an algorithmic model that learns from large amounts of data and generates predictions and answers for new data.

[1619] "Emotion understanding" is the process of detecting a user's emotional state from their voice, facial expressions, etc., and understanding that state.

[1620] "Notification" is the act of conveying information to a user in real time via a terminal.

[1621] The embodiments for carrying out the present invention will be described in detail below.

[1622] User Authentication

[1623] The server receives the authentication information entered from the user's device and compares it with the database. If the comparison is successful, it generates an authentication token and sends it to the device. The device receives this token and displays the authentication result to the user. This achieves highly secure access control.

[1624] Emotion understanding and environmental sensor data analysis

[1625] The device sends environmental data, such as temperature, humidity, and sound, collected from environmental sensors to a server. The server analyzes this data and determines whether there are any abnormalities. If an abnormality is detected, the server generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. The device also collects the user's voice and sends it to the server. The server uses a generative AI model to analyze this voice data and identify the user's emotions. The server then sends the results to the device and provides feedback appropriate to the user's situation.

[1626] Real-time reporting and notifications

[1627] The device collects trouble-related data from sensors and sends it to the server, which analyzes the data and generates a real-time report of specific troubles or abnormalities. This report is immediately sent to the device and notified to the user.

[1628] Integration with IoT devices

[1629] The server connects to various IoT devices, collects and analyzes data from these devices, and generates control signals as needed and sends them to the IoT devices via the terminals. This allows smart home appliances and security devices to automatically take appropriate action.

[1630] Assistance Features

[1631] When a user inputs a question or request into the device, the device sends it to the server. The server uses a generative AI model to generate the optimal answer to the user's question. This generated answer is displayed to the user via the device. For example, if a user asks, "Which route should I take to get home today?" the server will provide the optimal route taking into account the traffic conditions and weather for that day, and the device will display it.

[1632] Hardware and software used

[1633] The servers are operated using cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP).

[1634] For database management, relational databases such as MySQL and PostgreSQL are used.

[1635] The front-end devices used are iOS and Android smartphones and tablets.

[1636] Environmental sensors and IoT devices use microcontrollers such as Arduino and Raspberry Pi.

[1637] The generative AI model uses AI technologies such as OpenAI's GPT-4.

[1638] Example: Prompt sentence

[1639] For example, if a user requests, "Check if the door is open," the server analyzes data from the connected smart lock to determine the door's status and immediately notifies the user of the results.

[1640] In this way, the system of the present invention integrates multiple functions to provide users with safe and efficient information and control.

[1641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1642] Step 1:

[1643] The user enters authentication information (user name and password) into the terminal.

[1644] Input: Username and Password.

[1645] Output: Sending authentication information to the server.

[1646] Step 2:

[1647] The server checks the received authentication information against a database.

[1648] Input: The authentication information sent from the device.

[1649] Output: Authentication result (success or failure).

[1650] Step 3:

[1651] If the authentication is successful, the server generates an authentication token and sends it to the terminal.

[1652] Input: Authentication result (success).

[1653] Output: An authentication token.

[1654] Step 4:

[1655] The terminal stores the received authentication token and displays the authentication result to the user.

[1656] Input: Authentication token.

[1657] Output: Display of authentication result.

[1658] Step 5:

[1659] The device collects data from environmental sensors (temperature, humidity, sound, etc.) and sends it to a server.

[1660] Input: Data from environmental sensors.

[1661] Output: Sending environmental data to the server.

[1662] Step 6:

[1663] The server analyzes the collected environmental data and detects abnormalities.

[1664] Input: Environmental data sent from the device.

[1665] Output: Anomaly detection results.

[1666] Step 7:

[1667] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[1668] Input: Anomaly detection results.

[1669] Output: Recommended action.

[1670] Step 8:

[1671] The device displays the received recommended measures to the user.

[1672] Input: Recommended measures.

[1673] Output: Display of recommended actions.

[1674] Step 9:

[1675] The terminal collects trouble-related data (such as abnormal sounds) from sensors and sends it to the server.

[1676] Input: Trouble related data from sensors.

[1677] Output: Sending trouble-related data to the server.

[1678] Step 10:

[1679] The server analyzes the trouble-related data and generates real-time reports.

[1680] Input: Trouble-related data sent from the device.

[1681] Output: Real-time reports.

[1682] Step 11:

[1683] The server transmits the generated real-time report to the terminal and notifies the user.

[1684] Enter: Real-time reports.

[1685] Output: Real-time reports sent to terminal and user notifications.

[1686] Step 12:

[1687] The server collects and analyzes data from smart home appliances and security devices.

[1688] Input: Data from smart home appliances and security devices.

[1689] Output: Analysis results.

[1690] Step 13:

[1691] When necessary, the server generates control signals and sends them to the physical device via the terminal.

[1692] Input: Analysis results.

[1693] Output: Control signals to physical devices.

[1694] Step 14:

[1695] When a user enters a question or request into the terminal, the terminal sends it to the server.

[1696] Input: The user's question or request.

[1697] Output: Sending questions or requests to the server.

[1698] Step 15:

[1699] The server uses a generative AI model to generate the best answer to the user's question.

[1700] Input: The user's question or request.

[1701] Output: The generated answer.

[1702] Step 16:

[1703] The server sends the generated answer to the terminal, which displays it to the user.

[1704] Input: The generated answer.

[1705] Output: Display of answer to user.

[1706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1707] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of a system that combines emotion engines, and specific examples.

[1708] User Authentication

[1709] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[1710] Emotion understanding and environmental sensor data analysis

[1711] The device collects temperature, humidity, odor, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data with an emotion engine to identify the user's emotion and sends the result to the device. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[1712] Real-time reporting and notifications

[1713] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. For example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[1714] Integration with IoT devices

[1715] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends the data to the server, which then consolidates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if an abnormality is detected.

[1716] Assistance Features

[1717] A user inputs a question or request into a device. The device sends it to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends the answer to the device, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[1718] Emotion Engine

[1719] In the specific operation of the emotion engine, the device collects the user's voice and sends it to the server. The server passes the voice data to the emotion engine and analyzes the user's emotions. Once the analysis results are obtained, the server sends the emotion data to the device. The device displays the received emotion data to the user and may also provide an appropriate assistance plan. For example, if the user is angry, the emotion engine will recognize that emotion and suggest music to help them relax.

[1720] Furthermore, the server can generate control signals based on the emotion data to dynamically adjust the data from the environmental sensors. These control signals are used to change environmental conditions such as temperature and lighting. For example, if a user is feeling stressed, changing the room lighting to warmer colors can promote relaxation.

[1721] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[1722] The processing flow will be explained below.

[1723] User Authentication

[1724] Step 1:

[1725] The user enters a username and password into the terminal.

[1726] Step 2:

[1727] The terminal transmits the entered authentication information to the server.

[1728] Step 3:

[1729] The server compares the received authentication information with the database and generates an authentication result.

[1730] Step 4:

[1731] The server generates an authentication token based on the matching result and sends it to the terminal.

[1732] Step 5:

[1733] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[1734] Emotion understanding and environmental sensor data analysis

[1735] Step 1:

[1736] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[1737] Step 2:

[1738] The terminal transmits the collected environmental data to the server.

[1739] Step 3:

[1740] The server analyzes the received environmental data and detects abnormalities.

[1741] Step 4:

[1742] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[1743] Step 5:

[1744] The device displays recommended measures to the user.

[1745] Step 6:

[1746] The terminal collects the user's voice using a microphone and transmits it to the server.

[1747] Step 7:

[1748] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[1749] Step 8:

[1750] The server sends the results of the emotion analysis to the device.

[1751] Step 9:

[1752] The device displays the results of the emotion analysis to the user.

[1753] Real-time reporting and notifications

[1754] Step 1:

[1755] The terminal collects trouble-related data from sensors.

[1756] Step 2:

[1757] The trouble-related data collected by the terminal is sent to the server.

[1758] Step 3:

[1759] The server analyzes the trouble-related data and generates real-time reports.

[1760] Step 4:

[1761] The server sends the generated real-time report to the terminal.

[1762] Step 5:

[1763] The terminal notifies the user of real-time reports.

[1764] Step 6:

[1765] The device will display the report contents to the user and also notify them of recommended measures.

[1766] Integration with IoT devices

[1767] Step 1:

[1768] The terminal collects status data from various IoT devices.

[1769] Step 2:

[1770] The terminal transmits the collected status data to the server.

[1771] Step 3:

[1772] The server consolidates and analyzes data from IoT devices.

[1773] Step 4:

[1774] The server determines abnormal values ​​and necessary actions and generates control signals.

[1775] Step 5:

[1776] The server sends a control signal to the IoT device via the terminal.

[1777] Step 6:

[1778] The terminal displays the status of IoT devices and actions taken to the user.

[1779] Assistance Features

[1780] Step 1:

[1781] The user types a question or request into the terminal.

[1782] Step 2:

[1783] The device sends the user's questions or requests to the server.

[1784] Step 3:

[1785] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[1786] Step 4:

[1787] The server generates a response and sends it to the terminal.

[1788] Step 5:

[1789] The terminal displays the answer to the user.

[1790] Emotion Engine

[1791] Step 1:

[1792] The terminal collects the user's voice using a microphone.

[1793] Step 2:

[1794] The device sends the collected voice data to the server.

[1795] Step 3:

[1796] The server passes the voice data to the emotion engine.

[1797] Step 4:

[1798] The server performs emotion analysis using an emotion engine.

[1799] Step 5:

[1800] The server generates emotion data based on the analysis results.

[1801] Step 6:

[1802] The server transmits the emotion data to the terminal.

[1803] Step 7:

[1804] The device displays the results of the emotion analysis to the user.

[1805] Step 8:

[1806] The server generates an appropriate assistance plan based on the emotion data.

[1807] Step 9:

[1808] The server transmits the generated assist plan to the terminal.

[1809] Step 10:

[1810] The terminal displays the assistance plan to the user.

[1811] Step 11:

[1812] A server generates control signals for dynamic adjustment of the environment based on the emotion data.

[1813] Step 12:

[1814] The server sends control signals to various sensors and devices.

[1815] Step 13:

[1816] The device notifies the user of changes in the environment in real time.

[1817] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[1818] Example 2

[1819] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1820] Conventional authentication systems and systems for analyzing environmental data from sensors are often specialized in individual functions, and do not adequately consider user emotion analysis or the integration of multiple IoT devices. Furthermore, they lack the ability to respond in real time when a problem occurs or the ability to provide assistance in response to user requests. This forces users to use multiple systems and applications, which can be difficult to use.

[1821] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1822] In this invention, the server includes means for receiving authentication information from a user and transmitting it to the server, means for the server to compare the authentication information received with a database, means for the server to generate an authentication token based on the comparison result and transmit it to the terminal, means for the terminal to display the authentication result to the user, means for the server to collect the user's voice and transmit it to the server, means for the server to analyze the voice data and identify emotions, and means for the server to transmit the identification result to the terminal and display it to the user. This makes it possible to provide an environment in which users can use authentication, emotion analysis, environmental analysis, trouble notification, IoT device collaboration, and assistance functions in a unified manner.

[1823] "Authentication information" refers to information such as a username and password that a user uses to access a system.

[1824] A "server" is a computer system for processing and managing data.

[1825] A "database" is a system that stores an organized collection of data and allows it to be quickly searched and manipulated when needed.

[1826] An "authentication token" is a digital certificate that is generated by the server when a user is successfully authenticated, and that indicates that the user has been authenticated.

[1827] A "terminal" is a device (e.g., a smartphone, tablet, or personal computer) that a user directly operates to access the system.

[1828] "Environmental data" refers to information such as temperature, humidity, smell, and sound collected through sensors.

[1829] An "emotion engine" is an algorithm or software that analyzes voice data and facial expression data to identify a user's emotions.

[1830] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as a digital or analog signal.

[1831] "Abnormal" refers to a condition that is outside of expected ranges or outside of specified standards.

[1832] "Recommended measures" indicate appropriate methods and actions to be taken in response to detected abnormalities.

[1833] "Trouble-related data" is data that includes detailed information when the system detects an abnormality or problem.

[1834] A "real-time report" is a report that is automatically created and notified to the user immediately after data is generated.

[1835] An "IoT device" is a physical device that has the ability to communicate with other devices and systems over the Internet.

[1836] A "control signal" is a command signal that a system sends to an IoT device to cause it to perform a specific action.

[1837] An "assist plan" is appropriate advice or action plan provided to the user based on the user's emotions and environmental data.

[1838] This invention is an integrated system that includes functions for user authentication, environmental data analysis, emotion analysis, trouble notification, collaboration with IoT devices, and assistance in response to user requests. Specific implementation methods for each function are described below.

[1839] User Authentication

[1840] The user enters their username and password into their device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[1841] Environmental sensor data analysis

[1842] The device collects environmental data such as temperature, humidity, smell, and sound from sensors. The device then sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[1843] emotional understanding

[1844] The device collects the user's voice and sends it to the server. The server analyzes the voice data using an emotion engine to identify the user's emotion. The server sends the identification results to the device, which then displays the emotion result to the user. For example, if the user says, "I'm tired today," the server analyzes it using its emotion engine and identifies the emotion as "fatigue." It then suggests, "Would you like to play some music to relax?"

[1845] Trouble Notification

[1846] The device collects trouble-related data (for example, abnormal sound detection) from the sensor and sends it to the server. The server analyzes the data and generates a real-time report. The generated real-time report is sent to the device and notified to the user. For example, if an abnormal sound is detected by the sensor at night, that information is immediately sent to the user's smartphone.

[1847] Integration with IoT devices

[1848] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. The server generates control signals as needed and sends them to the IoT devices via the terminal. The terminal displays the current status of the IoT devices and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[1849] Assistance Features

[1850] A user inputs a question or request into a device. The device sends it to the server, which then references databases and Internet resources based on the request and generates an answer. The server then sends the answer to the device, which displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[1851] Examples of concrete examples and prompts

[1852] Example 1: User authentication

[1853] Example) Processing when a user logs in to a smartphone app:

[1854] Prompt: "Please explain what happens when a user logs in to your app. In particular, describe how authentication information is sent and received and the results are displayed."

[1855] Example 2: Environmental sensor data analysis

[1856] Example) What to do if the indoor temperature suddenly rises:

[1857] Prompt: "Describe the actions you would take if the room temperature suddenly spiked. Include collecting temperature data, analyzing it on a server, and generating and displaying recommended actions."

[1858] Example 3: Emotional Understanding

[1859] Example: What happens if the user says "I'm tired today"?

[1860] Prompt: "What is the process flow when the emotion engine recognizes that the user is angry? Describe how emotion data is collected, analyzed, and an appropriate assistance plan is presented."

[1861] Example 4: Trouble notification

[1862] Example) Detecting and notifying suspicious activity at night:

[1863] Prompt: "What happens when a sensor detects suspicious activity at night? From data collection to notification."

[1864] Example 5: Integration with IoT devices

[1865] Example) Checking the status of the smart lock and automatically locking it in case of an abnormality:

[1866] Prompt: "Please explain the process for checking whether the smart lock is locked properly. Please also mention how to generate and send the automatic locking signal in the event of an abnormality."

[1867] Example 6: Assistance features

[1868] Example) When a user gets the best route information:

[1869] Prompt: "Describe the flow of the assistance function when a user asks which way to go home. Describe receiving the request, database lookup, generating and displaying the answer."

[1870] The above is a specific embodiment of the present invention. This allows users to link with various sensors and IoT devices to realize safe and convenient daily life. In addition, by providing real-time abnormality notifications and assistance functions that respond to the user's emotions, it is expected that customer satisfaction will be improved.

[1871] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1872] User authentication process flow

[1873] Step 1:

[1874] The user enters a username and password into the terminal.

[1875] Input: Username, Password

[1876] Specific operation: The user enters their username and password into the login screen of the smartphone app and presses the "Login" button.

[1877] Output: Authentication information (username, password)

[1878] Step 2:

[1879] The terminal transmits the entered authentication information to the server.

[1880] Input: Credentials

[1881] Specific operation: The device encrypts the entered username and password and sends them to the server using HTTPS.

[1882] Output: Received authentication information (server side)

[1883] Step 3:

[1884] The server checks the received authentication information against a database.

[1885] Input: Received authentication information, user data in the database

[1886] What happens: The server compares the received username and password with the user data in its database.

[1887] Output: Match result (success or failure)

[1888] Step 4:

[1889] Based on the result of the match, the server generates an authentication token and sends it to the terminal.

[1890] Input: Matching result

[1891] Specific operation: If authentication is successful, the server generates a JWT (JSON Web Token), and if authentication fails, it generates an error message. Both are sent to the terminal.

[1892] Output: Authentication token or error message (terminal side)

[1893] Step 5:

[1894] The terminal displays the authentication result to the user.

[1895] Input: Authentication token or error message

[1896] Specific operation: The device saves the received authentication token and displays the home screen. If authentication fails, an error message is displayed to the user.

[1897] Output: Display of authentication result

[1898] Process flow for emotion understanding and environmental sensor data analysis

[1899] Step 1:

[1900] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[1901] Input: Environmental data (from sensors)

[1902] Specific operation: The device collects data in real time from sensors such as thermometers, hygrometers, odor sensors, and microphones.

[1903] Output: Collected environmental data

[1904] Step 2:

[1905] The device sends the collected sensor data to the server.

[1906] Input: Collected environmental data

[1907] Specific operation: The device organizes the collected data and sends it to the server using HTTPS.

[1908] Output: Received environment data (server side)

[1909] Step 3:

[1910] The server analyzes environmental data and detects abnormalities.

[1911] Input: Received environment data

[1912] How it works: The server uses Python's Pandas library to analyze the data and check for deviations from the reference value. For example, if the temperature exceeds 30°C, it is considered abnormal.

[1913] Output: Anomaly detection results

[1914] Step 4:

[1915] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[1916] Input: Anomaly detection results

[1917] Specific operation: When an abnormality is detected, the server generates a pre-defined countermeasure (e.g., turn on the air conditioner) and sends it to the terminal.

[1918] Output: Recommended Action

[1919] Step 5:

[1920] The device displays recommended measures to the user.

[1921] Input: Recommended Action

[1922] Specific behavior: The device notifies the user of the recommended action message received, for example, "The temperature is too high. Please turn on the air conditioner."

[1923] Output: Display of recommended actions

[1924] Step 6:

[1925] The terminal collects the user's voice and transmits it to the server.

[1926] Input: User voice

[1927] Specific operation: The device records the user's voice with a microphone and sends the data to the server.

[1928] Output: Received audio data (server side)

[1929] Step 7:

[1930] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[1931] Input: Audio data

[1932] Specific operation: The server analyzes the voice data using an emotion engine (e.g., IBM Watson Tone Analyzer) to identify emotions (e.g., anger, joy, sadness).

[1933] Output: Emotion identification result

[1934] Step 8:

[1935] The server transmits the identification result to the terminal, and the terminal displays the emotion result to the user.

[1936] Input: Emotion identification result

[1937] Specific operation: The server sends the emotion identification result to the device, and the device displays the emotion result (e.g., "You seem relaxed") to the user. If necessary, it suggests playing relaxing music.

[1938] Output: Display of emotion results

[1939] Real-time Reporting and Notifications Process

[1940] Step 1:

[1941] The device collects trouble-related data from the sensors and sends it to the server.

[1942] Input: Trouble-related data (from sensors)

[1943] Specific operation: The device collects trouble-related data such as abnormal sounds and movements from sensors and sends it to the server.

[1944] Output: Received trouble-related data (server side)

[1945] Step 2:

[1946] The server analyzes the trouble-related data and generates real-time reports.

[1947] Input: Trouble-related data

[1948] How it works: The server analyzes the received data using machine learning algorithms and generates reports in real time.

[1949] Output: Real-time reports

[1950] Step 3:

[1951] The server transmits the generated real-time report to the terminal, and the terminal notifies the user.

[1952] Input: Real-time Report

[1953] Specific operation: The server sends the generated report to the device, and the device notifies the user (e.g., "Suspicious activity was detected during the night").

[1954] Output: Real-time notification

[1955] Process flow for linking with IoT devices

[1956] Step 1:

[1957] The terminal collects status data from various IoT devices and sends it to the server.

[1958] Input: IoT device state data

[1959] Specific operation: The terminal collects status data from IoT devices such as smart locks and smart lights and sends it to the server.

[1960] Output: Received status data (server side)

[1961] Step 2:

[1962] The server integrates and analyzes the received data.

[1963] Input: State data

[1964] Specific operation: The server executes SQL queries to aggregate and analyze device status data.

[1965] Output: Analysis results

[1966] Step 3:

[1967] If necessary, the server generates a control signal and sends it to the IoT device via the terminal.

[1968] Input: Analysis results

[1969] Specific operation: If an abnormality is detected, the server generates an appropriate control signal (e.g., a locking signal) and sends it to the device via the terminal.

[1970] Output: Control signal

[1971] Step 4:

[1972] The terminal displays the current status of IoT devices and actions taken to the user.

[1973] Input: Result of the control signal

[1974] Specific operation: The device displays the result of the received control signal to the user (e.g., "The smart lock has been successfully locked").

[1975] Output: Shows the status and actions

[1976] Assistance function processing flow

[1977] Step 1:

[1978] The user enters a question or request into the terminal and sends it to the server.

[1979] Input: Question or request

[1980] Specific operation: The user enters a question (e.g., "Which way should I take to get home today?") into the input field of the device and presses the send button. The device then sends this to the server.

[1981] Output: Receiving the request (server side)

[1982] Step 2:

[1983] The server generates an answer based on the request by referencing databases and Internet resources.

[1984] Input: Request

[1985] Specific operation: The server refers to a database or API (e.g., Google Maps) and generates answers such as the best route.

[1986] Output: The generated answer

[1987] Step 3:

[1988] The server sends the generated answer to the terminal, which displays it to the user.

[1989] Input: Generated answer

[1990] Specific operation: The server sends the generated answer to the device, and the device displays it to the user (e.g., "The best route is Route A").

[1991] Output: Display the answer

[1992] The above are the specific processing steps of this system's program. This will enable safe and convenient daily life, and is expected to improve customer satisfaction.

[1993] (Application example 2)

[1994] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1995] In traditional brick-and-mortar stores, it is difficult to quickly respond to changes in customer emotions and the environment, which can delay appropriate responses to maintain customer satisfaction. In addition, because automation using environmental sensors and IoT devices is not fully implemented, adjustments to maintain customer comfort are time-consuming. As a result, the customer experience does not improve and efficiency in store operations is reduced.

[1996] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the received authentication information with a database; means for the server to generate an authentication token based on the comparison result and transmit it to the terminal; means for the terminal to display the authentication result to the user; means for the terminal to collect environmental data such as temperature and humidity from an environmental sensor and transmit it to the server; means for the server to analyze the environmental data, detect abnormalities, generate recommended measures, and transmit them to the terminal; means for the terminal to display the recommended measures to the user; means for collecting voice data, transmitting it to the server, analyzing it with an emotion engine, and transmitting the results to the terminal; and means for generating a control signal to automatically adjust environmental settings according to the customer's emotions and transmitting it to an IoT device. This enables rapid response to changes in customer emotions and the environment, significantly improving customer satisfaction. It also improves store operation efficiency and enables prompt appropriate responses.

[1997] "User Credentials" means information provided to identify a user when accessing a system.

[1998] A "server" is a computer system that stores, processes, and manages data over a network.

[1999] A "database" is a system for organizing, storing, and managing data according to a specific structure.

[2000] An "authentication token" is a temporary digital certificate that indicates that a user is authenticated.

[2001] A "terminal" is an electronic device that can be directly operated by a user, and includes, for example, a smartphone or a tablet.

[2002] An "environmental sensor" is a device for measuring physical environmental data such as temperature, humidity, light level, and sound volume.

[2003] "Environmental Data" is information about the physical environment collected by environmental sensors.

[2004] An "emotion engine" is software that analyzes voice data and other user data to identify a user's emotional state.

[2005] An "IoT device" is a physical device that is connected to the Internet and can communicate with other devices.

[2006] A "real-time report" is a report that instantly analyzes collected data and provides results immediately.

[2007] A "control signal" is an electronic command sent to perform a particular action.

[2008] A "recommended action" is a suggestion of the best course of action for a particular situation.

[2009] A "user notification" is a message sent from the system to present information to the user.

[2010] An "API" is an interface that allows different software applications to communicate with each other.

[2011] This paper describes a system and a program for implementing the present invention. Specifically, the system is designed to respond to changes in customer emotions and the environment in a physical store, thereby improving customer satisfaction and streamlining store operations.

[2012] Overall system configuration

[2013] To realize the invention, a device (smartphone) used by the user, a server for analyzing data, various environmental sensors, and IoT devices are required. The server runs in a cloud environment and analyzes data sent from the device and sensors. The emotion engine runs on the server and analyzes voice data to identify the user's emotions.

[2014] User Authentication

[2015] Users authenticate by scanning an identifier such as a QR code using their smartphone. The authentication information (user ID and password) is sent to the server, which compares it with a database to generate an authentication token and sends it to the device. The device then displays the authentication result to the user, completing the identification process.

[2016] Environmental data collection and analysis

[2017] Various environmental sensors (temperature, humidity, volume, etc.) are connected to the device. Data acquired from these sensors is sent to the server via the device. The server analyzes the received environmental data, and if an abnormality is detected, generates recommended measures and sends them to the device. The device then displays the recommended measures to the user.

[2018] Emotional understanding and regulation

[2019] The device collects voice data through a microphone and sends it to a server. The server then uses an emotion engine to analyze the voice data and identify the user's emotional state. Based on the results of the analysis, it generates control signals to automatically change environmental settings (such as changing music or adjusting lighting) if necessary, and sends them to the IoT device.

[2020] Real-time notifications

[2021] The server integrates the analysis results of the environmental sensors and the emotion engine, and generates a report if it detects a problem in real time. This report is sent to the device and notifies the user, allowing for a prompt response.

[2022] Examples of concrete examples and prompts

[2023] For example, when a customer enters a store, a device collects the customer's voice and analyzes their emotions. If the customer is dissatisfied, that information is sent to the staff's device in real time, prompting them to take appropriate action. Also, if the temperature in the store rises and it detects that the customer is feeling stressed, the device will automatically adjust the air conditioning and change the music to something more relaxing.

[2024] Example prompt sentence:

[2025] "Use the emotion engine API to analyze customer emotions from collected voice data, and if the emotion is stressful, generate code to change the music in the store to something relaxing using the IoT device API."

[2026] This makes it possible to respond quickly to changes in customer emotions and the environment, improving customer satisfaction and streamlining store operations.

[2027] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2028] Step 1:

[2029] The user enters authentication information using a terminal and sends it to the server. The entered authentication information is the user ID and password. The server compares the received authentication information with the database and generates an authentication token. If the comparison is positive, the authentication token is sent to the terminal, and the terminal displays the authentication result to the user. The input is the authentication information, and the output is the authentication result and authentication token.

[2030] Step 2:

[2031] The device collects environmental data from environmental sensors (temperature, humidity, volume) and sends it to the server. The server analyzes the received environmental data and generates recommended measures if an abnormality is detected. The generated recommended measures are sent to the device and displayed to the user. The input is environmental data and the output is recommended measures.

[2032] Step 3:

[2033] The terminal collects voice data and sends it to the server for analysis by the emotion engine. The server analyzes the voice data and identifies the user's emotional state. If the identification result indicates that environmental adjustment is necessary, the server generates a control signal and sends it to the IoT device. The input is the voice data and emotional state, and the output is a control signal for environmental adjustment.

[2034] Step 4:

[2035] The server generates real-time reports and immediately notifies the terminal if an abnormal situation is detected. The terminal displays the received real-time reports to the user and generates notifications to support a quick response. The input is trouble-related data, and the output is the real-time report and user notification.

[2036] Step 5:

[2037] If necessary, the server combines environmental sensor data with sentiment analysis results to perform detailed analysis to identify the root cause. If further action is required, the analysis results are sent to the device as recommended countermeasures. The input is environmental data and sentiment analysis results, and the output is detailed analysis results and recommended countermeasures.

[2038] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2039] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2040] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2041] [Fourth embodiment]

[2042] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2043] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2044] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2045] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2046] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2047] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2048] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2049] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2050] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2051] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2052] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2053] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2054] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2055] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[2056] User Authentication

[2057] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a specific example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[2058] Emotion understanding and environmental sensor data analysis

[2059] The device collects temperature, humidity, smell, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a specific example, if the indoor temperature suddenly rises, the data is analyzed by a cloud server and an immediate adjustment to the air conditioner is recommended.

[2060] Real-time reporting and notifications

[2061] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a specific example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[2062] Integration with IoT devices

[2063] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[2064] Assistance Features

[2065] A user inputs a question or request into a terminal. The terminal sends this to a server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[2066] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[2067] The processing flow will be explained below.

[2068] User Authentication

[2069] Step 1:

[2070] The user enters a username and password into the terminal.

[2071] Step 2:

[2072] The terminal transmits the entered authentication information to the server.

[2073] Step 3:

[2074] The server compares the received authentication information with the database and generates an authentication result.

[2075] Step 4:

[2076] The server generates an authentication token based on the matching result and sends it to the terminal.

[2077] Step 5:

[2078] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[2079] Emotion understanding and environmental sensor data analysis

[2080] Step 1:

[2081] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[2082] Step 2:

[2083] The terminal transmits the collected environmental data to the server.

[2084] Step 3:

[2085] The server analyzes the received environmental data and detects abnormalities.

[2086] Step 4:

[2087] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[2088] Step 5:

[2089] The device displays recommended measures to the user.

[2090] Step 6:

[2091] The terminal collects the user's voice using a microphone and transmits it to the server.

[2092] Step 7:

[2093] The server analyzes the voice data and identifies the user's emotions.

[2094] Step 8:

[2095] The server sends the results of the emotion analysis to the device.

[2096] Step 9:

[2097] The device displays the results of the emotion analysis to the user.

[2098] Real-time reporting and notifications

[2099] Step 1:

[2100] The terminal collects trouble-related data from sensors.

[2101] Step 2:

[2102] The trouble-related data collected by the terminal is sent to the server.

[2103] Step 3:

[2104] The server analyzes the trouble-related data and generates real-time reports.

[2105] Step 4:

[2106] The server sends the generated real-time report to the terminal.

[2107] Step 5:

[2108] The terminal notifies the user of real-time reports.

[2109] Step 6:

[2110] The device will display the report contents to the user and also notify them of recommended measures.

[2111] Integration with IoT devices

[2112] Step 1:

[2113] The terminal collects status data from various IoT devices.

[2114] Step 2:

[2115] The terminal transmits the collected status data to the server.

[2116] Step 3:

[2117] The server consolidates and analyzes data from IoT devices.

[2118] Step 4:

[2119] The server determines abnormal values ​​and necessary actions and generates control signals.

[2120] Step 5:

[2121] The server sends a control signal to the IoT device via the terminal.

[2122] Step 6:

[2123] The terminal displays the status of IoT devices and actions taken to the user.

[2124] Assistance Features

[2125] Step 1:

[2126] The user types a question or request into the terminal.

[2127] Step 2:

[2128] The device sends the user's questions or requests to the server.

[2129] Step 3:

[2130] Based on the request received, the server generates an answer by referencing databases and Internet resources.

[2131] Step 4:

[2132] The server generates a response and sends it to the terminal.

[2133] Step 5:

[2134] The terminal displays the answer to the user.

[2135] As a result, the present invention enables detailed user authentication, emotion understanding, environmental sensor data analysis, real-time reporting, collaboration with IoT devices, and assistance functions, providing a safe and convenient lifestyle.

[2136] Example 1

[2137] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2138] In conventional systems, user authentication, environmental data analysis, trouble detection, and integration with various devices were all performed separately, resulting in a lack of overall integration and efficiency. Using multiple systems that are not centrally managed is inconvenient for users, and system management and operation require significant resources. Furthermore, real-time data analysis and trouble notification may not be possible, potentially compromising user safety and convenience.

[2139] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2140] In this invention, the server includes a means for receiving authentication information from a user and transmitting it to the server, a means for the server to compare the received authentication information with a database, and a means for generating an authentication token based on the comparison result and transmitting it to the terminal, thereby enabling efficient user authentication in an integrated system.

[2141] The system also includes a means for the terminal to collect data from the environmental sensors and send it to the server, a means for the server to analyze the received environmental data, detect abnormalities, generate recommended countermeasures and send them to the terminal, and a means for the user to input questions or requests to the terminal, which the server receives and provides corresponding information. This enables real-time analysis of environmental data, prompt presentation of countermeasures to the user, and interactive information provision.

[2142] "User authentication" is the process in which the server receives the authentication information (user name, password, etc.) entered by the user into the terminal and compares it with a database to perform authentication.

[2143] "Authentication information" is information that identifies a user, such as a username and password.

[2144] A "terminal" is a device (smartphone, computer, etc.) operated by a user.

[2145] A "server" is a system that receives authentication information and environmental data, analyzes them, and returns the results to the terminal.

[2146] A "database" is a collection of information that stores user authentication information, environmental data, and the like.

[2147] An "authentication token" is a unique digital key issued to identify a user once authenticated.

[2148] An "environmental sensor" is a device that collects environmental data such as temperature, humidity, odor, and sound.

[2149] An "anomaly" is data or an event that deviates from the normal range.

[2150] "Recommended measures" are countermeasures that the system generates when an abnormality is detected.

[2151] "Data analysis" is the process of detecting anomalies and generating countermeasures based on collected data.

[2152] "Trouble-related data" is data related to troubles such as the detection of abnormal sounds and unauthorized intrusions.

[2153] A "real-time report" is a report that is generated based on analyzed data and provided to the user immediately.

[2154] "IoT devices" are various home appliances and devices connected to the Internet.

[2155] A "control signal" is an operation command generated by a server and sent to an IoT device.

[2156] An "assistance function" is a function in which a server provides answers to user questions or requests by referencing databases and Internet resources.

[2157] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of the system and a specific example.

[2158] First, let's explain the user authentication process. The user enters their username and password into their device. The device encrypts this authentication information and sends it to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. As a concrete example, when a user logs in to a smartphone app, the authentication information is compared with the cloud authentication service and the result is returned instantly.

[2159] Next, we will explain the processes of emotion understanding and environmental sensor data analysis. The device collects temperature, humidity, smell, sound, etc. from the environmental sensors. The device encrypts the collected data and sends it to the server. The server analyzes the received environmental data and determines whether there are any abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data to identify the user's emotions and sends the results to the device. As a concrete example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[2160] Next, we will explain the process of real-time reports and notifications. The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. As a concrete example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[2161] We will also explain the process of linking with IoT devices. The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends this data to a server, which then integrates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. As a concrete example, it checks whether a smart lock is locked properly and automatically sends a locking signal if there is an abnormality.

[2162] Finally, we will explain the assistance function. The user inputs a question or request into the terminal. The terminal sends this to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends this answer to the terminal, which then displays it to the user. As a concrete example, if the user asks, "Which route should I take to get home today?" the server provides optimal route information, which the terminal instantly displays.

[2163] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[2164] Prompt Sentence Examples

[2165] "Please explain how user authentication works in a smartphone app."

[2166] "Describe a system that uses environmental sensors and audio data to analyze situations and emotions."

[2167] "Please explain the procedure for using sensors to detect problems in real time and notify users."

[2168] "Please explain a system that improves home safety by linking with IoT devices."

[2169] "Please explain how the assistance feature works to provide information in response to a user's question."

[2170] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2171] (user authentication)

[2172] Step 1:

[2173] The user enters a username and password into the terminal.

[2174] Input: Username, Password.

[2175] Output: The authentication information sent to the device.

[2176] Specific operation: The user enters their username and password into the login screen on their smartphone.

[2177] Step 2:

[2178] The device encrypts the authentication information and sends it to the server.

[2179] Input: The credentials entered by the user.

[2180] Output: The encrypted authentication information sent to the server.

[2181] Specific operation: Authentication information is encrypted inside the terminal using the SSL / TLS protocol and sent to the server.

[2182] Step 3:

[2183] The server checks the received authentication information against a database.

[2184] Input: Encrypted credentials.

[2185] Output: Matching result (authentication success or failure).

[2186] What happens: The server decrypts the received authentication information, checks it against a database, and verifies the integrity of the password using a hash function.

[2187] Step 4:

[2188] The server generates an authentication token or an error message based on the result of the match and sends it to the terminal.

[2189] Input: Matching result.

[2190] Output: An authentication token or an error message.

[2191] Specific operation: If authentication is successful, the server generates a unique authentication token; if it fails, it generates an error message. The generated result is encrypted and sent to the terminal.

[2192] Step 5:

[2193] The terminal displays the authentication result to the user.

[2194] Input: An authentication token or an error message.

[2195] Output: The authentication result that is displayed to the user.

[2196] Specific operation: The device decrypts the received authentication token, and if authentication is successful, it transitions to the app's home screen, or if it fails, it displays an error message.

[2197] (Emotion understanding and environmental sensor data analysis)

[2198] Step 1:

[2199] The device collects data from environmental sensors.

[2200] Input: Measurement data from environmental sensors (temperature, humidity, odor, sound, etc.).

[2201] Output: Collected environmental data.

[2202] Specific operation: The device periodically collects data from connected thermometers, hygrometers, air quality sensors, etc.

[2203] Step 2:

[2204] The data collected by the device is encrypted and sent to the server.

[2205] Input: Collected environmental data.

[2206] Output: Encrypted environment data.

[2207] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[2208] Step 3:

[2209] The server analyzes the received environmental data.

[2210] Input: Encrypted environment data.

[2211] Output: Analysis result (normal or abnormal).

[2212] Specific operation: The server decrypts the received environmental data and detects outliers using statistical methods and machine learning models.

[2213] Step 4:

[2214] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[2215] Input: Analysis results (if anomaly is detected).

[2216] Output: Recommended action.

[2217] Specific operation: If an abnormality is detected, such as a sudden rise in temperature, the server generates recommended measures such as "adjusting the air conditioner," encrypts them, and sends them to the device.

[2218] Step 5:

[2219] The device displays recommended measures to the user.

[2220] Input: Recommended measures.

[2221] Output: Recommended action displayed to the user.

[2222] Specific operation: The device will decode the recommended measures received and display a pop-up message or notification area saying "The temperature is too high. Please adjust the air conditioner."

[2223] (Real-time reports and notifications)

[2224] Step 1:

[2225] The terminal collects trouble-related data from sensors.

[2226] Input: Sensor measurement data (abnormal sounds, suspicious movements, etc.).

[2227] Output: Collected trouble-related data.

[2228] How it works: The device collects data in real time from sensors that detect abnormal sounds and movements.

[2229] Step 2:

[2230] The device encrypts the collected data and sends it to the server.

[2231] Input: Collected trouble-related data.

[2232] Output: Encrypted trouble-related data.

[2233] Specific operation: The device immediately encrypts the collected data using the SSL / TLS protocol and sends it to the server.

[2234] Step 3:

[2235] The server analyzes the data and generates real-time reports.

[2236] Input: Encrypted trouble-related data.

[2237] Output: Real-time reports.

[2238] How it works: The server decrypts the data, analyzes it using machine learning models, and generates a real-time report detailing any anomalies detected.

[2239] Step 4:

[2240] The server encrypts the generated real-time report and sends it to the terminal.

[2241] Enter: Real-time reports.

[2242] Output: Encrypted real-time reports.

[2243] Specific operation: The generated report is encrypted and sent to the device.

[2244] Step 5:

[2245] The device notifies the user of real-time reports received and recommended measures.

[2246] Input: Encrypted real-time reports.

[2247] Output: Real-time reports and recommended actions notified to the user.

[2248] Specific behavior: The device will decode the report and notify the user via push notification or alert message, for example, "Suspicious activity was detected at 8:30 PM. Please check for details."

[2249] (Linkage with IoT devices)

[2250] Step 1:

[2251] The terminal collects data from IoT devices.

[2252] Input: IoT device state data.

[2253] Output: Collected IoT device data.

[2254] Specific operation: The device periodically collects status data from the home's smart locks, smart lights, etc.

[2255] Step 2:

[2256] The device encrypts the collected data and sends it to the server.

[2257] Input: Collected IoT device data.

[2258] Output: Encrypted IoT device data.

[2259] Specific operation: The terminal batch processes the collected data, encrypts it using the SSL / TLS protocol, and sends it to the server.

[2260] Step 3:

[2261] The server analyzes and integrates the data.

[2262] Input: Encrypted IoT device data.

[2263] Output: Analysis results.

[2264] How it works: The server decrypts the data, aggregates and analyzes the data from multiple IoT devices, for example, to check whether a door is locked.

[2265] Step 4:

[2266] The server generates the necessary control signals and sends them to the terminal.

[2267] Input: Analysis results.

[2268] Output: Control signal.

[2269] Specific operation: A control signal is generated based on the analysis result. For example, if the door is open, a lock signal is generated.

[2270] Step 5:

[2271] The terminal sends a control signal to the IoT device.

[2272] Input: Control signal.

[2273] Output: The control signal sent to the IoT device.

[2274] Specific operation: The terminal decodes the received control signal and sends it to the target IoT device, for example, sending a locking signal to a smart lock and confirming its execution.

[2275] (Application example 1)

[2276] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2277] With conventional systems, it was difficult to centrally manage data collected from multiple sensors and devices, detect anomalies in real time, and take countermeasures. Furthermore, delays in notifying users and providing countermeasures could result in insufficient security measures being implemented. Furthermore, it was difficult to understand users' emotions and situations and provide appropriate actions accordingly.

[2278] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2279] In this invention, the server includes: means for receiving authentication information from a user and transmitting it to the server; means for the server to compare the authentication information received with a database; means for generating an authentication token based on the comparison result and transmitting it to the terminal; means for the terminal to display the authentication result to the user; means for collecting environmental data from a sensor and transmitting it to the terminal; means for the terminal to transmit the collected environmental data to the server; means for the server to analyze the environmental data and detect anomalies; means for generating recommended measures when an abnormality is detected and transmitting them to the terminal; means for the terminal to display the recommended measures to the user; means for linking with various sensors and collecting trouble-related data and transmitting it to the server; means for the server to analyze the trouble-related data and generate a real-time report; means for transmitting the generated real-time report to the terminal and notifying the user; means for linking with physical devices such as smart home appliances and security devices; means for automatically controlling physical devices in the event of an abnormality; means for providing optimal information based on a user's question or request; means for generating an answer to a user's question using a generative AI model; and means for displaying the generated answer to the user. This makes it possible to detect anomalies in real time, implement countermeasures, and provide appropriate actions by understanding user emotions.

[2280] "User authentication" is the process of verifying a user's identity using authentication information such as a username and password when the user accesses a system.

[2281] A "terminal" is a device operated by a user, such as a PC, smartphone, or tablet.

[2282] A "server" is a central computing device that processes and stores data and communicates with terminals via a network.

[2283] A "database" is a system for organizing, storing, and managing large amounts of data.

[2284] An "authentication token" is an identifier issued by the server when user authentication is successful, and is used to simplify subsequent authentications.

[2285] An "environmental sensor" is a device that detects environmental data such as temperature, humidity, sound, and smell.

[2286] "Real-time reporting" refers to information that instantly processes data collected from sensors and devices and reports current conditions and abnormalities.

[2287] "Recommended measures" are specific countermeasures that are presented to the user when an abnormality or problem occurs.

[2288] An "IoT device" is a physical device that can connect to the Internet, collect data, and be controlled.

[2289] "Smart home appliances" are household electrical appliances that have internet connectivity and can be remotely controlled or automated.

[2290] A "physical device" is a hardware device that actually operates, such as a smart lock or security camera.

[2291] A "generative AI model" is an algorithmic model that learns from large amounts of data and generates predictions and answers for new data.

[2292] "Emotion understanding" is the process of detecting a user's emotional state from their voice, facial expressions, etc., and understanding that state.

[2293] "Notification" is the act of conveying information to a user in real time via a terminal.

[2294] The embodiments for carrying out the present invention will be described in detail below.

[2295] User Authentication

[2296] The server receives the authentication information entered from the user's device and compares it with the database. If the comparison is successful, it generates an authentication token and sends it to the device. The device receives this token and displays the authentication result to the user. This achieves highly secure access control.

[2297] Emotion understanding and environmental sensor data analysis

[2298] The device sends environmental data, such as temperature, humidity, and sound, collected from environmental sensors to a server. The server analyzes this data and determines whether there are any abnormalities. If an abnormality is detected, the server generates recommended measures and sends them to the device. The device then displays these recommended measures to the user. The device also collects the user's voice and sends it to the server. The server uses a generative AI model to analyze this voice data and identify the user's emotions. The server then sends the results to the device and provides feedback appropriate to the user's situation.

[2299] Real-time reporting and notifications

[2300] The device collects trouble-related data from sensors and sends it to the server, which analyzes the data and generates a real-time report of specific troubles or abnormalities. This report is immediately sent to the device and notified to the user.

[2301] Integration with IoT devices

[2302] The server connects to various IoT devices, collects and analyzes data from these devices, and generates control signals as needed and sends them to the IoT devices via the terminals. This allows smart home appliances and security devices to automatically take appropriate action.

[2303] Assistance Features

[2304] When a user inputs a question or request into the device, the device sends it to the server. The server uses a generative AI model to generate the optimal answer to the user's question. This generated answer is displayed to the user via the device. For example, if a user asks, "Which route should I take to get home today?" the server will provide the optimal route taking into account the traffic conditions and weather for that day, and the device will display it.

[2305] Hardware and software used

[2306] The servers are operated using cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP).

[2307] For database management, relational databases such as MySQL and PostgreSQL are used.

[2308] The front-end devices used are iOS and Android smartphones and tablets.

[2309] Environmental sensors and IoT devices use microcontrollers such as Arduino and Raspberry Pi.

[2310] The generative AI model uses AI technologies such as OpenAI's GPT-4.

[2311] Example: Prompt sentence

[2312] For example, if a user requests, "Check if the door is open," the server analyzes data from the connected smart lock to determine the door's status and immediately notifies the user of the results.

[2313] In this way, the system of the present invention integrates multiple functions to provide users with safe and efficient information and control.

[2314] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2315] Step 1:

[2316] The user enters authentication information (user name and password) into the terminal.

[2317] Input: Username and Password.

[2318] Output: Sending authentication information to the server.

[2319] Step 2:

[2320] The server checks the received authentication information against a database.

[2321] Input: The authentication information sent from the device.

[2322] Output: Authentication result (success or failure).

[2323] Step 3:

[2324] If the authentication is successful, the server generates an authentication token and sends it to the terminal.

[2325] Input: Authentication result (success).

[2326] Output: An authentication token.

[2327] Step 4:

[2328] The terminal stores the received authentication token and displays the authentication result to the user.

[2329] Input: Authentication token.

[2330] Output: Display of authentication result.

[2331] Step 5:

[2332] The device collects data from environmental sensors (temperature, humidity, sound, etc.) and sends it to a server.

[2333] Input: Data from environmental sensors.

[2334] Output: Sending environmental data to the server.

[2335] Step 6:

[2336] The server analyzes the collected environmental data and detects abnormalities.

[2337] Input: Environmental data sent from the device.

[2338] Output: Anomaly detection results.

[2339] Step 7:

[2340] If an anomaly is detected, the server generates recommended measures and sends them to the device.

[2341] Input: Anomaly detection results.

[2342] Output: Recommended action.

[2343] Step 8:

[2344] The device displays the received recommended measures to the user.

[2345] Input: Recommended measures.

[2346] Output: Display of recommended actions.

[2347] Step 9:

[2348] The terminal collects trouble-related data (such as abnormal sounds) from sensors and sends it to the server.

[2349] Input: Trouble related data from sensors.

[2350] Output: Sending trouble-related data to the server.

[2351] Step 10:

[2352] The server analyzes the trouble-related data and generates real-time reports.

[2353] Input: Trouble-related data sent from the device.

[2354] Output: Real-time reports.

[2355] Step 11:

[2356] The server transmits the generated real-time report to the terminal and notifies the user.

[2357] Enter: Real-time reports.

[2358] Output: Real-time reports sent to terminal and user notifications.

[2359] Step 12:

[2360] The server collects and analyzes data from smart home appliances and security devices.

[2361] Input: Data from smart home appliances and security devices.

[2362] Output: Analysis results.

[2363] Step 13:

[2364] When necessary, the server generates control signals and sends them to the physical device via the terminal.

[2365] Input: Analysis results.

[2366] Output: Control signals to physical devices.

[2367] Step 14:

[2368] When a user enters a question or request into the terminal, the terminal sends it to the server.

[2369] Input: The user's question or request.

[2370] Output: Sending questions or requests to the server.

[2371] Step 15:

[2372] The server uses a generative AI model to generate the best answer to the user's question.

[2373] Input: The user's question or request.

[2374] Output: The generated answer.

[2375] Step 16:

[2376] The server sends the generated answer to the terminal, which displays it to the user.

[2377] Input: The generated answer.

[2378] Output: Display of answer to user.

[2379] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2380] A specific embodiment of the present invention will be described below, based on the processing of a program that realizes each function of a system that combines emotion engines, and specific examples.

[2381] User Authentication

[2382] The user enters a username and password into the device. The device sends this authentication information to the server. The server compares the received authentication information with a database and generates an authentication result. If authentication is successful, the server generates an authentication token and sends it to the device. If authentication fails, the server sends an error message. The device displays the authentication result to the user. For example, when a user logs in to a smartphone app, the authentication information is compared with a cloud authentication service and the result is returned instantly.

[2383] Emotion understanding and environmental sensor data analysis

[2384] The device collects temperature, humidity, odor, sound, etc. from environmental sensors. The device sends the collected data to a server. The server analyzes the received environmental data and detects abnormalities. If an abnormality is detected, it generates recommended measures and sends them to the device. The device displays these recommended measures to the user. For emotion understanding, the device collects the user's voice and sends it to the server. The server analyzes this voice data with an emotion engine to identify the user's emotion and sends the result to the device. For example, if the indoor temperature suddenly rises, the data is analyzed by the cloud server and an immediate adjustment of the air conditioner is recommended.

[2385] Real-time reporting and notifications

[2386] The device collects trouble-related data (e.g., abnormal sound detection) from the sensor and sends it to the server. The server analyzes the trouble-related data and generates a real-time report. This report is sent to the device and notified to the user. The device displays the received real-time report and recommended countermeasures to the user. For example, if a sensor detects suspicious activity at night, that information is notified to the user's smartphone in real time.

[2387] Integration with IoT devices

[2388] The terminal collects status data from various IoT devices (e.g., smart locks, smart lights). The terminal sends the data to the server, which then consolidates and analyzes the received data. If necessary, the server generates a control signal and sends it to the IoT device via the terminal. The terminal displays the current status of the IoT device and any actions taken to the user. For example, it checks whether a smart lock is locked properly and automatically sends a locking signal if an abnormality is detected.

[2389] Assistance Features

[2390] A user inputs a question or request into a device. The device sends it to the server, which then generates an answer by referencing a database or Internet resource based on the request. The server then sends the answer to the device, which then displays it to the user. For example, if a user asks, "Which route should I take to get home today?" the server provides optimal route information, which the device instantly displays.

[2391] Emotion Engine

[2392] In the specific operation of the emotion engine, the device collects the user's voice and sends it to the server. The server passes the voice data to the emotion engine and analyzes the user's emotions. Once the analysis results are obtained, the server sends the emotion data to the device. The device displays the received emotion data to the user and may also provide an appropriate assistance plan. For example, if the user is angry, the emotion engine will recognize that emotion and suggest music to help them relax.

[2393] Furthermore, the server can generate control signals based on the emotion data to dynamically adjust the data from the environmental sensors. These control signals are used to change environmental conditions such as temperature and lighting. For example, if a user is feeling stressed, changing the room lighting to warmer colors can promote relaxation.

[2394] The above is a specific embodiment of the present invention, which is expected to realize safe and convenient daily life and improve customer satisfaction.

[2395] The processing flow will be explained below.

[2396] User Authentication

[2397] Step 1:

[2398] The user enters a username and password into the terminal.

[2399] Step 2:

[2400] The terminal transmits the entered authentication information to the server.

[2401] Step 3:

[2402] The server compares the received authentication information with the database and generates an authentication result.

[2403] Step 4:

[2404] The server generates an authentication token based on the matching result and sends it to the terminal.

[2405] Step 5:

[2406] The device displays the authentication result to the user. If authentication is successful, the device transitions to the home screen, and if authentication is unsuccessful, an error message is displayed.

[2407] Emotion understanding and environmental sensor data analysis

[2408] Step 1:

[2409] The device collects data such as temperature, humidity, smell, and sound from environmental sensors.

[2410] Step 2:

[2411] The terminal transmits the collected environmental data to the server.

[2412] Step 3:

[2413] The server analyzes the received environmental data and detects abnormalities.

[2414] Step 4:

[2415] If the server detects an abnormality, it generates recommended countermeasures and sends them to the terminal.

[2416] Step 5:

[2417] The device displays recommended measures to the user.

[2418] Step 6:

[2419] The terminal collects the user's voice using a microphone and transmits it to the server.

[2420] Step 7:

[2421] The server analyzes the voice data using an emotion engine to identify the user's emotion.

[2422] Step 8:

[2423] The server sends the results of the emotion analysis to the device.

[2424] Step 9:

[2425] The device displays the results of the emotion analysis to the user.

[2426] Real-time reporting and notifications

[2427] Step 1:

[2428] The terminal collects trouble-related data from sensors.

[2429] Step 2:

[2430] The trouble-related data collected by the terminal is sent to the server.

[2431] Step 3:

[2432] The server analyzes the trouble-related data and generates real-time reports.

[2433] Step 4:

[2434] The server sends the generated real-time report to the terminal.

[2435] Step 5:

[2436] The terminal notifies the user of real-time reports.

[2437] Step 6:

[2438] The device will display the report contents to the user and also notify them of recommended measures.

[2439] Integration with IoT devices

[2440] Step 1:

[2441] The terminal collects status data from various IoT devices.

[2442] Step 2:

[2443] The terminal transmits the collected status data to the server.

[2444] Step 3:

[2445] The server consolidates and analyzes data from IoT devices.

[2446] Step 4:

[2447] The server determines abnormal values ​​and necessary actions and generates control signals.

[2448] Step 5:

[2449] The server sends a control signal to the IoT device via the terminal.

[2450] Step 6:

[2451] The terminal displays the status of IoT devices and actions taken to the user.

[2452] Assistance Features

[2453] Step 1:

[2454] The user types a question or request into the terminal.

[2455] Step 2:

[2456] The device sends the user's questions or...

Claims

1. means for receiving authentication information from a user and transmitting it to a server; means for the server to check the received authentication information against a database; means for generating an authentication token based on the matching result and transmitting the token to the terminal; a means for the terminal to display the authentication result to the user; A system including:

2. means for collecting environmental data from the sensors and transmitting the data to the terminal; means for transmitting environmental data collected by the terminal to a server; A means for the server to analyze environmental data and detect abnormalities; A means for generating and transmitting recommended measures to the terminal when an anomaly is detected; a means for the device to display the recommended measures to the user; The system of claim 1 , comprising:

3. A means of linking with various sensors to collect trouble-related data and send it to a server; A means for the server to analyze the trouble-related data and generate a real-time report; a means for transmitting the generated real-time report to a terminal and notifying the user; The system of claim 1 , comprising:

Images