system

Abstract

We provide the system. [Solution] A means for registering real estate information, A means for users to input information about home appliances, A means of analyzing device information, A means of sending home appliance registration information to a central server, A means of configuring the device based on the transmitted information, A means of reflecting new device information based on information sent from the central server, A means of notifying users of home appliance information, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, household appliances and smart home devices have become popular, but there is a lack of a system for comprehensively managing them, and the problem is that the burden on users to manually set and manage individual devices is large. In addition, since there are few smart home systems in cooperation with real estate, the reconfiguration of devices associated with moving or purchasing a new property has become complicated. Therefore, there is a need to provide a method that allows users to more easily manage household appliances and the entire system.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a system that includes means for registering real estate information, means for the user to input home appliance information, means for analyzing device information, means for transmitting the registration information of home appliances to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, and means for notifying the user of home appliance information. Furthermore, the system includes means for recognizing voice commands and transmitting the analyzed content to a central server, means for generating control commands for home appliances based on the user's voice commands and transmitting them to the corresponding devices, means for receiving control commands sent from the central server and adjusting the status of home appliances, means for providing an application for the user to centrally manage real estate information and home appliance information, means for automatically generating a list of corresponding home appliances based on real estate property information, a robot for the user to issue voice commands, and means for transmitting the voice commands recognized by the robot to a central server. As a result, the user can intuitively centrally manage the entire home appliance and system, eliminating the complexity of setup and management.

[0006] "Real estate information" refers to information that includes details such as the property's location, size, layout, owner information, and contract terms.

[0007] A "user" is an individual or legal entity that uses the system to manage real estate information or home appliance information.

[0008] "Home appliance information" refers to information that includes details such as the model, model number, settings, and connection status of each home appliance.

[0009] "Device information" refers to the unique ID, connection status, and other related information of home appliances and smart devices.

[0010] A "central server" is a server that serves as the core of a system, responsible for data storage, analysis, and relaying commands.

[0011] "Device setup" refers to the process of adjusting and initializing the operating parameters of home appliances and smart devices.

[0012] "Notification" refers to the act of communicating information from a system to a user.

[0013] A "voice command" is a command that the system recognizes and analyzes when the user gives instructions by voice.

[0014] A "control command" is an instruction sent from a central server to each device that controls the operation of the device.

[0015] An "application" is software that users use to manage real estate information, home appliance information, and other similar data.

[0016] A "robot" is a device that recognizes voice commands through interaction with the user and transmits them to a central server.

[0017] "Analysis" is the process of understanding information collected within a system and deriving meaning from it. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing apparatus and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the language used in the following description will be explained.

[0021] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] The present invention is a system for integrated management and control of real estate information and home appliance information, designed to be intuitively operable by the user. Specific embodiments of the present invention are described below.

[0040] Program processing

[0041] Overall Overview

[0042] The system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands, and a terminal (smartphone) that receives user input. This allows for centralized management of property information and smart devices throughout the home.

[0043] User registers with the system

[0044] 1. User

[0045] When registering a new account using the smartphone app, you will need to enter the required information (name, address, email address, phone number, etc.) and submit it to the system.

[0046] 2. Server

[0047] The server receives user information and stores it in the database.

[0048] Property registration and device setup

[0049] 1. User

[0050] Register information about newly purchased or rented property in the app. Enter details such as the property's location, size, and floor plan.

[0051] Select a list of home appliances you plan to install in your home.

[0052] 2. Server

[0053] The server stores property information in a database and retrieves a list of suitable appliances for the user's home, along with related initial setup information.

[0054] The system analyzes information about the home appliances selected by the user and generates initial setup parameters corresponding to each device.

[0055] Home appliance and system management

[0056] 1. Device (smartphone)

[0057] Users scan the QR code (registered trademark) on their home appliances using the app, and the appliance information is added to the system.

[0058] Information about the added home appliance is sent to the server and registered in the database.

[0059] 2. Server

[0060] Based on the registered information of the home appliances, the initial setup is performed and applied to each device.

[0061] The registered information is automatically updated and reflected in the user's system.

[0062] 3. User

[0063] The status of all home appliances and systems can be checked in real time via the app.

[0064] Operation by voice command

[0065] 1. User

[0066] You can issue voice commands to the robot, such as "Turn up the living room lights."

[0067] 2. Butler Robot

[0068] It recognizes voice commands and analyzes their content.

[0069] The analyzed instructions are sent to the central server as commands.

[0070] 3. Server

[0071] The server receives the command and generates a control command for the corresponding home appliance.

[0072] The generated command is sent to the target device.

[0073] 4. Home appliances (lighting systems)

[0074] The lighting system adjusts its brightness based on commands received from the server.

[0075] 5. Users

[0076] The lighting is adjusted to the specified level, making the environment more comfortable.

[0077] Specific example

[0078] Listening: Adding home appliances

[0079] 1. User

[0080] If you want to install your newly purchased smart refrigerator in your home, launch the smartphone app and scan the QR code on the refrigerator.

[0081] 2. Device (smartphone)

[0082] The QR code information is analyzed, and the refrigerator's device information is obtained.

[0083] 3. Server

[0084] Device information is sent to the server and registration is performed.

[0085] The refrigerator's initial setup is performed and reflected in the system.

[0086] 4. Butler Robot

[0087] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[0088] Example 2: Adjusting the room lighting

[0089] 1. User

[0090] If you want to adjust the living room lighting by voice, you can say to the butler robot, "Brighten the living room lights."

[0091] 2. Butler Robot

[0092] It recognizes and analyzes voice commands.

[0093] Send the analysis results to the server.

[0094] 3. Server

[0095] It receives a command and generates a command to increase the brightness of the living room lighting system.

[0096] Send a command to the lighting system.

[0097] 4. Terminal (lighting system)

[0098] It receives commands from the server and adjusts the brightness of the lighting.

[0099] 5. Users

[0100] The lighting is adjusted to achieve the desired brightness.

[0101] These concrete examples demonstrate that users can intuitively and efficiently manage home appliances and systems, thereby improving their quality of life.

[0102] The following describes the processing flow.

[0103] User registers with the system

[0104] Step 1:

[0105] The user launches the app on their smartphone and taps the "New Registration" button.

[0106] Step 2:

[0107] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[0108] Step 3:

[0109] The device (smartphone) sends the entered information to the central server.

[0110] Step 4:

[0111] The server saves the received information to the database and creates user accounts.

[0112] Step 5:

[0113] The server returns a response to the terminal indicating successful account creation and notifies the user that account creation is complete.

[0114] Property registration and device setup

[0115] Step 1:

[0116] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[0117] Step 2:

[0118] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[0119] Step 3:

[0120] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[0121] Step 4:

[0122] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[0123] Step 5:

[0124] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[0125] Home appliance and system management

[0126] Step 1:

[0127] The user installs the new appliance and scans a QR code with a smartphone app.

[0128] Step 2:

[0129] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[0130] Step 3:

[0131] The terminal sends the information it has acquired to the central server.

[0132] Step 4:

[0133] The server receives home appliance information, saves it to a database, and configures the devices.

[0134] Step 5:

[0135] The server sends a notification to the device indicating that the setup is complete.

[0136] Step 6:

[0137] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[0138] Operation by voice command

[0139] Step 1:

[0140] The user issues voice commands to the butler robot, such as "Turn up the living room lights."

[0141] Step 2:

[0142] The butler robot recognizes voice commands, analyzes their content, and generates commands to control the lighting.

[0143] Step 3:

[0144] The butler robot sends the control commands it generates to the central server.

[0145] Step 4:

[0146] The server checks the device ID of the lighting system and generates a command to adjust the brightness.

[0147] Step 5:

[0148] The server generates commands and sends them to the lighting system devices.

[0149] Step 6:

[0150] The lighting system sets the brightness based on the command it receives.

[0151] Step 7:

[0152] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[0153] Step 8:

[0154] The butler robot notifies the user via voice, "I've brightened the living room lights."

[0155] This series of processing steps allows users to intuitively and efficiently manage their home appliances and the entire system.

[0156] (Example 1)

[0157] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0158] With the recent proliferation of smart home appliances, there is a growing need to efficiently manage numerous devices and property information. However, current systems require users to manage information individually using different platforms and applications, leading to cumbersome and inefficient systems. Furthermore, even with voice-controlled appliances, there is no unified procedure or system, and user convenience is not adequately considered. To address these challenges, it is necessary to provide a system that allows users to easily integrate and manage smart home appliances and use it intuitively.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0160] In this invention, the server includes means for a user to register a new account using a mobile terminal and transmit necessary information to a central server; means for a user to input property information and a list of home appliances using a mobile terminal and transmit them to a central server; means for the central server to store and analyze the received user information and property information in a database; means for scanning the QR code of a home appliance with a mobile terminal and transmitting device information to a central server; means for the central server to perform initial setup of the home appliance based on the received device information; means for the central server to transmit the generated setting information to the home appliance and adjust the status of the home appliance; means for a user to issue a voice command to a robot, for the robot to recognize and analyze the voice command and transmit it to a central server; means for the central server to generate control commands for the home appliance based on the analyzed voice command and transmit them to the home appliance; and means for a user to check and control the status of the home appliance in real time using a mobile terminal. This enables users to centrally manage a variety of home appliance and real estate information and operate them intuitively and efficiently.

[0161] "Account registration" is the process by which a user registers their information with the system, thereby obtaining a unique identifier and enabling them to use the system.

[0162] A "central server" is a central computer system that receives information sent by users, stores it in a database, and performs necessary analysis and processing.

[0163] "User information" refers to personally identifiable information such as the user's name, address, email address, and phone number.

[0164] A "database" is a collection of digital information used to efficiently store, manage, and retrieve information within a system.

[0165] A "mobile device" refers to a communication device that can be carried and used, such as a smartphone or tablet.

[0166] "Real estate information" refers to detailed information about a house or building, such as its location, size, and floor plan.

[0167] A "household appliance list" is a list of electrical appliances used in a home.

[0168] A "QR code" is a rectangular, two-dimensional barcode that contains various types of information, and by scanning it, data can be quickly retrieved.

[0169] "Initial setup" refers to the basic configuration process required to make a new device usable.

[0170] "Voice commands" refer to a method in which a user uses their voice to instruct a system or device to perform a specific action.

[0171] A "robot" is an automated mechanical device that has the function of receiving and recognizing voice commands and transmitting those commands to a central server.

[0172] A "control command" is an electronic instruction that tells a device to perform a specific action.

[0173] "Real-time" means that system or device operations and information updates occur instantly.

[0174] "Centralized management" refers to the integrated management of multiple devices and information through a single system or platform.

[0175] "Analysis" is the process of breaking down received data or information into smaller parts, understanding it, and using it for a specific purpose.

[0176] "Status adjustment" means changing or setting up the operating conditions of a device based on user instructions.

[0177] This invention relates to a system for the integrated management and control of real estate information and home appliance information. Designed for intuitive user operation, it consists of a mobile terminal, a central server, a home appliance control robot, and smart home appliances.

[0178] Program processing

[0179] Overall Overview

[0180] The system consists of a mobile application, a central server, a robot that receives user voice commands, and smart home appliances that receive user input. This system allows for centralized management of property information and smart devices throughout the home.

[0181] User account registration

[0182] Users register for a new account using their mobile device. Users enter information such as their name, address, email address, and phone number, and send it to a central server. The server receives the user information and stores it in a database. This database may use a database such as MySQL®.

[0183] Real estate and home appliance information registration

[0184] Users input new property information and appliance lists using their mobile devices and send them to a central server. The server stores the received property information in a database and performs analysis. Based on the property information, it automatically generates an appropriate appliance list and provides it to the user. A Python script could be used for this analysis.

[0185] Initial setup and management of home appliances

[0186] The user scans a QR code on a home appliance with their mobile device and sends the device information to a central server. The server uses the received device information to perform the initial setup of the home appliance and sends the generated configuration information to the appliance. This process uses the device manufacturer's API (e.g., Philips Hue API).

[0187] Controlling home appliances by voice commands

[0188] The user issues a voice command to the robot, such as "Brighten the living room lights." The robot recognizes the voice command, analyzes its content, and sends it to a central server. This recognition uses the Google® Speech-to-Text API. Based on the analyzed voice command, the server generates control commands for the appliance and sends them to the target appliance. The appliance adjusts its operation according to the received control commands.

[0189] Specific example

[0190] Example 1: Adding home appliances

[0191] When a user installs a newly purchased smart refrigerator at home, they launch the app on their mobile device and scan the refrigerator's QR code. The QR code information is analyzed, and the refrigerator's device information is retrieved. The device information is sent to a central server for registration. The refrigerator's initial setup is performed and reflected in the system. A robot retrieves the new device information and notifies the user that "the new refrigerator has been registered."

[0192] Example 2: Adjusting the room lighting

[0193] If a user wants to adjust the living room lighting by voice, they would say to the robot, "Brighten the living room lights." The robot would recognize and analyze the voice command. It would then send the analysis results to a central server. The server would receive the analysis results, generate a command to increase the brightness of the living room lighting system, and send it. The lighting system would receive the command and adjust the brightness. The user would then confirm the lighting change and see that the desired brightness had been achieved.

[0194] Example of a prompt

[0195] "Please explain how to add a new home appliance. In particular, please provide detailed instructions on scanning the QR code and the data processing on the server side."

[0196] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0197] Step 1: User account registration

[0198] Input: The user enters information such as their name, address, email address, and phone number using their mobile device and presses the "Register" button.

[0199] Output: The data is sent to the central server, and a registration success message is returned to the user's device.

[0200] Specific actions:

[0201] 1. The user launches the app on their mobile device and is directed to the account registration screen.

[0202] 2. The user enters their name, address, email address, and phone number, and presses the submit button. This sends an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / register).

[0203] 3. The server verifies the received information and saves it to the database. Upon success, a response containing status code 200 is returned, and the user receives the message "Registration complete."

[0204] Step 2: Register property and appliance information

[0205] Input: The user enters property information and a list of appliances using a mobile device and presses the submit button.

[0206] Output: The transmitted information is analyzed by a central server and stored in a database. A list of appropriate home appliances is also generated.

[0207] Specific actions:

[0208] 1. The user selects the "Real Estate Information Registration" screen from the app's menu.

[0209] 2. Enter property information such as location, size, and floor plan, and select the desired appliances from the appliance list.

[0210] 3. After entering the information, press the submit button to send an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / properties / register).

[0211] 4. The server analyzes the received information and stores it in a database (e.g., MySQL). Using multiple SQL queries, it registers real estate information and appliance list information, and generates appliance settings suitable for the property specified by the user.

[0212] Step 3: Initial setup and management of home appliances

[0213] Input: The user scans the QR code on the home appliance with their mobile device and sends the obtained device information.

[0214] Output: Device information is sent to the central server for registration and initial setup.

[0215] Specific actions:

[0216] 1. The user places the new home appliance and launches the app on their mobile device.

[0217] 2. Use the app's QR code scanner function (e.g., Zebra Crossing) to scan the QR code on the home appliance and obtain device information.

[0218] 3. Send the retrieved device information as an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / devices / register).

[0219] 4. The server stores the received device information in a database and generates an initial configuration file using a Python script or shell script, which is then distributed to each home appliance.

[0220] Step 4: Controlling home appliances with voice commands

[0221] Input: The user issues a voice command to the robot, such as "Brighten the living room lights."

[0222] Output: The robot recognizes voice commands, sends the analysis results to a central server, and generates control commands for home appliances based on those results.

[0223] Specific actions:

[0224] 1. The user issues a pre-set voice command to the home appliance control robot.

[0225] 2. The robot uses the Google Speech-to-Text API to recognize voice commands and convert their content into text format.

[0226] 3. Send the analysis results to the central server as an HTTP POST request. The API endpoint is (e.g., https: / / example.com / api / voicecommands).

[0227] 4. The server generates control commands for the target device based on the received analysis results. For example, it creates an SQL query such as "UPDATE devices SET brightness = ? WHERE location = ?" and sends the adjustment command to the smart lighting device.

[0228] 5. The lighting system adjusts the brightness based on the received command, and the user confirms the change.

[0229] (Application Example 1)

[0230] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0231] In modern factory operations, a wide variety of equipment and sensors are introduced, but it is difficult to manage and control them efficiently in a unified manner. Therefore, it is necessary to use separate management systems for each piece of equipment, leading to increased complexity and costs. Furthermore, there is a lack of adequate systems for real-time monitoring of equipment status and early detection of abnormalities. In particular, there is a need to quickly reflect information on newly introduced equipment in the system, and to perform continuous status monitoring and appropriate operation.

[0232] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0233] In this invention, the server includes means for registering real estate information, means for users to input equipment information, means for analyzing device information, means for transmitting equipment registration information to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of equipment information, means for recognizing voice commands and transmitting the analyzed content to the central server, means for generating equipment control commands based on the user's voice commands and transmitting them to the corresponding devices, means for receiving control commands sent from the central server and adjusting the equipment status, means for providing an application for users to centrally manage property information and equipment information, means for automatically generating a list of corresponding equipment based on property information, a robot for users to issue voice commands, means for transmitting voice commands recognized by the robot to the central server, means for notifying the user when new equipment is registered, and means for updating the operating status of equipment in real time. This makes it possible to centrally manage and efficiently control a wide variety of equipment and sensors in a factory. In addition, information on newly introduced equipment can be quickly reflected in the system, enabling real-time monitoring of operating status and appropriate operation.

[0234] (definition statement)

[0235] "Real estate information" is a general term for data related to properties owned or managed by a user. This includes information such as the property's location, size, and floor plan.

[0236] "Device information" refers to the collective data about devices and equipment owned or managed by a user. This includes information such as the type of device, its function, and its operating status.

[0237] "Device information" is a general term for detailed technical data about equipment and devices. This includes information such as the device's components, specifications, and operating status.

[0238] A "central server" refers to a central computer system that receives, manages, and controls data from a large number of users and devices.

[0239] "Voice commands" refer to instructions given by a user to a system using their voice. Examples include commands such as "Brighten the living room lights."

[0240] A "control command" refers to an instruction used to tell a device or equipment to perform a specific action. These commands are generated by a central server and sent to each device.

[0241] "Operating status" refers to information about the current state in which a piece of equipment or device is operating. This includes states such as normal operation, malfunction, and awaiting maintenance.

[0242] An "application" refers to software that allows users to centrally manage real estate and equipment information using devices such as smartphones and tablets.

[0243] A "robot" refers to an autonomous mechanical device that has the function of receiving voice commands from a user, analyzing them, and transmitting them to a central server.

[0244] "Notification" refers to the act of transmitting information from the system to the user. This includes changes in the registration status and operating status of equipment.

[0245] This invention relates to a system for centrally managing and efficiently controlling equipment and sensors within a factory. The system is designed for intuitive user operation and consists of a smartphone application, a central server, and a robot that receives voice commands. Detailed embodiments for carrying out this invention are described below.

[0246] User registers with the system

[0247] 1. User

[0248] When users register a new account using the smartphone application, they enter necessary information such as their name, address, email address, and phone number, and submit it to the system. This information is stored on the central server, as described below.

[0249] 2. Server

[0250] The server receives user information and stores it in the database. This is achieved using a database management library such as SQLAlchemy.

[0251] Device registration and device configuration

[0252] 1. User

[0253] Users register information about newly installed factory equipment using a smartphone application. By scanning a QR code, they can input detailed information such as the type of equipment, performance, and operating status into the system.

[0254] 2. Server

[0255] The server stores equipment information in a database and generates a list of equipment suitable for the user's factory, along with associated initial configuration information. The server then analyzes this information and generates initial configuration parameters for each device. This process is implemented by a server program combining Python's Flask and SQLAlchemy.

[0256] Equipment management and control

[0257] 1. Device (smartphone)

[0258] Users scan the device's QR code using a smartphone application and register their information. This information is sent to a server, and the appropriate device settings are configured.

[0259] 2. Server

[0260] The server performs the initial setup of the equipment based on the received information and reflects this in the database. Furthermore, the registered information is automatically updated and reflected in the user's system in real time.

[0261] 3. User

[0262] Users can check the operating status of all devices in real time using the application.

[0263] Operation by voice command

[0264] 1. User

[0265] Users can issue voice commands to the robot, such as "Check the status of a specific device." This allows users to operate the system without using their hands.

[0266] 2. Robots

[0267] The robot recognizes voice commands and analyzes their content. The analyzed commands are then sent to a central server. The Python SpeechRecognition library is used for this speech recognition.

[0268] 3. Server

[0269] The server receives the instruction and generates control commands for the corresponding device. These commands are then sent from the central server to the device.

[0270] 4. Equipment

[0271] The equipment adjusts its operation based on commands received from the server. For example, it may change the operating status of a specified piece of equipment.

[0272] 5. Users

[0273] When the system functions as expected, users experience improved ease of use.

[0274] Specific example

[0275] Adding equipment

[0276] When a user installs a newly purchased press machine in their factory, they launch a smartphone application and scan the press machine's QR code. The QR code information is analyzed, and the press machine's device information is retrieved. The device information is sent to a server and registered in a database. Subsequently, a robot retrieves the new device information and notifies the user that "the new equipment has been registered."

[0277] Monitoring and adjustment of equipment operating status

[0278] When a user wants to check the operating status of a specific device and operate it, they check the status from a smartphone application and give a voice command to the robot saying, "Check the status of this device." The voice command is recognized, and the analysis results are sent to the server. The server receives the command, generates and sends control commands for the device in question. The device receives the commands from the server and adjusts its operating status. This allows the user to monitor and operate the device's status in real time.

[0279] Example of a prompt

[0280] If you want to install newly purchased equipment in your factory, how do you scan the QR code and register it in the system?

[0281] How can I check the operating status of specific equipment within the factory?

[0282] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0283] Step 1:

[0284] The user newly registers an account using a smartphone application. The user inputs necessary information such as name, address, email address, and phone number and sends it to the system. The input data is sent from the application to the server. The server receives this data and stores it in the database.

[0285] Step 2:

[0286] The user registers information of newly introduced factory equipment in the smartphone application. The user scans the QR code of the factory equipment and inputs detailed information such as the type, performance, and operating status of the equipment. This data is sent from the smartphone application to the server. The server analyzes the received device information and stores it in the database.

[0287] Step 3:

[0288] The server generates initial setting parameters corresponding to each device. For this, an algorithm that automatically determines appropriate parameters based on the type and characteristics of the device is used. The generated initial setting parameters are registered in the database.

[0289] Step 4:

[0290] The user scans the QR code of the device using the smartphone application to complete the registration. As a result, information of the new device is added to the system. The added device information is sent to the server and registered in the database.

[0291] Step 5:

[0292] The server performs the initial setting of the device and reflects it on the device. The server generates and sends a setting command for each device based on the initial setting parameters for each device. The device applies the initial setting based on the received command.

[0293] Step 6:

[0294] Users can check the operating status of all equipment in real time using a smartphone application. The application retrieves the latest equipment information from the server and displays it to the user. This process allows users to always be aware of the status of equipment in the factory.

[0295] Step 7:

[0296] The user issues a voice command to the robot. For example, they might give the robot a command such as, "Check the status of a specific device." The robot recognizes this voice command and analyzes its content. The analyzed command is then sent to the server as a command.

[0297] Step 8:

[0298] The server generates control commands for the corresponding device based on the received voice commands. The generated control commands are sent from the central server to the device. The device adjusts its operation according to these control commands.

[0299] Step 9:

[0300] Users benefit from improved operational convenience when the system functions as expected. When control is functioning correctly, users can confidently continue their work within the factory.

[0301] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0302] The present invention is a system that integrates and manages real estate information and home appliance information, and combines it with an emotion engine that recognizes user emotions. The following describes specific embodiments of the present invention.

[0303] Program Processing

[0304] Overall Overview

[0305] This system consists of an application for managing real estate information and home appliance information, a central server, a robot that receives the user's voice commands and emotions, and a terminal (smartphone) that receives user operations. Thereby, property information and smart devices in the home can be integrated and managed, and the operation of the system can be adjusted according to the user's emotions.

[0306] User Registration with the System

[0307] 1. User

[0308] Launch the smartphone app and tap the "New Registration" button.

[0309] Enter the necessary personal information and press the "Send" button.

[0310] 2. Server

[0311] Save the received information in the database and create a user account.

[0312] Send a response indicating the success of account creation to the terminal and notify the user.

[0313] Real Estate Registration and Device Settings

[0314] 1. User

[0315] Register real estate information in the app and select the home appliances to be installed from the home appliance list.

[0316] <0​​​​​​​Home appliance and system management

[0319] 1. User

[0320] Install the new appliance and scan the QR code with a smartphone app.

[0321] 2. Device (smartphone)

[0322] The system analyzes QR code information and sends the home appliance information to a central server.

[0323] 3. Server

[0324] The system saves the received home appliance information to a database and configures the device settings.

[0325] A notification will be sent to the device and displayed upon completion of the setup.

[0326] Operation via voice commands and emotion recognition

[0327] 1. User

[0328] It issues voice commands such as "Turn up the living room lights" and also expresses emotions.

[0329] 2. Butler Robot

[0330] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[0331] Voice commands and emotional data are sent to a central server.

[0332] 3. Server

[0333] Based on the received voice commands and emotion data, the device ID of the lighting system is identified, and the appropriate control command is generated.

[0334] Sends a command to the lighting system device.

[0335] 4. Home appliances (lighting systems)

[0336] Adjustments are made based on the received commands.

[0337] 5. Users

[0338] Comfort is ensured by adjusting the lighting to a specified level and making appropriate environmental adjustments.

[0339] Specific example

[0340] Example 1: Adding home appliances

[0341] 1. User

[0342] I installed the newly purchased smart refrigerator in my home, launched the smartphone app, and scanned the QR code.

[0343] 2. Device (smartphone)

[0344] The system acquires and analyzes QR code information and sends device information to the server.

[0345] 3. Server

[0346] The system receives device information and registers home appliances in its database.

[0347] Perform the initial setup of the refrigerator and reflect the information in the system.

[0348] 4. Butler Robot

[0349] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[0350] Example 2: Lighting adjustment

[0351] 1. User

[0352] If you want to adjust the living room lighting using your voice, say "Brighten the living room lights" and express your emotion.

[0353] 2. Butler Robot

[0354] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[0355] It sends voice commands and emotional data to the server.

[0356] 3. Server

[0357] Based on voice commands and emotion data, the system identifies the device ID of the lighting system and generates appropriate control commands.

[0358] Send a command to the lighting system.

[0359] 4. Terminal (lighting system)

[0360] Receives commands and adjusts to the specified level.

[0361] 5. Users

[0362] The lighting is adjusted, and the environment is kept in an optimal state according to the user's emotions.

[0363] This allows users to intuitively and efficiently manage their home appliances and entire systems, and further improves their quality of life through emotion recognition.

[0364] The following describes the processing flow.

[0365] User registers with the system

[0366] Step 1:

[0367] The user launches the app on their smartphone and taps the "New Registration" button.

[0368] Step 2:

[0369] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[0370] Step 3:

[0371] The device (smartphone) sends the entered information to the central server.

[0372] Step 4:

[0373] The server saves the received information to the database and creates user accounts.

[0374] Step 5:

[0375] The server returns a response to the terminal indicating successful account creation, notifying the user that account creation is complete.

[0376] Property registration and device setup

[0377] Step 1:

[0378] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[0379] Step 2:

[0380] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[0381] Step 3:

[0382] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[0383] Step 4:

[0384] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[0385] Step 5:

[0386] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[0387] Home appliance and system management

[0388] Step 1:

[0389] The user installs the new appliance and scans a QR code with a smartphone app.

[0390] Step 2:

[0391] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[0392] Step 3:

[0393] The terminal sends the information it has acquired to the central server.

[0394] Step 4:

[0395] The server receives home appliance information, saves it to a database, and configures the devices.

[0396] Step 5:

[0397] The server sends a notification to the device indicating that the setup is complete.

[0398] Step 6:

[0399] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[0400] Operation via voice commands and emotion recognition

[0401] Step 1:

[0402] The user can issue voice commands such as "Turn up the living room lights" and express emotions.

[0403] Step 2:

[0404] The butler robot recognizes voice commands and emotions, and analyzes their content.

[0405] Step 3:

[0406] The butler robot sends voice commands and emotional data to a central server.

[0407] Step 4:

[0408] The server receives voice commands and emotion data, verifies the device ID of the lighting system, and generates appropriate control commands.

[0409] Step 5:

[0410] The server generates control commands which are then sent to the lighting system devices.

[0411] Step 6:

[0412] The lighting system sets the brightness based on the command it receives.

[0413] Step 7:

[0414] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[0415] Step 8:

[0416] The butler robot notifies the user via voice, "I've brightened the living room lights."

[0417] This allows users to efficiently and intuitively manage their home appliances and entire systems using voice commands and emotions.

[0418] (Example 2)

[0419] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0420] In modern homes and offices, numerous home appliances and devices exist, each operating as an independent system. This makes it cumbersome for users to manage and control individual devices, hindering efficient operation. Furthermore, conventional systems lack the ability to adjust their behavior based on user emotions, resulting in a failure to adequately improve user comfort.

[0421] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for providing a terminal for the user to operate, means for the user to register real estate information using the terminal, means for the user to input home appliance information using the terminal, means for analyzing information of home appliance devices using the terminal, means for transmitting the analyzed home appliance information to a central server using the terminal, means for the central server to configure the device based on the transmitted information, means for notifying the terminal that the device configuration is complete, means for reflecting new device information in the system based on the notified information, means for notifying the user of home appliance information, a robot for recognizing voice commands and analyzing emotion data along with the voice commands, means for transmitting the voice commands and emotion data analyzed by the robot to a central server, means for the central server to generate control commands for home appliances based on the received voice commands and emotion data and transmit them to the corresponding device, means for receiving control commands sent from the central server and adjusting the state of the home appliances, means for automatically generating a list of corresponding home appliances based on real estate property information, a robot for the user to issue voice commands, and means for transmitting the voice commands and emotion data recognized by the robot to a central server. This will allow users to centrally manage real estate and home appliance information, and create a comfortable living environment through emotion-based adjustments to the operation of home appliances.

[0422] A "user" refers to an individual or legal entity that uses the system to manage and control real estate information and home appliance information.

[0423] A "device" is a device used by a user to perform operations, and includes mobile devices such as smartphones and tablets.

[0424] "Real estate information" refers to data about a property, including information such as its location, overview, number of rooms, and size.

[0425] "Home appliance information" refers to data that includes information such as the name, model number, installation location, and usage status of home appliances.

[0426] A "central server" refers to a server device that receives, analyzes, stores, and processes data sent by users.

[0427] A "device" refers to an electronic device with a specific function, such as smart home appliances.

[0428] "Device setup" refers to the process of performing initial setup and adjusting operating parameters so that home appliances function correctly.

[0429] A "robot" refers to a device that recognizes a user's voice commands and has the ability to analyze emotional data.

[0430] "Voice commands" refer to instructions or requests that users make by voice, such as "Turn up the living room lights."

[0431] "Emotional data" refers to data that indicates a user's emotional state, including expressions such as smiles and sadness.

[0432] A "control command" is an instruction sent from a central server to a home appliance, and refers to a command to perform a specific action.

[0433] A "platform" refers to a software environment that allows users to centrally manage real estate information and home appliance information.

[0434] This invention is a system for integrated management and control of real estate information and home appliance information, combined with an emotion engine that recognizes user emotions. This system consists of a central server, terminals (mobile devices such as smartphones), and a robot that receives user voice commands and emotions.

[0435] Central server

[0436] The central server is responsible for receiving, analyzing, storing, and processing data sent from users. Specifically, it stores real estate and home appliance information sent from user-operated terminals in a database, and performs initial setup and adjustment of operating parameters for home appliance devices. It also generates control commands based on voice commands and emotion data and sends them to the relevant devices.

[0437] terminal

[0438] A terminal is a device used by the user for operation, such as a smartphone or tablet. The terminal provides an interface for the user to input real estate and home appliance information, and transmits the entered information to a central server. It also has the function of scanning QR codes to obtain information about home appliances and transmitting that information to the central server.

[0439] robot

[0440] The robot recognizes voice commands uttered by the user and further analyzes the user's emotions using an emotion engine. The analyzed voice commands and emotion data are then transmitted to a central server. The robot plays a crucial role in adjusting the operation of home appliances based on emotions.

[0441] Specific operation of the system

[0442] User registration

[0443] The user first launches the smartphone app and enters the required personal information on the new registration screen and submits it. The device sends the information to a central server, which creates a new user account. A notification is displayed on the device when registration is complete.

[0444] Entering real estate information and home appliance information

[0445] The user enters property information through a smartphone app and selects the appliances they plan to use from a list of appliances. The device sends this information to a central server, which generates initial setup parameters and returns them to the device.

[0446] Adding home appliances

[0447] The user installs a new home appliance and scans a QR code with a smartphone app. The device analyzes the information and sends it to a central server. The server stores the new appliance information in a database and performs the initial setup. A notification is sent to the user when the setup is complete.

[0448] Operation via voice commands and emotion recognition

[0449] The user issues voice commands to the robot, such as "Brighten the living room lights," and expresses their emotions. The robot analyzes these commands using an emotion engine and sends the voice commands and emotion data to a central server. The server generates control commands based on the received data and sends them to the corresponding device. The device then adjusts based on the commands to provide the optimal environment.

[0450] Specific example

[0451] For example, consider a scenario where a user installs a newly purchased smart refrigerator and scans a QR code. The device sends the QR code information to a central server, which registers the device in its database. A robot then issues a notification stating, "The new refrigerator has been registered."

[0452] Let's consider a scenario where a user gives a voice command to the robot, "Brighten the living room lights," and the robot responds with a smile. The robot's emotion engine analyzes the voice and emotion and sends the information to a central server. The server then sends a command to the lighting system, and the lights are adjusted to be brighter.

[0453] This allows users to comprehensively manage their home environment and create a comfortable living environment based on their emotions.

[0454] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0455] Step 1:

[0456] The user launches the smartphone app and taps the "New Registration" button. Next, they enter the required personal information such as their name and email address, and then press the "Submit" button.

[0457] Input: Personal information entered by the user (e.g., name, email address)

[0458] Output: Data containing personal information stored on the device and sent to the server.

[0459] Specific action: The user enters "Yamada Taro" and "taro@example.com" into the smartphone app and clicks the send button.

[0460] Step 2:

[0461] The terminal retrieves personal information entered by the user and sends it to a central server.

[0462] Input: Personal information entered by the user on the device.

[0463] Output: User information sent to the server

[0464] Specific operation: The terminal sends the entered "Yamada Taro" and "taro@example.com" to the central server.

[0465] Step 3:

[0466] The server stores the received personal information in its database and creates a new user account. Next, it sends a response to the terminal indicating successful account creation.

[0467] Input: User information sent to the server

[0468] Output: New user account registered in the database, success message sent to the terminal.

[0469] Specific operation: The server registers a new account in the database based on "Yamada Taro" and "taro@example.com", and sends a success message to the terminal.

[0470] Step 4:

[0471] Users enter the necessary information into a smartphone app to register their property details. Next, they select the appliances they plan to install from a list of appliances.

[0472] Input: Real estate information (e.g., property location, size), home appliance information (e.g., smart air conditioner)

[0473] Output: The entered information is saved on the terminal and sent to the server.

[0474] Specific operation: The user enters the information for "Yamada Mansion" into the smartphone app and selects "Smart Air Conditioner".

[0475] Step 5:

[0476] The terminal transmits the entered real estate information and home appliance information to a central server.

[0477] Input: Real estate information and home appliance information entered by the user on the device.

[0478] Output: Real estate information and home appliance information sent to the server

[0479] Specific operation: The terminal sends real estate information for "Yamada Mansion" and home appliance information for "Smart Air Conditioner" to the central server.

[0480] Step 6:

[0481] The server stores real estate information and home appliance information in a database, generates initial setup parameters, and sends them to the terminal.

[0482] Input: Real estate information and home appliance information sent from the terminal.

[0483] Output: Information stored in the database, generated initial settings parameters

[0484] Specific operation: The server saves information about "Yamada Mansion" and "Smart Air Conditioner" to the database, generates initial setup parameters, and sends them to the terminal.

[0485] Step 7:

[0486] Users install new home appliances and scan a QR code with a smartphone app.

[0487] Input: QR code for the new home appliance

[0488] Output: Acquired home appliance information

[0489] Specific operation: The user sets up the new smart refrigerator and scans the QR code.

[0490] Step 8:

[0491] The terminal analyzes the information in the QR code and sends the information about the home appliance to a central server.

[0492] Input: Home appliance information obtained from a QR code

[0493] Output: Home appliance information sent to the server

[0494] Specific operation: The device analyzes the smart refrigerator information obtained from the QR code and sends it to the central server.

[0495] Step 9:

[0496] The server saves the received home appliance information to its database and performs the initial setup. It then sends a notification to the terminal when the setup is complete.

[0497] Input: Home appliance information sent from the device

[0498] Output: Information stored in the database, configuration completion notification

[0499] Specific operation: The server saves information about the newly registered smart refrigerator to the database and performs the initial setup. A notification is sent to the device when the setup is complete.

[0500] Step 10:

[0501] Users can issue voice commands to the robot and express their emotions, such as "Brighten the living room lights."

[0502] Input: Voice commands, user sentiment data

[0503] Output: Voice commands and emotional data are recognized by the robot.

[0504] Specific action: The user tells the robot, "Turn up the living room lights," and the robot smiles.

[0505] Step 11:

[0506] The robot analyzes voice commands and user emotions using an emotion engine and sends the information to a central server.

[0507] Input: User voice commands and sentiment data

[0508] Output: Voice commands and emotion data are sent to the server.

[0509] Specific operation: The robot analyzes the command "Turn up the living room lights" and a smile, and sends that data to the server.

[0510] Step 12:

[0511] The server generates control commands based on the received voice commands and emotion data, and sends them to the corresponding device.

[0512] Input: Voice commands and emotion data sent to the server.

[0513] Output: A control command is generated and sent to the relevant device.

[0514] Specific operation: The server generates a command to "brighten the living room lights" and sends it to the corresponding lighting system.

[0515] Step 13:

[0516] The device (lighting system) makes adjustments based on the commands it receives.

[0517] Input: Control commands sent from the server

[0518] Output: The brightness of the lighting is adjusted.

[0519] Specific operation: The lighting system increases its brightness according to the received command.

[0520] Step 14:

[0521] Users can confirm that the lighting is adjusted to the specified level and experience comfort as the environment is appropriately adjusted.

[0522] Input: Lighting system adjustment results

[0523] Output: The user can see the change in lighting.

[0524] Specific action: The user confirms that the living room lights have brightened and realizes that a comfortable environment has been created.

[0525] (Application Example 2)

[0526] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0527] In modern brick-and-mortar stores, it is essential to appropriately control environmental elements such as lighting, air conditioning, and music to enhance customer comfort and satisfaction. However, manually adjusting these environmental factors is labor-intensive and time-consuming, and providing nuanced responses tailored to the emotions and needs of each individual customer is a difficult task. Furthermore, controlling individual devices requires specialized knowledge, and it is impractical for store operators to understand all of them. Therefore, there is a need for an automated and centralized environmental control system that responds to customer emotions.

[0528] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for registering real estate information, means for the user to input home appliance information, means for analyzing device information, means for transmitting the registration information of home appliances to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of home appliance information, means for analyzing the user's emotions using an emotion recognition engine, and means for adjusting home appliances and the environment based on the analyzed emotion data. As a result, environmental adjustments in response to customer emotions are performed automatically in real time, enabling store operators to provide an optimal store environment without specialized knowledge.

[0529] "Real estate information" refers to detailed information about properties such as buildings and land. This includes data such as the property's location, size, price, and owner.

[0530] "Home appliance information" refers to information about household electrical appliances. This includes data such as product name, model number, installation location, and function settings.

[0531] "Device information" refers to detailed information about individual home appliances and electronic devices connected to the system. This includes device identification information, function settings, and connection status.

[0532] A "central server" refers to a central computer system that manages device and user information on a network, and handles communication and data processing with each device.

[0533] An "emotion recognition engine" refers to software or algorithms that analyze and recognize a user's emotional state from data such as audio and video.

[0534] "Environmental adjustment" refers to the operation of changing the settings of various devices in order to optimize the in-store environment, such as lighting, air conditioning, and music.

[0535] "Voice commands" refer to voice data that users use to instruct specific actions. These include specific commands such as turning lights on or off or adjusting the temperature.

[0536] "Notification methods" refer to systems and techniques used to inform users and devices of information sent from a central server. This includes things like notifications from smartphone apps and voice guidance from robots.

[0537] The system for realizing this application consists of the following elements.

[0538] 1. Overview of the entire system

[0539] This system integrates and manages real estate and home appliance information, and analyzes user emotions using an emotion recognition engine to provide an optimal environment.

[0540] The hardware used includes smartphones, smart glasses, and Arduino.

[0541] The software used includes an emotion recognition API and a Python program.

[0542] 2. User actions

[0543] Users input their property information and appliance information into the system using their smartphones or smart glasses.

[0544] The system analyzes the user's voice commands and emotional responses to provide the optimal environment.

[0545] 3. Server Role

[0546] The server stores real estate and home appliance information submitted by users in its database.

[0547] The server uses an emotion recognition engine to analyze the user's emotions and generates control commands for lighting, air conditioning, music systems, and other functions based on the results.

[0548] The generated control commands are sent to the corresponding device, which then adjusts the environment.

[0549] 4. Hardware and Software Usage

[0550] Smartphones and smart glasses function as user interfaces and are used for voice input and acquiring emotional data.

[0551] Arduino is used to control devices such as lighting, air conditioning, and music systems.

[0552] The emotion recognition API is used to analyze the user's voice commands and emotions.

[0553] The Python program is responsible for sending and receiving data and generating control commands.

[0554] 5. Specific Examples

[0555] The following are some specific examples.

[0556] Example 1: Lighting adjustment

[0557] 1. The user says, "Please turn up the living room lights," indicating a desire to relax.

[0558] 2. The smart glasses send voice and emotion data to an emotion recognition API, and the analyzed information is sent to a server.

[0559] 3. The server generates control commands for the lighting system based on the analysis results and adjusts the lighting via Arduino.

[0560] Example 2: Adjusting the air conditioning

[0561] 1. The user says, "Make this room cooler," expressing an emotional need for comfort.

[0562] 2. The smartphone receives voice and emotion data and sends it to the server.

[0563] 3. The server generates control commands for the air conditioning system based on the emotion recognition results and voice data, and adjusts the air conditioning using Arduino.

[0564] Example of a prompt

[0565] Example of a prompt message regarding lighting adjustments: "Brighten the living room lights."

[0566] Example of a prompt message regarding air conditioning adjustment: "Cool this room down."

[0567] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0568] Step 1:

[0569] The user inputs voice commands and emotions using a smartphone or smart glasses. For example, they might say, "Brighten the living room lights," while also expressing a relaxed emotion. The input here consists of voice data and emotion data. The device collects this data and prepares it to be sent to an emotion recognition API.

[0570] Step 2:

[0571] The device sends voice data and emotion data to the emotion recognition API. The voice data and emotion data sent from the device are input into the emotion recognition engine on the server side and analyzed. The processing here converts voice commands into text and outputs emotions as numerical data.

[0572] Step 3:

[0573] The server receives the analysis results returned from the emotion recognition API. The input data includes the text format of the voice command and the emotion analysis results. Based on this data, the server determines which device to operate and generates control commands. Specifically, it analyzes the obtained text data, retrieves the ID of the lighting system corresponding to the interpreted command, and determines the optimal lighting level based on the emotion data.

[0574] Step 4:

[0575] The server generates control commands which are then sent to the corresponding device via the Arduino. The transmitted control commands specify the brightness of the lighting system, for example, in the format "Set brightness to 70%".

[0576] Step 5:

[0577] The Arduino executes the control commands it receives. The input data consists of control commands sent from the server. The Arduino analyzes these commands, converts them into specific device control signals, and sends them to the device. The lighting system then receives the commands and adjusts to the specified brightness.

[0578] Step 6:

[0579] The system notifies the user of the results of the environment adjustments. A notification appears on the device to inform the user that the lighting has been adjusted to the specified brightness. Furthermore, it provides feedback on the overall system status and the next necessary actions.

[0580] By performing these processing steps, users can easily and intuitively optimize the store environment according to their emotions.

[0581] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0582] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0584] [Second Embodiment]

[0585] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0586] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0587] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0589] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0591] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0592] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0593] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0594] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0595] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0596] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0597] The present invention is a system for integrated management and control of real estate information and home appliance information, designed to be intuitively operable by the user. Specific embodiments of the present invention are described below.

[0598] Program processing

[0599] Overall Overview

[0600] The system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands, and a terminal (smartphone) that receives user input. This allows for centralized management of property information and smart devices throughout the home.

[0601] User registers with the system

[0602] 1. User

[0603] When registering a new account using the smartphone app, you will need to enter the required information (name, address, email address, phone number, etc.) and submit it to the system.

[0604] 2. Server

[0605] The server receives user information and stores it in the database.

[0606] Property registration and device setup

[0607] 1. User

[0608] Register information about newly purchased or rented property in the app. Enter details such as the property's location, size, and floor plan.

[0609] Select a list of home appliances you plan to install in your home.

[0610] 2. Server

[0611] The server stores property information in a database and retrieves a list of suitable appliances for the user's home, along with related initial setup information.

[0612] The system analyzes information about the home appliances selected by the user and generates initial setup parameters corresponding to each device.

[0613] Home appliance and system management

[0614] 1. Device (smartphone)

[0615] Users scan QR codes on home appliances using the app and add the appliance information to the system.

[0616] Information about the added home appliance is sent to the server and registered in the database.

[0617] 2. Server

[0618] Based on the registered information of the home appliances, the initial setup is performed and applied to each device.

[0619] The registered information is automatically updated and reflected in the user's system.

[0620] 3. User

[0621] The status of all home appliances and systems can be checked in real time via the app.

[0622] Operation by voice command

[0623] 1. User

[0624] You can issue voice commands to the robot, such as "Turn up the living room lights."

[0625] 2. Butler Robot

[0626] It recognizes voice commands and analyzes their content.

[0627] The analyzed instructions are sent to the central server as commands.

[0628] 3. Server

[0629] The server receives the command and generates a control command for the corresponding home appliance.

[0630] The generated command is sent to the target device.

[0631] 4. Home appliances (lighting systems)

[0632] The lighting system adjusts its brightness based on commands received from the server.

[0633] 5. Users

[0634] The lighting is adjusted to the specified level, making the environment more comfortable.

[0635] Specific example

[0636] Listening: Adding home appliances

[0637] 1. User

[0638] If you want to install your newly purchased smart refrigerator in your home, launch the smartphone app and scan the QR code on the refrigerator.

[0639] 2. Device (smartphone)

[0640] The QR code information is analyzed, and the refrigerator's device information is obtained.

[0641] 3. Server

[0642] Device information is sent to the server and registration is performed.

[0643] The refrigerator's initial setup is performed and reflected in the system.

[0644] 4. Butler Robot

[0645] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[0646] Example 2: Adjusting the room lighting

[0647] 1. User

[0648] If you want to adjust the living room lighting by voice, you can say to the butler robot, "Brighten the living room lights."

[0649] 2. Butler Robot

[0650] It recognizes and analyzes voice commands.

[0651] Send the analysis results to the server.

[0652] 3. Server

[0653] It receives a command and generates a command to increase the brightness of the living room lighting system.

[0654] Send a command to the lighting system.

[0655] 4. Terminal (lighting system)

[0656] It receives commands from the server and adjusts the brightness of the lighting.

[0657] 5. Users

[0658] The lighting is adjusted to achieve the desired brightness.

[0659] These concrete examples demonstrate that users can intuitively and efficiently manage home appliances and systems, thereby improving their quality of life.

[0660] The following describes the processing flow.

[0661] User registers with the system

[0662] Step 1:

[0663] The user launches the app on their smartphone and taps the "New Registration" button.

[0664] Step 2:

[0665] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[0666] Step 3:

[0667] The device (smartphone) sends the entered information to the central server.

[0668] Step 4:

[0669] The server saves the received information to the database and creates user accounts.

[0670] Step 5:

[0671] The server returns a response to the terminal indicating successful account creation and notifies the user that account creation is complete.

[0672] Property registration and device setup

[0673] Step 1:

[0674] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[0675] Step 2:

[0676] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[0677] Step 3:

[0678] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[0679] Step 4:

[0680] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[0681] Step 5:

[0682] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[0683] Home appliance and system management

[0684] Step 1:

[0685] The user installs the new appliance and scans a QR code with a smartphone app.

[0686] Step 2:

[0687] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[0688] Step 3:

[0689] The terminal sends the information it has acquired to the central server.

[0690] Step 4:

[0691] The server receives home appliance information, saves it to a database, and configures the devices.

[0692] Step 5:

[0693] The server sends a notification to the device indicating that the setup is complete.

[0694] Step 6:

[0695] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[0696] Operation by voice command

[0697] Step 1:

[0698] The user issues voice commands to the butler robot, such as "Turn up the living room lights."

[0699] Step 2:

[0700] The butler robot recognizes voice commands, analyzes their content, and generates commands to control the lighting.

[0701] Step 3:

[0702] The butler robot sends the control commands it generates to the central server.

[0703] Step 4:

[0704] The server checks the device ID of the lighting system and generates a command to adjust the brightness.

[0705] Step 5:

[0706] The server generates commands and sends them to the lighting system devices.

[0707] Step 6:

[0708] The lighting system sets the brightness based on the command it receives.

[0709] Step 7:

[0710] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[0711] Step 8:

[0712] The butler robot notifies the user via voice, "I've brightened the living room lights."

[0713] This series of processing steps allows users to intuitively and efficiently manage their home appliances and the entire system.

[0714] (Example 1)

[0715] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0716] With the recent proliferation of smart home appliances, there is a growing need to efficiently manage numerous devices and property information. However, current systems require users to manage information individually using different platforms and applications, leading to cumbersome and inefficient systems. Furthermore, even with voice-controlled appliances, there is no unified procedure or system, and user convenience is not adequately considered. To address these challenges, it is necessary to provide a system that allows users to easily integrate and manage smart home appliances and use it intuitively.

[0717] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0718] In this invention, the server includes means for a user to register a new account using a mobile terminal and transmit necessary information to a central server; means for a user to input property information and a list of home appliances using a mobile terminal and transmit them to a central server; means for the central server to store and analyze the received user information and property information in a database; means for scanning the QR code of a home appliance with a mobile terminal and transmitting device information to a central server; means for the central server to perform initial setup of the home appliance based on the received device information; means for the central server to transmit the generated setting information to the home appliance and adjust the status of the home appliance; means for a user to issue a voice command to a robot, for the robot to recognize and analyze the voice command and transmit it to a central server; means for the central server to generate control commands for the home appliance based on the analyzed voice command and transmit them to the home appliance; and means for a user to check and control the status of the home appliance in real time using a mobile terminal. This enables users to centrally manage a variety of home appliance and real estate information and operate them intuitively and efficiently.

[0719] "Account registration" is the process by which a user registers their information with the system, thereby obtaining a unique identifier and enabling them to use the system.

[0720] A "central server" is a central computer system that receives information sent by users, stores it in a database, and performs necessary analysis and processing.

[0721] "User information" refers to personally identifiable information such as the user's name, address, email address, and phone number.

[0722] A "database" is a collection of digital information used to efficiently store, manage, and retrieve information within a system.

[0723] A "mobile device" refers to a communication device that can be carried and used, such as a smartphone or tablet.

[0724] "Real estate information" refers to detailed information about a house or building, such as its location, size, and floor plan.

[0725] A "household appliance list" is a list of electrical appliances used in a home.

[0726] A "QR code" is a rectangular, two-dimensional barcode that contains various types of information, and by scanning it, data can be quickly retrieved.

[0727] "Initial setup" refers to the basic configuration process required to make a new device usable.

[0728] "Voice commands" refer to a method in which a user uses their voice to instruct a system or device to perform a specific action.

[0729] A "robot" is an automated mechanical device that has the function of receiving and recognizing voice commands and transmitting those commands to a central server.

[0730] A "control command" is an electronic instruction that tells a device to perform a specific action.

[0731] "Real-time" means that system or device operations and information updates occur instantly.

[0732] "Centralized management" refers to the integrated management of multiple devices and information through a single system or platform.

[0733] "Analysis" is the process of breaking down received data or information into smaller parts, understanding it, and using it for a specific purpose.

[0734] "Status adjustment" means changing or setting up the operating conditions of a device based on user instructions.

[0735] This invention relates to a system for the integrated management and control of real estate information and home appliance information. Designed for intuitive user operation, it consists of a mobile terminal, a central server, a home appliance control robot, and smart home appliances.

[0736] Program processing

[0737] Overall Overview

[0738] The system consists of a mobile application, a central server, a robot that receives user voice commands, and smart home appliances that receive user input. This system allows for centralized management of property information and smart devices throughout the home.

[0739] User account registration

[0740] A user registers a new account using a mobile device. The user enters information such as their name, address, email address, and phone number, and sends it to a central server. The server receives the user information and stores it in a database. This database could be something like MySQL.

[0741] Real estate and home appliance information registration

[0742] Users input new property information and appliance lists using their mobile devices and send them to a central server. The server stores the received property information in a database and performs analysis. Based on the property information, it automatically generates an appropriate appliance list and provides it to the user. A Python script could be used for this analysis.

[0743] Initial setup and management of home appliances

[0744] The user scans a QR code on a home appliance with their mobile device and sends the device information to a central server. The server uses the received device information to perform the initial setup of the home appliance and sends the generated configuration information to the appliance. This process uses the device manufacturer's API (e.g., Philips Hue API).

[0745] Controlling home appliances by voice commands

[0746] The user issues a voice command to the robot, such as "Brighten the living room lights." The robot recognizes the voice command, analyzes its content, and sends it to a central server. This recognition uses the Google Speech-to-Text API. Based on the analyzed voice command, the server generates control commands for the appliance and sends them to the target appliance. The appliance adjusts its operation according to the received control commands.

[0747] Specific example

[0748] Example 1: Adding home appliances

[0749] When a user installs a newly purchased smart refrigerator at home, they launch the app on their mobile device and scan the refrigerator's QR code. The QR code information is analyzed, and the refrigerator's device information is retrieved. The device information is sent to a central server for registration. The refrigerator's initial setup is performed and reflected in the system. A robot retrieves the new device information and notifies the user that "the new refrigerator has been registered."

[0750] Example 2: Adjusting the room lighting

[0751] If a user wants to adjust the living room lighting by voice, they would say to the robot, "Brighten the living room lights." The robot would recognize and analyze the voice command. It would then send the analysis results to a central server. The server would receive the analysis results, generate a command to increase the brightness of the living room lighting system, and send it. The lighting system would receive the command and adjust the brightness. The user would then confirm the lighting change and see that the desired brightness had been achieved.

[0752] Example of a prompt

[0753] "Please explain how to add a new home appliance. In particular, please provide detailed instructions on scanning the QR code and the data processing on the server side."

[0754] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0755] Step 1: User account registration

[0756] Input: The user enters information such as their name, address, email address, and phone number using their mobile device and presses the "Register" button.

[0757] Output: The data is sent to the central server, and a registration success message is returned to the user's device.

[0758] Specific actions:

[0759] 1. The user launches the app on their mobile device and is directed to the account registration screen.

[0760] 2. The user enters their name, address, email address, and phone number, and presses the submit button. This sends an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / register).

[0761] 3. The server verifies the received information and saves it to the database. Upon success, a response containing status code 200 is returned, and the user receives the message "Registration complete."

[0762] Step 2: Register property and appliance information

[0763] Input: The user enters property information and a list of appliances using a mobile device and presses the submit button.

[0764] Output: The transmitted information is analyzed by a central server and stored in a database. A list of appropriate home appliances is also generated.

[0765] Specific actions:

[0766] 1. The user selects the "Real Estate Information Registration" screen from the app's menu.

[0767] 2. Enter property information such as location, size, and floor plan, and select the desired appliances from the appliance list.

[0768] 3. After entering the information, press the submit button to send an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / properties / register).

[0769] 4. The server analyzes the received information and stores it in a database (e.g., MySQL). Using multiple SQL queries, it registers real estate information and appliance list information, and generates appliance settings suitable for the property specified by the user.

[0770] Step 3: Initial setup and management of home appliances

[0771] Input: The user scans the QR code on the home appliance with their mobile device and sends the obtained device information.

[0772] Output: Device information is sent to the central server for registration and initial setup.

[0773] Specific actions:

[0774] 1. The user places the new home appliance and launches the app on their mobile device.

[0775] 2. Use the app's QR code scanner function (e.g., Zebra Crossing) to scan the QR code on the home appliance and obtain device information.

[0776] 3. Send the retrieved device information as an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / devices / register).

[0777] 4. The server stores the received device information in a database and generates an initial configuration file using a Python script or shell script, which is then distributed to each home appliance.

[0778] Step 4: Controlling home appliances with voice commands

[0779] Input: The user issues a voice command to the robot, such as "Brighten the living room lights."

[0780] Output: The robot recognizes voice commands, sends the analysis results to a central server, and generates control commands for home appliances based on those results.

[0781] Specific actions:

[0782] 1. The user issues a pre-set voice command to the home appliance control robot.

[0783] 2. The robot uses the Google Speech-to-Text API to recognize voice commands and convert their content into text format.

[0784] 3. Send the analysis results to the central server as an HTTP POST request. The API endpoint is (e.g., https: / / example.com / api / voicecommands).

[0785] 4. The server generates control commands for the target device based on the received analysis results. For example, it creates an SQL query such as "UPDATE devices SET brightness = ? WHERE location = ?" and sends the adjustment command to the smart lighting device.

[0786] 5. The lighting system adjusts the brightness based on the received command, and the user confirms the change.

[0787] (Application Example 1)

[0788] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0789] In modern factory operations, a wide variety of equipment and sensors are introduced, but it is difficult to manage and control them efficiently in a unified manner. Therefore, it is necessary to use separate management systems for each piece of equipment, leading to increased complexity and costs. Furthermore, there is a lack of adequate systems for real-time monitoring of equipment status and early detection of abnormalities. In particular, there is a need to quickly reflect information on newly introduced equipment in the system, and to perform continuous status monitoring and appropriate operation.

[0790] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0791] In this invention, the server includes means for registering real estate information, means for users to input equipment information, means for analyzing device information, means for transmitting equipment registration information to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of equipment information, means for recognizing voice commands and transmitting the analyzed content to the central server, means for generating equipment control commands based on the user's voice commands and transmitting them to the corresponding devices, means for receiving control commands sent from the central server and adjusting the equipment status, means for providing an application for users to centrally manage property information and equipment information, means for automatically generating a list of corresponding equipment based on property information, a robot for users to issue voice commands, means for transmitting voice commands recognized by the robot to the central server, means for notifying the user when new equipment is registered, and means for updating the operating status of equipment in real time. This makes it possible to centrally manage and efficiently control a wide variety of equipment and sensors in a factory. In addition, information on newly introduced equipment can be quickly reflected in the system, enabling real-time monitoring of operating status and appropriate operation.

[0792] (definition statement)

[0793] "Real estate information" is a general term for data related to properties owned or managed by a user. This includes information such as the property's location, size, and floor plan.

[0794] "Device information" refers to the collective data about devices and equipment owned or managed by a user. This includes information such as the type of device, its function, and its operating status.

[0795] "Device information" is a general term for detailed technical data about equipment and devices. This includes information such as the device's components, specifications, and operating status.

[0796] A "central server" refers to a central computer system that receives, manages, and controls data from a large number of users and devices.

[0797] "Voice commands" refer to instructions given by a user to a system using their voice. Examples include commands such as "Brighten the living room lights."

[0798] A "control command" refers to an instruction used to tell a device or equipment to perform a specific action. These commands are generated by a central server and sent to each device.

[0799] "Operating status" refers to information about the current state in which a piece of equipment or device is operating. This includes states such as normal operation, malfunction, and awaiting maintenance.

[0800] An "application" refers to software that allows users to centrally manage real estate and equipment information using devices such as smartphones and tablets.

[0801] A "robot" refers to an autonomous mechanical device that has the function of receiving voice commands from a user, analyzing them, and transmitting them to a central server.

[0802] "Notification" refers to the act of transmitting information from the system to the user. This includes changes in the registration status and operating status of equipment.

[0803] This invention relates to a system for centrally managing and efficiently controlling equipment and sensors within a factory. The system is designed for intuitive user operation and consists of a smartphone application, a central server, and a robot that receives voice commands. Detailed embodiments for carrying out this invention are described below.

[0804] User registers with the system

[0805] 1. User

[0806] When users register a new account using the smartphone application, they enter necessary information such as their name, address, email address, and phone number, and submit it to the system. This information is stored on the central server, as described below.

[0807] 2. Server

[0808] The server receives user information and stores it in the database. This is achieved using a database management library such as SQLAlchemy.

[0809] Device registration and device configuration

[0810] 1. User

[0811] Users register information about newly installed factory equipment using a smartphone application. By scanning a QR code, they can input detailed information such as the type of equipment, performance, and operating status into the system.

[0812] 2. Server

[0813] The server stores equipment information in a database and generates a list of equipment suitable for the user's factory, along with associated initial configuration information. The server then analyzes this information and generates initial configuration parameters for each device. This process is implemented by a server program combining Python's Flask and SQLAlchemy.

[0814] Equipment management and control

[0815] 1. Device (smartphone)

[0816] Users scan the device's QR code using a smartphone application and register their information. This information is sent to a server, and the appropriate device settings are configured.

[0817] 2. Server

[0818] The server performs the initial setup of the equipment based on the received information and reflects this in the database. Furthermore, the registered information is automatically updated and reflected in the user's system in real time.

[0819] 3. User

[0820] Users can check the operating status of all devices in real time using the application.

[0821] Operation by voice command

[0822] 1. User

[0823] Users can issue voice commands to the robot, such as "Check the status of a specific device." This allows users to operate the system without using their hands.

[0824] 2. Robots

[0825] The robot recognizes voice commands and analyzes their content. The analyzed commands are then sent to a central server. The Python SpeechRecognition library is used for this speech recognition.

[0826] 3. Server

[0827] The server receives the instruction and generates control commands for the corresponding device. These commands are then sent from the central server to the device.

[0828] 4. Equipment

[0829] The equipment adjusts its operation based on commands received from the server. For example, it may change the operating status of a specified piece of equipment.

[0830] 5. Users

[0831] When the system functions as expected, users experience improved ease of use.

[0832] Specific example

[0833] Adding equipment

[0834] When a user installs a newly purchased press machine in their factory, they launch a smartphone application and scan the press machine's QR code. The QR code information is analyzed, and the press machine's device information is retrieved. The device information is sent to a server and registered in a database. Subsequently, a robot retrieves the new device information and notifies the user that "the new equipment has been registered."

[0835] Monitoring and adjustment of equipment operating status

[0836] When a user wants to check the operating status of a specific device and operate it, they check the status from a smartphone application and give a voice command to the robot saying, "Check the status of this device." The voice command is recognized, and the analysis results are sent to the server. The server receives the command, generates and sends control commands for the device in question. The device receives the commands from the server and adjusts its operating status. This allows the user to monitor and operate the device's status in real time.

[0837] Example of a prompt

[0838] If you want to install newly purchased equipment in your factory, how do you scan the QR code and register it in the system?

[0839] How can I check the operating status of specific equipment within the factory?

[0840] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0841] Step 1:

[0842] A user registers a new account using a smartphone application. The user enters required information such as their name, address, email address, and phone number, and submits it to the system. The entered data is sent from the application to the server. The server receives this data and stores it in its database.

[0843] Step 2:

[0844] Users register information about newly installed factory equipment using a smartphone application. Users scan the QR code of the factory equipment and input detailed information such as the type of equipment, performance, and operating status. This data is sent from the smartphone application to the server. The server analyzes the received device information and stores it in a database.

[0845] Step 3:

[0846] The server generates initial configuration parameters for each device. This uses an algorithm that automatically determines appropriate parameters based on the type and characteristics of the device. The generated initial configuration parameters are then registered in a database.

[0847] Step 4:

[0848] The user scans the device's QR code using a smartphone application to complete registration. This adds the new device's information to the system. The added device information is sent to the server and registered in the database.

[0849] Step 5:

[0850] The server performs the initial setup of the devices and applies these settings to them. Based on the initial setup parameters for each device, the server generates and sends configuration commands to each device. The devices apply the initial setup based on the commands they receive.

[0851] Step 6:

[0852] Users can check the operating status of all equipment in real time using a smartphone application. The application retrieves the latest equipment information from the server and displays it to the user. This process allows users to always be aware of the status of equipment in the factory.

[0853] Step 7:

[0854] The user issues a voice command to the robot. For example, they might give the robot a command such as, "Check the status of a specific device." The robot recognizes this voice command and analyzes its content. The analyzed command is then sent to the server as a command.

[0855] Step 8:

[0856] The server generates control commands for the corresponding device based on the received voice commands. The generated control commands are sent from the central server to the device. The device adjusts its operation according to these control commands.

[0857] Step 9:

[0858] Users benefit from improved operational convenience when the system functions as expected. When control is functioning correctly, users can confidently continue their work within the factory.

[0859] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0860] The present invention is a system that integrates and manages real estate information and home appliance information, and combines it with an emotion engine that recognizes user emotions. The following describes specific embodiments of the present invention.

[0861] Program processing

[0862] Overall Overview

[0863] This system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands and emotions, and a terminal (smartphone) that receives user input. This allows for integrated management of property information and smart devices throughout the home, and enables the system's operation to be adjusted according to the user's emotions.

[0864] User registers with the system

[0865] 1. User

[0866] Launch the app on your smartphone and tap the "New Registration" button.

[0867] Enter the required personal information and press the "Submit" button.

[0868] 2. Server

[0869] The received information is saved to the database, and a user account is created.

[0870] A response indicating successful account creation is sent to the device, notifying the user.

[0871] Property registration and device setup

[0872] 1. User

[0873] Register your property information using the app and select the appliances you plan to install from the appliance list.

[0874] 2. Server

[0875] Based on real estate information and selected appliance information, initial setup parameters are generated and sent to the terminal.

[0876] Home appliance and system management

[0877] 1. User

[0878] Install the new appliance and scan the QR code with a smartphone app.

[0879] 2. Device (smartphone)

[0880] The system analyzes QR code information and sends the home appliance information to a central server.

[0881] 3. Server

[0882] The system saves the received home appliance information to a database and configures the device settings.

[0883] A notification will be sent to the device and displayed upon completion of the setup.

[0884] Operation via voice commands and emotion recognition

[0885] 1. User

[0886] It issues voice commands such as "Turn up the living room lights" and also expresses emotions.

[0887] 2. Butler Robot

[0888] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[0889] Voice commands and emotional data are sent to a central server.

[0890] 3. Server

[0891] Based on the received voice commands and emotion data, the device ID of the lighting system is identified, and the appropriate control command is generated.

[0892] Sends a command to the lighting system device.

[0893] 4. Home appliances (lighting systems)

[0894] Adjustments are made based on the received commands.

[0895] 5. Users

[0896] Comfort is ensured by adjusting the lighting to a specified level and making appropriate environmental adjustments.

[0897] Specific example

[0898] Example 1: Adding home appliances

[0899] 1. User

[0900] I installed the newly purchased smart refrigerator in my home, launched the smartphone app, and scanned the QR code.

[0901] 2. Device (smartphone)

[0902] The system acquires and analyzes QR code information and sends device information to the server.

[0903] 3. Server

[0904] The system receives device information and registers home appliances in its database.

[0905] Perform the initial setup of the refrigerator and reflect the information in the system.

[0906] 4. Butler Robot

[0907] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[0908] Example 2: Lighting adjustment

[0909] 1. User

[0910] If you want to adjust the living room lighting using your voice, say "Brighten the living room lights" and express your emotion.

[0911] 2. Butler Robot

[0912] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[0913] It sends voice commands and emotional data to the server.

[0914] 3. Server

[0915] Based on voice commands and emotion data, the system identifies the device ID of the lighting system and generates appropriate control commands.

[0916] Send a command to the lighting system.

[0917] 4. Terminal (lighting system)

[0918] Receives commands and adjusts to the specified level.

[0919] 5. Users

[0920] The lighting is adjusted, and the environment is kept in an optimal state according to the user's emotions.

[0921] This allows users to intuitively and efficiently manage their home appliances and entire systems, and further improves their quality of life through emotion recognition.

[0922] The following describes the processing flow.

[0923] User registers with the system

[0924] Step 1:

[0925] The user launches the app on their smartphone and taps the "New Registration" button.

[0926] Step 2:

[0927] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[0928] Step 3:

[0929] The device (smartphone) sends the entered information to the central server.

[0930] Step 4:

[0931] The server saves the received information to the database and creates user accounts.

[0932] Step 5:

[0933] The server returns a response to the terminal indicating successful account creation, notifying the user that account creation is complete.

[0934] Property registration and device setup

[0935] Step 1:

[0936] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[0937] Step 2:

[0938] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[0939] Step 3:

[0940] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[0941] Step 4:

[0942] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[0943] Step 5:

[0944] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[0945] Home appliance and system management

[0946] Step 1:

[0947] The user installs the new appliance and scans a QR code with a smartphone app.

[0948] Step 2:

[0949] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[0950] Step 3:

[0951] The terminal sends the information it has acquired to the central server.

[0952] Step 4:

[0953] The server receives home appliance information, saves it to a database, and configures the devices.

[0954] Step 5:

[0955] The server sends a notification to the device indicating that the setup is complete.

[0956] Step 6:

[0957] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[0958] Operation via voice commands and emotion recognition

[0959] Step 1:

[0960] The user can issue voice commands such as "Turn up the living room lights" and express emotions.

[0961] Step 2:

[0962] The butler robot recognizes voice commands and emotions, and analyzes their content.

[0963] Step 3:

[0964] The butler robot sends voice commands and emotional data to a central server.

[0965] Step 4:

[0966] The server receives voice commands and emotion data, verifies the device ID of the lighting system, and generates appropriate control commands.

[0967] Step 5:

[0968] The server generates control commands which are then sent to the lighting system devices.

[0969] Step 6:

[0970] The lighting system sets the brightness based on the command it receives.

[0971] Step 7:

[0972] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[0973] Step 8:

[0974] The butler robot notifies the user via voice, "I've brightened the living room lights."

[0975] This allows users to efficiently and intuitively manage their home appliances and entire systems using voice commands and emotions.

[0976] (Example 2)

[0977] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0978] In modern homes and offices, numerous home appliances and devices exist, each operating as an independent system. This makes it cumbersome for users to manage and control individual devices, hindering efficient operation. Furthermore, conventional systems lack the ability to adjust their behavior based on user emotions, resulting in a failure to adequately improve user comfort.

[0979] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for providing a terminal for the user to operate, means for the user to register real estate information using the terminal, means for the user to input home appliance information using the terminal, means for analyzing information of home appliance devices using the terminal, means for transmitting the analyzed home appliance information to a central server using the terminal, means for the central server to configure the device based on the transmitted information, means for notifying the terminal that the device configuration is complete, means for reflecting new device information in the system based on the notified information, means for notifying the user of home appliance information, a robot for recognizing voice commands and analyzing emotion data along with the voice commands, means for transmitting the voice commands and emotion data analyzed by the robot to a central server, means for the central server to generate control commands for home appliances based on the received voice commands and emotion data and transmit them to the corresponding device, means for receiving control commands sent from the central server and adjusting the state of the home appliances, means for automatically generating a list of corresponding home appliances based on real estate property information, a robot for the user to issue voice commands, and means for transmitting the voice commands and emotion data recognized by the robot to a central server. This will allow users to centrally manage real estate and home appliance information, and create a comfortable living environment through emotion-based adjustments to the operation of home appliances.

[0980] A "user" refers to an individual or legal entity that uses the system to manage and control real estate information and home appliance information.

[0981] A "device" is a device used by a user to perform operations, and includes mobile devices such as smartphones and tablets.

[0982] "Real estate information" refers to data about a property, including information such as its location, overview, number of rooms, and size.

[0983] "Home appliance information" refers to data that includes information such as the name, model number, installation location, and usage status of home appliances.

[0984] A "central server" refers to a server device that receives, analyzes, stores, and processes data sent by users.

[0985] A "device" refers to an electronic device with a specific function, such as smart home appliances.

[0986] "Device setup" refers to the process of performing initial setup and adjusting operating parameters so that home appliances function correctly.

[0987] A "robot" refers to a device that recognizes a user's voice commands and has the ability to analyze emotional data.

[0988] "Voice commands" refer to instructions or requests that users make by voice, such as "Turn up the living room lights."

[0989] "Emotional data" refers to data that indicates a user's emotional state, including expressions such as smiles and sadness.

[0990] A "control command" is an instruction sent from a central server to a home appliance, and refers to a command to perform a specific action.

[0991] A "platform" refers to a software environment that allows users to centrally manage real estate information and home appliance information.

[0992] This invention is a system for integrated management and control of real estate information and home appliance information, combined with an emotion engine that recognizes user emotions. This system consists of a central server, terminals (mobile devices such as smartphones), and a robot that receives user voice commands and emotions.

[0993] Central server

[0994] The central server is responsible for receiving, analyzing, storing, and processing data sent from users. Specifically, it stores real estate and home appliance information sent from user-operated terminals in a database, and performs initial setup and adjustment of operating parameters for home appliance devices. It also generates control commands based on voice commands and emotion data and sends them to the relevant devices.

[0995] terminal

[0996] A terminal is a device used by the user for operation, such as a smartphone or tablet. The terminal provides an interface for the user to input real estate and home appliance information, and transmits the entered information to a central server. It also has the function of scanning QR codes to obtain information about home appliances and transmitting that information to the central server.

[0997] robot

[0998] The robot recognizes voice commands uttered by the user and further analyzes the user's emotions using an emotion engine. The analyzed voice commands and emotion data are then transmitted to a central server. The robot plays a crucial role in adjusting the operation of home appliances based on emotions.

[0999] Specific operation of the system

[1000] User registration

[1001] The user first launches the smartphone app and enters the required personal information on the new registration screen and submits it. The device sends the information to a central server, which creates a new user account. A notification is displayed on the device when registration is complete.

[1002] Entering real estate information and home appliance information

[1003] The user enters property information through a smartphone app and selects the appliances they plan to use from a list of appliances. The device sends this information to a central server, which generates initial setup parameters and returns them to the device.

[1004] Adding home appliances

[1005] The user installs a new home appliance and scans a QR code with a smartphone app. The device analyzes the information and sends it to a central server. The server stores the new appliance information in a database and performs the initial setup. A notification is sent to the user when the setup is complete.

[1006] Operation via voice commands and emotion recognition

[1007] The user issues voice commands to the robot, such as "Brighten the living room lights," and expresses their emotions. The robot analyzes these commands using an emotion engine and sends the voice commands and emotion data to a central server. The server generates control commands based on the received data and sends them to the corresponding device. The device then adjusts based on the commands to provide the optimal environment.

[1008] Specific example

[1009] For example, consider a scenario where a user installs a newly purchased smart refrigerator and scans a QR code. The device sends the QR code information to a central server, which registers the device in its database. A robot then issues a notification stating, "The new refrigerator has been registered."

[1010] Let's consider a scenario where a user gives a voice command to the robot, "Brighten the living room lights," and the robot responds with a smile. The robot's emotion engine analyzes the voice and emotion and sends the information to a central server. The server then sends a command to the lighting system, and the lights are adjusted to be brighter.

[1011] This allows users to comprehensively manage their home environment and create a comfortable living environment based on their emotions.

[1012] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1013] Step 1:

[1014] The user launches the smartphone app and taps the "New Registration" button. Next, they enter the required personal information such as their name and email address, and then press the "Submit" button.

[1015] Input: Personal information entered by the user (e.g., name, email address)

[1016] Output: Data containing personal information stored on the device and sent to the server.

[1017] Specific action: The user enters "Yamada Taro" and "taro@example.com" into the smartphone app and clicks the send button.

[1018] Step 2:

[1019] The terminal retrieves personal information entered by the user and sends it to a central server.

[1020] Input: Personal information entered by the user on the device.

[1021] Output: User information sent to the server

[1022] Specific operation: The terminal sends the entered "Yamada Taro" and "taro@example.com" to the central server.

[1023] Step 3:

[1024] The server stores the received personal information in its database and creates a new user account. Next, it sends a response to the terminal indicating successful account creation.

[1025] Input: User information sent to the server

[1026] Output: New user account registered in the database, success message sent to the terminal.

[1027] Specific operation: The server registers a new account in the database based on "Yamada Taro" and "taro@example.com", and sends a success message to the terminal.

[1028] Step 4:

[1029] Users enter the necessary information into a smartphone app to register their property details. Next, they select the appliances they plan to install from a list of appliances.

[1030] Input: Real estate information (e.g., property location, size), home appliance information (e.g., smart air conditioner)

[1031] Output: The entered information is saved on the terminal and sent to the server.

[1032] Specific operation: The user enters the information for "Yamada Mansion" into the smartphone app and selects "Smart Air Conditioner".

[1033] Step 5:

[1034] The terminal transmits the entered real estate information and home appliance information to a central server.

[1035] Input: Real estate information and home appliance information entered by the user on the device.

[1036] Output: Real estate information and home appliance information sent to the server

[1037] Specific operation: The terminal sends real estate information for "Yamada Mansion" and home appliance information for "Smart Air Conditioner" to the central server.

[1038] Step 6:

[1039] The server stores real estate information and home appliance information in a database, generates initial setup parameters, and sends them to the terminal.

[1040] Input: Real estate information and home appliance information sent from the terminal.

[1041] Output: Information stored in the database, generated initial settings parameters

[1042] Specific operation: The server saves information about "Yamada Mansion" and "Smart Air Conditioner" to the database, generates initial setup parameters, and sends them to the terminal.

[1043] Step 7:

[1044] Users install new home appliances and scan a QR code with a smartphone app.

[1045] Input: QR code for the new home appliance

[1046] Output: Acquired home appliance information

[1047] Specific operation: The user sets up the new smart refrigerator and scans the QR code.

[1048] Step 8:

[1049] The terminal analyzes the information in the QR code and sends the information about the home appliance to a central server.

[1050] Input: Home appliance information obtained from a QR code

[1051] Output: Home appliance information sent to the server

[1052] Specific operation: The device analyzes the smart refrigerator information obtained from the QR code and sends it to the central server.

[1053] Step 9:

[1054] The server saves the received home appliance information to its database and performs the initial setup. It then sends a notification to the terminal when the setup is complete.

[1055] Input: Home appliance information sent from the device

[1056] Output: Information stored in the database, configuration completion notification

[1057] Specific operation: The server saves information about the newly registered smart refrigerator to the database and performs the initial setup. A notification is sent to the device when the setup is complete.

[1058] Step 10:

[1059] Users can issue voice commands to the robot and express their emotions, such as "Brighten the living room lights."

[1060] Input: Voice commands, user sentiment data

[1061] Output: Voice commands and emotional data are recognized by the robot.

[1062] Specific action: The user tells the robot, "Turn up the living room lights," and the robot smiles.

[1063] Step 11:

[1064] The robot analyzes voice commands and user emotions using an emotion engine and sends the information to a central server.

[1065] Input: User voice commands and sentiment data

[1066] Output: Voice commands and emotion data are sent to the server.

[1067] Specific operation: The robot analyzes the command "Turn up the living room lights" and a smile, and sends that data to the server.

[1068] Step 12:

[1069] The server generates control commands based on the received voice commands and emotion data, and sends them to the corresponding device.

[1070] Input: Voice commands and emotion data sent to the server.

[1071] Output: A control command is generated and sent to the relevant device.

[1072] Specific operation: The server generates a command to "brighten the living room lights" and sends it to the corresponding lighting system.

[1073] Step 13:

[1074] The device (lighting system) makes adjustments based on the commands it receives.

[1075] Input: Control commands sent from the server

[1076] Output: The brightness of the lighting is adjusted.

[1077] Specific operation: The lighting system increases its brightness according to the received command.

[1078] Step 14:

[1079] Users can confirm that the lighting is adjusted to the specified level and experience comfort as the environment is appropriately adjusted.

[1080] Input: Lighting system adjustment results

[1081] Output: The user can see the change in lighting.

[1082] Specific action: The user confirms that the living room lights have brightened and realizes that a comfortable environment has been created.

[1083] (Application Example 2)

[1084] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1085] In modern brick-and-mortar stores, it is essential to appropriately control environmental elements such as lighting, air conditioning, and music to enhance customer comfort and satisfaction. However, manually adjusting these environmental factors is labor-intensive and time-consuming, and providing nuanced responses tailored to the emotions and needs of each individual customer is a difficult task. Furthermore, controlling individual devices requires specialized knowledge, and it is impractical for store operators to understand all of them. Therefore, there is a need for an automated and centralized environmental control system that responds to customer emotions.

[1086] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for registering real estate information, means for the user to input home appliance information, means for analyzing device information, means for transmitting the registration information of home appliances to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of home appliance information, means for analyzing the user's emotions using an emotion recognition engine, and means for adjusting home appliances and the environment based on the analyzed emotion data. As a result, environmental adjustments in response to customer emotions are performed automatically in real time, enabling store operators to provide an optimal store environment without specialized knowledge.

[1087] "Real estate information" refers to detailed information about properties such as buildings and land. This includes data such as the property's location, size, price, and owner.

[1088] "Home appliance information" refers to information about household electrical appliances. This includes data such as product name, model number, installation location, and function settings.

[1089] "Device information" refers to detailed information about individual home appliances and electronic devices connected to the system. This includes device identification information, function settings, and connection status.

[1090] A "central server" refers to a central computer system that manages device and user information on a network, and handles communication and data processing with each device.

[1091] An "emotion recognition engine" refers to software or algorithms that analyze and recognize a user's emotional state from data such as audio and video.

[1092] "Environmental adjustment" refers to the operation of changing the settings of various devices in order to optimize the in-store environment, such as lighting, air conditioning, and music.

[1093] "Voice commands" refer to voice data that users use to instruct specific actions. These include specific commands such as turning lights on or off or adjusting the temperature.

[1094] "Notification methods" refer to systems and techniques used to inform users and devices of information sent from a central server. This includes things like notifications from smartphone apps and voice guidance from robots.

[1095] The system for realizing this application consists of the following elements.

[1096] 1. Overview of the entire system

[1097] This system integrates and manages real estate and home appliance information, and analyzes user emotions using an emotion recognition engine to provide an optimal environment.

[1098] The hardware used includes smartphones, smart glasses, and Arduino.

[1099] The software used includes an emotion recognition API and a Python program.

[1100] 2. User actions

[1101] Users input their property information and appliance information into the system using their smartphones or smart glasses.

[1102] The system analyzes the user's voice commands and emotional responses to provide the optimal environment.

[1103] 3. Server Role

[1104] The server stores real estate and home appliance information submitted by users in its database.

[1105] The server uses an emotion recognition engine to analyze the user's emotions and generates control commands for lighting, air conditioning, music systems, and other functions based on the results.

[1106] The generated control commands are sent to the corresponding device, which then adjusts the environment.

[1107] 4. Hardware and Software Usage

[1108] Smartphones and smart glasses function as user interfaces and are used for voice input and acquiring emotional data.

[1109] Arduino is used to control devices such as lighting, air conditioning, and music systems.

[1110] The emotion recognition API is used to analyze the user's voice commands and emotions.

[1111] The Python program is responsible for sending and receiving data and generating control commands.

[1112] 5. Specific Examples

[1113] The following are some specific examples.

[1114] Example 1: Lighting adjustment

[1115] 1. The user says, "Please turn up the living room lights," indicating a desire to relax.

[1116] 2. The smart glasses send voice and emotion data to an emotion recognition API, and the analyzed information is sent to a server.

[1117] 3. The server generates control commands for the lighting system based on the analysis results and adjusts the lighting via Arduino.

[1118] Example 2: Adjusting the air conditioning

[1119] 1. The user says, "Make this room cooler," expressing an emotional need for comfort.

[1120] 2. The smartphone receives voice and emotion data and sends it to the server.

[1121] 3. The server generates control commands for the air conditioning system based on the emotion recognition results and voice data, and adjusts the air conditioning using Arduino.

[1122] Example of a prompt

[1123] Example of a prompt message regarding lighting adjustments: "Brighten the living room lights."

[1124] Example of a prompt message regarding air conditioning adjustment: "Cool this room down."

[1125] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1126] Step 1:

[1127] The user inputs voice commands and emotions using a smartphone or smart glasses. For example, they might say, "Brighten the living room lights," while also expressing a relaxed emotion. The input here consists of voice data and emotion data. The device collects this data and prepares it to be sent to an emotion recognition API.

[1128] Step 2:

[1129] The device sends voice data and emotion data to the emotion recognition API. The voice data and emotion data sent from the device are input into the emotion recognition engine on the server side and analyzed. The processing here converts voice commands into text and outputs emotions as numerical data.

[1130] Step 3:

[1131] The server receives the analysis results returned from the emotion recognition API. The input data includes the text format of the voice command and the emotion analysis results. Based on this data, the server determines which device to operate and generates control commands. Specifically, it analyzes the obtained text data, retrieves the ID of the lighting system corresponding to the interpreted command, and determines the optimal lighting level based on the emotion data.

[1132] Step 4:

[1133] The server generates control commands which are then sent to the corresponding device via the Arduino. The transmitted control commands specify the brightness of the lighting system, for example, in the format "Set brightness to 70%".

[1134] Step 5:

[1135] The Arduino executes the control commands it receives. The input data consists of control commands sent from the server. The Arduino analyzes these commands, converts them into specific device control signals, and sends them to the device. The lighting system then receives the commands and adjusts to the specified brightness.

[1136] Step 6:

[1137] The system notifies the user of the results of the environment adjustments. A notification appears on the device to inform the user that the lighting has been adjusted to the specified brightness. Furthermore, it provides feedback on the overall system status and the next necessary actions.

[1138] By performing these processing steps, users can easily and intuitively optimize the store environment according to their emotions.

[1139] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1140] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1141] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1142] [Third Embodiment]

[1143] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1144] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1145] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1146] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1147] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1148] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1149] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1150] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1151] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1152] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1153] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1154] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1155] The present invention is a system for integrated management and control of real estate information and home appliance information, designed to be intuitively operable by the user. Specific embodiments of the present invention are described below.

[1156] Program processing

[1157] Overall Overview

[1158] The system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands, and a terminal (smartphone) that receives user input. This allows for centralized management of property information and smart devices throughout the home.

[1159] User registers with the system

[1160] 1. User

[1161] When registering a new account using the smartphone app, you will need to enter the required information (name, address, email address, phone number, etc.) and submit it to the system.

[1162] 2. Server

[1163] The server receives user information and stores it in the database.

[1164] Property registration and device setup

[1165] 1. User

[1166] Register information about newly purchased or rented property in the app. Enter details such as the property's location, size, and floor plan.

[1167] Select a list of home appliances you plan to install in your home.

[1168] 2. Server

[1169] The server stores property information in a database and retrieves a list of suitable appliances for the user's home, along with related initial setup information.

[1170] The system analyzes information about the home appliances selected by the user and generates initial setup parameters corresponding to each device.

[1171] Home appliance and system management

[1172] 1. Device (smartphone)

[1173] Users scan QR codes on home appliances using the app and add the appliance information to the system.

[1174] Information about the added home appliance is sent to the server and registered in the database.

[1175] 2. Server

[1176] Based on the registered information of the home appliances, the initial setup is performed and applied to each device.

[1177] The registered information is automatically updated and reflected in the user's system.

[1178] 3. User

[1179] The status of all home appliances and systems can be checked in real time via the app.

[1180] Operation by voice command

[1181] 1. User

[1182] You can issue voice commands to the robot, such as "Turn up the living room lights."

[1183] 2. Butler Robot

[1184] It recognizes voice commands and analyzes their content.

[1185] The analyzed instructions are sent to the central server as commands.

[1186] 3. Server

[1187] The server receives the command and generates a control command for the corresponding home appliance.

[1188] The generated command is sent to the target device.

[1189] 4. Home appliances (lighting systems)

[1190] The lighting system adjusts its brightness based on commands received from the server.

[1191] 5. Users

[1192] The lighting is adjusted to the specified level, making the environment more comfortable.

[1193] Specific example

[1194] Listening: Adding home appliances

[1195] 1. User

[1196] If you want to install your newly purchased smart refrigerator in your home, launch the smartphone app and scan the QR code on the refrigerator.

[1197] 2. Device (smartphone)

[1198] The QR code information is analyzed, and the refrigerator's device information is obtained.

[1199] 3. Server

[1200] Device information is sent to the server and registration is performed.

[1201] The refrigerator's initial setup is performed and reflected in the system.

[1202] 4. Butler Robot

[1203] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[1204] Example 2: Adjusting the room lighting

[1205] 1. User

[1206] If you want to adjust the living room lighting by voice, you can say to the butler robot, "Brighten the living room lights."

[1207] 2. Butler Robot

[1208] It recognizes and analyzes voice commands.

[1209] Send the analysis results to the server.

[1210] 3. Server

[1211] It receives a command and generates a command to increase the brightness of the living room lighting system.

[1212] Send a command to the lighting system.

[1213] 4. Terminal (lighting system)

[1214] It receives commands from the server and adjusts the brightness of the lighting.

[1215] 5. Users

[1216] The lighting is adjusted to achieve the desired brightness.

[1217] These concrete examples demonstrate that users can intuitively and efficiently manage home appliances and systems, thereby improving their quality of life.

[1218] The following describes the processing flow.

[1219] User registers with the system

[1220] Step 1:

[1221] The user launches the app on their smartphone and taps the "New Registration" button.

[1222] Step 2:

[1223] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[1224] Step 3:

[1225] The device (smartphone) sends the entered information to the central server.

[1226] Step 4:

[1227] The server saves the received information to the database and creates user accounts.

[1228] Step 5:

[1229] The server returns a response to the terminal indicating successful account creation and notifies the user that account creation is complete.

[1230] Property registration and device setup

[1231] Step 1:

[1232] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[1233] Step 2:

[1234] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[1235] Step 3:

[1236] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[1237] Step 4:

[1238] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[1239] Step 5:

[1240] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[1241] Home appliance and system management

[1242] Step 1:

[1243] The user installs the new appliance and scans a QR code with a smartphone app.

[1244] Step 2:

[1245] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[1246] Step 3:

[1247] The terminal sends the information it has acquired to the central server.

[1248] Step 4:

[1249] The server receives home appliance information, saves it to a database, and configures the devices.

[1250] Step 5:

[1251] The server sends a notification to the device indicating that the setup is complete.

[1252] Step 6:

[1253] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[1254] Operation by voice command

[1255] Step 1:

[1256] The user issues voice commands to the butler robot, such as "Turn up the living room lights."

[1257] Step 2:

[1258] The butler robot recognizes voice commands, analyzes their content, and generates commands to control the lighting.

[1259] Step 3:

[1260] The butler robot sends the control commands it generates to the central server.

[1261] Step 4:

[1262] The server checks the device ID of the lighting system and generates a command to adjust the brightness.

[1263] Step 5:

[1264] The server generates commands and sends them to the lighting system devices.

[1265] Step 6:

[1266] The lighting system sets the brightness based on the command it receives.

[1267] Step 7:

[1268] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[1269] Step 8:

[1270] The butler robot notifies the user via voice, "I've brightened the living room lights."

[1271] This series of processing steps allows users to intuitively and efficiently manage their home appliances and the entire system.

[1272] (Example 1)

[1273] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1274] With the recent proliferation of smart home appliances, there is a growing need to efficiently manage numerous devices and property information. However, current systems require users to manage information individually using different platforms and applications, leading to cumbersome and inefficient systems. Furthermore, even with voice-controlled appliances, there is no unified procedure or system, and user convenience is not adequately considered. To address these challenges, it is necessary to provide a system that allows users to easily integrate and manage smart home appliances and use it intuitively.

[1275] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1276] In this invention, the server includes means for a user to register a new account using a mobile terminal and transmit necessary information to a central server; means for a user to input property information and a list of home appliances using a mobile terminal and transmit them to a central server; means for the central server to store and analyze the received user information and property information in a database; means for scanning the QR code of a home appliance with a mobile terminal and transmitting device information to a central server; means for the central server to perform initial setup of the home appliance based on the received device information; means for the central server to transmit the generated setting information to the home appliance and adjust the status of the home appliance; means for a user to issue a voice command to a robot, for the robot to recognize and analyze the voice command and transmit it to a central server; means for the central server to generate control commands for the home appliance based on the analyzed voice command and transmit them to the home appliance; and means for a user to check and control the status of the home appliance in real time using a mobile terminal. This enables users to centrally manage a variety of home appliance and real estate information and operate them intuitively and efficiently.

[1277] "Account registration" is the process by which a user registers their information with the system, thereby obtaining a unique identifier and enabling them to use the system.

[1278] A "central server" is a central computer system that receives information sent by users, stores it in a database, and performs necessary analysis and processing.

[1279] "User information" refers to personally identifiable information such as the user's name, address, email address, and phone number.

[1280] A "database" is a collection of digital information used to efficiently store, manage, and retrieve information within a system.

[1281] A "mobile device" refers to a communication device that can be carried and used, such as a smartphone or tablet.

[1282] "Real estate information" refers to detailed information about a house or building, such as its location, size, and floor plan.

[1283] A "household appliance list" is a list of electrical appliances used in a home.

[1284] A "QR code" is a rectangular, two-dimensional barcode that contains various types of information, and by scanning it, data can be quickly retrieved.

[1285] "Initial setup" refers to the basic configuration process required to make a new device usable.

[1286] "Voice commands" refer to a method in which a user uses their voice to instruct a system or device to perform a specific action.

[1287] A "robot" is an automated mechanical device that has the function of receiving and recognizing voice commands and transmitting those commands to a central server.

[1288] A "control command" is an electronic instruction that tells a device to perform a specific action.

[1289] "Real-time" means that system or device operations and information updates occur instantly.

[1290] "Centralized management" refers to the integrated management of multiple devices and information through a single system or platform.

[1291] "Analysis" is the process of breaking down received data or information into smaller parts, understanding it, and using it for a specific purpose.

[1292] "Status adjustment" means changing or setting up the operating conditions of a device based on user instructions.

[1293] This invention relates to a system for the integrated management and control of real estate information and home appliance information. Designed for intuitive user operation, it consists of a mobile terminal, a central server, a home appliance control robot, and smart home appliances.

[1294] Program processing

[1295] Overall Overview

[1296] The system consists of a mobile application, a central server, a robot that receives user voice commands, and smart home appliances that receive user input. This system allows for centralized management of property information and smart devices throughout the home.

[1297] User account registration

[1298] A user registers a new account using a mobile device. The user enters information such as their name, address, email address, and phone number, and sends it to a central server. The server receives the user information and stores it in a database. This database could be something like MySQL.

[1299] Real estate and home appliance information registration

[1300] Users input new property information and appliance lists using their mobile devices and send them to a central server. The server stores the received property information in a database and performs analysis. Based on the property information, it automatically generates an appropriate appliance list and provides it to the user. A Python script could be used for this analysis.

[1301] Initial setup and management of home appliances

[1302] The user scans a QR code on a home appliance with their mobile device and sends the device information to a central server. The server uses the received device information to perform the initial setup of the home appliance and sends the generated configuration information to the appliance. This process uses the device manufacturer's API (e.g., Philips Hue API).

[1303] Controlling home appliances by voice commands

[1304] The user issues a voice command to the robot, such as "Brighten the living room lights." The robot recognizes the voice command, analyzes its content, and sends it to a central server. This recognition uses the Google Speech-to-Text API. Based on the analyzed voice command, the server generates control commands for the appliance and sends them to the target appliance. The appliance adjusts its operation according to the received control commands.

[1305] Specific example

[1306] Example 1: Adding home appliances

[1307] When a user installs a newly purchased smart refrigerator at home, they launch the app on their mobile device and scan the refrigerator's QR code. The QR code information is analyzed, and the refrigerator's device information is retrieved. The device information is sent to a central server for registration. The refrigerator's initial setup is performed and reflected in the system. A robot retrieves the new device information and notifies the user that "the new refrigerator has been registered."

[1308] Example 2: Adjusting the room lighting

[1309] If a user wants to adjust the living room lighting by voice, they would say to the robot, "Brighten the living room lights." The robot would recognize and analyze the voice command. It would then send the analysis results to a central server. The server would receive the analysis results, generate a command to increase the brightness of the living room lighting system, and send it. The lighting system would receive the command and adjust the brightness. The user would then confirm the lighting change and see that the desired brightness had been achieved.

[1310] Example of a prompt

[1311] "Please explain how to add a new home appliance. In particular, please provide detailed instructions on scanning the QR code and the data processing on the server side."

[1312] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1313] Step 1: User account registration

[1314] Input: The user enters information such as their name, address, email address, and phone number using their mobile device and presses the "Register" button.

[1315] Output: The data is sent to the central server, and a registration success message is returned to the user's device.

[1316] Specific actions:

[1317] 1. The user launches the app on their mobile device and is directed to the account registration screen.

[1318] 2. The user enters their name, address, email address, and phone number, and presses the submit button. This sends an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / register).

[1319] 3. The server verifies the received information and saves it to the database. Upon success, a response containing status code 200 is returned, and the user receives the message "Registration complete."

[1320] Step 2: Register property and appliance information

[1321] Input: The user enters property information and a list of appliances using a mobile device and presses the submit button.

[1322] Output: The transmitted information is analyzed by a central server and stored in a database. A list of appropriate home appliances is also generated.

[1323] Specific actions:

[1324] 1. The user selects the "Real Estate Information Registration" screen from the app's menu.

[1325] 2. Enter property information such as location, size, and floor plan, and select the desired appliances from the appliance list.

[1326] 3. After entering the information, press the submit button to send an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / properties / register).

[1327] 4. The server analyzes the received information and stores it in a database (e.g., MySQL). Using multiple SQL queries, it registers real estate information and appliance list information, and generates appliance settings suitable for the property specified by the user.

[1328] Step 3: Initial setup and management of home appliances

[1329] Input: The user scans the QR code on the home appliance with their mobile device and sends the obtained device information.

[1330] Output: Device information is sent to the central server for registration and initial setup.

[1331] Specific actions:

[1332] 1. The user places the new home appliance and launches the app on their mobile device.

[1333] 2. Use the app's QR code scanner function (e.g., Zebra Crossing) to scan the QR code on the home appliance and obtain device information.

[1334] 3. Send the retrieved device information as an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / devices / register).

[1335] 4. The server stores the received device information in a database and generates an initial configuration file using a Python script or shell script, which is then distributed to each home appliance.

[1336] Step 4: Controlling home appliances with voice commands

[1337] Input: The user issues a voice command to the robot, such as "Brighten the living room lights."

[1338] Output: The robot recognizes voice commands, sends the analysis results to a central server, and generates control commands for home appliances based on those results.

[1339] Specific actions:

[1340] 1. The user issues a pre-set voice command to the home appliance control robot.

[1341] 2. The robot uses the Google Speech-to-Text API to recognize voice commands and convert their content into text format.

[1342] 3. Send the analysis results to the central server as an HTTP POST request. The API endpoint is (e.g., https: / / example.com / api / voicecommands).

[1343] 4. The server generates control commands for the target device based on the received analysis results. For example, it creates an SQL query such as "UPDATE devices SET brightness = ? WHERE location = ?" and sends the adjustment command to the smart lighting device.

[1344] 5. The lighting system adjusts the brightness based on the received command, and the user confirms the change.

[1345] (Application Example 1)

[1346] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1347] In modern factory operations, a wide variety of equipment and sensors are introduced, but it is difficult to manage and control them efficiently in a unified manner. Therefore, it is necessary to use separate management systems for each piece of equipment, leading to increased complexity and costs. Furthermore, there is a lack of adequate systems for real-time monitoring of equipment status and early detection of abnormalities. In particular, there is a need to quickly reflect information on newly introduced equipment in the system, and to perform continuous status monitoring and appropriate operation.

[1348] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1349] In this invention, the server includes means for registering real estate information, means for users to input equipment information, means for analyzing device information, means for transmitting equipment registration information to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of equipment information, means for recognizing voice commands and transmitting the analyzed content to the central server, means for generating equipment control commands based on the user's voice commands and transmitting them to the corresponding devices, means for receiving control commands sent from the central server and adjusting the equipment status, means for providing an application for users to centrally manage property information and equipment information, means for automatically generating a list of corresponding equipment based on property information, a robot for users to issue voice commands, means for transmitting voice commands recognized by the robot to the central server, means for notifying the user when new equipment is registered, and means for updating the operating status of equipment in real time. This makes it possible to centrally manage and efficiently control a wide variety of equipment and sensors in a factory. In addition, information on newly introduced equipment can be quickly reflected in the system, enabling real-time monitoring of operating status and appropriate operation.

[1350] (definition statement)

[1351] "Real estate information" is a general term for data related to properties owned or managed by a user. This includes information such as the property's location, size, and floor plan.

[1352] "Device information" refers to the collective data about devices and equipment owned or managed by a user. This includes information such as the type of device, its function, and its operating status.

[1353] "Device information" is a general term for detailed technical data about equipment and devices. This includes information such as the device's components, specifications, and operating status.

[1354] A "central server" refers to a central computer system that receives, manages, and controls data from a large number of users and devices.

[1355] "Voice commands" refer to instructions given by a user to a system using their voice. Examples include commands such as "Brighten the living room lights."

[1356] A "control command" refers to an instruction used to tell a device or equipment to perform a specific action. These commands are generated by a central server and sent to each device.

[1357] "Operating status" refers to information about the current state in which a piece of equipment or device is operating. This includes states such as normal operation, malfunction, and awaiting maintenance.

[1358] An "application" refers to software that allows users to centrally manage real estate and equipment information using devices such as smartphones and tablets.

[1359] A "robot" refers to an autonomous mechanical device that has the function of receiving voice commands from a user, analyzing them, and transmitting them to a central server.

[1360] "Notification" refers to the act of transmitting information from the system to the user. This includes changes in the registration status and operating status of equipment.

[1361] This invention relates to a system for centrally managing and efficiently controlling equipment and sensors within a factory. The system is designed for intuitive user operation and consists of a smartphone application, a central server, and a robot that receives voice commands. Detailed embodiments for carrying out this invention are described below.

[1362] User registers with the system

[1363] 1. User

[1364] When users register a new account using the smartphone application, they enter necessary information such as their name, address, email address, and phone number, and submit it to the system. This information is stored on the central server, as described below.

[1365] 2. Server

[1366] The server receives user information and stores it in the database. This is achieved using a database management library such as SQLAlchemy.

[1367] Device registration and device configuration

[1368] 1. User

[1369] Users register information about newly installed factory equipment using a smartphone application. By scanning a QR code, they can input detailed information such as the type of equipment, performance, and operating status into the system.

[1370] 2. Server

[1371] The server stores equipment information in a database and generates a list of equipment suitable for the user's factory, along with associated initial configuration information. The server then analyzes this information and generates initial configuration parameters for each device. This process is implemented by a server program combining Python's Flask and SQLAlchemy.

[1372] Equipment management and control

[1373] 1. Device (smartphone)

[1374] Users scan the device's QR code using a smartphone application and register their information. This information is sent to a server, and the appropriate device settings are configured.

[1375] 2. Server

[1376] The server performs the initial setup of the equipment based on the received information and reflects this in the database. Furthermore, the registered information is automatically updated and reflected in the user's system in real time.

[1377] 3. User

[1378] Users can check the operating status of all devices in real time using the application.

[1379] Operation by voice command

[1380] 1. User

[1381] Users can issue voice commands to the robot, such as "Check the status of a specific device." This allows users to operate the system without using their hands.

[1382] 2. Robots

[1383] The robot recognizes voice commands and analyzes their content. The analyzed commands are then sent to a central server. The Python SpeechRecognition library is used for this speech recognition.

[1384] 3. Server

[1385] The server receives the instruction and generates control commands for the corresponding device. These commands are then sent from the central server to the device.

[1386] 4. Equipment

[1387] The equipment adjusts its operation based on commands received from the server. For example, it may change the operating status of a specified piece of equipment.

[1388] 5. Users

[1389] When the system functions as expected, users experience improved ease of use.

[1390] Specific example

[1391] Adding equipment

[1392] When a user installs a newly purchased press machine in their factory, they launch a smartphone application and scan the press machine's QR code. The QR code information is analyzed, and the press machine's device information is retrieved. The device information is sent to a server and registered in a database. Subsequently, a robot retrieves the new device information and notifies the user that "the new equipment has been registered."

[1393] Monitoring and adjustment of equipment operating status

[1394] When a user wants to check the operating status of a specific device and operate it, they check the status from a smartphone application and give a voice command to the robot saying, "Check the status of this device." The voice command is recognized, and the analysis results are sent to the server. The server receives the command, generates and sends control commands for the device in question. The device receives the commands from the server and adjusts its operating status. This allows the user to monitor and operate the device's status in real time.

[1395] Example of a prompt

[1396] If you want to install newly purchased equipment in your factory, how do you scan the QR code and register it in the system?

[1397] How can I check the operating status of specific equipment within the factory?

[1398] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1399] Step 1:

[1400] A user registers a new account using a smartphone application. The user enters required information such as their name, address, email address, and phone number, and submits it to the system. The entered data is sent from the application to the server. The server receives this data and stores it in its database.

[1401] Step 2:

[1402] Users register information about newly installed factory equipment using a smartphone application. Users scan the QR code of the factory equipment and input detailed information such as the type of equipment, performance, and operating status. This data is sent from the smartphone application to the server. The server analyzes the received device information and stores it in a database.

[1403] Step 3:

[1404] The server generates initial configuration parameters for each device. This uses an algorithm that automatically determines appropriate parameters based on the type and characteristics of the device. The generated initial configuration parameters are then registered in a database.

[1405] Step 4:

[1406] The user scans the device's QR code using a smartphone application to complete registration. This adds the new device's information to the system. The added device information is sent to the server and registered in the database.

[1407] Step 5:

[1408] The server performs the initial setup of the devices and applies these settings to them. Based on the initial setup parameters for each device, the server generates and sends configuration commands to each device. The devices apply the initial setup based on the commands they receive.

[1409] Step 6:

[1410] Users can check the operating status of all equipment in real time using a smartphone application. The application retrieves the latest equipment information from the server and displays it to the user. This process allows users to always be aware of the status of equipment in the factory.

[1411] Step 7:

[1412] The user issues a voice command to the robot. For example, they might give the robot a command such as, "Check the status of a specific device." The robot recognizes this voice command and analyzes its content. The analyzed command is then sent to the server as a command.

[1413] Step 8:

[1414] The server generates control commands for the corresponding device based on the received voice commands. The generated control commands are sent from the central server to the device. The device adjusts its operation according to these control commands.

[1415] Step 9:

[1416] Users benefit from improved operational convenience when the system functions as expected. When control is functioning correctly, users can confidently continue their work within the factory.

[1417] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1418] The present invention is a system that integrates and manages real estate information and home appliance information, and combines it with an emotion engine that recognizes user emotions. The following describes specific embodiments of the present invention.

[1419] Program processing

[1420] Overall Overview

[1421] This system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands and emotions, and a terminal (smartphone) that receives user input. This allows for integrated management of property information and smart devices throughout the home, and enables the system's operation to be adjusted according to the user's emotions.

[1422] User registers with the system

[1423] 1. User

[1424] Launch the app on your smartphone and tap the "New Registration" button.

[1425] Enter the required personal information and press the "Submit" button.

[1426] 2. Server

[1427] The received information is saved to the database, and a user account is created.

[1428] A response indicating successful account creation is sent to the device, notifying the user.

[1429] Property registration and device setup

[1430] 1. User

[1431] Register your property information using the app and select the appliances you plan to install from the appliance list.

[1432] 2. Server

[1433] Based on real estate information and selected appliance information, initial setup parameters are generated and sent to the terminal.

[1434] Home appliance and system management

[1435] 1. User

[1436] Install the new appliance and scan the QR code with a smartphone app.

[1437] 2. Device (smartphone)

[1438] The system analyzes QR code information and sends the home appliance information to a central server.

[1439] 3. Server

[1440] The system saves the received home appliance information to a database and configures the device settings.

[1441] A notification will be sent to the device and displayed upon completion of the setup.

[1442] Operation via voice commands and emotion recognition

[1443] 1. User

[1444] It issues voice commands such as "Turn up the living room lights" and also expresses emotions.

[1445] 2. Butler Robot

[1446] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[1447] Voice commands and emotional data are sent to a central server.

[1448] 3. Server

[1449] Based on the received voice commands and emotion data, the device ID of the lighting system is identified, and the appropriate control command is generated.

[1450] Sends a command to the lighting system device.

[1451] 4. Home appliances (lighting systems)

[1452] Adjustments are made based on the received commands.

[1453] 5. Users

[1454] Comfort is ensured by adjusting the lighting to a specified level and making appropriate environmental adjustments.

[1455] Specific example

[1456] Example 1: Adding home appliances

[1457] 1. User

[1458] I installed the newly purchased smart refrigerator in my home, launched the smartphone app, and scanned the QR code.

[1459] 2. Device (smartphone)

[1460] The system acquires and analyzes QR code information and sends device information to the server.

[1461] 3. Server

[1462] The system receives device information and registers home appliances in its database.

[1463] Perform the initial setup of the refrigerator and reflect the information in the system.

[1464] 4. Butler Robot

[1465] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[1466] Example 2: Lighting adjustment

[1467] 1. User

[1468] If you want to adjust the living room lighting using your voice, say "Brighten the living room lights" and express your emotion.

[1469] 2. Butler Robot

[1470] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[1471] It sends voice commands and emotional data to the server.

[1472] 3. Server

[1473] Based on voice commands and emotion data, the system identifies the device ID of the lighting system and generates appropriate control commands.

[1474] Send a command to the lighting system.

[1475] 4. Terminal (lighting system)

[1476] Receives commands and adjusts to the specified level.

[1477] 5. Users

[1478] The lighting is adjusted, and the environment is kept in an optimal state according to the user's emotions.

[1479] This allows users to intuitively and efficiently manage their home appliances and entire systems, and further improves their quality of life through emotion recognition.

[1480] The following describes the processing flow.

[1481] User registers with the system

[1482] Step 1:

[1483] The user launches the app on their smartphone and taps the "New Registration" button.

[1484] Step 2:

[1485] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[1486] Step 3:

[1487] The device (smartphone) sends the entered information to the central server.

[1488] Step 4:

[1489] The server saves the received information to the database and creates user accounts.

[1490] Step 5:

[1491] The server returns a response to the terminal indicating successful account creation, notifying the user that account creation is complete.

[1492] Property registration and device setup

[1493] Step 1:

[1494] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[1495] Step 2:

[1496] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[1497] Step 3:

[1498] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[1499] Step 4:

[1500] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[1501] Step 5:

[1502] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[1503] Home appliance and system management

[1504] Step 1:

[1505] The user installs the new appliance and scans a QR code with a smartphone app.

[1506] Step 2:

[1507] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[1508] Step 3:

[1509] The terminal sends the information it has acquired to the central server.

[1510] Step 4:

[1511] The server receives home appliance information, saves it to a database, and configures the devices.

[1512] Step 5:

[1513] The server sends a notification to the device indicating that the setup is complete.

[1514] Step 6:

[1515] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[1516] Operation via voice commands and emotion recognition

[1517] Step 1:

[1518] The user can issue voice commands such as "Turn up the living room lights" and express emotions.

[1519] Step 2:

[1520] The butler robot recognizes voice commands and emotions, and analyzes their content.

[1521] Step 3:

[1522] The butler robot sends voice commands and emotional data to a central server.

[1523] Step 4:

[1524] The server receives voice commands and emotion data, verifies the device ID of the lighting system, and generates appropriate control commands.

[1525] Step 5:

[1526] The server generates control commands which are then sent to the lighting system devices.

[1527] Step 6:

[1528] The lighting system sets the brightness based on the command it receives.

[1529] Step 7:

[1530] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[1531] Step 8:

[1532] The butler robot notifies the user via voice, "I've brightened the living room lights."

[1533] This allows users to efficiently and intuitively manage their home appliances and entire systems using voice commands and emotions.

[1534] (Example 2)

[1535] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1536] In modern homes and offices, numerous home appliances and devices exist, each operating as an independent system. This makes it cumbersome for users to manage and control individual devices, hindering efficient operation. Furthermore, conventional systems lack the ability to adjust their behavior based on user emotions, resulting in a failure to adequately improve user comfort.

[1537] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for providing a terminal for the user to operate, means for the user to register real estate information using the terminal, means for the user to input home appliance information using the terminal, means for analyzing information of home appliance devices using the terminal, means for transmitting the analyzed home appliance information to a central server using the terminal, means for the central server to configure the device based on the transmitted information, means for notifying the terminal that the device configuration is complete, means for reflecting new device information in the system based on the notified information, means for notifying the user of home appliance information, a robot for recognizing voice commands and analyzing emotion data along with the voice commands, means for transmitting the voice commands and emotion data analyzed by the robot to a central server, means for the central server to generate control commands for home appliances based on the received voice commands and emotion data and transmit them to the corresponding device, means for receiving control commands sent from the central server and adjusting the state of the home appliances, means for automatically generating a list of corresponding home appliances based on real estate property information, a robot for the user to issue voice commands, and means for transmitting the voice commands and emotion data recognized by the robot to a central server. This will allow users to centrally manage real estate and home appliance information, and create a comfortable living environment through emotion-based adjustments to the operation of home appliances.

[1538] A "user" refers to an individual or legal entity that uses the system to manage and control real estate information and home appliance information.

[1539] A "device" is a device used by a user to perform operations, and includes mobile devices such as smartphones and tablets.

[1540] "Real estate information" refers to data about a property, including information such as its location, overview, number of rooms, and size.

[1541] "Home appliance information" refers to data that includes information such as the name, model number, installation location, and usage status of home appliances.

[1542] A "central server" refers to a server device that receives, analyzes, stores, and processes data sent by users.

[1543] A "device" refers to an electronic device with a specific function, such as smart home appliances.

[1544] "Device setup" refers to the process of performing initial setup and adjusting operating parameters so that home appliances function correctly.

[1545] A "robot" refers to a device that recognizes a user's voice commands and has the ability to analyze emotional data.

[1546] "Voice commands" refer to instructions or requests that users make by voice, such as "Turn up the living room lights."

[1547] "Emotional data" refers to data that indicates a user's emotional state, including expressions such as smiles and sadness.

[1548] A "control command" is an instruction sent from a central server to a home appliance, and refers to a command to perform a specific action.

[1549] A "platform" refers to a software environment that allows users to centrally manage real estate information and home appliance information.

[1550] This invention is a system for integrated management and control of real estate information and home appliance information, combined with an emotion engine that recognizes user emotions. This system consists of a central server, terminals (mobile devices such as smartphones), and a robot that receives user voice commands and emotions.

[1551] Central server

[1552] The central server is responsible for receiving, analyzing, storing, and processing data sent from users. Specifically, it stores real estate and home appliance information sent from user-operated terminals in a database, and performs initial setup and adjustment of operating parameters for home appliance devices. It also generates control commands based on voice commands and emotion data and sends them to the relevant devices.

[1553] terminal

[1554] A terminal is a device used by the user for operation, such as a smartphone or tablet. The terminal provides an interface for the user to input real estate and home appliance information, and transmits the entered information to a central server. It also has the function of scanning QR codes to obtain information about home appliances and transmitting that information to the central server.

[1555] robot

[1556] The robot recognizes voice commands uttered by the user and further analyzes the user's emotions using an emotion engine. The analyzed voice commands and emotion data are then transmitted to a central server. The robot plays a crucial role in adjusting the operation of home appliances based on emotions.

[1557] Specific operation of the system

[1558] User registration

[1559] The user first launches the smartphone app and enters the required personal information on the new registration screen and submits it. The device sends the information to a central server, which creates a new user account. A notification is displayed on the device when registration is complete.

[1560] Entering real estate information and home appliance information

[1561] The user enters property information through a smartphone app and selects the appliances they plan to use from a list of appliances. The device sends this information to a central server, which generates initial setup parameters and returns them to the device.

[1562] Adding home appliances

[1563] The user installs a new home appliance and scans a QR code with a smartphone app. The device analyzes the information and sends it to a central server. The server stores the new appliance information in a database and performs the initial setup. A notification is sent to the user when the setup is complete.

[1564] Operation via voice commands and emotion recognition

[1565] The user issues voice commands to the robot, such as "Brighten the living room lights," and expresses their emotions. The robot analyzes these commands using an emotion engine and sends the voice commands and emotion data to a central server. The server generates control commands based on the received data and sends them to the corresponding device. The device then adjusts based on the commands to provide the optimal environment.

[1566] Specific example

[1567] For example, consider a scenario where a user installs a newly purchased smart refrigerator and scans a QR code. The device sends the QR code information to a central server, which registers the device in its database. A robot then issues a notification stating, "The new refrigerator has been registered."

[1568] Let's consider a scenario where a user gives a voice command to the robot, "Brighten the living room lights," and the robot responds with a smile. The robot's emotion engine analyzes the voice and emotion and sends the information to a central server. The server then sends a command to the lighting system, and the lights are adjusted to be brighter.

[1569] This allows users to comprehensively manage their home environment and create a comfortable living environment based on their emotions.

[1570] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1571] Step 1:

[1572] The user launches the smartphone app and taps the "New Registration" button. Next, they enter the required personal information such as their name and email address, and then press the "Submit" button.

[1573] Input: Personal information entered by the user (e.g., name, email address)

[1574] Output: Data containing personal information stored on the device and sent to the server.

[1575] Specific action: The user enters "Yamada Taro" and "taro@example.com" into the smartphone app and clicks the send button.

[1576] Step 2:

[1577] The terminal retrieves personal information entered by the user and sends it to a central server.

[1578] Input: Personal information entered by the user on the device.

[1579] Output: User information sent to the server

[1580] Specific operation: The terminal sends the entered "Yamada Taro" and "taro@example.com" to the central server.

[1581] Step 3:

[1582] The server stores the received personal information in its database and creates a new user account. Next, it sends a response to the terminal indicating successful account creation.

[1583] Input: User information sent to the server

[1584] Output: New user account registered in the database, success message sent to the terminal.

[1585] Specific operation: The server registers a new account in the database based on "Yamada Taro" and "taro@example.com", and sends a success message to the terminal.

[1586] Step 4:

[1587] Users enter the necessary information into a smartphone app to register their property details. Next, they select the appliances they plan to install from a list of appliances.

[1588] Input: Real estate information (e.g., property location, size), home appliance information (e.g., smart air conditioner)

[1589] Output: The entered information is saved on the terminal and sent to the server.

[1590] Specific operation: The user enters the information for "Yamada Mansion" into the smartphone app and selects "Smart Air Conditioner".

[1591] Step 5:

[1592] The terminal transmits the entered real estate information and home appliance information to a central server.

[1593] Input: Real estate information and home appliance information entered by the user on the device.

[1594] Output: Real estate information and home appliance information sent to the server

[1595] Specific operation: The terminal sends real estate information for "Yamada Mansion" and home appliance information for "Smart Air Conditioner" to the central server.

[1596] Step 6:

[1597] The server stores real estate information and home appliance information in a database, generates initial setup parameters, and sends them to the terminal.

[1598] Input: Real estate information and home appliance information sent from the terminal.

[1599] Output: Information stored in the database, generated initial settings parameters

[1600] Specific operation: The server saves information about "Yamada Mansion" and "Smart Air Conditioner" to the database, generates initial setup parameters, and sends them to the terminal.

[1601] Step 7:

[1602] Users install new home appliances and scan a QR code with a smartphone app.

[1603] Input: QR code for the new home appliance

[1604] Output: Acquired home appliance information

[1605] Specific operation: The user sets up the new smart refrigerator and scans the QR code.

[1606] Step 8:

[1607] The terminal analyzes the information in the QR code and sends the information about the home appliance to a central server.

[1608] Input: Home appliance information obtained from a QR code

[1609] Output: Home appliance information sent to the server

[1610] Specific operation: The device analyzes the smart refrigerator information obtained from the QR code and sends it to the central server.

[1611] Step 9:

[1612] The server saves the received home appliance information to its database and performs the initial setup. It then sends a notification to the terminal when the setup is complete.

[1613] Input: Home appliance information sent from the device

[1614] Output: Information stored in the database, configuration completion notification

[1615] Specific operation: The server saves information about the newly registered smart refrigerator to the database and performs the initial setup. A notification is sent to the device when the setup is complete.

[1616] Step 10:

[1617] Users can issue voice commands to the robot and express their emotions, such as "Brighten the living room lights."

[1618] Input: Voice commands, user sentiment data

[1619] Output: Voice commands and emotional data are recognized by the robot.

[1620] Specific action: The user tells the robot, "Turn up the living room lights," and the robot smiles.

[1621] Step 11:

[1622] The robot analyzes voice commands and user emotions using an emotion engine and sends the information to a central server.

[1623] Input: User voice commands and sentiment data

[1624] Output: Voice commands and emotion data are sent to the server.

[1625] Specific operation: The robot analyzes the command "Turn up the living room lights" and a smile, and sends that data to the server.

[1626] Step 12:

[1627] The server generates control commands based on the received voice commands and emotion data, and sends them to the corresponding device.

[1628] Input: Voice commands and emotion data sent to the server.

[1629] Output: A control command is generated and sent to the relevant device.

[1630] Specific operation: The server generates a command to "brighten the living room lights" and sends it to the corresponding lighting system.

[1631] Step 13:

[1632] The device (lighting system) makes adjustments based on the commands it receives.

[1633] Input: Control commands sent from the server

[1634] Output: The brightness of the lighting is adjusted.

[1635] Specific operation: The lighting system increases its brightness according to the received command.

[1636] Step 14:

[1637] Users can confirm that the lighting is adjusted to the specified level and experience comfort as the environment is appropriately adjusted.

[1638] Input: Lighting system adjustment results

[1639] Output: The user can see the change in lighting.

[1640] Specific action: The user confirms that the living room lights have brightened and realizes that a comfortable environment has been created.

[1641] (Application Example 2)

[1642] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1643] In modern brick-and-mortar stores, it is essential to appropriately control environmental elements such as lighting, air conditioning, and music to enhance customer comfort and satisfaction. However, manually adjusting these environmental factors is labor-intensive and time-consuming, and providing nuanced responses tailored to the emotions and needs of each individual customer is a difficult task. Furthermore, controlling individual devices requires specialized knowledge, and it is impractical for store operators to understand all of them. Therefore, there is a need for an automated and centralized environmental control system that responds to customer emotions.

[1644] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for registering real estate information, means for the user to input home appliance information, means for analyzing device information, means for transmitting the registration information of home appliances to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of home appliance information, means for analyzing the user's emotions using an emotion recognition engine, and means for adjusting home appliances and the environment based on the analyzed emotion data. As a result, environmental adjustments in response to customer emotions are performed automatically in real time, enabling store operators to provide an optimal store environment without specialized knowledge.

[1645] "Real estate information" refers to detailed information about properties such as buildings and land. This includes data such as the property's location, size, price, and owner.

[1646] "Home appliance information" refers to information about household electrical appliances. This includes data such as product name, model number, installation location, and function settings.

[1647] "Device information" refers to detailed information about individual home appliances and electronic devices connected to the system. This includes device identification information, function settings, and connection status.

[1648] A "central server" refers to a central computer system that manages device and user information on a network, and handles communication and data processing with each device.

[1649] An "emotion recognition engine" refers to software or algorithms that analyze and recognize a user's emotional state from data such as audio and video.

[1650] "Environmental adjustment" refers to the operation of changing the settings of various devices in order to optimize the in-store environment, such as lighting, air conditioning, and music.

[1651] "Voice commands" refer to voice data that users use to instruct specific actions. These include specific commands such as turning lights on or off or adjusting the temperature.

[1652] "Notification methods" refer to systems and techniques used to inform users and devices of information sent from a central server. This includes things like notifications from smartphone apps and voice guidance from robots.

[1653] The system for realizing this application consists of the following elements.

[1654] 1. Overview of the entire system

[1655] This system integrates and manages real estate and home appliance information, and analyzes user emotions using an emotion recognition engine to provide an optimal environment.

[1656] The hardware used includes smartphones, smart glasses, and Arduino.

[1657] The software used includes an emotion recognition API and a Python program.

[1658] 2. User actions

[1659] Users input their property information and appliance information into the system using their smartphones or smart glasses.

[1660] The system analyzes the user's voice commands and emotional responses to provide the optimal environment.

[1661] 3. Server Role

[1662] The server stores real estate and home appliance information submitted by users in its database.

[1663] The server uses an emotion recognition engine to analyze the user's emotions and generates control commands for lighting, air conditioning, music systems, and other functions based on the results.

[1664] The generated control commands are sent to the corresponding device, which then adjusts the environment.

[1665] 4. Hardware and Software Usage

[1666] Smartphones and smart glasses function as user interfaces and are used for voice input and acquiring emotional data.

[1667] Arduino is used to control devices such as lighting, air conditioning, and music systems.

[1668] The emotion recognition API is used to analyze the user's voice commands and emotions.

[1669] The Python program is responsible for sending and receiving data and generating control commands.

[1670] 5. Specific Examples

[1671] The following are some specific examples.

[1672] Example 1: Lighting adjustment

[1673] 1. The user says, "Please turn up the living room lights," indicating a desire to relax.

[1674] 2. The smart glasses send voice and emotion data to an emotion recognition API, and the analyzed information is sent to a server.

[1675] 3. The server generates control commands for the lighting system based on the analysis results and adjusts the lighting via Arduino.

[1676] Example 2: Adjusting the air conditioning

[1677] 1. The user says, "Make this room cooler," expressing an emotional need for comfort.

[1678] 2. The smartphone receives voice and emotion data and sends it to the server.

[1679] 3. The server generates control commands for the air conditioning system based on the emotion recognition results and voice data, and adjusts the air conditioning using Arduino.

[1680] Example of a prompt

[1681] Example of a prompt message regarding lighting adjustments: "Brighten the living room lights."

[1682] Example of a prompt message regarding air conditioning adjustment: "Cool this room down."

[1683] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1684] Step 1:

[1685] The user inputs voice commands and emotions using a smartphone or smart glasses. For example, they might say, "Brighten the living room lights," while also expressing a relaxed emotion. The input here consists of voice data and emotion data. The device collects this data and prepares it to be sent to an emotion recognition API.

[1686] Step 2:

[1687] The device sends voice data and emotion data to the emotion recognition API. The voice data and emotion data sent from the device are input into the emotion recognition engine on the server side and analyzed. The processing here converts voice commands into text and outputs emotions as numerical data.

[1688] Step 3:

[1689] The server receives the analysis results returned from the emotion recognition API. The input data includes the text format of the voice command and the emotion analysis results. Based on this data, the server determines which device to operate and generates control commands. Specifically, it analyzes the obtained text data, retrieves the ID of the lighting system corresponding to the interpreted command, and determines the optimal lighting level based on the emotion data.

[1690] Step 4:

[1691] The server generates control commands which are then sent to the corresponding device via the Arduino. The transmitted control commands specify the brightness of the lighting system, for example, in the format "Set brightness to 70%".

[1692] Step 5:

[1693] The Arduino executes the control commands it receives. The input data consists of control commands sent from the server. The Arduino analyzes these commands, converts them into specific device control signals, and sends them to the device. The lighting system then receives the commands and adjusts to the specified brightness.

[1694] Step 6:

[1695] The system notifies the user of the results of the environment adjustments. A notification appears on the device to inform the user that the lighting has been adjusted to the specified brightness. Furthermore, it provides feedback on the overall system status and the next necessary actions.

[1696] By performing these processing steps, users can easily and intuitively optimize the store environment according to their emotions.

[1697] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1698] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1699] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1700] [Fourth Embodiment]

[1701] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1702] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1703] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1704] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1705] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1706] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1707] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1708] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1709] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1710] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1711] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1712] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1713] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1714] The present invention is a system for integrated management and control of real estate information and home appliance information, designed to be intuitively operable by the user. Specific embodiments of the present invention are described below.

[1715] Program processing

[1716] Overall Overview

[1717] The system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands, and a terminal (smartphone) that receives user input. This allows for centralized management of property information and smart devices throughout the home.

[1718] User registers with the system

[1719] 1. User

[1720] When registering a new account using the smartphone app, you will need to enter the required information (name, address, email address, phone number, etc.) and submit it to the system.

[1721] 2. Server

[1722] The server receives user information and stores it in the database.

[1723] Property registration and device setup

[1724] 1. User

[1725] Register information about newly purchased or rented property in the app. Enter details such as the property's location, size, and floor plan.

[1726] Select a list of home appliances you plan to install in your home.

[1727] 2. Server

[1728] The server stores property information in a database and retrieves a list of suitable appliances for the user's home, along with related initial setup information.

[1729] The system analyzes information about the home appliances selected by the user and generates initial setup parameters corresponding to each device.

[1730] Home appliance and system management

[1731] 1. Device (smartphone)

[1732] Users scan QR codes on home appliances using the app and add the appliance information to the system.

[1733] Information about the added home appliance is sent to the server and registered in the database.

[1734] 2. Server

[1735] Based on the registered information of the home appliances, the initial setup is performed and applied to each device.

[1736] The registered information is automatically updated and reflected in the user's system.

[1737] 3. User

[1738] The status of all home appliances and systems can be checked in real time via the app.

[1739] Operation by voice command

[1740] 1. User

[1741] You can issue voice commands to the robot, such as "Turn up the living room lights."

[1742] 2. Butler Robot

[1743] It recognizes voice commands and analyzes their content.

[1744] The analyzed instructions are sent to the central server as commands.

[1745] 3. Server

[1746] The server receives the command and generates a control command for the corresponding home appliance.

[1747] The generated command is sent to the target device.

[1748] 4. Home appliances (lighting systems)

[1749] The lighting system adjusts its brightness based on commands received from the server.

[1750] 5. Users

[1751] The lighting is adjusted to the specified level, making the environment more comfortable.

[1752] Specific example

[1753] Listening: Adding home appliances

[1754] 1. User

[1755] If you want to install your newly purchased smart refrigerator in your home, launch the smartphone app and scan the QR code on the refrigerator.

[1756] 2. Device (smartphone)

[1757] The QR code information is analyzed, and the refrigerator's device information is obtained.

[1758] 3. Server

[1759] Device information is sent to the server and registration is performed.

[1760] The refrigerator's initial setup is performed and reflected in the system.

[1761] 4. Butler Robot

[1762] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[1763] Example 2: Adjusting the room lighting

[1764] 1. User

[1765] If you want to adjust the living room lighting by voice, you can say to the butler robot, "Brighten the living room lights."

[1766] 2. Butler Robot

[1767] It recognizes and analyzes voice commands.

[1768] Send the analysis results to the server.

[1769] 3. Server

[1770] It receives a command and generates a command to increase the brightness of the living room lighting system.

[1771] Send a command to the lighting system.

[1772] 4. Terminal (lighting system)

[1773] It receives commands from the server and adjusts the brightness of the lighting.

[1774] 5. Users

[1775] The lighting is adjusted to achieve the desired brightness.

[1776] These concrete examples demonstrate that users can intuitively and efficiently manage home appliances and systems, thereby improving their quality of life.

[1777] The following describes the processing flow.

[1778] User registers with the system

[1779] Step 1:

[1780] The user launches the app on their smartphone and taps the "New Registration" button.

[1781] Step 2:

[1782] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[1783] Step 3:

[1784] The device (smartphone) sends the entered information to the central server.

[1785] Step 4:

[1786] The server saves the received information to the database and creates user accounts.

[1787] Step 5:

[1788] The server returns a response to the terminal indicating successful account creation and notifies the user that account creation is complete.

[1789] Property registration and device setup

[1790] Step 1:

[1791] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[1792] Step 2:

[1793] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[1794] Step 3:

[1795] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[1796] Step 4:

[1797] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[1798] Step 5:

[1799] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[1800] Home appliance and system management

[1801] Step 1:

[1802] The user installs the new appliance and scans a QR code with a smartphone app.

[1803] Step 2:

[1804] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[1805] Step 3:

[1806] The terminal sends the information it has acquired to the central server.

[1807] Step 4:

[1808] The server receives home appliance information, saves it to a database, and configures the devices.

[1809] Step 5:

[1810] The server sends a notification to the device indicating that the setup is complete.

[1811] Step 6:

[1812] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[1813] Operation by voice command

[1814] Step 1:

[1815] The user issues voice commands to the butler robot, such as "Turn up the living room lights."

[1816] Step 2:

[1817] The butler robot recognizes voice commands, analyzes their content, and generates commands to control the lighting.

[1818] Step 3:

[1819] The butler robot sends the control commands it generates to the central server.

[1820] Step 4:

[1821] The server checks the device ID of the lighting system and generates a command to adjust the brightness.

[1822] Step 5:

[1823] The server generates commands and sends them to the lighting system devices.

[1824] Step 6:

[1825] The lighting system sets the brightness based on the command it receives.

[1826] Step 7:

[1827] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[1828] Step 8:

[1829] The butler robot notifies the user via voice, "I've brightened the living room lights."

[1830] This series of processing steps allows users to intuitively and efficiently manage their home appliances and the entire system.

[1831] (Example 1)

[1832] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1833] With the recent proliferation of smart home appliances, there is a growing need to efficiently manage numerous devices and property information. However, current systems require users to manage information individually using different platforms and applications, leading to cumbersome and inefficient systems. Furthermore, even with voice-controlled appliances, there is no unified procedure or system, and user convenience is not adequately considered. To address these challenges, it is necessary to provide a system that allows users to easily integrate and manage smart home appliances and use it intuitively.

[1834] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1835] In this invention, the server includes means for a user to register a new account using a mobile terminal and transmit necessary information to a central server; means for a user to input property information and a list of home appliances using a mobile terminal and transmit them to a central server; means for the central server to store and analyze the received user information and property information in a database; means for scanning the QR code of a home appliance with a mobile terminal and transmitting device information to a central server; means for the central server to perform initial setup of the home appliance based on the received device information; means for the central server to transmit the generated setting information to the home appliance and adjust the status of the home appliance; means for a user to issue a voice command to a robot, for the robot to recognize and analyze the voice command and transmit it to a central server; means for the central server to generate control commands for the home appliance based on the analyzed voice command and transmit them to the home appliance; and means for a user to check and control the status of the home appliance in real time using a mobile terminal. This enables users to centrally manage a variety of home appliance and real estate information and operate them intuitively and efficiently.

[1836] "Account registration" is the process by which a user registers their information with the system, thereby obtaining a unique identifier and enabling them to use the system.

[1837] A "central server" is a central computer system that receives information sent by users, stores it in a database, and performs necessary analysis and processing.

[1838] "User information" refers to personally identifiable information such as the user's name, address, email address, and phone number.

[1839] A "database" is a collection of digital information used to efficiently store, manage, and retrieve information within a system.

[1840] A "mobile device" refers to a communication device that can be carried and used, such as a smartphone or tablet.

[1841] "Real estate information" refers to detailed information about a house or building, such as its location, size, and floor plan.

[1842] A "household appliance list" is a list of electrical appliances used in a home.

[1843] A "QR code" is a rectangular, two-dimensional barcode that contains various types of information, and by scanning it, data can be quickly retrieved.

[1844] "Initial setup" refers to the basic configuration process required to make a new device usable.

[1845] "Voice commands" refer to a method in which a user uses their voice to instruct a system or device to perform a specific action.

[1846] A "robot" is an automated mechanical device that has the function of receiving and recognizing voice commands and transmitting those commands to a central server.

[1847] A "control command" is an electronic instruction that tells a device to perform a specific action.

[1848] "Real-time" means that system or device operations and information updates occur instantly.

[1849] "Centralized management" refers to the integrated management of multiple devices and information through a single system or platform.

[1850] "Analysis" is the process of breaking down received data or information into smaller parts, understanding it, and using it for a specific purpose.

[1851] "Status adjustment" means changing or setting up the operating conditions of a device based on user instructions.

[1852] This invention relates to a system for the integrated management and control of real estate information and home appliance information. Designed for intuitive user operation, it consists of a mobile terminal, a central server, a home appliance control robot, and smart home appliances.

[1853] Program processing

[1854] Overall Overview

[1855] The system consists of a mobile application, a central server, a robot that receives user voice commands, and smart home appliances that receive user input. This system allows for centralized management of property information and smart devices throughout the home.

[1856] User account registration

[1857] A user registers a new account using a mobile device. The user enters information such as their name, address, email address, and phone number, and sends it to a central server. The server receives the user information and stores it in a database. This database could be something like MySQL.

[1858] Real estate and home appliance information registration

[1859] Users input new property information and appliance lists using their mobile devices and send them to a central server. The server stores the received property information in a database and performs analysis. Based on the property information, it automatically generates an appropriate appliance list and provides it to the user. A Python script could be used for this analysis.

[1860] Initial setup and management of home appliances

[1861] The user scans a QR code on a home appliance with their mobile device and sends the device information to a central server. The server uses the received device information to perform the initial setup of the home appliance and sends the generated configuration information to the appliance. This process uses the device manufacturer's API (e.g., Philips Hue API).

[1862] Controlling home appliances by voice commands

[1863] The user issues a voice command to the robot, such as "Brighten the living room lights." The robot recognizes the voice command, analyzes its content, and sends it to a central server. This recognition uses the Google Speech-to-Text API. Based on the analyzed voice command, the server generates control commands for the appliance and sends them to the target appliance. The appliance adjusts its operation according to the received control commands.

[1864] Specific example

[1865] Example 1: Adding home appliances

[1866] When a user installs a newly purchased smart refrigerator at home, they launch the app on their mobile device and scan the refrigerator's QR code. The QR code information is analyzed, and the refrigerator's device information is retrieved. The device information is sent to a central server for registration. The refrigerator's initial setup is performed and reflected in the system. A robot retrieves the new device information and notifies the user that "the new refrigerator has been registered."

[1867] Example 2: Adjusting the room lighting

[1868] If a user wants to adjust the living room lighting by voice, they would say to the robot, "Brighten the living room lights." The robot would recognize and analyze the voice command. It would then send the analysis results to a central server. The server would receive the analysis results, generate a command to increase the brightness of the living room lighting system, and send it. The lighting system would receive the command and adjust the brightness. The user would then confirm the lighting change and see that the desired brightness had been achieved.

[1869] Example of a prompt

[1870] "Please explain how to add a new home appliance. In particular, please provide detailed instructions on scanning the QR code and the data processing on the server side."

[1871] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1872] Step 1: User account registration

[1873] Input: The user enters information such as their name, address, email address, and phone number using their mobile device and presses the "Register" button.

[1874] Output: The data is sent to the central server, and a registration success message is returned to the user's device.

[1875] Specific actions:

[1876] 1. The user launches the app on their mobile device and is directed to the account registration screen.

[1877] 2. The user enters their name, address, email address, and phone number, and presses the submit button. This sends an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / register).

[1878] 3. The server verifies the received information and saves it to the database. Upon success, a response containing status code 200 is returned, and the user receives the message "Registration complete."

[1879] Step 2: Register property and appliance information

[1880] Input: The user enters property information and a list of appliances using a mobile device and presses the submit button.

[1881] Output: The transmitted information is analyzed by a central server and stored in a database. A list of appropriate home appliances is also generated.

[1882] Specific actions:

[1883] 1. The user selects the "Real Estate Information Registration" screen from the app's menu.

[1884] 2. Enter property information such as location, size, and floor plan, and select the desired appliances from the appliance list.

[1885] 3. After entering the information, press the submit button to send an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / properties / register).

[1886] 4. The server analyzes the received information and stores it in a database (e.g., MySQL). Using multiple SQL queries, it registers real estate information and appliance list information, and generates appliance settings suitable for the property specified by the user.

[1887] Step 3: Initial setup and management of home appliances

[1888] Input: The user scans the QR code on the home appliance with their mobile device and sends the obtained device information.

[1889] Output: Device information is sent to the central server for registration and initial setup.

[1890] Specific actions:

[1891] 1. The user places the new home appliance and launches the app on their mobile device.

[1892] 2. Use the app's QR code scanner function (e.g., Zebra Crossing) to scan the QR code on the home appliance and obtain device information.

[1893] 3. Send the retrieved device information as an HTTP POST request to the API endpoint (e.g., https: / / example.com / api / devices / register).

[1894] 4. The server stores the received device information in a database and generates an initial configuration file using a Python script or shell script, which is then distributed to each home appliance.

[1895] Step 4: Controlling home appliances with voice commands

[1896] Input: The user issues a voice command to the robot, such as "Brighten the living room lights."

[1897] Output: The robot recognizes voice commands, sends the analysis results to a central server, and generates control commands for home appliances based on those results.

[1898] Specific actions:

[1899] 1. The user issues a pre-set voice command to the home appliance control robot.

[1900] 2. The robot uses the Google Speech-to-Text API to recognize voice commands and convert their content into text format.

[1901] 3. Send the analysis results to the central server as an HTTP POST request. The API endpoint is (e.g., https: / / example.com / api / voicecommands).

[1902] 4. The server generates control commands for the target device based on the received analysis results. For example, it creates an SQL query such as "UPDATE devices SET brightness = ? WHERE location = ?" and sends the adjustment command to the smart lighting device.

[1903] 5. The lighting system adjusts the brightness based on the received command, and the user confirms the change.

[1904] (Application Example 1)

[1905] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1906] In modern factory operations, a wide variety of equipment and sensors are introduced, but it is difficult to manage and control them efficiently in a unified manner. Therefore, it is necessary to use separate management systems for each piece of equipment, leading to increased complexity and costs. Furthermore, there is a lack of adequate systems for real-time monitoring of equipment status and early detection of abnormalities. In particular, there is a need to quickly reflect information on newly introduced equipment in the system, and to perform continuous status monitoring and appropriate operation.

[1907] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1908] In this invention, the server includes means for registering real estate information, means for users to input equipment information, means for analyzing device information, means for transmitting equipment registration information to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of equipment information, means for recognizing voice commands and transmitting the analyzed content to the central server, means for generating equipment control commands based on the user's voice commands and transmitting them to the corresponding devices, means for receiving control commands sent from the central server and adjusting the equipment status, means for providing an application for users to centrally manage property information and equipment information, means for automatically generating a list of corresponding equipment based on property information, a robot for users to issue voice commands, means for transmitting voice commands recognized by the robot to the central server, means for notifying the user when new equipment is registered, and means for updating the operating status of equipment in real time. This makes it possible to centrally manage and efficiently control a wide variety of equipment and sensors in a factory. In addition, information on newly introduced equipment can be quickly reflected in the system, enabling real-time monitoring of operating status and appropriate operation.

[1909] (definition statement)

[1910] "Real estate information" is a general term for data related to properties owned or managed by a user. This includes information such as the property's location, size, and floor plan.

[1911] "Device information" refers to the collective data about devices and equipment owned or managed by a user. This includes information such as the type of device, its function, and its operating status.

[1912] "Device information" is a general term for detailed technical data about equipment and devices. This includes information such as the device's components, specifications, and operating status.

[1913] A "central server" refers to a central computer system that receives, manages, and controls data from a large number of users and devices.

[1914] "Voice commands" refer to instructions given by a user to a system using their voice. Examples include commands such as "Brighten the living room lights."

[1915] A "control command" refers to an instruction used to tell a device or equipment to perform a specific action. These commands are generated by a central server and sent to each device.

[1916] "Operating status" refers to information about the current state in which a piece of equipment or device is operating. This includes states such as normal operation, malfunction, and awaiting maintenance.

[1917] An "application" refers to software that allows users to centrally manage real estate and equipment information using devices such as smartphones and tablets.

[1918] A "robot" refers to an autonomous mechanical device that has the function of receiving voice commands from a user, analyzing them, and transmitting them to a central server.

[1919] "Notification" refers to the act of transmitting information from the system to the user. This includes changes in the registration status and operating status of equipment.

[1920] This invention relates to a system for centrally managing and efficiently controlling equipment and sensors within a factory. The system is designed for intuitive user operation and consists of a smartphone application, a central server, and a robot that receives voice commands. Detailed embodiments for carrying out this invention are described below.

[1921] User registers with the system

[1922] 1. User

[1923] When users register a new account using the smartphone application, they enter necessary information such as their name, address, email address, and phone number, and submit it to the system. This information is stored on the central server, as described below.

[1924] 2. Server

[1925] The server receives user information and stores it in the database. This is achieved using a database management library such as SQLAlchemy.

[1926] Device registration and device configuration

[1927] 1. User

[1928] Users register information about newly installed factory equipment using a smartphone application. By scanning a QR code, they can input detailed information such as the type of equipment, performance, and operating status into the system.

[1929] 2. Server

[1930] The server stores equipment information in a database and generates a list of equipment suitable for the user's factory, along with associated initial configuration information. The server then analyzes this information and generates initial configuration parameters for each device. This process is implemented by a server program combining Python's Flask and SQLAlchemy.

[1931] Equipment management and control

[1932] 1. Device (smartphone)

[1933] Users scan the device's QR code using a smartphone application and register their information. This information is sent to a server, and the appropriate device settings are configured.

[1934] 2. Server

[1935] The server performs the initial setup of the equipment based on the received information and reflects this in the database. Furthermore, the registered information is automatically updated and reflected in the user's system in real time.

[1936] 3. User

[1937] Users can check the operating status of all devices in real time using the application.

[1938] Operation by voice command

[1939] 1. User

[1940] Users can issue voice commands to the robot, such as "Check the status of a specific device." This allows users to operate the system without using their hands.

[1941] 2. Robots

[1942] The robot recognizes voice commands and analyzes their content. The analyzed commands are then sent to a central server. The Python SpeechRecognition library is used for this speech recognition.

[1943] 3. Server

[1944] The server receives the instruction and generates control commands for the corresponding device. These commands are then sent from the central server to the device.

[1945] 4. Equipment

[1946] The equipment adjusts its operation based on commands received from the server. For example, it may change the operating status of a specified piece of equipment.

[1947] 5. Users

[1948] When the system functions as expected, users experience improved ease of use.

[1949] Specific example

[1950] Adding equipment

[1951] When a user installs a newly purchased press machine in their factory, they launch a smartphone application and scan the press machine's QR code. The QR code information is analyzed, and the press machine's device information is retrieved. The device information is sent to a server and registered in a database. Subsequently, a robot retrieves the new device information and notifies the user that "the new equipment has been registered."

[1952] Monitoring and adjustment of equipment operating status

[1953] When a user wants to check the operating status of a specific device and operate it, they check the status from a smartphone application and give a voice command to the robot saying, "Check the status of this device." The voice command is recognized, and the analysis results are sent to the server. The server receives the command, generates and sends control commands for the device in question. The device receives the commands from the server and adjusts its operating status. This allows the user to monitor and operate the device's status in real time.

[1954] Example of a prompt

[1955] If you want to install newly purchased equipment in your factory, how do you scan the QR code and register it in the system?

[1956] How can I check the operating status of specific equipment within the factory?

[1957] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1958] Step 1:

[1959] A user registers a new account using a smartphone application. The user enters required information such as their name, address, email address, and phone number, and submits it to the system. The entered data is sent from the application to the server. The server receives this data and stores it in its database.

[1960] Step 2:

[1961] Users register information about newly installed factory equipment using a smartphone application. Users scan the QR code of the factory equipment and input detailed information such as the type of equipment, performance, and operating status. This data is sent from the smartphone application to the server. The server analyzes the received device information and stores it in a database.

[1962] Step 3:

[1963] The server generates initial configuration parameters for each device. This uses an algorithm that automatically determines appropriate parameters based on the type and characteristics of the device. The generated initial configuration parameters are then registered in a database.

[1964] Step 4:

[1965] The user scans the device's QR code using a smartphone application to complete registration. This adds the new device's information to the system. The added device information is sent to the server and registered in the database.

[1966] Step 5:

[1967] The server performs the initial setup of the devices and applies these settings to them. Based on the initial setup parameters for each device, the server generates and sends configuration commands to each device. The devices apply the initial setup based on the commands they receive.

[1968] Step 6:

[1969] Users can check the operating status of all equipment in real time using a smartphone application. The application retrieves the latest equipment information from the server and displays it to the user. This process allows users to always be aware of the status of equipment in the factory.

[1970] Step 7:

[1971] The user issues a voice command to the robot. For example, they might give the robot a command such as, "Check the status of a specific device." The robot recognizes this voice command and analyzes its content. The analyzed command is then sent to the server as a command.

[1972] Step 8:

[1973] The server generates control commands for the corresponding device based on the received voice commands. The generated control commands are sent from the central server to the device. The device adjusts its operation according to these control commands.

[1974] Step 9:

[1975] Users benefit from improved operational convenience when the system functions as expected. When control is functioning correctly, users can confidently continue their work within the factory.

[1976] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1977] The present invention is a system that integrates and manages real estate information and home appliance information, and combines it with an emotion engine that recognizes user emotions. The following describes specific embodiments of the present invention.

[1978] Program processing

[1979] Overall Overview

[1980] This system consists of an application for managing real estate and home appliance information, a central server, a robot that receives user voice commands and emotions, and a terminal (smartphone) that receives user input. This allows for integrated management of property information and smart devices throughout the home, and enables the system's operation to be adjusted according to the user's emotions.

[1981] User registers with the system

[1982] 1. User

[1983] Launch the app on your smartphone and tap the "New Registration" button.

[1984] Enter the required personal information and press the "Submit" button.

[1985] 2. Server

[1986] The received information is saved to the database, and a user account is created.

[1987] A response indicating successful account creation is sent to the device, notifying the user.

[1988] Property registration and device setup

[1989] 1. User

[1990] Register your property information using the app and select the appliances you plan to install from the appliance list.

[1991] 2. Server

[1992] Based on real estate information and selected appliance information, initial setup parameters are generated and sent to the terminal.

[1993] Home appliance and system management

[1994] 1. User

[1995] Install the new appliance and scan the QR code with a smartphone app.

[1996] 2. Device (smartphone)

[1997] The system analyzes QR code information and sends the home appliance information to a central server.

[1998] 3. Server

[1999] The system saves the received home appliance information to a database and configures the device settings.

[2000] A notification will be sent to the device and displayed upon completion of the setup.

[2001] Operation via voice commands and emotion recognition

[2002] 1. User

[2003] It issues voice commands such as "Turn up the living room lights" and also expresses emotions.

[2004] 2. Butler Robot

[2005] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[2006] Voice commands and emotional data are sent to a central server.

[2007] 3. Server

[2008] Based on the received voice commands and emotion data, the device ID of the lighting system is identified, and the appropriate control command is generated.

[2009] Sends a command to the lighting system device.

[2010] 4. Home appliances (lighting systems)

[2011] Adjustments are made based on the received commands.

[2012] 5. Users

[2013] Comfort is ensured by adjusting the lighting to a specified level and making appropriate environmental adjustments.

[2014] Specific example

[2015] Example 1: Adding home appliances

[2016] 1. User

[2017] I installed the newly purchased smart refrigerator in my home, launched the smartphone app, and scanned the QR code.

[2018] 2. Device (smartphone)

[2019] The system acquires and analyzes QR code information and sends device information to the server.

[2020] 3. Server

[2021] The system receives device information and registers home appliances in its database.

[2022] Perform the initial setup of the refrigerator and reflect the information in the system.

[2023] 4. Butler Robot

[2024] The system retrieves new device information and notifies the user with the message, "A new refrigerator has been registered."

[2025] Example 2: Lighting adjustment

[2026] 1. User

[2027] If you want to adjust the living room lighting using your voice, say "Brighten the living room lights" and express your emotion.

[2028] 2. Butler Robot

[2029] Along with voice commands, the system uses an emotion engine to recognize and analyze the user's emotions.

[2030] It sends voice commands and emotional data to the server.

[2031] 3. Server

[2032] Based on voice commands and emotion data, the system identifies the device ID of the lighting system and generates appropriate control commands.

[2033] Send a command to the lighting system.

[2034] 4. Terminal (lighting system)

[2035] Receives commands and adjusts to the specified level.

[2036] 5. Users

[2037] The lighting is adjusted, and the environment is kept in an optimal state according to the user's emotions.

[2038] This allows users to intuitively and efficiently manage their home appliances and entire systems, and further improves their quality of life through emotion recognition.

[2039] The following describes the processing flow.

[2040] User registers with the system

[2041] Step 1:

[2042] The user launches the app on their smartphone and taps the "New Registration" button.

[2043] Step 2:

[2044] The user enters the required personal information such as their name, email address, phone number, and address, and then presses the "Submit" button.

[2045] Step 3:

[2046] The device (smartphone) sends the entered information to the central server.

[2047] Step 4:

[2048] The server saves the received information to the database and creates user accounts.

[2049] Step 5:

[2050] The server returns a response to the terminal indicating successful account creation, notifying the user that account creation is complete.

[2051] Property registration and device setup

[2052] Step 1:

[2053] The user selects the "Property Registration" option within the app and enters detailed information such as the property's location, size, and floor plan.

[2054] Step 2:

[2055] The user selects the appliance they intend to register from the appliance list and presses the "Next" button.

[2056] Step 3:

[2057] The device (smartphone) sends real estate information and selected home appliance information to a central server.

[2058] Step 4:

[2059] The server records the received real estate information in a database and generates corresponding initial configuration parameters.

[2060] Step 5:

[2061] The server generates initial setup information for home appliances that is suitable for the user and sends it to the terminal.

[2062] Home appliance and system management

[2063] Step 1:

[2064] The user installs the new appliance and scans a QR code with a smartphone app.

[2065] Step 2:

[2066] The device (smartphone) retrieves detailed information about the home appliance (model, model number, device ID, etc.) from the QR code.

[2067] Step 3:

[2068] The terminal sends the information it has acquired to the central server.

[2069] Step 4:

[2070] The server receives home appliance information, saves it to a database, and configures the devices.

[2071] Step 5:

[2072] The server sends a notification to the device indicating that the setup is complete.

[2073] Step 6:

[2074] The device displays a notification to the user that the setup is complete and informs the butler robot that a new appliance has been added.

[2075] Operation via voice commands and emotion recognition

[2076] Step 1:

[2077] The user can issue voice commands such as "Turn up the living room lights" and express emotions.

[2078] Step 2:

[2079] The butler robot recognizes voice commands and emotions, and analyzes their content.

[2080] Step 3:

[2081] The butler robot sends voice commands and emotional data to a central server.

[2082] Step 4:

[2083] The server receives voice commands and emotion data, verifies the device ID of the lighting system, and generates appropriate control commands.

[2084] Step 5:

[2085] The server generates control commands which are then sent to the lighting system devices.

[2086] Step 6:

[2087] The lighting system sets the brightness based on the command it receives.

[2088] Step 7:

[2089] The server checks the status of the lighting system and notifies the user and the robot when the adjustments are complete.

[2090] Step 8:

[2091] The butler robot notifies the user via voice, "I've brightened the living room lights."

[2092] This allows users to efficiently and intuitively manage their home appliances and entire systems using voice commands and emotions.

[2093] (Example 2)

[2094] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2095] In modern homes and offices, numerous home appliances and devices exist, each operating as an independent system. This makes it cumbersome for users to manage and control individual devices, hindering efficient operation. Furthermore, conventional systems lack the ability to adjust their behavior based on user emotions, resulting in a failure to adequately improve user comfort.

[2096] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for providing a terminal for the user to operate, means for the user to register real estate information using the terminal, means for the user to input home appliance information using the terminal, means for analyzing information of home appliance devices using the terminal, means for transmitting the analyzed home appliance information to a central server using the terminal, means for the central server to configure the device based on the transmitted information, means for notifying the terminal that the device configuration is complete, means for reflecting new device information in the system based on the notified information, means for notifying the user of home appliance information, a robot for recognizing voice commands and analyzing emotion data along with the voice commands, means for transmitting the voice commands and emotion data analyzed by the robot to a central server, means for the central server to generate control commands for home appliances based on the received voice commands and emotion data and transmit them to the corresponding device, means for receiving control commands sent from the central server and adjusting the state of the home appliances, means for automatically generating a list of corresponding home appliances based on real estate property information, a robot for the user to issue voice commands, and means for transmitting the voice commands and emotion data recognized by the robot to a central server. This will allow users to centrally manage real estate and home appliance information, and create a comfortable living environment through emotion-based adjustments to the operation of home appliances.

[2097] A "user" refers to an individual or legal entity that uses the system to manage and control real estate information and home appliance information.

[2098] A "device" is a device used by a user to perform operations, and includes mobile devices such as smartphones and tablets.

[2099] "Real estate information" refers to data about a property, including information such as its location, overview, number of rooms, and size.

[2100] "Home appliance information" refers to data that includes information such as the name, model number, installation location, and usage status of home appliances.

[2101] A "central server" refers to a server device that receives, analyzes, stores, and processes data sent by users.

[2102] A "device" refers to an electronic device with a specific function, such as smart home appliances.

[2103] "Device setup" refers to the process of performing initial setup and adjusting operating parameters so that home appliances function correctly.

[2104] A "robot" refers to a device that recognizes a user's voice commands and has the ability to analyze emotional data.

[2105] "Voice commands" refer to instructions or requests that users make by voice, such as "Turn up the living room lights."

[2106] "Emotional data" refers to data that indicates a user's emotional state, including expressions such as smiles and sadness.

[2107] A "control command" is an instruction sent from a central server to a home appliance, and refers to a command to perform a specific action.

[2108] A "platform" refers to a software environment that allows users to centrally manage real estate information and home appliance information.

[2109] This invention is a system for integrated management and control of real estate information and home appliance information, combined with an emotion engine that recognizes user emotions. This system consists of a central server, terminals (mobile devices such as smartphones), and a robot that receives user voice commands and emotions.

[2110] Central server

[2111] The central server is responsible for receiving, analyzing, storing, and processing data sent from users. Specifically, it stores real estate and home appliance information sent from user-operated terminals in a database, and performs initial setup and adjustment of operating parameters for home appliance devices. It also generates control commands based on voice commands and emotion data and sends them to the relevant devices.

[2112] terminal

[2113] A terminal is a device used by the user for operation, such as a smartphone or tablet. The terminal provides an interface for the user to input real estate and home appliance information, and transmits the entered information to a central server. It also has the function of scanning QR codes to obtain information about home appliances and transmitting that information to the central server.

[2114] robot

[2115] The robot recognizes voice commands uttered by the user and further analyzes the user's emotions using an emotion engine. The analyzed voice commands and emotion data are then transmitted to a central server. The robot plays a crucial role in adjusting the operation of home appliances based on emotions.

[2116] Specific operation of the system

[2117] User registration

[2118] The user first launches the smartphone app and enters the required personal information on the new registration screen and submits it. The device sends the information to a central server, which creates a new user account. A notification is displayed on the device when registration is complete.

[2119] Entering real estate information and home appliance information

[2120] The user enters property information through a smartphone app and selects the appliances they plan to use from a list of appliances. The device sends this information to a central server, which generates initial setup parameters and returns them to the device.

[2121] Adding home appliances

[2122] The user installs a new home appliance and scans a QR code with a smartphone app. The device analyzes the information and sends it to a central server. The server stores the new appliance information in a database and performs the initial setup. A notification is sent to the user when the setup is complete.

[2123] Operation via voice commands and emotion recognition

[2124] The user issues voice commands to the robot, such as "Brighten the living room lights," and expresses their emotions. The robot analyzes these commands using an emotion engine and sends the voice commands and emotion data to a central server. The server generates control commands based on the received data and sends them to the corresponding device. The device then adjusts based on the commands to provide the optimal environment.

[2125] Specific example

[2126] For example, consider a scenario where a user installs a newly purchased smart refrigerator and scans a QR code. The device sends the QR code information to a central server, which registers the device in its database. A robot then issues a notification stating, "The new refrigerator has been registered."

[2127] Let's consider a scenario where a user gives a voice command to the robot, "Brighten the living room lights," and the robot responds with a smile. The robot's emotion engine analyzes the voice and emotion and sends the information to a central server. The server then sends a command to the lighting system, and the lights are adjusted to be brighter.

[2128] This allows users to comprehensively manage their home environment and create a comfortable living environment based on their emotions.

[2129] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2130] Step 1:

[2131] The user launches the smartphone app and taps the "New Registration" button. Next, they enter the required personal information such as their name and email address, and then press the "Submit" button.

[2132] Input: Personal information entered by the user (e.g., name, email address)

[2133] Output: Data containing personal information stored on the device and sent to the server.

[2134] Specific action: The user enters "Yamada Taro" and "taro@example.com" into the smartphone app and clicks the send button.

[2135] Step 2:

[2136] The terminal retrieves personal information entered by the user and sends it to a central server.

[2137] Input: Personal information entered by the user on the device.

[2138] Output: User information sent to the server

[2139] Specific operation: The terminal sends the entered "Yamada Taro" and "taro@example.com" to the central server.

[2140] Step 3:

[2141] The server stores the received personal information in its database and creates a new user account. Next, it sends a response to the terminal indicating successful account creation.

[2142] Input: User information sent to the server

[2143] Output: New user account registered in the database, success message sent to the terminal.

[2144] Specific operation: The server registers a new account in the database based on "Yamada Taro" and "taro@example.com", and sends a success message to the terminal.

[2145] Step 4:

[2146] Users enter the necessary information into a smartphone app to register their property details. Next, they select the appliances they plan to install from a list of appliances.

[2147] Input: Real estate information (e.g., property location, size), home appliance information (e.g., smart air conditioner)

[2148] Output: The entered information is saved on the terminal and sent to the server.

[2149] Specific operation: The user enters the information for "Yamada Mansion" into the smartphone app and selects "Smart Air Conditioner".

[2150] Step 5:

[2151] The terminal transmits the entered real estate information and home appliance information to a central server.

[2152] Input: Real estate information and home appliance information entered by the user on the device.

[2153] Output: Real estate information and home appliance information sent to the server

[2154] Specific operation: The terminal sends real estate information for "Yamada Mansion" and home appliance information for "Smart Air Conditioner" to the central server.

[2155] Step 6:

[2156] The server stores real estate information and home appliance information in a database, generates initial setup parameters, and sends them to the terminal.

[2157] Input: Real estate information and home appliance information sent from the terminal.

[2158] Output: Information stored in the database, generated initial settings parameters

[2159] Specific operation: The server saves information about "Yamada Mansion" and "Smart Air Conditioner" to the database, generates initial setup parameters, and sends them to the terminal.

[2160] Step 7:

[2161] Users install new home appliances and scan a QR code with a smartphone app.

[2162] Input: QR code for the new home appliance

[2163] Output: Acquired home appliance information

[2164] Specific operation: The user sets up the new smart refrigerator and scans the QR code.

[2165] Step 8:

[2166] The terminal analyzes the information in the QR code and sends the information about the home appliance to a central server.

[2167] Input: Home appliance information obtained from a QR code

[2168] Output: Home appliance information sent to the server

[2169] Specific operation: The device analyzes the smart refrigerator information obtained from the QR code and sends it to the central server.

[2170] Step 9:

[2171] The server saves the received home appliance information to its database and performs the initial setup. It then sends a notification to the terminal when the setup is complete.

[2172] Input: Home appliance information sent from the device

[2173] Output: Information stored in the database, configuration completion notification

[2174] Specific operation: The server saves information about the newly registered smart refrigerator to the database and performs the initial setup. A notification is sent to the device when the setup is complete.

[2175] Step 10:

[2176] Users can issue voice commands to the robot and express their emotions, such as "Brighten the living room lights."

[2177] Input: Voice commands, user sentiment data

[2178] Output: Voice commands and emotional data are recognized by the robot.

[2179] Specific action: The user tells the robot, "Turn up the living room lights," and the robot smiles.

[2180] Step 11:

[2181] The robot analyzes voice commands and user emotions using an emotion engine and sends the information to a central server.

[2182] Input: User voice commands and sentiment data

[2183] Output: Voice commands and emotion data are sent to the server.

[2184] Specific operation: The robot analyzes the command "Turn up the living room lights" and a smile, and sends that data to the server.

[2185] Step 12:

[2186] The server generates control commands based on the received voice commands and emotion data, and sends them to the corresponding device.

[2187] Input: Voice commands and emotion data sent to the server.

[2188] Output: A control command is generated and sent to the relevant device.

[2189] Specific operation: The server generates a command to "brighten the living room lights" and sends it to the corresponding lighting system.

[2190] Step 13:

[2191] The device (lighting system) makes adjustments based on the commands it receives.

[2192] Input: Control commands sent from the server

[2193] Output: The brightness of the lighting is adjusted.

[2194] Specific operation: The lighting system increases its brightness according to the received command.

[2195] Step 14:

[2196] Users can confirm that the lighting is adjusted to the specified level and experience comfort as the environment is appropriately adjusted.

[2197] Input: Lighting system adjustment results

[2198] Output: The user can see the change in lighting.

[2199] Specific action: The user confirms that the living room lights have brightened and realizes that a comfortable environment has been created.

[2200] (Application Example 2)

[2201] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2202] In modern brick-and-mortar stores, it is essential to appropriately control environmental elements such as lighting, air conditioning, and music to enhance customer comfort and satisfaction. However, manually adjusting these environmental factors is labor-intensive and time-consuming, and providing nuanced responses tailored to the emotions and needs of each individual customer is a difficult task. Furthermore, controlling individual devices requires specialized knowledge, and it is impractical for store operators to understand all of them. Therefore, there is a need for an automated and centralized environmental control system that responds to customer emotions.

[2203] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for registering real estate information, means for the user to input home appliance information, means for analyzing device information, means for transmitting the registration information of home appliances to a central server, means for configuring devices based on the transmitted information, means for reflecting new device information based on information sent from the central server, means for notifying the user of home appliance information, means for analyzing the user's emotions using an emotion recognition engine, and means for adjusting home appliances and the environment based on the analyzed emotion data. As a result, environmental adjustments in response to customer emotions are performed automatically in real time, enabling store operators to provide an optimal store environment without specialized knowledge.

[2204] "Real estate information" refers to detailed information about properties such as buildings and land. This includes data such as the property's location, size, price, and owner.

[2205] "Home appliance information" refers to information about household electrical appliances. This includes data such as product name, model number, installation location, and function settings.

[2206] "Device information" refers to detailed information about individual home appliances and electronic devices connected to the system. This includes device identification information, function settings, and connection status.

[2207] A "central server" refers to a central computer system that manages device and user information on a network, and handles communication and data processing with each device.

[2208] An "emotion recognition engine" refers to software or algorithms that analyze and recognize a user's emotional state from data such as audio and video.

[2209] "Environmental adjustment" refers to the operation of changing the settings of various devices in order to optimize the in-store environment, such as lighting, air conditioning, and music.

[2210] "Voice commands" refer to voice data that users use to instruct specific actions. These include specific commands such as turning lights on or off or adjusting the temperature.

[2211] "Notification methods" refer to systems and techniques used to inform users and devices of information sent from a central server. This includes things like notifications from smartphone apps and voice guidance from robots.

[2212] The system for realizing this application consists of the following elements.

[2213] 1. Overview of the entire system

[2214] This system integrates and manages real estate and home appliance information, and analyzes user emotions using an emotion recognition engine to provide an optimal environment.

[2215] The hardware used includes smartphones, smart glasses, and Arduino.

[2216] The software used includes an emotion recognition API and a Python program.

[2217] 2. User actions

[2218] Users input their property information and appliance information into the system using their smartphones or smart glasses.

[2219] The system analyzes the user's voice commands and emotional responses to provide the optimal environment.

[2220] 3. Server Role

[2221] The server stores real estate and home appliance information submitted by users in its database.

[2222] The server uses an emotion recognition engine to analyze the user's emotions and generates control commands for lighting, air conditioning, music systems, and other functions based on the results.

[2223] The generated control commands are sent to the corresponding device, which then adjusts the environment.

[2224] 4. Hardware and Software Usage

[2225] Smartphones and smart glasses function as user interfaces and are used for voice input and acquiring emotional data.

[2226] Arduino is used to control devices such as lighting, air conditioning, and music systems.

[2227] The emotion recognition API is used to analyze the user's voice commands and emotions.

[2228] The Python program is responsible for sending and receiving data and generating control commands.

[2229] 5. Specific Examples

[2230] The following are some specific examples.

[2231] Example 1: Lighting adjustment

[2232] 1. The user says, "Please turn up the living room lights," indicating a desire to relax.

[2233] 2. The smart glasses send voice and emotion data to an emotion recognition API, and the analyzed information is sent to a server.

[2234] 3. The server generates control commands for the lighting system based on the analysis results and adjusts the lighting via Arduino.

[2235] Example 2: Adjusting the air conditioning

[2236] 1. The user says, "Make this room cooler," expressing an emotional need for comfort.

[2237] 2. The smartphone receives voice and emotion data and sends it to the server.

[2238] 3. The server generates control commands for the air conditioning system based on the emotion recognition results and voice data, and adjusts the air conditioning using Arduino.

[2239] Example of a prompt

[2240] Example of a prompt message regarding lighting adjustments: "Brighten the living room lights."

[2241] Example of a prompt message regarding air conditioning adjustment: "Cool this room down."

[2242] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2243] Step 1:

[2244] The user inputs voice commands and emotions using a smartphone or smart glasses. For example, they might say, "Brighten the living room lights," while also expressing a relaxed emotion. The input here consists of voice data and emotion data. The device collects this data and prepares it to be sent to an emotion recognition API.

[2245] Step 2:

[2246] The device sends voice data and emotion data to the emotion recognition API. The voice data and emotion data sent from the device are input into the emotion recognition engine on the server side and analyzed. The processing here converts voice commands into text and outputs emotions as numerical data.

[2247] Step 3:

[2248] The server receives the analysis results returned from the emotion recognition API. The input data includes the text format of the voice command and the emotion analysis results. Based on this data, the server determines which device to operate and generates control commands. Specifically, it analyzes the obtained text data, retrieves the ID of the lighting system corresponding to the interpreted command, and determines the optimal lighting level based on the emotion data.

[2249] Step 4:

[2250] The server generates control commands which are then sent to the corresponding device via the Arduino. The transmitted control commands specify the brightness of the lighting system, for example, in the format "Set brightness to 70%".

[2251] Step 5:

[2252] The Arduino executes the control commands it receives. The input data consists of control commands sent from the server. The Arduino analyzes these commands, converts them into specific devic...

Claims

1. Methods for registering real estate information, A means for users to input information about home appliances, A means of analyzing device information, A means of sending home appliance registration information to a central server, A means of configuring the device based on the transmitted information, A means of reflecting new device information based on information sent from the central server, A means of notifying users of home appliance information, A system that includes this.

2. A means for recognizing voice commands and transmitting the analyzed content to a central server, A means for generating control commands for home appliances based on user voice commands and sending them to the relevant device, The system according to claim 1, comprising means for receiving control commands sent from a central server and adjusting the status of home appliances.

3. A means of providing an application for users to centrally manage real estate information and home appliance information, A method for automatically generating a list of compatible home appliances based on real estate property information, A robot for users to issue voice commands, The system according to claim 1, comprising means for transmitting voice commands recognized by a robot to a central server.

Images