System

Abstract

To provide a system.SOLUTION: The system includes means for displaying the operation interface of an IoT apparatus specified by performing image recognition about video data captured by a camera together with the video of the IoT apparatus, means for transmitting operation information for the operation interface to a server, and means for controlling the IoT apparatus on the basis of a control signal transmitted from the server on the basis of the operation information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, IoT home appliances are mainly controlled through smartphones or smart speakers, but these are very indirect interactions and cannot be said to have an intuitive UI. [Means for solving the problem]

[0005] By combining MR (Mixed Reality) and generative AI, we propose a new UI that allows direct communication with IoT appliances. Users can talk directly to the IoT appliances themselves, and intuitively and freely operate them through conversation and MR-GUI. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a system that uses mixed reality (MR) technology to enable direct communication with IoT home appliances. Specifically, a user wears special goggles or a headset to visually recognize the IoT home appliance. At this time, MR technology displays a direct operation interface on the IoT home appliance. For example, in the case of an air conditioner, a temperature control slider and airflow control buttons are displayed on the air conditioner body using MR, and the user can change the air conditioner settings by directly operating these.

[0029] "Example 2"

[0030] Furthermore, one embodiment of the present invention is a system that uses generative AI to enable conversations between users and IoT appliances. Specifically, when a user speaks to an IoT appliance, the generative AI understands the instruction and transmits the appropriate operation to the IoT appliance. For example, if a user says, "Heat the room," the generative AI understands the instruction and instructs the air conditioner to operate in heating mode.

[0031] "Example 3"

[0032] Another embodiment of the present invention is a system that allows users to intuitively and freely operate IoT home appliances using an MR-GUI. Specifically, users operate IoT home appliances using a GUI displayed using MR technology. For example, an air conditioner's control panel displayed using MR displays a temperature control slider, buttons for adjusting airflow, and buttons for switching operation modes, allowing users to freely change the air conditioner's settings by directly operating these.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user puts on the MR-enabled goggles or headset.

[0036] Step 2: The user visually recognizes the IoT appliance. At this time, a direct operation interface is displayed on the IoT appliance using mixed reality technology.

[0037] Step 3: The user changes the settings of the IoT appliance by directly operating the operation interface displayed in MR.

[0038] "Example 2"

[0039] Step 1: The user talks to the IoT appliance.

[0040] Step 2: The generative AI understands the user's instructions.

[0041] Step 3: The generative AI communicates the understood instructions to the IoT appliance, which then performs the operation according to the instructions.

[0042] "Example 3"

[0043] Step 1: The user puts on the MR-enabled goggles or headset.

[0044] Step 2: The user operates the IoT appliance using the GUI displayed by MR technology.

[0045] Step 3: The user can freely change the settings of the IoT home appliance by directly operating the control panel displayed in MR.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional methods of operating IoT devices typically involve using smartphones or remote controls, which require unintuitive operation and complex settings. Furthermore, there are limited ways for users to directly communicate with IoT devices, creating issues with operability and convenience. Furthermore, the transmission and analysis of real-time operational information is insufficient, making rapid control difficult. To resolve these issues, a system that allows for more intuitive and flexible operation is needed.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, means for enabling conversation between the user and the IoT device using generative AI, means for the user to intuitively and freely operate the IoT device using MR-GUI, means for visually recognizing the IoT device using a visual device worn by the user and displaying an operation interface, and means for transmitting user operation information to the server in real time, which then analyzes the operation information and transmits control signals to the corresponding IoT device. This allows the user to intuitively and freely operate the IoT device, achieving rapid control in real time.

[0051] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0052] An "IoT device" is an electronic device that can communicate with other devices and systems via the Internet and send, receive, and control data.

[0053] "Generative AI" is a type of artificial intelligence technology that refers to algorithms and models for interacting with users and generating data.

[0054] "MR-GUI" is a graphical user interface that uses Mixed Reality technology and is designed to allow users to operate it intuitively.

[0055] "Visual devices" are devices worn by users that provide visual information and blend the real world with the virtual world. Examples include goggles and headsets.

[0056] An "operation interface" is a visual control element that users use to operate IoT devices, including sliders and buttons.

[0057] A "server" is a computer system that processes and manages data on a network, and is responsible for analyzing user operation information and sending control signals to corresponding IoT devices.

[0058] A "control signal" is a signal sent from a server to an IoT device, and includes instructions for changing the device's operation or settings.

[0059] This invention is a system that enables direct communication with IoT devices using Mixed Reality (MR) technology. The user wears special visual devices (goggles or headset) and visually recognizes the IoT devices. The visual devices use MR technology to display the IoT devices in the user's field of vision and overlay an operation interface on top of them.

[0060] Hardware and software used

[0061] Visual equipment: goggles or headset

[0062] Image recognition software: OpenCV, etc.

[0063] MR development platform: Unity, etc.

[0064] Server-side programming languages: Python, Node.js, etc.

[0065] Communication technologies: Wi-Fi, Bluetooth, MQTT, HTTP, etc.

[0066] Explanation of program processing

[0067] When a user wears the vision device, the device recognizes objects in the user's field of vision and identifies IoT devices. For example, to recognize an air conditioner in the living room, the device uses the goggles' built-in camera and image recognition software (e.g., OpenCV). The device displays an operation interface overlaid on the recognized IoT device. In the case of an air conditioner, a slider for adjusting the temperature and buttons for adjusting the airflow are displayed. This interface is generated using an MR development platform such as Unity.

[0068] The user operates the device with their hands or eyes while looking at the operation interface displayed in their field of vision. For example, they change the temperature setting by dragging a temperature control slider with their finger. The device captures the user's operation information and transmits it to the server in real time. This communication uses wireless communication technologies such as Wi-Fi and Bluetooth.

[0069] The server analyzes the received operation information and generates an appropriate control signal for the corresponding IoT device. For example, if a user sets the temperature control slider to 25 degrees, the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. This process uses server-side programming languages ​​such as Python and Node.js. The IoT device receives the control signal sent from the server and changes the setting. In the case of an air conditioner, its internal control system changes the temperature setting to 25 degrees. Protocols such as MQTT and HTTP are used for this communication.

[0070] Specific examples

[0071] A user wants to operate the air conditioner in their living room. The user puts on MR goggles and visually recognizes the air conditioner in the living room. Using MR technology, a temperature control slider and airflow control buttons are displayed on the air conditioner itself. The user operates the temperature control slider to change the set temperature to 25 degrees. This operation information is sent from the goggles to the server, and the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. The air conditioner receives this signal, and its internal control system changes the temperature setting to 25 degrees.

[0072] Prompt Sentence Examples

[0073] "Describe a scenario in which a user puts on mixed reality goggles and operates the air conditioner in their living room. Please describe in detail the process by which the user operates the temperature control slider to change the set temperature to 25 degrees."

[0074] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0075] Step 1:

[0076] The user puts on the MR goggles.

[0077] Input: User puts on MR goggles and turns them on.

[0078] Output: The goggles are up and ready to capture the user's view.

[0079] How it works: The user puts on the MR goggles and presses the power button to turn them on. The camera built into the goggles begins capturing the user's field of vision.

[0080] Step 2:

[0081] The terminal displays IoT devices in the user's field of view.

[0082] Input: Video data captured by the goggles' built-in camera.

[0083] Output: The IoT device is displayed in the user's field of view.

[0084] Specific operation: The device analyzes the video data from the camera and identifies IoT devices using image recognition software (e.g., OpenCV). Identified IoT devices are displayed in the user's field of view.

[0085] Step 3:

[0086] The terminal displays the operation interface.

[0087] Input: Location and type of IoT device.

[0088] Output: The operation interface is displayed overlaid on the IoT device.

[0089] Specific operation: The terminal uses an MR development platform (e.g., Unity) to generate and display operation interfaces such as temperature control sliders and airflow control buttons on the IoT device.

[0090] Step 4:

[0091] The user operates the operation interface.

[0092] Input: User's hand and eye movements.

[0093] Output: Operation information is generated.

[0094] Specific actions: The user operates the interface displayed in the field of view with their hands or eyes. For example, they change the temperature setting by dragging the temperature control slider with their finger.

[0095] Step 5:

[0096] The terminal transmits operation information to the server.

[0097] Input: User operation information.

[0098] Output: Operation information is sent to the server.

[0099] How it works: The device captures user operation information and transmits it to the server in real time. This communication is carried out using wireless communication technologies such as Wi-Fi and Bluetooth.

[0100] Step 6:

[0101] The server analyzes the operation information and sends a control signal to the corresponding IoT device.

[0102] Input: Operation information sent from the device.

[0103] Output: Control signal for IoT devices.

[0104] How it works: The server analyzes the received operation information and generates the appropriate control signal for the corresponding IoT device. For example, if a user sets the temperature control slider to 25 degrees, the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. This process is performed using server-side programming languages ​​such as Python and Node.js.

[0105] Step 7:

[0106] The IoT device receives the control signal and changes its settings.

[0107] Input: Control signal sent from the server.

[0108] Output: The settings of the IoT device are changed.

[0109] Specific operation: The IoT device receives a control signal sent from the server and changes the setting. In the case of an air conditioner, the internal control system changes the temperature setting to 25 degrees. Protocols such as MQTT and HTTP are used for this communication.

[0110] (Application example 1)

[0111] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] Conventional IoT home appliances are typically operated via smartphones or remote controls, which makes intuitive operation difficult. Furthermore, there are limited ways to efficiently obtain product and promotion information in physical stores, creating a need to improve the user shopping experience.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0114] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliances using generative AI, a means for the user to intuitively and freely operate the IoT home appliances using MR-GUI, and a means for visually displaying product information and promotion information in a physical store using smart glasses and providing an interface that the user can operate directly. This not only enables the user to intuitively operate the IoT home appliances, but also improves the shopping experience in the physical store.

[0115] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0116] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[0117] "Generative AI" is an artificial intelligence technology that generates appropriate responses and actions based on user input and environmental data.

[0118] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[0119] "Smart glasses" are wearable devices equipped with a display, camera, and sensors that allow users to experience visual information as augmented reality.

[0120] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[0121] "Product information" refers to information necessary for users to make purchasing decisions, such as product price, specifications, and stock status.

[0122] "Promotion information" refers to information aimed at promoting product sales, such as discounts and campaigns.

[0123] An "interface" is the means or method by which a user interacts with a system or device.

[0124] A system for implementing the present invention is configured as follows.

[0125] First, the server uses mixed reality (MR) technology to enable direct communication with IoT appliances. When a user puts on smart glasses and visually recognizes an IoT appliance, MR technology displays an operation interface on the appliance. For example, in the case of an air conditioner, the temperature control slider and airflow control buttons are displayed on the air conditioner body using MR, allowing the user to change the air conditioner settings by directly operating them.

[0126] Next, generative AI is used to enable conversations between users and IoT appliances. When a user gives a command via voice, the generative AI analyzes the command and executes the appropriate operation. For example, if the command is "Set the air conditioner temperature to 25 degrees," the generative AI will understand the command and set the temperature to 25 degrees.

[0127] Furthermore, MR-GUI allows users to intuitively and freely operate IoT home appliances. MR-GUI provides a visual interface that allows users to easily change various settings of home appliances.

[0128] The smart glasses also visually display product and promotion information from physical stores, providing an interface that users can operate directly. When a user looks at a specific product in a physical store, the camera in the smart glasses recognizes it and displays product and promotion information. Users can tap on this interface to check detailed information or complete the purchase process.

[0129] The following hardware and software are used to realize this system. The hardware includes smart glasses (e.g., Microsoft® HoloLens®) and IoT home appliances. The software includes OpenCV (image processing), TENSORFLOW® / Keras (AI model for product recognition), and ARToolkit (MR interface generation).

[0130] As a concrete example, imagine a user wearing smart glasses in a brick-and-mortar store. When the user looks at a particular product (e.g., a refrigerator), the smart glasses' camera recognizes the product and displays an MR interface. The interface displays the product's price, availability, and promotional information, and the user can tap to view more information or check out.

[0131] An example prompt is, "When you wear the smart glasses and look at a product in a physical store, detailed information and promotional information about the product will be displayed as an MR interface. Users can tap on this interface to view more information about the product or to complete the purchase."

[0132] In this way, users can not only intuitively operate IoT home appliances, but also improve their shopping experience in physical stores.

[0133] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0134] Step 1:

[0135] A user wears smart glasses and walks around a physical store. The camera in the smart glasses captures the surroundings in real time. The input is the camera image, and the output is the captured video data.

[0136] Step 2:

[0137] The device (smart glasses) inputs the captured video data into an AI model for product recognition. The AI ​​model uses TensorFlow / Keras to recognize products from the video data. The input is the video data, and the output is the ID of the recognized product.

[0138] Step 3:

[0139] The server receives the ID of the recognized product and retrieves the corresponding product information and promotion information from the database. The input is the product ID, and the output is the product information and promotion information.

[0140] Step 4:

[0141] Based on the product information and promotion information acquired by the server, an MR interface is generated using ARToolkit. The input is product information and promotion information, and the output is the MR interface.

[0142] Step 5:

[0143] The device (smart glasses) displays the generated MR interface in the user's field of view. The input is the MR interface, and the output is the interface displayed in the user's field of view.

[0144] Step 6:

[0145] The user operates the MR interface to check detailed information and carry out purchase procedures. The input is the user's operation, and the output is the operation result.

[0146] Step 7:

[0147] The terminal (smart glasses) sends the results of the user's operations to the server, which then executes the necessary processing. For example, in the case of a purchase, the server works with an electronic payment system to complete the payment. The input is the result of the user's operations, and the output is the processing result.

[0148] Step 8:

[0149] The server sends the processing results to the device (smart glasses) and notifies the user. The input is the processing results, and the output is the notification to the user.

[0150] In this way, users can intuitively obtain and manipulate product information and promotional information in physical stores through smart glasses.

[0151] Example 2

[0152] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] Conventional methods of operating IoT home appliances require users to use dedicated remote controls or smartphone apps, which often require unintuitive operation. While systems using voice recognition technology exist, there are issues with the accuracy of interpreting voice commands and the speed of operation. This makes it difficult for users to comfortably operate IoT home appliances.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0155] In this invention, the server includes a means for a user to issue a voice instruction, a means for a terminal to capture the voice and send it to the server, a means for the server to convert the voice data into text data, a means for the server to analyze the text data using a generative AI model and generate an operation command, a means for the server to send the operation command to the IoT home appliance, and a means for the IoT home appliance to operate in accordance with the operation command. This enables a user to intuitively and quickly operate the IoT home appliance through natural conversation.

[0156] "User" refers to a person who uses the system to operate IoT home appliances.

[0157] "Device" refers to a device that captures a user's voice and transmits it to a server, such as a smart speaker or smartphone.

[0158] "Server" refers to a computer system that converts voice data into text data, analyzes it using a generative AI model, generates operation commands, and sends them to IoT home appliances.

[0159] "Voice data" refers to data that digitally represents the voice uttered by the user to the terminal.

[0160] "Text data" refers to character string data obtained by analyzing voice data.

[0161] A "generative AI model" refers to an artificial intelligence model that analyzes text data and generates appropriate operation commands.

[0162] An "operation command" refers to an instruction to instruct an IoT home appliance to perform a specific action.

[0163] "IoT home appliances" refer to home appliances that are connected to the Internet and can be controlled remotely. Examples include air conditioners, lighting, and smart locks.

[0164] "Behavior" refers to the specific action that an IoT home appliance performs in accordance with an operation command.

[0165] This invention is a system that enables users to operate IoT home appliances through natural conversation. This system consists of a user, a terminal, a server, and IoT home appliances.

[0166] First, the user issues a voice command, for example, "heat the room." The device (e.g., a smart speaker or smartphone) then captures the user's voice and sends the voice data to a server. The server then uses voice recognition software (e.g., Google® Speech-to-Text API) to convert the received voice data into text data.

[0167] The server then analyzes the text data using a generative AI model (e.g., GPT-4 (registered trademark)) and generates an appropriate operation command. For example, from the text data "warm the room," it generates the operation command "set the air conditioner to heating mode." The server uses an IoT platform (e.g., AWS (registered trademark) IoT Core) to send the generated operation command to the IoT home appliance.

[0168] Finally, the IoT appliance (e.g., air conditioner) operates according to the received operation command. Specifically, the air conditioner starts operating in heating mode, raising the temperature in the living room.

[0169] As a concrete example, consider a scenario in which a user says, "Heat the room." In this case, the device captures the voice and sends it to the server. The server converts the voice into text, analyzes the instruction "Heat the room" using a generative AI model, and generates an operation command such as "Set the air conditioner to heating mode." The server then sends the operation command to the IoT appliance, and the air conditioner starts operating in heating mode.

[0170] An example of a prompt to input to a generative AI model is as follows:

[0171] When the user says "heat the room," generate an operation command to set the air conditioner to heating mode.

[0172] In this way, users can intuitively and quickly operate IoT home appliances through natural conversation.

[0173] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0174] Step 1:

[0175] The user issues a voice command.

[0176] A specific example of this behavior is when the user is in the living room and says, "Warm up the room."

[0177] Input: User's voice commands

[0178] Output: Audio data

[0179] Step 2:

[0180] The device captures the audio and sends it to the server.

[0181] Specifically, the smart speaker captures the user's voice and sends the voice data to a server via the Internet.

[0182] Input: Audio data

[0183] Output: Audio data sent to the server

[0184] Step 3:

[0185] The server converts the voice data into text data.

[0186] Specifically, the server uses voice recognition software (for example, Google Speech-to-Text API) to convert the received voice data into text data such as "Warm up the room."

[0187] Input: Audio data sent to the server

[0188] Output: Text data

[0189] Step 4:

[0190] The server analyzes the text data using a generative AI model and generates operation commands.

[0191] Specifically, the server uses a generative AI model (e.g., GPT-4) to analyze the text data "heat the room" and generate an operation command such as "set the air conditioner to heating mode."

[0192] Input: Text data

[0193] Output: Operation command

[0194] Step 5:

[0195] The server sends operation commands to the IoT appliances.

[0196] Specifically, the server uses an IoT platform (e.g., AWS IoT Core) to send the generated operation command to the air conditioner.

[0197] Input: Operation command

[0198] Output: Operation commands sent to IoT appliances

[0199] Step 6:

[0200] IoT home appliances operate according to operational commands.

[0201] Specifically, the air conditioner will start operating in heating mode in response to the received command, raising the temperature in the living room and providing a comfortable environment for the user.

[0202] Input: Operation command sent to IoT appliances

[0203] Output: The air conditioner starts operating in heating mode

[0204] (Application example 2)

[0205] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] In conventional brick-and-mortar stores, it was difficult for customers to intuitively and freely operate IoT devices in the store. Furthermore, customers could not directly interact with IoT devices, making operation cumbersome. Furthermore, there was a lack of a system for understanding and appropriately executing instructions for operating IoT devices, creating a need for an improved customer experience.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0208] In this invention, the server includes means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, means for enabling conversation between the user and the IoT device using generative AI, means for the user to intuitively and freely operate the IoT device using MR-GUI, means for enabling customers in a physical store to talk to the IoT device using a smartphone, and means for the generative AI to understand the customer's instructions and transmit appropriate operations to the IoT device. This enables customers to intuitively and freely operate the IoT device in a physical store and talk to it directly through the generative AI.

[0209] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0210] "IoT devices" are electronic devices that are interconnected via the Internet and are capable of sending, receiving, and controlling data.

[0211] "Generative AI" is an artificial intelligence technology that generates natural language based on user input and provides appropriate responses and instructions.

[0212] "MR-GUI" is a graphical user interface that uses MR technology and allows users to operate it intuitively.

[0213] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[0214] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use applications.

[0215] A "customer" is a consumer who visits a physical store to purchase a product or service.

[0216] "Communication" is the act of exchanging information and communicating intentions.

[0217] "Operation" is the act of controlling a device or system to perform a desired action.

[0218] "Instruction" refers to an order to perform a specific action or operation.

[0219] To implement the present invention, the following system configuration and program are required.

[0220] System Configuration

[0221] 1. Hardware:

[0222] Smartphone: A device that customers use in a physical store and that is capable of voice input and internet connectivity.

[0223] IoT devices: Electronic devices that can be connected to the internet, such as digital signage, smart lighting, and sound systems, installed in physical stores.

[0224] Server: Hosts the generative AI model and manages communication between smartphones and IoT devices.

[0225] 2. Software:

[0226] Speech Recognition Library: Use the speech_recognition library to capture audio from your smartphone's microphone and convert it into text.

[0227] Generative AI models: Generative AI models, such as OpenAI's GPT-3, are used to understand customer instructions and generate appropriate responses.

[0228] HTTP Request Library: Use the requests library to send the generated instructions to the IoT device.

[0229] Program processing explanation

[0230] 1. Voice input and recognition:

[0231] When customers speak into their smartphone's microphone, a speech recognition library converts the speech into text.

[0232] For example, if a customer says, "Turn up the lights in this area," the speech is converted into text.

[0233] 2. Use of generative AI models:

[0234] The converted text is sent as a prompt to a generative AI model, which generates appropriate operating instructions for the IoT device.

[0235] Example prompt: "User instructions: Increase the lighting in this area and suggest appropriate IoT device actions."

[0236] 3. Sending instructions to IoT devices:

[0237] The generated instructions are sent to the IoT device as an HTTP request.

[0238] For example, a generative AI model generates an instruction to "brighten the lights" and sends it to a smart light.

[0239] Specific examples

[0240] When a customer uses their smartphone in a brick-and-mortar store to say, "Turn up the lights in this area," a speech recognition library converts that speech into text. A generative AI model then understands the text and generates the command "Turn up the lights." Finally, that command is sent as an HTTP request to the smart lighting, which then turns on the lights.

[0241] In this way, customers can intuitively and freely operate IoT devices in physical stores and can speak directly to them through generative AI.

[0242] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0243] Step 1:

[0244] The user speaks into the smartphone's microphone.

[0245] Input: User's voice

[0246] Output: Audio data

[0247] Specific action: The user says, "Turn up the lights in this area."

[0248] Step 2:

[0249] The device uses a speech recognition library to convert the voice data into text.

[0250] Input: Audio data

[0251] Output: Text data

[0252] Specific operation: The speech recognition library (speech_recognition) analyzes the voice data and converts it into text such as "Turn on the lights in this area."

[0253] Step 3:

[0254] The device sends text data as prompts to the generative AI model, which generates appropriate operating instructions for the IoT device.

[0255] Input: Text data

[0256] Output: Generated instructions

[0257] Specific operation: A generative AI model (OpenAI's GPT-3) receives text data and generates the instruction "Turn the lights on."

[0258] Example prompt: "User instructions: Increase the lighting in this area and suggest appropriate IoT device actions."

[0259] Step 4:

[0260] The server sends the generated instructions to the IoT device as an HTTP request.

[0261] Input: Generated instructions

[0262] Output: HTTP request

[0263] What happens: The server formats the generated instructions as an HTTP request and sends it to the smart lighting.

[0264] Step 5:

[0265] The IoT device performs an action based on the received instructions.

[0266] Input: HTTP request

[0267] Output: IoT device operation

[0268] Specific behavior: The smart lighting receives the HTTP request and performs the action of brightening the light.

[0269] In this way, users can intuitively and freely operate IoT devices using their smartphones in physical stores, and can speak directly to them through generative AI, improving the customer experience.

[0270] Example 3

[0271] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] Conventional methods for operating IoT devices often had complex and unintuitive user interfaces, making it difficult for users to operate them. Furthermore, interaction with and operation of IoT devices was limited, making it difficult to accurately reflect the user's intentions. Furthermore, the lack of an operating environment that combined the real world and the virtual world did not improve the user experience.

[0273] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for enabling direct communication with the IoT device using MR technology, means for enabling interaction between the user and the IoT device using a generation AI, means for the user to intuitively and freely operate the IoT device using an MR-GUI, and means for the user to wear the MR device, the server to generate the MR-GUI, the terminal to display the GUI, the user to operate the GUI, the terminal to send operation information to the server, and the server to control the IoT device. This allows the user to intuitively and freely operate the IoT device, significantly improving the user experience.

[0274] "MR technology" is a technology that provides visual information that combines the real world and the virtual world.

[0275] An "IoT device" is an electronic device that communicates via the Internet and can be remotely controlled and collect data.

[0276] "Generative AI" is an artificial intelligence technology that interacts with users and generates data.

[0277] "MR-GUI" is a graphical user interface displayed using MR technology.

[0278] A "server" is a computer system that processes data and provides services over a network.

[0279] A "terminal" is a device that a user can directly operate, including MR devices.

[0280] A "user" is a person who operates the system.

[0281] "Operation information" refers to the data and instructions entered by the user through the terminal.

[0282] "Communication" is the process of sending and receiving data or information.

[0283] "Dialogue" is the process by which a user and a system exchange information with each other.

[0284] "Control" is the process by which a system manages and coordinates the behavior of IoT devices.

[0285] This invention is a system that allows users to intuitively and freely operate IoT devices using mixed reality (MR) technology. Specifically, the user operates the IoT device using an MR-GUI displayed by MR technology. Detailed embodiments of this system are described below.

[0286] Hardware and software used

[0287] 1. MR device:

[0288] Example: Microsoft HoloLens

[0289] Function: Provides visual information that blends the real and virtual worlds.

[0290] 2. Server:

[0291] Function: Generates MR-GUI, processes operation information, and controls IoT devices.

[0292] 3. IoT devices:

[0293] Examples: air conditioners, lighting, smart locks, etc.

[0294] Function: Communicates via the Internet, allowing remote control and data collection.

[0295] 4. Generation AI:

[0296] Function: Interacts with the user and generates data.

[0297] System Operation

[0298] When a user puts on an MR device, the server acquires information about the user's environment and generates an MR-GUI based on that information. The MR device displays the GUI received from the server in the user's field of view. The user intuitively operates the GUI displayed through the MR device. For example, the user can change the air conditioner temperature by moving the temperature control slider, or adjust the air volume by pressing the air volume control button. The MR device transmits the user's operation information to the server in real time. The server controls the IoT devices based on the received operation information.

[0299] Specific examples

[0300] For example, if a user wants to set the air conditioner temperature to 25 degrees, they would follow these steps:

[0301] 1. The user wears the MR device.

[0302] 2. The server generates an MR-GUI that includes the air conditioner control panel.

[0303] 3. The MR device displays the operation panel in the user's field of view.

[0304] 4. The user sets the temperature slider to 25 degrees.

[0305] 5. The MR device sends this operation information to the server.

[0306] 6. The server sends a command to the air conditioner to set the temperature to 25 degrees.

[0307] Prompt Sentence Examples

[0308] A system description can be generated by inputting a prompt like the one below into the generative AI model.

[0309] "Please explain the system process in which users operate IoT devices using mixed reality technology. Please clearly state what kind of data processing and calculations are performed, including the names of specific hardware and software. Also, please provide specific examples."

[0310] By using this prompt sentence, the generative AI model can generate a detailed explanation of the system. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0311] Step 1:

[0312] The user puts on the MR device.

[0313] Input: The user puts on a mixed reality device (e.g., Microsoft HoloLens).

[0314] Data processing: The MR device acquires the user's location information and surrounding environmental data.

[0315] Output: The MR device sends the user's location and environmental data to the server.

[0316] Specific Action: The user puts the HoloLens on their head and turns on the device.

[0317] Step 2:

[0318] The server generates the MR-GUI.

[0319] Input: User location and environmental data sent from the MR device.

[0320] Data processing: The server analyzes the data received and generates an MR-GUI that is suitable for the user's environment.

[0321] Output: Send the generated MR-GUI data to the MR device.

[0322] Specific operation: The server generates an air conditioner control panel based on the user's location information and surrounding environmental data.

[0323] Step 3:

[0324] The terminal displays the GUI.

[0325] Input: MR-GUI data sent from the server.

[0326] Data processing: The MR device renders the received GUI data to display in the user's field of view.

[0327] Output: The MR-GUI displayed in the user's field of view.

[0328] Specific operation: HoloLens displays the air conditioner control panel overlaid on the user's field of vision.

[0329] Step 4:

[0330] The user operates the GUI.

[0331] Input: The user operates the MR-GUI displayed in the field of view.

[0332] Data processing: The MR device acquires the user's operation information and analyzes the operation content.

[0333] Output: Sends the parsed operation information to the server.

[0334] What happens: The user moves their hand to set a slider displayed in the HoloLens' field of view to 25 degrees.

[0335] Step 5:

[0336] The terminal transmits operation information to the server.

[0337] Input: User operation information.

[0338] Data processing: The MR device packets the operation information and sends it to the server via the network.

[0339] Output: Operation information sent to the server.

[0340] Specific operation: HoloLens sends user operation information (e.g., changing the temperature setting to 25 degrees) to the server.

[0341] Step 6:

[0342] The server controls the IoT devices.

[0343] Input: Operation information sent from the MR device.

[0344] Data processing: The server analyzes the operation information and generates control commands for the corresponding IoT devices.

[0345] Output: Control commands sent to IoT devices.

[0346] Specific operation: The server sends a command to the air conditioner to "set the temperature to 25 degrees," and the air conditioner adjusts the temperature according to that command.

[0347] (Application example 3)

[0348] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] Conventional IoT home appliances are generally operated using remote controls or smartphone apps, which makes intuitive operation difficult. Furthermore, in brick-and-mortar stores, there is a lack of means to efficiently manage and operate multiple IoT devices, which makes it difficult for store staff to respond quickly. To solve these issues, a more intuitive and efficient operation method is needed.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0351] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between a user and IoT home appliances using generative AI, a means for a user to intuitively and freely operate IoT home appliances using MR-GUI, a means for intuitively operating IoT devices in a physical store using smart glasses, and a means for detecting hand movements to operate the MR-GUI, thereby enabling a user to intuitively operate IoT home appliances and IoT devices in a physical store.

[0352] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0353] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[0354] "Generative AI" is an artificial intelligence technology that generates natural conversations and responses based on user input.

[0355] "MR-GUI" is a graphical user interface displayed using Mixed Reality technology, and is an interface that users can operate intuitively.

[0356] "Smart glasses" are wearable devices that have built-in displays, cameras, and sensors, allowing users to visually check information.

[0357] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[0358] An "IoT device" is an electronic device that is connected to the Internet and is capable of data communication and remote control.

[0359] "Means for detecting hand movements" refers to technology that uses cameras and sensors to recognize the position and movement of the user's hands and perform operations based on that.

[0360] This invention relates to a system that uses mixed reality (MR) technology to intuitively operate IoT home appliances and IoT devices in brick-and-mortar stores. Specific embodiments for realizing this system are described below.

[0361] System Configuration

[0362] The system consists of the following major components:

[0363] 1. Server:

[0364] Generative AI: Generates natural-sounding conversations and responses based on user input.

[0365] Database: Stores configuration information for IoT home appliances and IoT devices, as well as user operation history.

[0366] 2. Terminal:

[0367] Smart glasses: Wearable devices with built-in displays, cameras, and sensors that allow users to visually view information.

[0368] Camera: Used to detect the user's hand movements.

[0369] MR-GUI: A graphical user interface displayed using Mixed Reality technology.

[0370] Program processing

[0371] The server uses generative AI to enable conversations between users and IoT appliances and devices. The user wears smart glasses and operates the MR-GUI through a camera that detects hand movements. Specifically, the following process is performed:

[0372] 1. Hand movement detection:

[0373] The camera recognizes the position and movement of the user's hands and detects hand landmarks.

[0374] The sliders and buttons of the MR-GUI are operated based on the detected hand movements.

[0375] 2. Display MR-GUI:

[0376] The smart glasses display shows the operation panels of IoT home appliances and devices.

[0377] For example, the air conditioner control panel displays a temperature control slider, a button to adjust the airflow, and a button to switch between operating modes.

[0378] 3. Generative AI Conversation:

[0379] When a user gives a command via voice, the generative AI analyzes the command and generates an appropriate response.

[0380] For example, in response to an instruction such as "Set the air conditioner temperature to 24 degrees," the generative AI will change the air conditioner temperature.

[0381] Specific examples

[0382] Consider a scenario in which a store clerk wears smart glasses and adjusts the temperature of the air conditioning in a physical store. The clerk moves his or her hand to operate the slider on the MR-GUI, changing the temperature from 22 degrees to 24 degrees. In this case, the generative AI analyzes the clerk's voice instructions and generates an appropriate response.

[0383] Prompt Sentence Examples

[0384] "Put on the smart glasses and use your hand to operate the slider on the MR-GUI to change the temperature of the store's air conditioning from 22 degrees to 24 degrees."

[0385] In this way, the present invention enables users to intuitively operate IoT home appliances and IoT devices in physical stores.

[0386] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0387] Step 1:

[0388] The user puts on the smart glasses and starts the system.

[0389] Input: Smart glasses activation signal

[0390] Output: MR-GUI is displayed on the smart glasses display.

[0391] Specific operation: When the user turns on the smart glasses, the system starts up and the MR-GUI is displayed on the display, allowing the user to visually confirm the operation panel.

[0392] Step 2:

[0393] The device's camera detects the user's hand movements.

[0394] Input: Camera image

[0395] Output: Hand landmark data

[0396] How it works: The camera captures the position and movement of the user's hands in real time and detects hand landmarks, which allows the system to recognize the user's hand movements.

[0397] Step 3:

[0398] The device analyzes the hand landmark data and operates the MR-GUI.

[0399] Input: Hand landmark data

[0400] Output: MR-GUI operation instructions

[0401] Specific operation: Sliders and buttons in the MR-GUI are operated based on the hand landmark data. For example, when a user moves their hand to operate a slider, the slider's position changes accordingly.

[0402] Step 4:

[0403] The user issues a command by voice.

[0404] Input: Voice command

[0405] Output: Audio data

[0406] How it works: When a user gives a voice command such as "Set the air conditioner temperature to 24 degrees," the voice is captured by the microphone in the smart glasses.

[0407] Step 5:

[0408] The server uses generative AI to analyze voice commands and generate an appropriate response.

[0409] Input: Audio data

[0410] Output: Operation instruction data

[0411] Specific operation: The server analyzes the voice data and understands the user's instructions. The generative AI generates an appropriate response and generates operation instruction data to set the air conditioner temperature to 24 degrees.

[0412] Step 6:

[0413] The terminal operates IoT home appliances based on the operation instruction data.

[0414] Input: Operation instruction data

[0415] Output: Change the settings of IoT appliances

[0416] Specific operation: Based on the operation instruction data, the air conditioner temperature is set to 24 degrees. This executes the user's instruction.

[0417] Step 7:

[0418] The server provides feedback on the operation results to the user.

[0419] Input: IoT appliance settings change results

[0420] Output: Feedback message

[0421] Specific operation: The server confirms that the air conditioner temperature is set to 24 degrees, generates a feedback message for the user such as "The air conditioner temperature has been set to 24 degrees," and displays it on the smart glasses display.

[0422] In this way, users can intuitively operate IoT home appliances and IoT devices in physical stores through smart glasses.

[0423] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0424] "Example 1"

[0425] One embodiment of the present invention is a system in which a user wears mixed reality (MR) goggles or a headset and communicates directly with IoT appliances. This system includes an emotion engine that recognizes the user's emotions. Specifically, when the user speaks to an IoT appliance, the system analyzes the user's emotions from the tone of voice and facial expressions, and reflects the results in the operation of the IoT appliance. For example, if the user expresses anger, the IoT appliance can quickly perform the appropriate operation, enabling it to operate in accordance with the user's emotions.

[0426] "Example 2"

[0427] Another example is a system that uses generative AI to enable conversations between users and IoT appliances. This system also includes an emotion engine that recognizes the user's emotions. Specifically, it analyzes the user's speech and reflects the results in the operation of the IoT appliance. For example, if the user expresses joy, the IoT appliance will operate in a joyful manner, enabling it to operate in accordance with the user's emotions.

[0428] "Example 3"

[0429] Another example is a system that uses MR-GUI to allow users to intuitively and freely operate IoT appliances. This system also includes an emotion engine that recognizes the user's emotions. Specifically, it analyzes the user's emotions from their operations and reflects the results in the operation of the IoT appliance. For example, if the user expresses surprise, the IoT appliance can operate in a gentle manner, allowing it to respond to the user's emotions.

[0430] The processing flow of each embodiment will be described below.

[0431] "Example 1"

[0432] Step 1: The user puts on MR (Mixed Reality) goggles or a headset.

[0433] Step 2: The user talks to the IoT appliance.

[0434] Step 3: The emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[0435] Step 4: Reflect the analysis results in the operation of IoT home appliances.

[0436] "Example 2"

[0437] Step 1: The user speaks to the IoT appliance via generative AI.

[0438] Step 2: The emotion engine analyzes the emotion from the user's speech.

[0439] Step 3: Reflect the analysis results in the operation of IoT home appliances.

[0440] "Example 3"

[0441] Step 1: The user operates the IoT appliance using the MR-GUI.

[0442] Step 2: The emotion engine analyzes the emotions from the user's operations.

[0443] Step 3: Reflect the analysis results in the operation of IoT home appliances.

[0444] Example 1

[0445] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0446] Conventional IoT devices require the use of dedicated remote controls or smartphone apps, making them difficult to operate intuitively. Furthermore, since operations cannot take into account the user's emotions, it is difficult to improve the user experience. Furthermore, communication with IoT devices is limited, and users cannot directly talk to them, limiting the degree of freedom in operation.

[0447] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0448] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, and a means for recognizing the user's emotions and optimizing the operation of the IoT device based on those emotions. This allows the user to intuitively and freely operate the IoT device, and further realizes optimal operation according to the user's emotions.

[0449] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0450] "IoT devices" are home appliances and devices that are connected to each other via the Internet and can send, receive, and control data.

[0451] "Generative AI" is a system that uses artificial intelligence technology to generate natural conversations and interactions with users.

[0452] "MR-GUI" is a graphical user interface that uses Mixed Reality technology, providing an interface that users can operate intuitively.

[0453] "Emotion recognition" is a technology that analyzes emotions from the user's tone of voice, facial expressions, etc. and obtains the results.

[0454] An "operation interface" is a visual or physical means by which a user operates an IoT device.

[0455] "Optimization" refers to adjusting something to the most effective state for specific conditions or purposes.

[0456] This invention is a system that enables direct communication with IoT devices using MR (Mixed Reality) technology. This system starts when a user wears MR-compatible goggles or a headset and visually recognizes the IoT device. Specifically, the user wears special goggles or a headset and visually recognizes the IoT device. At this time, a direct operation interface is displayed on the IoT device using MR technology.

[0457] Hardware and software used

[0458] Hardware: Mixed reality goggles or headset (e.g., Microsoft HoloLens)

[0459] Software: Emotion recognition engine (e.g., Affectiva SDK), IoT device control software (e.g., Home Assistant)

[0460] Data processing and calculation

[0461] 1. User visual recognition:

[0462] The user puts on MR-compatible goggles or a headset.

[0463] The cameras in the goggles or headset recognize surrounding IoT devices and use MR technology to display an operating interface on the device.

[0464] Example: The air conditioner itself displays a temperature control slider and airflow control buttons.

[0465] 2. Emotion recognition:

[0466] The user speaks to the IoT device.

[0467] The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize their emotions.

[0468] Example: If a user expresses an angry emotion, an emotion recognition engine will analyze that information.

[0469] 3. Operation reflection:

[0470] IoT devices are operated based on the analysis results of the emotion recognition engine.

[0471] Example: If the user is showing signs of anger, quickly change the air conditioning settings.

[0472] Specific examples

[0473] Example 1: Air conditioner operation

[0474] Scenario: A user wants to lower the temperature in their room.

[0475] Operation steps:

[0476] 1. The user puts on the MR compatible goggles.

[0477] 2. The goggles will recognize the air conditioner and a temperature control slider will appear on the air conditioner itself.

[0478] 3. The user operates the slider to lower the temperature.

[0479] 4. The device analyzes the user's facial expressions to confirm satisfaction.

[0480] 5. The server changes the air conditioning settings, lowering the room temperature.

[0481] Example 2: Lighting control

[0482] Scenario: A user wants to dim the lights in their room.

[0483] Operation steps:

[0484] 1. The user puts on the mixed reality headset.

[0485] 2. The headset will recognize the lighting and a button to adjust the lighting brightness will appear.

[0486] 3. The user presses a button to decrease the brightness.

[0487] 4. The device analyzes the user's tone of voice to ensure they are relaxed.

[0488] 5. The server changes the lighting settings and the room becomes less bright.

[0489] Prompt Sentence Examples

[0490] Prompt 1: "I want to turn down the air conditioner. If you show any signs of anger, I will change the setting immediately."

[0491] Prompt 2: "I want the lights in the room to be dimmed. If I'm relaxing, I slowly lower the brightness."

[0492] The above is an embodiment of the present invention.

[0493] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0494] Step 1:

[0495] The user puts on MR-compatible goggles or a headset.

[0496] Input: User puts on mixed reality goggles or headset.

[0497] Data processing: The goggles or headset activates and displays the MR environment in the user's field of vision.

[0498] Output: The mixed reality environment is displayed in the user's field of view.

[0499] Specific operation: When the user puts on the goggles and turns them on, the MR environment is displayed on the goggles' display.

[0500] Step 2:

[0501] The device recognizes surrounding IoT devices.

[0502] Input: Cameras in goggles or headset scan the surrounding environment.

[0503] Data processing: The device recognizes IoT devices (e.g., air conditioners, lighting) and obtains their location information.

[0504] Output: Location information of recognized IoT devices.

[0505] How it works: The goggles' camera scans the room and recognizes the air conditioner and lights.

[0506] Step 3:

[0507] The device displays the operating interface using MR technology.

[0508] Input: Location information of the recognized IoT device.

[0509] Data processing: Using MR technology, the device displays an operation interface on the IoT device it recognizes.

[0510] Output: Operation interface displayed on the IoT device.

[0511] Specific operation: The goggles' display will show an air conditioning temperature control slider and lighting brightness adjustment buttons.

[0512] Step 4:

[0513] The user operates the IoT device using the operation interface.

[0514] Input: The user operates a visually displayed control interface.

[0515] Data processing: IoT device settings are changed based on user actions.

[0516] Output: The modified IoT device configuration.

[0517] Specific action: The user moves their hand to operate a slider on the goggles' display to set the air conditioner temperature.

[0518] Step 5:

[0519] The device recognizes the user's emotions.

[0520] Input: The user's tone of voice and facial expressions.

[0521] Data processing: The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize their emotions.

[0522] Output: Perceived user emotion.

[0523] How it works: The goggles' camera captures the user's facial expressions, which are then analyzed by the emotion recognition engine.

[0524] Step 6:

[0525] The server optimizes the operation of IoT devices based on the emotion recognition results.

[0526] Input: Perceived user emotion.

[0527] Data processing: The server receives the emotion recognition results and determines the optimal operation based on the user's emotion.

[0528] Output: Optimized IoT device operation instructions.

[0529] Specific operation: The server issues instructions to immediately change the air conditioner settings based on the emotion recognition results.

[0530] (Application example 1)

[0531] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0532] Conventional IoT device operation methods require users to use smartphones or remote controls, which is often unintuitive. Furthermore, they lack the ability to take user emotions into account, resulting in a lack of user experience. Furthermore, even in physical stores, shopping requires manual searches to obtain detailed product information, which is inefficient.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0534] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, and a means for analyzing the user's emotions using an emotion engine and reflecting the results in the operation of the IoT device. This allows the user to operate the IoT device intuitively and according to their emotions, improving the shopping experience in physical stores.

[0535] "MR (Mixed Reality)" is a technology that combines the real world and the virtual world, allowing users to experience both elements simultaneously.

[0536] "IoT devices" are electronic devices that are interconnected via the Internet and are capable of sending, receiving, and controlling data.

[0537] "Generative AI" is an artificial intelligence technology that generates natural conversations with users.

[0538] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[0539] The "emotion engine" is a technology that analyzes emotions from the user's tone of voice and facial expressions, and reflects the results in the system's operation.

[0540] A system for implementing the present invention includes the following major components:

[0541] 1. Hardware Configuration

[0542] MR-compatible goggles and headsets: These are devices worn by users that allow them to blend the real world with the virtual world.

[0543] Camera: A device for capturing images for visual recognition by the user.

[0544] Server: A computer system that processes data, runs generative AI, and executes emotion engines.

[0545] 2. Software Configuration

[0546] MRDisplay Library: Software for displaying graphical user interfaces using MR technology.

[0547] EmotionEngine Library: Software for analyzing emotions from a user's tone of voice and facial expressions.

[0548] Generative AI: An artificial intelligence technology that generates natural conversations with users.

[0549] OpenCV: A library for image processing.

[0550] 3. System Operation

[0551] The server recognizes products and IoT devices using images captured by the camera. Detailed information about the recognized products and IoT devices is displayed on MR-compatible goggles or headsets using the MRDisplay library. When the user talks to the products or IoT devices, the generative AI analyzes the conversation and generates an appropriate response. The EmotionEngine library also analyzes the user's emotions and reflects the results in the system's behavior.

[0552] 4. Specific Examples

[0553] For example, a user puts on MR-compatible goggles in a physical store and takes a photo of a product with the camera. The server recognizes the product and uses MR technology to display detailed product information on the goggles' display. When the user talks about the product, the generative AI analyzes the conversation and generates an appropriate response. Furthermore, the Emotion Engine library analyzes the user's emotions and makes appropriate product suggestions based on the results.

[0554] 5. Examples of prompts

[0555] "The user puts on MR-compatible goggles and takes a photo of a product with the camera. The system recognizes the product and uses MR technology to display detailed product information on the goggles' display. When the user talks about the product, the emotion engine analyzes their emotions from their tone of voice and facial expressions and makes appropriate product suggestions."

[0556] In this way, users can operate IoT devices intuitively and emotionally, improving the shopping experience in physical stores.

[0557] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0558] Step 1:

[0559] The user puts on MR-compatible goggles and takes a photo of a product or IoT device with the camera.

[0560] Input: Image data captured by a camera

[0561] Output: Image data is sent to the server

[0562] Specific operation: The user puts on the goggles and takes pictures of products and IoT devices that come into view with the camera. The captured image data is sent to the server in real time.

[0563] Step 2:

[0564] The server receives the image data and recognizes the product or IoT device.

[0565] Input: Image data sent from the camera

[0566] Output: ID of recognized product or IoT device

[0567] Specific operation: The server processes images using the OpenCV library and recognizes products and IoT devices. An ID is generated for the recognized products and IoT devices.

[0568] Step 3:

[0569] The server obtains detailed information about recognized products and IoT devices and displays it on the goggles using MR technology.

[0570] Input: ID of recognized product or IoT device

[0571] Output: Detailed information displayed on MR-compatible goggles

[0572] Specific operation: The server retrieves detailed information about products and IoT devices from the database and displays it on the goggles' display using the MRDisplay library.

[0573] Step 4:

[0574] Users talk to products and IoT devices.

[0575] Input: User's voice data

[0576] Output: The audio data is sent to the server.

[0577] Specific operation: The user talks to the product or IoT device while wearing the goggles. The voice data is sent to the server in real time.

[0578] Step 5:

[0579] The server analyzes the voice data and uses generative AI to generate an appropriate response.

[0580] Input: User's voice data

[0581] Output: The generated response

[0582] How it works: The server uses generative AI to analyze the user's voice data and generate an appropriate response, which is then sent to the goggles.

[0583] Step 6:

[0584] The server uses an emotion engine to analyze the user's emotions and makes appropriate product suggestions based on the results.

[0585] Input: User's voice data and facial expression data

[0586] Output: Sentiment analysis results and product recommendations

[0587] Specific operation: The server uses the EmotionEngine library to analyze the user's emotions from their voice and facial expression data. Based on the analysis results, it generates appropriate product suggestions and displays them on the goggles.

[0588] Step 7:

[0589] The user reviews the suggested products and takes action as necessary.

[0590] Input: Details of the proposed product

[0591] Output: User operation instructions

[0592] Specific operation: The user checks the details of the suggested products displayed on the goggles and performs operations as necessary. Operation instructions are sent to the server in real time.

[0593] Example 2

[0594] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0595] Conventional IoT home appliance control systems have faced the challenge of making it difficult for users to operate home appliances in an intuitive and natural way. In particular, when operating using voice input, it is necessary to accurately understand and appropriately reflect the user's instructions. Furthermore, there is also the problem that the user experience is limited because the system is unable to take the user's emotions into account.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0597] In this invention, the server includes a means for a user to input voice, a means for a terminal to acquire voice data and send it to the server, a means for the server to convert the voice data to text, a means for the server to analyze instructions using a generative AI model, a means for the server to analyze emotions using an emotion engine, a means for the server to send operation commands to the IoT home appliance, and a means for the IoT home appliance to operate in accordance with the operation commands. This allows the user to operate the IoT home appliance in an intuitive and natural way, and also enables operation that takes the user's emotions into consideration.

[0598] "User" refers to a person who uses the system to operate IoT home appliances.

[0599] "Voice input" refers to the act of a user giving instructions to a system using voice.

[0600] "Terminal" refers to a device for acquiring and transmitting voice data to a server.

[0601] "Voice data" refers to a digital recording of a user's voice input.

[0602] A "server" refers to a computer system that processes voice data, analyzes instructions using generative AI models and emotion engines, and sends operational commands to IoT home appliances.

[0603] "Text data" refers to data obtained by converting voice data into character information.

[0604] A "generative AI model" refers to an artificial intelligence model that analyzes user instructions and generates appropriate operation commands.

[0605] An "emotion engine" refers to software that analyzes emotions from the content of a user's speech.

[0606] An "operation command" refers to an instruction to instruct an IoT home appliance to perform a specific action.

[0607] "IoT home appliances" refer to home appliances that are connected to the Internet and can be controlled remotely.

[0608] "Behavior" refers to the specific action that an IoT home appliance performs in accordance with an operation command.

[0609] This invention relates to a system for operating IoT home appliances by having a user input voice data, a terminal acquiring the voice data, and transmitting the data to a server. A specific embodiment of this system will be described below.

[0610] Hardware and software used

[0611] Hardware: IoT home appliances (e.g., air conditioners, lighting, smart speakers)

[0612] Software: Generative AI models (e.g., GPT-4), emotion engines (e.g., Affectiva), speech recognition software (e.g., Google Speech-to-Text)

[0613] Data processing and calculation

[0614] 1. Acquiring voice input:

[0615] The user speaks commands into the smart speaker, such as "heat the room" or "brighten the room."

[0616] Specific action: A user sits in the living room and asks their smart speaker to "heat the room."

[0617] 2. Acquiring and sending audio data:

[0618] The device (smart speaker) captures the user's voice with a microphone and saves it as voice data.

[0619] The device sends the audio data to a server via the Internet.

[0620] How it works: The smart speaker records the user's voice and sends the data to a cloud server via Wi-Fi.

[0621] 3. Audio to text conversion:

[0622] The server uses speech recognition software (e.g., Google Speech-to-Text) to convert the voice data into text data.

[0623] What happens: The server receives the voice data and calls speech recognition software to generate the text "heat the room."

[0624] 4. Generative AI instruction analysis:

[0625] The server uses a generative AI model (e.g., GPT-4) to parse user instructions from text data.

[0626] The generative AI model analyzes the instruction "heat the room" and generates an operation command to operate the air conditioner in heating mode.

[0627] Specific operation: The server inputs the text "heat the room" into the generative AI model, which generates a command to instruct the air conditioner to operate in heating mode.

[0628] 5. Emotion analysis using emotion engine:

[0629] The server uses an emotion engine (e.g., Affectiva) to analyze the emotion from the user's speech.

[0630] The emotion engine detects emotions such as joy or anger from the user's tone of voice and language.

[0631] Specific operation: The server inputs the text "warm the room" into the emotion engine and analyzes that the user is relaxed.

[0632] 6. Sending operation commands to IoT appliances:

[0633] The server generates operation commands and sends them to the IoT appliances.

[0634] Specific operation: The server sends a command to the air conditioner to "operate in heating mode."

[0635] 7. IoT appliances in action:

[0636] The IoT appliance (e.g., air conditioner) begins to operate according to the received operation command.

[0637] Specific operation: The air conditioner starts operating in heating mode and warms the room.

[0638] Specific examples

[0639] Example 1: When the user says "Heat the room"

[0640] A user sits in the living room and asks a smart speaker to "heat the room."

[0641] The smart speaker records the audio and sends it to a server.

[0642] The server converts the voice data into text and analyzes it using a generative AI model.

[0643] The server sends a command to the air conditioner to "operate in heating mode."

[0644] The air conditioner will start operating in heating mode and warm the room.

[0645] Example 2: User expresses happiness and says "Turn on the lights in the room"

[0646] A user sits in the living room and asks a smart speaker to "brighten the room."

[0647] The smart speaker records the audio and sends it to a server.

[0648] The server converts the voice data into text and analyzes it using a generative AI model.

[0649] The server uses an emotion engine to analyze the user's emotions of joy.

[0650] The server sends a command to the light to "turn it on" and adds emotional information.

[0651] The lights come on brightly, lighting up the room.

[0652] Prompt Sentence Examples

[0653] "When the user says 'heat the room,' generate an operation command to operate the air conditioner in heating mode."

[0654] "If the user expresses happiness and says 'Turn on the lights in the room,' generate an operation command to turn on the lights and attach the emotional information."

[0655] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0656] Step 1:

[0657] The user provides voice input.

[0658] Specific action: The user speaks commands into the smart speaker, such as "heat the room" or "light up the room."

[0659] Input: User's voice command.

[0660] Output: Audio data.

[0661] Step 2:

[0662] The terminal acquires the voice data and transmits it to the server.

[0663] How it works: The smart speaker captures the user's voice with a microphone and saves it as audio data. The device then sends the audio data to a server via the internet.

[0664] Input: Audio data.

[0665] Output: The audio data sent to the server.

[0666] Step 3:

[0667] The server converts the audio data into text.

[0668] What happens: The server uses speech recognition software (e.g., Google Speech-to-Text) to convert the voice data into text data.

[0669] Input: The audio data sent to the server.

[0670] Output: Text data.

[0671] Step 4:

[0672] The server analyzes the instructions using a generative AI model.

[0673] How it works: The server uses a generative AI model (e.g., GPT-4) to analyze the user's instructions from the text data. The generative AI model analyzes the instruction "heat the room" and generates an operation command to operate the air conditioner in heating mode.

[0674] Input: Text data.

[0675] Output: Operation command.

[0676] Step 5:

[0677] The server analyzes emotions using an emotion engine.

[0678] How it works: The server uses an emotion engine (e.g., Affectiva) to analyze the emotions from the user's speech. The emotion engine detects emotions such as joy or anger from the user's tone of voice and choice of words.

[0679] Input: Text data.

[0680] Output: Emotion analysis results.

[0681] Step 6:

[0682] The server sends operation commands to the IoT appliances.

[0683] Specific operation: The server generates an operation command and sends it to the IoT appliance.

[0684] Input: Operation command, emotion analysis result.

[0685] Output: Operation commands sent to IoT appliances.

[0686] Step 7:

[0687] IoT home appliances operate according to operational commands.

[0688] Specific operation: The IoT appliance (e.g., air conditioner) begins to operate according to the received operation command.

[0689] Input: Operation command sent to IoT appliances.

[0690] Output: IoT appliance operation.

[0691] (Application example 2)

[0692] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0693] Conventional IoT device interfaces have been difficult for users to operate intuitively and have had issues with not being able to provide appropriate information based on emotions. Furthermore, there has been a lack of interfaces that allow users to directly talk to IoT devices, limiting the user experience.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0695] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, a means for recognizing the user's emotions and providing information according to those emotions, and a means for displaying information to the user using smart glasses. This allows the user to not only intuitively operate the IoT device but also receive appropriate information according to their emotions.

[0696] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0697] An "IoT device" is an electronic device that can communicate with other devices and systems via the Internet and send and receive data and perform other control functions.

[0698] "Generative AI" is an artificial intelligence technology that generates responses in natural language based on user input.

[0699] "MR-GUI" is a graphical user interface that uses Mixed Reality technology and allows users to operate it intuitively.

[0700] "Emotion recognition" is a technology that analyzes and recognizes emotions from a user's speech, facial expressions, behavior, etc.

[0701] "Smart glasses" are wearable devices that can visually display information to users when worn.

[0702] As an embodiment of the present invention, a smart shopping assistant system will be described as an example. This system is configured using MR (Mixed Reality) technology, generative AI, MR-GUI, emotion recognition, and smart glasses.

[0703] System configuration

[0704] 1. Hardware

[0705] Smart glasses: Wearable devices worn by a user that visually display information. An example is Google Glass®.

[0706] Server: A computer system that runs generative AI models and emotion recognition engines.

[0707] 2. Software

[0708] Generative AI model: An artificial intelligence technology that generates responses in natural language based on user input. For example, OpenAI's API is used.

[0709] Emotion recognition engine: A technology that analyzes and recognizes emotions from a user's speech, facial expressions, actions, etc. An example is the Emotion Recognition API.

[0710] IoT Device Control API: Software for controlling IoT devices.

[0711] System Operation

[0712] 1. Processing User Input

[0713] When a user speaks into the smart glasses, the voice input is converted into text.

[0714] The server uses an emotion recognition engine to analyze emotions from the user's speech.

[0715] 2. Response generation using generative AI

[0716] The server uses a generative AI model to generate appropriate responses based on user input and emotions.

[0717] The generated response is displayed on the smart glasses.

[0718] 3. Controlling IoT devices

[0719] Based on the generated response, the server operates the IoT device as needed using the IoT device control API.

[0720] Specific examples

[0721] When a user says, "Tell me about this product," the smart glasses will display, "This product is made from high-quality materials."

[0722] When a user says, "I feel tired today," the smart glasses will display, "We have aroma candles and massage services to help you relax."

[0723] Prompt Sentence Examples

[0724] User Input: Tell us about this product

[0725] Emotion: Interest

[0726] Generate an appropriate response.

[0727] In this way, the smart shopping assistant system can provide appropriate information according to the user's questions and emotions, improving the shopping experience in physical stores.

[0728] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0729] Step 1:

[0730] When a user speaks into the smart glasses, the voice input is converted into text. The smart glasses use voice recognition software to convert the voice data into text, which then becomes the input for the next processing step.

[0731] Step 2:

[0732] The server receives the text data and uses an emotion recognition engine to analyze the user's emotions from the user's speech. The emotion recognition engine analyzes the text data and identifies the user's emotions (e.g., interest, fatigue, joy, etc.). This emotion data becomes the input for the next processing step.

[0733] Step 3:

[0734] The server uses a generative AI model to generate an appropriate response based on the user's input text and emotion data. The generative AI model creates a prompt and generates a response in natural language based on the user's input and emotion. This generated response serves as input for the next processing step.

[0735] Step 4:

[0736] The server transmits the generated response to the smart glasses, which visually display the response to the user, and the smart glasses display the received text data on a display to provide information to the user.

[0737] Step 5:

[0738] The server uses the IoT device control API to operate the IoT device as needed based on the generated response. For example, if the user says, "Heat the room," the server instructs the air conditioner to operate in heating mode. This instruction is sent to the IoT device, and the IoT device operates appropriately.

[0739] In this way, the smart shopping assistant system can provide appropriate information and operate IoT devices according to the user's questions and emotions.

[0740] Example 3

[0741] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0742] Conventional IoT home appliance operation systems have the problem that they are difficult for users to operate intuitively and cannot respond to the user's emotions. Furthermore, there is a demand for systems that allow users to speak directly to IoT home appliances and reflect emotion-based operations. This will improve the user experience and provide a more comfortable living environment.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0744] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliances using generative AI, a means for the user to intuitively and freely operate the IoT home appliances using MR-GUI, and a means for analyzing emotions from the user's operating actions and reflecting the results in the operation of the IoT home appliances. This allows the user to not only intuitively operate the IoT home appliances but also to operate them according to their emotions, providing a more comfortable living environment.

[0745] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both elements simultaneously.

[0746] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[0747] "Generative AI" is a type of artificial intelligence technology that refers to algorithms and models for interacting with users and generating data.

[0748] "MR-GUI" is a graphical user interface displayed using MR technology, and is designed to allow users to operate it intuitively.

[0749] An "emotion engine" is software or an algorithm that analyzes emotions from a user's operations, facial expressions, etc., and outputs the results.

[0750] An "operation panel" is an interface that a user uses to operate a home appliance, and includes operation elements such as sliders and buttons.

[0751] "Communication means" refers to the technologies and protocols used to send and receive data, including the Internet and wireless communications.

[0752] "Analysis results" refers to information about the user's emotions analyzed by the emotion engine.

[0753] "Operation instructions" refer to specific operation commands for IoT home appliances that the server generates based on the analysis results.

[0754] This invention is a system that allows users to intuitively and freely operate IoT home appliances through a GUI (Graphical User Interface) using MR (Mixed Reality) technology. Furthermore, it includes an emotion engine that recognizes the user's emotions, enabling operation according to the user's emotions.

[0755] Hardware and software used

[0756] Hardware: Mixed reality device (e.g. head-mounted display)

[0757] Software: MR-GUI, emotion engine, IoT home appliance control software

[0758] Data processing and calculation

[0759] 1. Display MR-GUI:

[0760] The server sends data to the MR device to display the air conditioner's control panel.

[0761] Based on the data received, the terminal (MR device) displays a temperature control slider, airflow control buttons, operation mode switching buttons, etc.

[0762] 2. User Action Recognition:

[0763] The user directly operates the displayed control panel through the MR device.

[0764] The terminal (MR device) detects the user's operations and sends that information to the server.

[0765] 3. Emotion Analysis:

[0766] The server uses an emotion engine to analyze emotions from the user's operations.

[0767] For example, if a user expresses surprise, the emotion engine outputs that information as an analysis result.

[0768] 4. Controlling IoT appliances:

[0769] Based on the analysis results, the server generates operating instructions for the IoT home appliances.

[0770] For example, if the user shows signs of surprise, instructions to operate the air conditioner gently are generated.

[0771] The server sends the generated operation instructions to the IoT appliance, which then operates according to those instructions.

[0772] Specific examples

[0773] Example 1: When a user adjusts the temperature of an air conditioner

[0774] The user wears the MR device and the air conditioner control panel is displayed in their field of vision.

[0775] The user operates the temperature control slider to set the desired temperature.

[0776] The terminal (MR device) sends operation information to the server, and the server sends an instruction to change the temperature to the air conditioner.

[0777] The IoT appliance (air conditioner) will follow instructions and change the set temperature.

[0778] Example 2: When a user expresses surprise

[0779] The user expresses surprise while operating the air conditioner control panel.

[0780] The terminal (MR device) transmits the user's emotional data along with the operation information to the server.

[0781] The server uses an emotion engine to analyze the emotion of surprise and generate instructions to operate the air conditioner gently.

[0782] IoT home appliances (air conditioners) follow instructions and operate gently.

[0783] Prompt Sentence Examples

[0784] Prompt 1: "Describe a scenario in which a user adjusts the temperature of an air conditioner through a mixed reality device."

[0785] Prompt 2: "Describe how an IoT appliance operates when a user expresses surprise."

[0786] This system allows users to intuitively and emotionally respond to their operations, providing a more comfortable experience using IoT home appliances. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0787] Step 1:

[0788] MR-GUI display

[0789] The server generates data to display the air conditioner control panel and sends it to the MR device.

[0790] Input: Air conditioner control panel configuration data

[0791] Data processing: Converting the configuration data of the operation panel into a format that can be interpreted by the MR device

[0792] Output: Operation panel data sent to the MR device

[0793] The terminal (MR device) displays the operation panel based on the received data.

[0794] Input: Operation panel data sent from the server

[0795] Data calculation: Rendering process for visually displaying operation panel data

[0796] Output: The control panel that is visible to the user

[0797] Specific operation: The MR device displays temperature control sliders, airflow control buttons, operation mode switching buttons, etc. in the user's field of vision.

[0798] Step 2:

[0799] Recognizing user actions

[0800] The user directly operates the operation panel displayed through the MR device.

[0801] Input: Operation panel displayed on MR device

[0802] Data processing: Tracking the user's hand movements and eye gaze position

[0803] Output: Operation information

[0804] The terminal (MR device) detects the user's operations and sends the information to the server.

[0805] Input: User operation information

[0806] Data calculation: Converts the operation information into a format that can be interpreted by the server

[0807] Output: Operation information sent to the server

[0808] Specific operation: The MR device tracks the user's hand movements and gaze and sends them to the server as operation information.

[0809] Step 3:

[0810] Emotion Analysis

[0811] The server uses an emotion engine to analyze emotions from the user's operations.

[0812] Input: User operation information

[0813] Data calculation: Analyzes operation speed and movement patterns and inputs them into the emotion engine

[0814] Output: Emotion analysis results

[0815] The emotion engine analyzes the user's emotions and returns the results to the server.

[0816] Input: Operation speed and movement pattern

[0817] Data Computing: Identifying emotions using sentiment analysis algorithms

[0818] Output: Emotion analysis results

[0819] Specific behavior: The emotion engine determines whether the user is expressing surprise emotion and returns the result to the server.

[0820] Step 4:

[0821] Controlling IoT appliances

[0822] The server generates operating instructions for IoT home appliances based on the analysis results.

[0823] Input: Sentiment analysis results

[0824] Data calculation: Generate operational instructions based on the results of sentiment analysis

[0825] Output: Operation instructions sent to IoT appliances

[0826] The server sends the generated operation instructions to the IoT home appliance.

[0827] Input: Operation instructions

[0828] Data processing: Converting operation instructions into a format that can be interpreted by IoT appliances

[0829] Output: Operation instructions sent to IoT appliances

[0830] The IoT home appliance (air conditioner) operates according to the instructions it receives.

[0831] Input: Operation instructions sent from the server

[0832] Data calculation: Controlling operations based on operational instructions

[0833] Output: Actual air conditioner operation

[0834] Specific behavior: The air conditioner gradually changes the set temperature and operates according to the user's emotions.

[0835] (Application example 3)

[0836] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0837] Conventional IoT home appliance operation systems have the problem that they are difficult for users to operate intuitively and do not take into account the user's emotions. Furthermore, there are also a lack of easy-to-use methods for users to operate IoT devices in physical stores. This has led to a poor user experience and the inability to fully utilize the convenience of IoT home appliances and devices.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0839] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliance using generative AI, a means for the user to intuitively and freely operate the IoT home appliance using MR-GUI, a means for recognizing the user's emotions and adjusting the operation of the IoT home appliance according to those emotions, and a means for operating IoT devices in a physical store using smart glasses. This allows the user to intuitively operate IoT home appliances and devices and further enables operation according to emotions, improving the user experience and maximizing the convenience of IoT home appliances and devices.

[0840] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0841] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled or automatically controlled.

[0842] "Generative AI" is a type of artificial intelligence that has the ability to generate natural conversations and interactions with users.

[0843] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[0844] "Emotion recognition" is a technology that analyzes a user's facial expressions and movements to identify their emotional state.

[0845] "Smart glasses" are wearable devices that have built-in displays, cameras, and sensors, allowing users to operate them while visually checking information.

[0846] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[0847] An "IoT device" is a piece of equipment or device that is connected to the Internet and can send and receive data and be remotely controlled.

[0848] A system for implementing this invention is configured using MR (Mixed Reality) technology, generative AI, MR-GUI, emotion recognition technology, and smart glasses. Specific embodiments of each element are described below.

[0849] System configuration

[0850] 1. Hardware

[0851] Smart glasses: A wearable device equipped with a display, camera, and sensors that allows users to visually view information and operate the device.

[0852] Camera: Used to capture the user's facial expressions and movements.

[0853] IoT devices: Lighting, displays, sound systems, and other equipment that are connected to the internet and can be controlled remotely.

[0854] 2. Software

[0855] OpenCV: An image processing library used to analyze images captured by the camera.

[0856] EmotionRecognizer: An emotion recognition library that analyzes emotions from the user's facial expressions and movements.

[0857] IoTDeviceController: A library for controlling IoT devices.

[0858] MRGUI: A library that provides a graphical user interface displayed using MR technology.

[0859] Processing flow

[0860] The server captures the user's face using the camera built into the smart glasses and processes the image using OpenCV. Next, it recognizes the user's emotions using EmotionRecognizer. The recognized emotional information is sent to the MRGUI and displayed on the smart glasses' display. The user visually checks the operation panel of the IoT device through the MRGUI and operates it. The user's operation information is sent to the IoTDeviceController, which adjusts the operation of the IoT device based on the emotional information.

[0861] Specific examples

[0862] For example, if a user wears smart glasses in a physical store and adjusts the brightness of the lights, if the user expresses surprise, the brightness of the lights will be adjusted more gently. This allows users to operate IoT devices intuitively and enables operation according to their emotions, improving the user experience.

[0863] Prompt Sentence Examples

[0864] "If a user wearing smart glasses operates an MR-GUI to adjust the brightness of lights in a physical store and displays an emotion of surprise, design a system that adjusts the brightness of the lights gently."

[0865] The above is a specific embodiment for carrying out the present invention.

[0866] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0867] Step 1:

[0868] The server captures the user's face using the camera built into the smart glasses. The input is the camera image, and the output is the captured image data. This image data becomes the basis for analyzing the user's facial expressions and movements.

[0869] Step 2:

[0870] The server processes the captured image data using OpenCV. The input is the image data, and the output is preprocessed image data. This preprocessing includes image resizing and noise removal.

[0871] Step 3:

[0872] The server inputs the preprocessed image data into EmotionRecognizer to recognize the user's emotions. The input is the preprocessed image data, and the output is the user's emotional information (e.g., surprise, joy, sadness, etc.). EmotionRecognizer extracts features from the image data and classifies the emotions.

[0873] Step 4:

[0874] The server sends the recognized emotion information to the MRGUI and displays it on the display of the smart glasses. The input is emotion information, and the output is an operation panel displayed on the MRGUI. The user visually checks this operation panel and performs operations.

[0875] Step 5:

[0876] The user operates the operation panel of the IoT device through the MRGUI. The input is the user's operation information, and the output is the operation command. The user changes the settings of the IoT device using the buttons and sliders on the operation panel.

[0877] Step 6:

[0878] The server controls IoT devices using IoTDeviceController based on the user's operation information and emotional information. The input is the operation command and emotional information, and the output is the behavior of the IoT device. For example, if the user shows surprise, the brightness of the lights is gently adjusted.

[0879] Step 7:

[0880] The server feeds back the operation results of the IoT device to the user. The input is the operation result of the IoT device, and the output is the feedback information. The user can check the current status of the IoT device through the display of the smart glasses.

[0881] The above are the specific processing steps of this system.

[0882] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0883] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0884] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0885] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0886] [Second embodiment]

[0887] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0888] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0889] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0890] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0891] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0892] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0893] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0894] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0895] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0896] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0897] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0898] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0899] "Example 1"

[0900] One embodiment of the present invention is a system that uses mixed reality (MR) technology to enable direct communication with IoT home appliances. Specifically, a user wears special goggles or a headset to visually recognize the IoT home appliance. At this time, MR technology displays a direct operation interface on the IoT home appliance. For example, in the case of an air conditioner, a temperature control slider and airflow control buttons are displayed on the air conditioner body using MR, and the user can change the air conditioner settings by directly operating these.

[0901] "Example 2"

[0902] Furthermore, one embodiment of the present invention is a system that uses generative AI to enable conversations between users and IoT appliances. Specifically, when a user speaks to an IoT appliance, the generative AI understands the instruction and transmits the appropriate operation to the IoT appliance. For example, if a user says, "Heat the room," the generative AI understands the instruction and instructs the air conditioner to operate in heating mode.

[0903] "Example 3"

[0904] Another embodiment of the present invention is a system that allows users to intuitively and freely operate IoT home appliances using an MR-GUI. Specifically, users operate IoT home appliances using a GUI displayed using MR technology. For example, an air conditioner's control panel displayed using MR displays a temperature control slider, buttons for adjusting airflow, and buttons for switching operation modes, allowing users to freely change the air conditioner's settings by directly operating these.

[0905] The processing flow of each embodiment will be described below.

[0906] "Example 1"

[0907] Step 1: The user puts on the MR-enabled goggles or headset.

[0908] Step 2: The user visually recognizes the IoT appliance. At this time, a direct operation interface is displayed on the IoT appliance using mixed reality technology.

[0909] Step 3: The user changes the settings of the IoT appliance by directly operating the operation interface displayed in MR.

[0910] "Example 2"

[0911] Step 1: The user talks to the IoT appliance.

[0912] Step 2: The generative AI understands the user's instructions.

[0913] Step 3: The generative AI communicates the understood instructions to the IoT appliance, which then performs the operation according to the instructions.

[0914] "Example 3"

[0915] Step 1: The user puts on the MR-enabled goggles or headset.

[0916] Step 2: The user operates the IoT appliance using the GUI displayed by MR technology.

[0917] Step 3: The user can freely change the settings of the IoT home appliance by directly operating the control panel displayed in MR.

[0918] Example 1

[0919] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0920] Conventional methods of operating IoT devices typically involve using smartphones or remote controls, which require unintuitive operation and complex settings. Furthermore, there are limited ways for users to directly communicate with IoT devices, creating issues with operability and convenience. Furthermore, the transmission and analysis of real-time operational information is insufficient, making rapid control difficult. To resolve these issues, a system that allows for more intuitive and flexible operation is needed.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0922] In this invention, the server includes means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, means for enabling conversation between the user and the IoT device using generative AI, means for the user to intuitively and freely operate the IoT device using MR-GUI, means for visually recognizing the IoT device using a visual device worn by the user and displaying an operation interface, and means for transmitting user operation information to the server in real time, which then analyzes the operation information and transmits control signals to the corresponding IoT device. This allows the user to intuitively and freely operate the IoT device, achieving rapid control in real time.

[0923] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0924] An "IoT device" is an electronic device that can communicate with other devices and systems via the Internet and send, receive, and control data.

[0925] "Generative AI" is a type of artificial intelligence technology that refers to algorithms and models for interacting with users and generating data.

[0926] "MR-GUI" is a graphical user interface that uses Mixed Reality technology and is designed to allow users to operate it intuitively.

[0927] "Visual devices" are devices worn by users that provide visual information and blend the real world with the virtual world. Examples include goggles and headsets.

[0928] An "operation interface" is a visual control element that users use to operate IoT devices, including sliders and buttons.

[0929] A "server" is a computer system that processes and manages data on a network, and is responsible for analyzing user operation information and sending control signals to corresponding IoT devices.

[0930] A "control signal" is a signal sent from a server to an IoT device, and includes instructions for changing the device's operation or settings.

[0931] This invention is a system that enables direct communication with IoT devices using Mixed Reality (MR) technology. The user wears special visual devices (goggles or headset) and visually recognizes the IoT devices. The visual devices use MR technology to display the IoT devices in the user's field of vision and overlay an operation interface on top of them.

[0932] Hardware and software used

[0933] Visual equipment: goggles or headset

[0934] Image recognition software: OpenCV, etc.

[0935] MR development platform: Unity, etc.

[0936] Server-side programming languages: Python, Node.js, etc.

[0937] Communication technologies: Wi-Fi, Bluetooth, MQTT, HTTP, etc.

[0938] Explanation of program processing

[0939] When a user wears the vision device, the device recognizes objects in the user's field of vision and identifies IoT devices. For example, to recognize an air conditioner in the living room, the device uses the goggles' built-in camera and image recognition software (e.g., OpenCV). The device displays an operation interface overlaid on the recognized IoT device. In the case of an air conditioner, a slider for adjusting the temperature and buttons for adjusting the airflow are displayed. This interface is generated using an MR development platform such as Unity.

[0940] The user operates the device with their hands or eyes while looking at the operation interface displayed in their field of vision. For example, they change the temperature setting by dragging a temperature control slider with their finger. The device captures the user's operation information and transmits it to the server in real time. This communication uses wireless communication technologies such as Wi-Fi and Bluetooth.

[0941] The server analyzes the received operation information and generates an appropriate control signal for the corresponding IoT device. For example, if a user sets the temperature control slider to 25 degrees, the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. This process uses server-side programming languages ​​such as Python and Node.js. The IoT device receives the control signal sent from the server and changes the setting. In the case of an air conditioner, its internal control system changes the temperature setting to 25 degrees. Protocols such as MQTT and HTTP are used for this communication.

[0942] Specific examples

[0943] A user wants to operate the air conditioner in their living room. The user puts on MR goggles and visually recognizes the air conditioner in the living room. Using MR technology, a temperature control slider and airflow control buttons are displayed on the air conditioner itself. The user operates the temperature control slider to change the set temperature to 25 degrees. This operation information is sent from the goggles to the server, and the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. The air conditioner receives this signal, and its internal control system changes the temperature setting to 25 degrees.

[0944] Prompt Sentence Examples

[0945] "Describe a scenario in which a user puts on mixed reality goggles and operates the air conditioner in their living room. Please describe in detail the process by which the user operates the temperature control slider to change the set temperature to 25 degrees."

[0946] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0947] Step 1:

[0948] The user puts on the MR goggles.

[0949] Input: User puts on MR goggles and turns them on.

[0950] Output: The goggles are up and ready to capture the user's view.

[0951] How it works: The user puts on the MR goggles and presses the power button to turn them on. The camera built into the goggles begins capturing the user's field of vision.

[0952] Step 2:

[0953] The terminal displays IoT devices in the user's field of view.

[0954] Input: Video data captured by the goggles' built-in camera.

[0955] Output: The IoT device is displayed in the user's field of view.

[0956] Specific operation: The device analyzes the video data from the camera and identifies IoT devices using image recognition software (e.g., OpenCV). Identified IoT devices are displayed in the user's field of view.

[0957] Step 3:

[0958] The terminal displays the operation interface.

[0959] Input: Location and type of IoT device.

[0960] Output: The operation interface is displayed overlaid on the IoT device.

[0961] Specific operation: The terminal uses an MR development platform (e.g., Unity) to generate and display operation interfaces such as temperature control sliders and airflow control buttons on the IoT device.

[0962] Step 4:

[0963] The user operates the operation interface.

[0964] Input: User's hand and eye movements.

[0965] Output: Operation information is generated.

[0966] Specific actions: The user operates the interface displayed in the field of view with their hands or eyes. For example, they change the temperature setting by dragging the temperature control slider with their finger.

[0967] Step 5:

[0968] The terminal transmits operation information to the server.

[0969] Input: User operation information.

[0970] Output: Operation information is sent to the server.

[0971] How it works: The device captures user operation information and transmits it to the server in real time. This communication is carried out using wireless communication technologies such as Wi-Fi and Bluetooth.

[0972] Step 6:

[0973] The server analyzes the operation information and sends a control signal to the corresponding IoT device.

[0974] Input: Operation information sent from the device.

[0975] Output: Control signal for IoT devices.

[0976] How it works: The server analyzes the received operation information and generates the appropriate control signal for the corresponding IoT device. For example, if a user sets the temperature control slider to 25 degrees, the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. This process is performed using server-side programming languages ​​such as Python and Node.js.

[0977] Step 7:

[0978] The IoT device receives the control signal and changes its settings.

[0979] Input: Control signal sent from the server.

[0980] Output: The settings of the IoT device are changed.

[0981] Specific operation: The IoT device receives a control signal sent from the server and changes the setting. In the case of an air conditioner, the internal control system changes the temperature setting to 25 degrees. Protocols such as MQTT and HTTP are used for this communication.

[0982] (Application example 1)

[0983] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] Conventional IoT home appliances are typically operated via smartphones or remote controls, which makes intuitive operation difficult. Furthermore, there are limited ways to efficiently obtain product and promotion information in physical stores, creating a need to improve the user shopping experience.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0986] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliances using generative AI, a means for the user to intuitively and freely operate the IoT home appliances using MR-GUI, and a means for visually displaying product information and promotion information in a physical store using smart glasses and providing an interface that the user can operate directly. This not only enables the user to intuitively operate the IoT home appliances, but also improves the shopping experience in the physical store.

[0987] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[0988] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[0989] "Generative AI" is an artificial intelligence technology that generates appropriate responses and actions based on user input and environmental data.

[0990] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[0991] "Smart glasses" are wearable devices equipped with a display, camera, and sensors that allow users to experience visual information as augmented reality.

[0992] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[0993] "Product information" refers to information necessary for users to make purchasing decisions, such as product price, specifications, and stock status.

[0994] "Promotion information" refers to information aimed at promoting product sales, such as discounts and campaigns.

[0995] An "interface" is the means or method by which a user interacts with a system or device.

[0996] A system for implementing the present invention is configured as follows.

[0997] First, the server uses mixed reality (MR) technology to enable direct communication with IoT appliances. When a user puts on smart glasses and visually recognizes an IoT appliance, MR technology displays an operation interface on the appliance. For example, in the case of an air conditioner, the temperature control slider and airflow control buttons are displayed on the air conditioner body using MR, allowing the user to change the air conditioner settings by directly operating them.

[0998] Next, generative AI is used to enable conversations between users and IoT appliances. When a user gives a command via voice, the generative AI analyzes the command and executes the appropriate operation. For example, if the command is "Set the air conditioner temperature to 25 degrees," the generative AI will understand the command and set the temperature to 25 degrees.

[0999] Furthermore, MR-GUI allows users to intuitively and freely operate IoT home appliances. MR-GUI provides a visual interface that allows users to easily change various settings of home appliances.

[1000] The smart glasses also visually display product and promotion information from physical stores, providing an interface that users can operate directly. When a user looks at a specific product in a physical store, the camera in the smart glasses recognizes it and displays product and promotion information. Users can tap on this interface to check detailed information or complete the purchase process.

[1001] The following hardware and software are required to realize this system: For hardware, smart glasses (e.g., Microsoft HoloLens) and IoT home appliances are required. For software, OpenCV (image processing), TensorFlow / Keras (AI model for product recognition), and ARToolkit (MR interface generation) are used.

[1002] As a concrete example, imagine a user wearing smart glasses in a brick-and-mortar store. When the user looks at a particular product (e.g., a refrigerator), the smart glasses' camera recognizes the product and displays an MR interface. The interface displays the product's price, availability, and promotional information, and the user can tap to view more information or check out.

[1003] An example prompt is, "When you wear the smart glasses and look at a product in a physical store, detailed information and promotional information about the product will be displayed as an MR interface. Users can tap on this interface to view more information about the product or to complete the purchase."

[1004] In this way, users can not only intuitively operate IoT home appliances, but also improve their shopping experience in physical stores.

[1005] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1006] Step 1:

[1007] A user wears smart glasses and walks around a physical store. The camera in the smart glasses captures the surroundings in real time. The input is the camera image, and the output is the captured video data.

[1008] Step 2:

[1009] The device (smart glasses) inputs the captured video data into an AI model for product recognition. The AI ​​model uses TensorFlow / Keras to recognize products from the video data. The input is the video data, and the output is the ID of the recognized product.

[1010] Step 3:

[1011] The server receives the ID of the recognized product and retrieves the corresponding product information and promotion information from the database. The input is the product ID, and the output is the product information and promotion information.

[1012] Step 4:

[1013] Based on the product information and promotion information acquired by the server, an MR interface is generated using ARToolkit. The input is product information and promotion information, and the output is the MR interface.

[1014] Step 5:

[1015] The device (smart glasses) displays the generated MR interface in the user's field of view. The input is the MR interface, and the output is the interface displayed in the user's field of view.

[1016] Step 6:

[1017] The user operates the MR interface to check detailed information and carry out purchase procedures. The input is the user's operation, and the output is the operation result.

[1018] Step 7:

[1019] The terminal (smart glasses) sends the results of the user's operations to the server, which then executes the necessary processing. For example, in the case of a purchase, the server works with an electronic payment system to complete the payment. The input is the result of the user's operations, and the output is the processing result.

[1020] Step 8:

[1021] The server sends the processing results to the device (smart glasses) and notifies the user. The input is the processing results, and the output is the notification to the user.

[1022] In this way, users can intuitively obtain and manipulate product information and promotional information in physical stores through smart glasses.

[1023] Example 2

[1024] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1025] Conventional methods of operating IoT home appliances require users to use dedicated remote controls or smartphone apps, which often require unintuitive operation. While systems using voice recognition technology exist, there are issues with the accuracy of interpreting voice commands and the speed of operation. This makes it difficult for users to comfortably operate IoT home appliances.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1027] In this invention, the server includes a means for a user to issue a voice instruction, a means for a terminal to capture the voice and send it to the server, a means for the server to convert the voice data into text data, a means for the server to analyze the text data using a generative AI model and generate an operation command, a means for the server to send the operation command to the IoT home appliance, and a means for the IoT home appliance to operate in accordance with the operation command. This enables a user to intuitively and quickly operate the IoT home appliance through natural conversation.

[1028] "User" refers to a person who uses the system to operate IoT home appliances.

[1029] "Device" refers to a device that captures a user's voice and transmits it to a server, such as a smart speaker or smartphone.

[1030] "Server" refers to a computer system that converts voice data into text data, analyzes it using a generative AI model, generates operation commands, and sends them to IoT home appliances.

[1031] "Voice data" refers to data that digitally represents the voice uttered by the user to the terminal.

[1032] "Text data" refers to character string data obtained by analyzing voice data.

[1033] A "generative AI model" refers to an artificial intelligence model that analyzes text data and generates appropriate operation commands.

[1034] An "operation command" refers to an instruction to instruct an IoT home appliance to perform a specific action.

[1035] "IoT home appliances" refer to home appliances that are connected to the Internet and can be controlled remotely. Examples include air conditioners, lighting, and smart locks.

[1036] "Behavior" refers to the specific action that an IoT home appliance performs in accordance with an operation command.

[1037] This invention is a system that enables users to operate IoT home appliances through natural conversation. This system consists of a user, a terminal, a server, and IoT home appliances.

[1038] First, the user issues a voice command, for example, "heat the room." Next, the device (e.g., a smart speaker or smartphone) captures the user's voice and sends the voice data to a server. The server then uses voice recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data.

[1039] The server then analyzes the text data using a generative AI model (e.g., GPT-4) and generates an appropriate operation command. For example, from the text data "heat the room," it generates the operation command "set the air conditioner to heating mode." The server then uses an IoT platform (e.g., AWS IoT Core) to send the generated operation command to the IoT home appliance.

[1040] Finally, the IoT appliance (e.g., air conditioner) operates according to the received operation command. Specifically, the air conditioner starts operating in heating mode, raising the temperature in the living room.

[1041] As a concrete example, consider a scenario in which a user says, "Heat the room." In this case, the device captures the voice and sends it to the server. The server converts the voice into text, analyzes the instruction "Heat the room" using a generative AI model, and generates an operation command such as "Set the air conditioner to heating mode." The server then sends the operation command to the IoT appliance, and the air conditioner starts operating in heating mode.

[1042] An example of a prompt to input to a generative AI model is as follows:

[1043] When the user says "heat the room," generate an operation command to set the air conditioner to heating mode.

[1044] In this way, users can intuitively and quickly operate IoT home appliances through natural conversation.

[1045] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1046] Step 1:

[1047] The user issues a voice command.

[1048] A specific example of this behavior is when the user is in the living room and says, "Warm up the room."

[1049] Input: User's voice commands

[1050] Output: Audio data

[1051] Step 2:

[1052] The device captures the audio and sends it to the server.

[1053] Specifically, the smart speaker captures the user's voice and sends the voice data to a server via the Internet.

[1054] Input: Audio data

[1055] Output: Audio data sent to the server

[1056] Step 3:

[1057] The server converts the voice data into text data.

[1058] Specifically, the server uses voice recognition software (for example, Google Speech-to-Text API) to convert the received voice data into text data such as "Warm up the room."

[1059] Input: Audio data sent to the server

[1060] Output: Text data

[1061] Step 4:

[1062] The server analyzes the text data using a generative AI model and generates operation commands.

[1063] Specifically, the server uses a generative AI model (e.g., GPT-4) to analyze the text data "heat the room" and generate an operation command such as "set the air conditioner to heating mode."

[1064] Input: Text data

[1065] Output: Operation command

[1066] Step 5:

[1067] The server sends operation commands to the IoT appliances.

[1068] Specifically, the server uses an IoT platform (e.g., AWS IoT Core) to send the generated operation command to the air conditioner.

[1069] Input: Operation command

[1070] Output: Operation commands sent to IoT appliances

[1071] Step 6:

[1072] IoT home appliances operate according to operational commands.

[1073] Specifically, the air conditioner will start operating in heating mode in response to the received command, raising the temperature in the living room and providing a comfortable environment for the user.

[1074] Input: Operation command sent to IoT appliances

[1075] Output: The air conditioner starts operating in heating mode

[1076] (Application example 2)

[1077] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1078] In conventional brick-and-mortar stores, it was difficult for customers to intuitively and freely operate IoT devices in the store. Furthermore, customers could not directly interact with IoT devices, making operation cumbersome. Furthermore, there was a lack of a system for understanding and appropriately executing instructions for operating IoT devices, creating a need for an improved customer experience.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1080] In this invention, the server includes means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, means for enabling conversation between the user and the IoT device using generative AI, means for the user to intuitively and freely operate the IoT device using MR-GUI, means for enabling customers in a physical store to talk to the IoT device using a smartphone, and means for the generative AI to understand the customer's instructions and transmit appropriate operations to the IoT device. This enables customers to intuitively and freely operate the IoT device in a physical store and talk to it directly through the generative AI.

[1081] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1082] "IoT devices" are electronic devices that are interconnected via the Internet and are capable of sending, receiving, and controlling data.

[1083] "Generative AI" is an artificial intelligence technology that generates natural language based on user input and provides appropriate responses and instructions.

[1084] "MR-GUI" is a graphical user interface that uses MR technology and allows users to operate it intuitively.

[1085] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[1086] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use applications.

[1087] A "customer" is a consumer who visits a physical store to purchase a product or service.

[1088] "Communication" is the act of exchanging information and communicating intentions.

[1089] "Operation" is the act of controlling a device or system to perform a desired action.

[1090] "Instruction" refers to an order to perform a specific action or operation.

[1091] To implement the present invention, the following system configuration and program are required.

[1092] System Configuration

[1093] 1. Hardware:

[1094] Smartphone: A device that customers use in a physical store and that is capable of voice input and internet connectivity.

[1095] IoT devices: Electronic devices that can be connected to the internet, such as digital signage, smart lighting, and sound systems, installed in physical stores.

[1096] Server: Hosts the generative AI model and manages communication between smartphones and IoT devices.

[1097] 2. Software:

[1098] Speech Recognition Library: Use the speech_recognition library to capture audio from your smartphone's microphone and convert it into text.

[1099] Generative AI models: Use generative AI models, such as OpenAI's GPT-3, to understand customer instructions and generate appropriate responses.

[1100] HTTP Request Library: Use the requests library to send the generated instructions to the IoT device.

[1101] Program processing explanation

[1102] 1. Voice input and recognition:

[1103] When customers speak into their smartphone's microphone, a speech recognition library converts the speech into text.

[1104] For example, if a customer says, "Turn up the lights in this area," the speech is converted into text.

[1105] 2. Use of generative AI models:

[1106] The converted text is sent as a prompt to a generative AI model, which generates appropriate operating instructions for the IoT device.

[1107] Example prompt: "User instructions: Increase the lighting in this area and suggest appropriate IoT device actions."

[1108] 3. Sending instructions to IoT devices:

[1109] The generated instructions are sent to the IoT device as an HTTP request.

[1110] For example, a generative AI model generates an instruction to "brighten the lights" and sends it to a smart light.

[1111] Specific examples

[1112] When a customer uses their smartphone in a brick-and-mortar store to say, "Turn up the lights in this area," a speech recognition library converts that speech into text. A generative AI model then understands the text and generates the command "Turn up the lights." Finally, that command is sent as an HTTP request to the smart lighting, which then turns on the lights.

[1113] In this way, customers can intuitively and freely operate IoT devices in physical stores and can speak directly to them through generative AI.

[1114] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1115] Step 1:

[1116] The user speaks into the smartphone's microphone.

[1117] Input: User's voice

[1118] Output: Audio data

[1119] Specific action: The user says, "Turn up the lights in this area."

[1120] Step 2:

[1121] The device uses a speech recognition library to convert the voice data into text.

[1122] Input: Audio data

[1123] Output: Text data

[1124] Specific operation: The speech recognition library (speech_recognition) analyzes the voice data and converts it into text such as "Turn on the lights in this area."

[1125] Step 3:

[1126] The device sends text data as prompts to the generative AI model, which generates appropriate operating instructions for the IoT device.

[1127] Input: Text data

[1128] Output: Generated instructions

[1129] Specific operation: A generative AI model (OpenAI's GPT-3) receives text data and generates the instruction "Turn the lights on."

[1130] Example prompt: "User instructions: Increase the lighting in this area and suggest appropriate IoT device actions."

[1131] Step 4:

[1132] The server sends the generated instructions to the IoT device as an HTTP request.

[1133] Input: Generated instructions

[1134] Output: HTTP request

[1135] What happens: The server formats the generated instructions as an HTTP request and sends it to the smart lighting.

[1136] Step 5:

[1137] The IoT device performs an action based on the received instructions.

[1138] Input: HTTP request

[1139] Output: IoT device operation

[1140] Specific behavior: The smart lighting receives the HTTP request and performs the action of brightening the light.

[1141] In this way, users can intuitively and freely operate IoT devices using their smartphones in physical stores, and can speak directly to them through generative AI, improving the customer experience.

[1142] Example 3

[1143] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1144] Conventional methods for operating IoT devices often had complex and unintuitive user interfaces, making it difficult for users to operate them. Furthermore, interaction with and operation of IoT devices was limited, making it difficult to accurately reflect the user's intentions. Furthermore, the lack of an operating environment that combined the real world and the virtual world did not improve the user experience.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for enabling direct communication with the IoT device using MR technology, means for enabling interaction between the user and the IoT device using a generation AI, means for the user to intuitively and freely operate the IoT device using an MR-GUI, and means for the user to wear the MR device, the server to generate the MR-GUI, the terminal to display the GUI, the user to operate the GUI, the terminal to send operation information to the server, and the server to control the IoT device. This allows the user to intuitively and freely operate the IoT device, significantly improving the user experience.

[1146] "MR technology" is a technology that provides visual information that combines the real world and the virtual world.

[1147] An "IoT device" is an electronic device that communicates via the Internet and can be remotely controlled and collect data.

[1148] "Generative AI" is an artificial intelligence technology that interacts with users and generates data.

[1149] "MR-GUI" is a graphical user interface displayed using MR technology.

[1150] A "server" is a computer system that processes data and provides services over a network.

[1151] A "terminal" is a device that a user can directly operate, including MR devices.

[1152] A "user" is a person who operates the system.

[1153] "Operation information" refers to the data and instructions entered by the user through the terminal.

[1154] "Communication" is the process of sending and receiving data or information.

[1155] "Dialogue" is the process by which a user and a system exchange information with each other.

[1156] "Control" is the process by which a system manages and coordinates the behavior of IoT devices.

[1157] This invention is a system that allows users to intuitively and freely operate IoT devices using mixed reality (MR) technology. Specifically, the user operates the IoT device using an MR-GUI displayed by MR technology. Detailed embodiments of this system are described below.

[1158] Hardware and software used

[1159] 1. MR device:

[1160] Example: Microsoft HoloLens

[1161] Function: Provides visual information that blends the real and virtual worlds.

[1162] 2. Server:

[1163] Function: Generates MR-GUI, processes operation information, and controls IoT devices.

[1164] 3. IoT devices:

[1165] Examples: air conditioners, lighting, smart locks, etc.

[1166] Function: Communicates via the Internet, allowing remote control and data collection.

[1167] 4. Generation AI:

[1168] Function: Interacts with the user and generates data.

[1169] System Operation

[1170] When a user puts on an MR device, the server acquires information about the user's environment and generates an MR-GUI based on that information. The MR device displays the GUI received from the server in the user's field of view. The user intuitively operates the GUI displayed through the MR device. For example, the user can change the air conditioner temperature by moving the temperature control slider, or adjust the air volume by pressing the air volume control button. The MR device transmits the user's operation information to the server in real time. The server controls the IoT devices based on the received operation information.

[1171] Specific examples

[1172] For example, if a user wants to set the air conditioner temperature to 25 degrees, they would follow these steps:

[1173] 1. The user wears the MR device.

[1174] 2. The server generates an MR-GUI that includes the air conditioner control panel.

[1175] 3. The MR device displays the operation panel in the user's field of view.

[1176] 4. The user sets the temperature slider to 25 degrees.

[1177] 5. The MR device sends this operation information to the server.

[1178] 6. The server sends a command to the air conditioner to set the temperature to 25 degrees.

[1179] Prompt Sentence Examples

[1180] A system description can be generated by inputting a prompt like the one below into the generative AI model.

[1181] "Please explain the system process in which users operate IoT devices using mixed reality technology. Please clearly state what kind of data processing and calculations are performed, including the names of specific hardware and software. Also, please provide specific examples."

[1182] By using this prompt sentence, the generative AI model can generate a detailed explanation of the system. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1183] Step 1:

[1184] The user puts on the MR device.

[1185] Input: The user puts on a mixed reality device (e.g., Microsoft HoloLens).

[1186] Data processing: The MR device acquires the user's location information and surrounding environmental data.

[1187] Output: The MR device sends the user's location and environmental data to the server.

[1188] Specific Action: The user puts the HoloLens on their head and turns on the device.

[1189] Step 2:

[1190] The server generates the MR-GUI.

[1191] Input: User location and environmental data sent from the MR device.

[1192] Data processing: The server analyzes the data received and generates an MR-GUI that is suitable for the user's environment.

[1193] Output: Send the generated MR-GUI data to the MR device.

[1194] Specific operation: The server generates an air conditioner control panel based on the user's location information and surrounding environmental data.

[1195] Step 3:

[1196] The terminal displays the GUI.

[1197] Input: MR-GUI data sent from the server.

[1198] Data processing: The MR device renders the received GUI data to display in the user's field of view.

[1199] Output: The MR-GUI displayed in the user's field of view.

[1200] Specific operation: HoloLens displays the air conditioner control panel overlaid on the user's field of vision.

[1201] Step 4:

[1202] The user operates the GUI.

[1203] Input: The user operates the MR-GUI displayed in the field of view.

[1204] Data processing: The MR device acquires the user's operation information and analyzes the operation content.

[1205] Output: Sends the parsed operation information to the server.

[1206] What happens: The user moves their hand to set a slider displayed in the HoloLens' field of view to 25 degrees.

[1207] Step 5:

[1208] The terminal transmits operation information to the server.

[1209] Input: User operation information.

[1210] Data processing: The MR device packets the operation information and sends it to the server via the network.

[1211] Output: Operation information sent to the server.

[1212] Specific operation: HoloLens sends user operation information (e.g., changing the temperature setting to 25 degrees) to the server.

[1213] Step 6:

[1214] The server controls the IoT devices.

[1215] Input: Operation information sent from the MR device.

[1216] Data processing: The server analyzes the operation information and generates control commands for the corresponding IoT devices.

[1217] Output: Control commands sent to IoT devices.

[1218] Specific operation: The server sends a command to the air conditioner to "set the temperature to 25 degrees," and the air conditioner adjusts the temperature according to that command.

[1219] (Application example 3)

[1220] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1221] Conventional IoT home appliances are generally operated using remote controls or smartphone apps, which makes intuitive operation difficult. Furthermore, in brick-and-mortar stores, there is a lack of means to efficiently manage and operate multiple IoT devices, which makes it difficult for store staff to respond quickly. To solve these issues, a more intuitive and efficient operation method is needed.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1223] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between a user and IoT home appliances using generative AI, a means for a user to intuitively and freely operate IoT home appliances using MR-GUI, a means for intuitively operating IoT devices in a physical store using smart glasses, and a means for detecting hand movements to operate the MR-GUI, thereby enabling a user to intuitively operate IoT home appliances and IoT devices in a physical store.

[1224] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1225] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[1226] "Generative AI" is an artificial intelligence technology that generates natural conversations and responses based on user input.

[1227] "MR-GUI" is a graphical user interface displayed using Mixed Reality technology, and is an interface that users can operate intuitively.

[1228] "Smart glasses" are wearable devices that have built-in displays, cameras, and sensors, allowing users to visually check information.

[1229] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[1230] An "IoT device" is an electronic device that is connected to the Internet and is capable of data communication and remote control.

[1231] "Means for detecting hand movements" refers to technology that uses cameras and sensors to recognize the position and movement of the user's hands and perform operations based on that.

[1232] This invention relates to a system that uses mixed reality (MR) technology to intuitively operate IoT home appliances and IoT devices in brick-and-mortar stores. Specific embodiments for realizing this system are described below.

[1233] System Configuration

[1234] The system consists of the following major components:

[1235] 1. Server:

[1236] Generative AI: Generates natural-sounding conversations and responses based on user input.

[1237] Database: Stores configuration information for IoT home appliances and IoT devices, as well as user operation history.

[1238] 2. Terminal:

[1239] Smart glasses: Wearable devices with built-in displays, cameras, and sensors that allow users to visually view information.

[1240] Camera: Used to detect the user's hand movements.

[1241] MR-GUI: A graphical user interface displayed using Mixed Reality technology.

[1242] Program processing

[1243] The server uses generative AI to enable conversations between users and IoT appliances and devices. The user wears smart glasses and operates the MR-GUI through a camera that detects hand movements. Specifically, the following process is performed:

[1244] 1. Hand movement detection:

[1245] The camera recognizes the position and movement of the user's hands and detects hand landmarks.

[1246] The sliders and buttons of the MR-GUI are operated based on the detected hand movements.

[1247] 2. Display MR-GUI:

[1248] The smart glasses display shows the operation panels of IoT home appliances and devices.

[1249] For example, the air conditioner control panel displays a temperature control slider, a button to adjust the airflow, and a button to switch between operating modes.

[1250] 3. Generative AI Conversation:

[1251] When a user gives a command via voice, the generative AI analyzes the command and generates an appropriate response.

[1252] For example, in response to an instruction such as "Set the air conditioner temperature to 24 degrees," the generative AI will change the air conditioner temperature.

[1253] Specific examples

[1254] Consider a scenario in which a store clerk wears smart glasses and adjusts the temperature of the air conditioning in a physical store. The clerk moves his or her hand to operate the slider on the MR-GUI, changing the temperature from 22 degrees to 24 degrees. In this case, the generative AI analyzes the clerk's voice instructions and generates an appropriate response.

[1255] Prompt Sentence Examples

[1256] "Put on the smart glasses and use your hand to operate the slider on the MR-GUI to change the temperature of the store's air conditioning from 22 degrees to 24 degrees."

[1257] In this way, the present invention enables users to intuitively operate IoT home appliances and IoT devices in physical stores.

[1258] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1259] Step 1:

[1260] The user puts on the smart glasses and starts the system.

[1261] Input: Smart glasses activation signal

[1262] Output: MR-GUI is displayed on the smart glasses display.

[1263] Specific operation: When the user turns on the smart glasses, the system starts up and the MR-GUI is displayed on the display, allowing the user to visually confirm the operation panel.

[1264] Step 2:

[1265] The device's camera detects the user's hand movements.

[1266] Input: Camera image

[1267] Output: Hand landmark data

[1268] How it works: The camera captures the position and movement of the user's hands in real time and detects hand landmarks, which allows the system to recognize the user's hand movements.

[1269] Step 3:

[1270] The device analyzes the hand landmark data and operates the MR-GUI.

[1271] Input: Hand landmark data

[1272] Output: MR-GUI operation instructions

[1273] Specific operation: Sliders and buttons in the MR-GUI are operated based on the hand landmark data. For example, when a user moves their hand to operate a slider, the slider's position changes accordingly.

[1274] Step 4:

[1275] The user issues a command by voice.

[1276] Input: Voice command

[1277] Output: Audio data

[1278] How it works: When a user gives a voice command such as "Set the air conditioner temperature to 24 degrees," the voice is captured by the microphone in the smart glasses.

[1279] Step 5:

[1280] The server uses generative AI to analyze voice commands and generate an appropriate response.

[1281] Input: Audio data

[1282] Output: Operation instruction data

[1283] How it works: The server analyzes the voice data and understands the user's instructions. The generative AI generates an appropriate response and generates operation instruction data to set the air conditioner temperature to 24 degrees.

[1284] Step 6:

[1285] The terminal operates IoT home appliances based on the operation instruction data.

[1286] Input: Operation instruction data

[1287] Output: Change the settings of IoT appliances

[1288] Specific operation: Based on the operation instruction data, the air conditioner temperature is set to 24 degrees. This executes the user's instruction.

[1289] Step 7:

[1290] The server provides feedback on the operation results to the user.

[1291] Input: IoT appliance settings change results

[1292] Output: Feedback message

[1293] Specific operation: The server confirms that the air conditioner temperature is set to 24 degrees, generates a feedback message for the user such as "The air conditioner temperature has been set to 24 degrees," and displays it on the smart glasses display.

[1294] In this way, users can intuitively operate IoT home appliances and IoT devices in physical stores through smart glasses.

[1295] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1296] "Example 1"

[1297] One embodiment of the present invention is a system in which a user wears mixed reality (MR) goggles or a headset and communicates directly with IoT appliances. This system includes an emotion engine that recognizes the user's emotions. Specifically, when the user speaks to an IoT appliance, the system analyzes the user's emotions from the tone of voice and facial expressions, and reflects the results in the operation of the IoT appliance. For example, if the user expresses anger, the IoT appliance can quickly perform the appropriate operation, enabling it to operate in accordance with the user's emotions.

[1298] "Example 2"

[1299] Another example is a system that uses generative AI to enable conversations between users and IoT appliances. This system also includes an emotion engine that recognizes the user's emotions. Specifically, it analyzes the user's speech and reflects the results in the operation of the IoT appliance. For example, if the user expresses joy, the IoT appliance will operate in a joyful manner, enabling it to operate in accordance with the user's emotions.

[1300] "Example 3"

[1301] Another example is a system that uses MR-GUI to allow users to intuitively and freely operate IoT appliances. This system also includes an emotion engine that recognizes the user's emotions. Specifically, it analyzes the user's emotions from their operations and reflects the results in the operation of the IoT appliance. For example, if the user expresses surprise, the IoT appliance can operate in a gentle manner, allowing it to respond to the user's emotions.

[1302] The processing flow of each embodiment will be described below.

[1303] "Example 1"

[1304] Step 1: The user puts on MR (Mixed Reality) goggles or a headset.

[1305] Step 2: The user talks to the IoT appliance.

[1306] Step 3: The emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[1307] Step 4: Reflect the analysis results in the operation of IoT home appliances.

[1308] "Example 2"

[1309] Step 1: The user speaks to the IoT appliance via generative AI.

[1310] Step 2: The emotion engine analyzes the emotion from the user's speech.

[1311] Step 3: Reflect the analysis results in the operation of IoT home appliances.

[1312] "Example 3"

[1313] Step 1: The user operates the IoT appliance using the MR-GUI.

[1314] Step 2: The emotion engine analyzes the emotions from the user's operations.

[1315] Step 3: Reflect the analysis results in the operation of IoT home appliances.

[1316] Example 1

[1317] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1318] Conventional IoT devices require the use of dedicated remote controls or smartphone apps, making them difficult to operate intuitively. Furthermore, since operations cannot take into account the user's emotions, it is difficult to improve the user experience. Furthermore, communication with IoT devices is limited, and users cannot directly talk to them, limiting the degree of freedom in operation.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1320] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, and a means for recognizing the user's emotions and optimizing the operation of the IoT device based on those emotions. This allows the user to intuitively and freely operate the IoT device, and further realizes optimal operation according to the user's emotions.

[1321] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1322] "IoT devices" are home appliances and devices that are connected to each other via the Internet and can send, receive, and control data.

[1323] "Generative AI" is a system that uses artificial intelligence technology to generate natural conversations and interactions with users.

[1324] "MR-GUI" is a graphical user interface that uses Mixed Reality technology, providing an interface that users can operate intuitively.

[1325] "Emotion recognition" is a technology that analyzes emotions from the user's tone of voice, facial expressions, etc. and obtains the results.

[1326] An "operation interface" is a visual or physical means by which a user operates an IoT device.

[1327] "Optimization" refers to adjusting something to the most effective state for specific conditions or purposes.

[1328] This invention is a system that enables direct communication with IoT devices using MR (Mixed Reality) technology. This system starts when a user wears MR-compatible goggles or a headset and visually recognizes the IoT device. Specifically, the user wears special goggles or a headset and visually recognizes the IoT device. At this time, a direct operation interface is displayed on the IoT device using MR technology.

[1329] Hardware and software used

[1330] Hardware: Mixed reality goggles or headset (e.g., Microsoft HoloLens)

[1331] Software: Emotion recognition engine (e.g., Affectiva SDK), IoT device control software (e.g., Home Assistant)

[1332] Data processing and calculation

[1333] 1. User visual recognition:

[1334] The user puts on MR-compatible goggles or a headset.

[1335] The cameras in the goggles or headset recognize surrounding IoT devices and use MR technology to display an operating interface on the device.

[1336] Example: The air conditioner itself displays a temperature control slider and airflow control buttons.

[1337] 2. Emotion recognition:

[1338] The user speaks to the IoT device.

[1339] The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize their emotions.

[1340] Example: If a user expresses an angry emotion, an emotion recognition engine will analyze that information.

[1341] 3. Operation reflection:

[1342] IoT devices are operated based on the analysis results of the emotion recognition engine.

[1343] Example: If the user is showing signs of anger, quickly change the air conditioning settings.

[1344] Specific examples

[1345] Example 1: Air conditioner operation

[1346] Scenario: A user wants to lower the temperature in their room.

[1347] Operation steps:

[1348] 1. The user puts on the MR compatible goggles.

[1349] 2. The goggles will recognize the air conditioner and a temperature control slider will appear on the air conditioner itself.

[1350] 3. The user operates the slider to lower the temperature.

[1351] 4. The device analyzes the user's facial expressions to confirm satisfaction.

[1352] 5. The server changes the air conditioning settings, lowering the room temperature.

[1353] Example 2: Lighting control

[1354] Scenario: A user wants to dim the lights in their room.

[1355] Operation steps:

[1356] 1. The user puts on the mixed reality headset.

[1357] 2. The headset will recognize the lighting and a button to adjust the lighting brightness will appear.

[1358] 3. The user presses a button to decrease the brightness.

[1359] 4. The device analyzes the user's tone of voice to ensure they are relaxed.

[1360] 5. The server changes the lighting settings and the room becomes less bright.

[1361] Prompt Sentence Examples

[1362] Prompt 1: "I want to turn down the air conditioner. If you show any signs of anger, I will change the setting immediately."

[1363] Prompt 2: "I want the lights in the room to be dimmed. If I'm relaxing, I slowly lower the brightness."

[1364] The above is an embodiment of the present invention.

[1365] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1366] Step 1:

[1367] The user puts on MR-compatible goggles or a headset.

[1368] Input: User puts on mixed reality goggles or headset.

[1369] Data processing: The goggles or headset activates and displays the MR environment in the user's field of vision.

[1370] Output: The mixed reality environment is displayed in the user's field of view.

[1371] Specific operation: When the user puts on the goggles and turns them on, the MR environment is displayed on the goggles' display.

[1372] Step 2:

[1373] The device recognizes surrounding IoT devices.

[1374] Input: Cameras in goggles or headset scan the surrounding environment.

[1375] Data processing: The device recognizes IoT devices (e.g., air conditioners, lighting) and obtains their location information.

[1376] Output: Location information of recognized IoT devices.

[1377] How it works: The goggles' camera scans the room and recognizes the air conditioner and lights.

[1378] Step 3:

[1379] The device displays the operating interface using MR technology.

[1380] Input: Location information of the recognized IoT device.

[1381] Data processing: Using MR technology, the device displays an operation interface on the IoT device it recognizes.

[1382] Output: Operation interface displayed on the IoT device.

[1383] Specific operation: The goggles' display will show an air conditioning temperature control slider and lighting brightness adjustment buttons.

[1384] Step 4:

[1385] The user operates the IoT device using the operation interface.

[1386] Input: The user operates a visually displayed control interface.

[1387] Data processing: IoT device settings are changed based on user actions.

[1388] Output: The modified IoT device configuration.

[1389] Specific action: The user moves their hand to operate a slider on the goggles' display to set the air conditioner temperature.

[1390] Step 5:

[1391] The device recognizes the user's emotions.

[1392] Input: The user's tone of voice and facial expressions.

[1393] Data processing: The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize their emotions.

[1394] Output: Perceived user emotion.

[1395] How it works: The goggles' camera captures the user's facial expressions, which are then analyzed by the emotion recognition engine.

[1396] Step 6:

[1397] The server optimizes the operation of IoT devices based on the emotion recognition results.

[1398] Input: Perceived user emotion.

[1399] Data processing: The server receives the emotion recognition results and determines the optimal operation based on the user's emotion.

[1400] Output: Optimized IoT device operation instructions.

[1401] Specific operation: The server issues instructions to immediately change the air conditioner settings based on the emotion recognition results.

[1402] (Application example 1)

[1403] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1404] Conventional IoT device operation methods require users to use smartphones or remote controls, which is often unintuitive. Furthermore, they lack the ability to take user emotions into account, resulting in a lack of user experience. Furthermore, even in physical stores, shopping requires manual searches to obtain detailed product information, which is inefficient.

[1405] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1406] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, and a means for analyzing the user's emotions using an emotion engine and reflecting the results in the operation of the IoT device. This allows the user to operate the IoT device intuitively and according to their emotions, improving the shopping experience in physical stores.

[1407] "MR (Mixed Reality)" is a technology that combines the real world and the virtual world, allowing users to experience both elements simultaneously.

[1408] "IoT devices" are electronic devices that are interconnected via the Internet and are capable of sending, receiving, and controlling data.

[1409] "Generative AI" is an artificial intelligence technology that generates natural conversations with users.

[1410] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[1411] The "emotion engine" is a technology that analyzes emotions from the user's tone of voice and facial expressions, and reflects the results in the system's operation.

[1412] A system for implementing the present invention includes the following major components:

[1413] 1. Hardware Configuration

[1414] MR-compatible goggles and headsets: These are devices worn by users that allow them to blend the real world with the virtual world.

[1415] Camera: A device for capturing images for visual recognition by the user.

[1416] Server: A computer system that processes data, runs generative AI, and executes emotion engines.

[1417] 2. Software Configuration

[1418] MRDisplay Library: Software for displaying graphical user interfaces using MR technology.

[1419] EmotionEngine Library: Software for analyzing emotions from a user's tone of voice and facial expressions.

[1420] Generative AI: An artificial intelligence technology that generates natural conversations with users.

[1421] OpenCV: A library for image processing.

[1422] 3. System Operation

[1423] The server recognizes products and IoT devices using images captured by the camera. Detailed information about the recognized products and IoT devices is displayed on MR-compatible goggles or headsets using the MRDisplay library. When the user talks to the products or IoT devices, the generative AI analyzes the conversation and generates an appropriate response. The EmotionEngine library also analyzes the user's emotions and reflects the results in the system's behavior.

[1424] 4. Specific Examples

[1425] For example, a user puts on MR-compatible goggles in a physical store and takes a photo of a product with the camera. The server recognizes the product and uses MR technology to display detailed product information on the goggles' display. When the user talks about the product, the generative AI analyzes the conversation and generates an appropriate response. Furthermore, the Emotion Engine library analyzes the user's emotions and makes appropriate product suggestions based on the results.

[1426] 5. Examples of prompts

[1427] "The user puts on MR-compatible goggles and takes a photo of a product with the camera. The system recognizes the product and uses MR technology to display detailed product information on the goggles' display. When the user talks about the product, the emotion engine analyzes their emotions from their tone of voice and facial expressions and makes appropriate product suggestions."

[1428] In this way, users can operate IoT devices intuitively and emotionally, improving the shopping experience in physical stores.

[1429] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1430] Step 1:

[1431] The user puts on MR-compatible goggles and takes a photo of a product or IoT device with the camera.

[1432] Input: Image data captured by a camera

[1433] Output: Image data is sent to the server

[1434] Specific operation: The user puts on the goggles and takes pictures of products and IoT devices that come into view with the camera. The captured image data is sent to the server in real time.

[1435] Step 2:

[1436] The server receives the image data and recognizes the product or IoT device.

[1437] Input: Image data sent from the camera

[1438] Output: ID of recognized product or IoT device

[1439] Specific operation: The server processes images using the OpenCV library and recognizes products and IoT devices. An ID is generated for the recognized products and IoT devices.

[1440] Step 3:

[1441] The server obtains detailed information about recognized products and IoT devices and displays it on the goggles using MR technology.

[1442] Input: ID of recognized product or IoT device

[1443] Output: Detailed information displayed on MR-compatible goggles

[1444] Specific operation: The server retrieves detailed information about products and IoT devices from the database and displays it on the goggles' display using the MRDisplay library.

[1445] Step 4:

[1446] Users talk to products and IoT devices.

[1447] Input: User's voice data

[1448] Output: The audio data is sent to the server.

[1449] Specific operation: The user talks to the product or IoT device while wearing the goggles. The voice data is sent to the server in real time.

[1450] Step 5:

[1451] The server analyzes the voice data and uses generative AI to generate an appropriate response.

[1452] Input: User's voice data

[1453] Output: The generated response

[1454] How it works: The server uses generative AI to analyze the user's voice data and generate an appropriate response, which is then sent to the goggles.

[1455] Step 6:

[1456] The server uses an emotion engine to analyze the user's emotions and makes appropriate product suggestions based on the results.

[1457] Input: User's voice data and facial expression data

[1458] Output: Sentiment analysis results and product recommendations

[1459] Specific operation: The server uses the EmotionEngine library to analyze the user's emotions from their voice and facial expression data. Based on the analysis results, it generates appropriate product suggestions and displays them on the goggles.

[1460] Step 7:

[1461] The user reviews the suggested products and takes action as necessary.

[1462] Input: Details of the proposed product

[1463] Output: User operation instructions

[1464] Specific operation: The user checks the details of the suggested products displayed on the goggles and performs operations as necessary. Operation instructions are sent to the server in real time.

[1465] Example 2

[1466] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1467] Conventional IoT home appliance control systems have faced the challenge of making it difficult for users to operate home appliances in an intuitive and natural way. In particular, when operating using voice input, it is necessary to accurately understand and appropriately reflect the user's instructions. Furthermore, there is also the problem that the user experience is limited because the system is unable to take the user's emotions into account.

[1468] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1469] In this invention, the server includes a means for a user to input voice, a means for a terminal to acquire voice data and send it to the server, a means for the server to convert the voice data to text, a means for the server to analyze instructions using a generative AI model, a means for the server to analyze emotions using an emotion engine, a means for the server to send operation commands to the IoT home appliance, and a means for the IoT home appliance to operate in accordance with the operation commands. This allows the user to operate the IoT home appliance in an intuitive and natural way, and also enables operation that takes the user's emotions into consideration.

[1470] "User" refers to a person who uses the system to operate IoT home appliances.

[1471] "Voice input" refers to the act of a user giving instructions to a system using voice.

[1472] "Terminal" refers to a device for acquiring and transmitting voice data to a server.

[1473] "Voice data" refers to a digital recording of a user's voice input.

[1474] A "server" refers to a computer system that processes voice data, analyzes instructions using generative AI models and emotion engines, and sends operational commands to IoT home appliances.

[1475] "Text data" refers to data obtained by converting voice data into character information.

[1476] A "generative AI model" refers to an artificial intelligence model that analyzes user instructions and generates appropriate operation commands.

[1477] An "emotion engine" refers to software that analyzes emotions from the content of a user's speech.

[1478] An "operation command" refers to an instruction to instruct an IoT home appliance to perform a specific action.

[1479] "IoT home appliances" refer to home appliances that are connected to the Internet and can be controlled remotely.

[1480] "Behavior" refers to the specific action that an IoT home appliance performs in accordance with an operation command.

[1481] This invention relates to a system for operating IoT home appliances by having a user input voice data, a terminal acquiring the voice data, and transmitting the data to a server. A specific embodiment of this system will be described below.

[1482] Hardware and software used

[1483] Hardware: IoT home appliances (e.g., air conditioners, lighting, smart speakers)

[1484] Software: Generative AI models (e.g., GPT-4), emotion engines (e.g., Affectiva), speech recognition software (e.g., Google Speech-to-Text)

[1485] Data processing and calculation

[1486] 1. Acquiring voice input:

[1487] The user speaks commands into the smart speaker, such as "heat the room" or "brighten the room."

[1488] Specific action: A user sits in the living room and asks their smart speaker to "heat the room."

[1489] 2. Acquiring and sending audio data:

[1490] The device (smart speaker) captures the user's voice with a microphone and saves it as voice data.

[1491] The device sends the audio data to a server via the Internet.

[1492] How it works: The smart speaker records the user's voice and sends the data to a cloud server via Wi-Fi.

[1493] 3. Audio to text conversion:

[1494] The server uses speech recognition software (e.g., Google Speech-to-Text) to convert the voice data into text data.

[1495] What happens: The server receives the voice data and calls speech recognition software to generate the text "heat the room."

[1496] 4. Generative AI instruction analysis:

[1497] The server uses a generative AI model (e.g., GPT-4) to parse user instructions from text data.

[1498] The generative AI model analyzes the instruction "heat the room" and generates an operation command to operate the air conditioner in heating mode.

[1499] Specific operation: The server inputs the text "heat the room" into the generative AI model, which generates a command to instruct the air conditioner to operate in heating mode.

[1500] 5. Emotion analysis using emotion engine:

[1501] The server uses an emotion engine (e.g., Affectiva) to analyze the emotion from the user's speech.

[1502] The emotion engine detects emotions such as joy or anger from the user's tone of voice and language.

[1503] Specific operation: The server inputs the text "warm the room" into the emotion engine and analyzes that the user is relaxed.

[1504] 6. Sending operation commands to IoT appliances:

[1505] The server generates operation commands and sends them to the IoT appliances.

[1506] Specific operation: The server sends a command to the air conditioner to "operate in heating mode."

[1507] 7. IoT appliances in action:

[1508] The IoT appliance (e.g., air conditioner) begins to operate according to the received operation command.

[1509] Specific operation: The air conditioner starts operating in heating mode and warms the room.

[1510] Specific examples

[1511] Example 1: When the user says "Heat the room"

[1512] A user sits in the living room and asks a smart speaker to "heat the room."

[1513] The smart speaker records the audio and sends it to a server.

[1514] The server converts the voice data into text and analyzes it using a generative AI model.

[1515] The server sends a command to the air conditioner to "operate in heating mode."

[1516] The air conditioner will start operating in heating mode and warm the room.

[1517] Example 2: User expresses happiness and says "Turn on the lights in the room"

[1518] A user sits in the living room and asks a smart speaker to "brighten the room."

[1519] The smart speaker records the audio and sends it to a server.

[1520] The server converts the voice data into text and analyzes it using a generative AI model.

[1521] The server uses an emotion engine to analyze the user's emotions of joy.

[1522] The server sends a command to the light to "turn it on" and adds emotional information.

[1523] The lights come on brightly, lighting up the room.

[1524] Prompt Sentence Examples

[1525] "When the user says 'heat the room,' generate an operation command to operate the air conditioner in heating mode."

[1526] "If the user expresses happiness and says 'Turn on the lights in the room,' generate an operation command to turn on the lights and attach the emotional information."

[1527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1528] Step 1:

[1529] The user provides voice input.

[1530] Specific action: The user speaks commands into the smart speaker, such as "heat the room" or "light up the room."

[1531] Input: User's voice command.

[1532] Output: Audio data.

[1533] Step 2:

[1534] The terminal acquires the voice data and transmits it to the server.

[1535] How it works: The smart speaker captures the user's voice with a microphone and saves it as audio data. The device then sends the audio data to a server via the internet.

[1536] Input: Audio data.

[1537] Output: The audio data sent to the server.

[1538] Step 3:

[1539] The server converts the audio data into text.

[1540] What happens: The server uses speech recognition software (e.g., Google Speech-to-Text) to convert the voice data into text data.

[1541] Input: The audio data sent to the server.

[1542] Output: Text data.

[1543] Step 4:

[1544] The server analyzes the instructions using a generative AI model.

[1545] How it works: The server uses a generative AI model (e.g., GPT-4) to analyze the user's instructions from the text data. The generative AI model analyzes the instruction "heat the room" and generates an operation command to operate the air conditioner in heating mode.

[1546] Input: Text data.

[1547] Output: Operation command.

[1548] Step 5:

[1549] The server analyzes emotions using an emotion engine.

[1550] How it works: The server uses an emotion engine (e.g., Affectiva) to analyze the emotions from the user's speech. The emotion engine detects emotions such as joy or anger from the user's tone of voice and choice of words.

[1551] Input: Text data.

[1552] Output: Emotion analysis results.

[1553] Step 6:

[1554] The server sends operation commands to the IoT appliances.

[1555] Specific operation: The server generates an operation command and sends it to the IoT appliance.

[1556] Input: Operation command, emotion analysis result.

[1557] Output: Operation commands sent to IoT appliances.

[1558] Step 7:

[1559] IoT home appliances operate according to operational commands.

[1560] Specific operation: The IoT appliance (e.g., air conditioner) begins to operate according to the received operation command.

[1561] Input: Operation command sent to IoT appliances.

[1562] Output: IoT appliance operation.

[1563] (Application example 2)

[1564] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1565] Conventional IoT device interfaces have been difficult for users to operate intuitively and have had issues with not being able to provide appropriate information based on emotions. Furthermore, there has been a lack of interfaces that allow users to directly talk to IoT devices, limiting the user experience.

[1566] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1567] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, a means for recognizing the user's emotions and providing information according to those emotions, and a means for displaying information to the user using smart glasses. This allows the user to not only intuitively operate the IoT device but also receive appropriate information according to their emotions.

[1568] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1569] An "IoT device" is an electronic device that can communicate with other devices and systems via the Internet and send and receive data and perform other control functions.

[1570] "Generative AI" is an artificial intelligence technology that generates responses in natural language based on user input.

[1571] "MR-GUI" is a graphical user interface that uses Mixed Reality technology and allows users to operate it intuitively.

[1572] "Emotion recognition" is a technology that analyzes and recognizes emotions from a user's speech, facial expressions, behavior, etc.

[1573] "Smart glasses" are wearable devices that can visually display information to users when worn.

[1574] As an embodiment of the present invention, a smart shopping assistant system will be described as an example. This system is configured using MR (Mixed Reality) technology, generative AI, MR-GUI, emotion recognition, and smart glasses.

[1575] System configuration

[1576] 1. Hardware

[1577] Smart glasses: Wearable devices worn by the user that visually display information. An example is Google Glass.

[1578] Server: A computer system that runs generative AI models and emotion recognition engines.

[1579] 2. Software

[1580] Generative AI model: An artificial intelligence technology that generates responses in natural language based on user input. For example, OpenAI's API is used.

[1581] Emotion recognition engine: A technology that analyzes and recognizes emotions from a user's speech, facial expressions, actions, etc. An example is the Emotion Recognition API.

[1582] IoT Device Control API: Software for controlling IoT devices.

[1583] System Operation

[1584] 1. Processing User Input

[1585] When a user speaks into the smart glasses, the voice input is converted into text.

[1586] The server uses an emotion recognition engine to analyze emotions from the user's speech.

[1587] 2. Response generation using generative AI

[1588] The server uses a generative AI model to generate appropriate responses based on user input and emotions.

[1589] The generated response is displayed on the smart glasses.

[1590] 3. Controlling IoT devices

[1591] Based on the generated response, the server operates the IoT device as needed using the IoT device control API.

[1592] Specific examples

[1593] When a user says, "Tell me about this product," the smart glasses will display, "This product is made from high-quality materials."

[1594] When a user says, "I feel tired today," the smart glasses will display, "We have aroma candles and massage services to help you relax."

[1595] Prompt Sentence Examples

[1596] User Input: Tell us about this product

[1597] Emotion: Interest

[1598] Generate an appropriate response.

[1599] In this way, the smart shopping assistant system can provide appropriate information according to the user's questions and emotions, improving the shopping experience in physical stores.

[1600] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1601] Step 1:

[1602] When a user speaks into the smart glasses, the voice input is converted into text. The smart glasses use voice recognition software to convert the voice data into text, which then becomes the input for the next processing step.

[1603] Step 2:

[1604] The server receives the text data and uses an emotion recognition engine to analyze the user's emotions from the user's speech. The emotion recognition engine analyzes the text data and identifies the user's emotions (e.g., interest, fatigue, joy, etc.). This emotion data becomes the input for the next processing step.

[1605] Step 3:

[1606] The server uses a generative AI model to generate an appropriate response based on the user's input text and emotion data. The generative AI model creates a prompt and generates a response in natural language based on the user's input and emotion. This generated response serves as input for the next processing step.

[1607] Step 4:

[1608] The server transmits the generated response to the smart glasses, which visually display the response to the user, and the smart glasses display the received text data on a display to provide information to the user.

[1609] Step 5:

[1610] The server uses the IoT device control API to operate the IoT device as needed based on the generated response. For example, if the user says, "Heat the room," the server instructs the air conditioner to operate in heating mode. This instruction is sent to the IoT device, and the IoT device operates appropriately.

[1611] In this way, the smart shopping assistant system can provide appropriate information and operate IoT devices according to the user's questions and emotions.

[1612] Example 3

[1613] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1614] Conventional IoT home appliance operation systems have the problem that they are difficult for users to operate intuitively and cannot respond to the user's emotions. Furthermore, there is a demand for systems that allow users to speak directly to IoT home appliances and reflect emotion-based operations. This will improve the user experience and provide a more comfortable living environment.

[1615] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1616] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliances using generative AI, a means for the user to intuitively and freely operate the IoT home appliances using MR-GUI, and a means for analyzing emotions from the user's operating actions and reflecting the results in the operation of the IoT home appliances. This allows the user to not only intuitively operate the IoT home appliances but also to operate them according to their emotions, providing a more comfortable living environment.

[1617] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both elements simultaneously.

[1618] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[1619] "Generative AI" is a type of artificial intelligence technology that refers to algorithms and models for interacting with users and generating data.

[1620] "MR-GUI" is a graphical user interface displayed using MR technology, and is designed to allow users to operate it intuitively.

[1621] An "emotion engine" is software or an algorithm that analyzes emotions from a user's operations, facial expressions, etc., and outputs the results.

[1622] An "operation panel" is an interface that a user uses to operate a home appliance, and includes operation elements such as sliders and buttons.

[1623] "Communication means" refers to the technologies and protocols used to send and receive data, including the Internet and wireless communications.

[1624] "Analysis results" refers to information about the user's emotions analyzed by the emotion engine.

[1625] "Operation instructions" refer to specific operation commands for IoT home appliances that the server generates based on the analysis results.

[1626] This invention is a system that allows users to intuitively and freely operate IoT home appliances through a GUI (Graphical User Interface) using MR (Mixed Reality) technology. Furthermore, it includes an emotion engine that recognizes the user's emotions, enabling operation according to the user's emotions.

[1627] Hardware and software used

[1628] Hardware: Mixed reality device (e.g. head-mounted display)

[1629] Software: MR-GUI, emotion engine, IoT home appliance control software

[1630] Data processing and calculation

[1631] 1. Display MR-GUI:

[1632] The server sends data to the MR device to display the air conditioner's control panel.

[1633] Based on the data received, the terminal (MR device) displays a temperature control slider, airflow control buttons, operation mode switching buttons, etc.

[1634] 2. User Action Recognition:

[1635] The user directly operates the displayed control panel through the MR device.

[1636] The terminal (MR device) detects the user's operations and sends that information to the server.

[1637] 3. Emotion Analysis:

[1638] The server uses an emotion engine to analyze emotions from the user's operations.

[1639] For example, if a user expresses surprise, the emotion engine outputs that information as an analysis result.

[1640] 4. Controlling IoT appliances:

[1641] Based on the analysis results, the server generates operating instructions for the IoT home appliances.

[1642] For example, if the user shows signs of surprise, instructions to operate the air conditioner gently are generated.

[1643] The server sends the generated operation instructions to the IoT appliance, which then operates according to those instructions.

[1644] Specific examples

[1645] Example 1: When a user adjusts the temperature of an air conditioner

[1646] The user wears the MR device and the air conditioner control panel is displayed in their field of vision.

[1647] The user operates the temperature control slider to set the desired temperature.

[1648] The terminal (MR device) sends operation information to the server, and the server sends an instruction to change the temperature to the air conditioner.

[1649] The IoT appliance (air conditioner) will follow instructions and change the set temperature.

[1650] Example 2: When a user expresses surprise

[1651] The user expresses surprise while operating the air conditioner control panel.

[1652] The terminal (MR device) transmits the user's emotional data along with the operation information to the server.

[1653] The server uses an emotion engine to analyze the emotion of surprise and generate instructions to operate the air conditioner gently.

[1654] IoT home appliances (air conditioners) follow instructions and operate gently.

[1655] Prompt Sentence Examples

[1656] Prompt 1: "Describe a scenario in which a user adjusts the temperature of an air conditioner through a mixed reality device."

[1657] Prompt 2: "Describe how an IoT appliance operates when a user expresses surprise."

[1658] This system allows users to intuitively and emotionally respond to their operations, providing a more comfortable experience using IoT home appliances. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1659] Step 1:

[1660] MR-GUI display

[1661] The server generates data to display the air conditioner control panel and sends it to the MR device.

[1662] Input: Air conditioner control panel configuration data

[1663] Data processing: Converting the configuration data of the operation panel into a format that can be interpreted by the MR device

[1664] Output: Operation panel data sent to the MR device

[1665] The terminal (MR device) displays the operation panel based on the received data.

[1666] Input: Operation panel data sent from the server

[1667] Data calculation: Rendering process for visually displaying operation panel data

[1668] Output: The control panel that is visible to the user

[1669] Specific operation: The MR device displays temperature control sliders, airflow control buttons, operation mode switching buttons, etc. in the user's field of vision.

[1670] Step 2:

[1671] Recognizing user actions

[1672] The user directly operates the operation panel displayed through the MR device.

[1673] Input: Operation panel displayed on MR device

[1674] Data processing: Tracking the user's hand movements and eye gaze position

[1675] Output: Operation information

[1676] The terminal (MR device) detects the user's operations and sends the information to the server.

[1677] Input: User operation information

[1678] Data calculation: Converts the operation information into a format that can be interpreted by the server

[1679] Output: Operation information sent to the server

[1680] Specific operation: The MR device tracks the user's hand movements and gaze and sends them to the server as operation information.

[1681] Step 3:

[1682] Emotion Analysis

[1683] The server uses an emotion engine to analyze emotions from the user's operations.

[1684] Input: User operation information

[1685] Data calculation: Analyzes operation speed and movement patterns and inputs them into the emotion engine

[1686] Output: Emotion analysis results

[1687] The emotion engine analyzes the user's emotions and returns the results to the server.

[1688] Input: Operation speed and movement pattern

[1689] Data Computing: Identifying emotions using sentiment analysis algorithms

[1690] Output: Emotion analysis results

[1691] Specific behavior: The emotion engine determines whether the user is expressing surprise emotion and returns the result to the server.

[1692] Step 4:

[1693] Controlling IoT appliances

[1694] The server generates operating instructions for IoT home appliances based on the analysis results.

[1695] Input: Sentiment analysis results

[1696] Data calculation: Generate operational instructions based on the results of sentiment analysis

[1697] Output: Operation instructions sent to IoT appliances

[1698] The server sends the generated operation instructions to the IoT home appliance.

[1699] Input: Operation instructions

[1700] Data processing: Converting operation instructions into a format that can be interpreted by IoT appliances

[1701] Output: Operation instructions sent to IoT appliances

[1702] The IoT home appliance (air conditioner) operates according to the instructions it receives.

[1703] Input: Operation instructions sent from the server

[1704] Data calculation: Controlling operations based on operational instructions

[1705] Output: Actual air conditioner operation

[1706] Specific behavior: The air conditioner gradually changes the set temperature and operates according to the user's emotions.

[1707] (Application example 3)

[1708] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1709] Conventional IoT home appliance operation systems have the problem that they are difficult for users to operate intuitively and do not take into account the user's emotions. Furthermore, there are also a lack of easy-to-use methods for users to operate IoT devices in physical stores. This has led to a poor user experience and the inability to fully utilize the convenience of IoT home appliances and devices.

[1710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1711] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliance using generative AI, a means for the user to intuitively and freely operate the IoT home appliance using MR-GUI, a means for recognizing the user's emotions and adjusting the operation of the IoT home appliance according to those emotions, and a means for operating IoT devices in a physical store using smart glasses. This allows the user to intuitively operate IoT home appliances and devices and further enables operation according to emotions, improving the user experience and maximizing the convenience of IoT home appliances and devices.

[1712] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1713] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled or automatically controlled.

[1714] "Generative AI" is a type of artificial intelligence that has the ability to generate natural conversations and interactions with users.

[1715] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[1716] "Emotion recognition" is a technology that analyzes a user's facial expressions and movements to identify their emotional state.

[1717] "Smart glasses" are wearable devices that have built-in displays, cameras, and sensors, allowing users to operate them while visually checking information.

[1718] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[1719] An "IoT device" is a piece of equipment or device that is connected to the Internet and can send and receive data and be remotely controlled.

[1720] A system for implementing this invention is configured using MR (Mixed Reality) technology, generative AI, MR-GUI, emotion recognition technology, and smart glasses. Specific embodiments of each element are described below.

[1721] System configuration

[1722] 1. Hardware

[1723] Smart glasses: A wearable device equipped with a display, camera, and sensors that allows users to visually view information and operate the device.

[1724] Camera: Used to capture the user's facial expressions and movements.

[1725] IoT devices: Lighting, displays, sound systems, and other equipment that are connected to the internet and can be controlled remotely.

[1726] 2. Software

[1727] OpenCV: An image processing library used to analyze images captured by the camera.

[1728] EmotionRecognizer: An emotion recognition library that analyzes emotions from the user's facial expressions and movements.

[1729] IoTDeviceController: A library for controlling IoT devices.

[1730] MRGUI: A library that provides a graphical user interface displayed using MR technology.

[1731] Processing flow

[1732] The server captures the user's face using the camera built into the smart glasses and processes the image using OpenCV. Next, it recognizes the user's emotions using EmotionRecognizer. The recognized emotional information is sent to the MRGUI and displayed on the smart glasses' display. The user visually checks the operation panel of the IoT device through the MRGUI and operates it. The user's operation information is sent to the IoTDeviceController, which adjusts the operation of the IoT device based on the emotional information.

[1733] Specific examples

[1734] For example, if a user wears smart glasses in a physical store and adjusts the brightness of the lights, if the user expresses surprise, the brightness of the lights will be adjusted more gently. This allows users to operate IoT devices intuitively and enables operation according to their emotions, improving the user experience.

[1735] Prompt Sentence Examples

[1736] "If a user wearing smart glasses operates an MR-GUI to adjust the brightness of lights in a physical store and displays an emotion of surprise, design a system that adjusts the brightness of the lights gently."

[1737] The above is a specific embodiment for carrying out the present invention.

[1738] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1739] Step 1:

[1740] The server captures the user's face using the camera built into the smart glasses. The input is the camera image, and the output is the captured image data. This image data becomes the basis for analyzing the user's facial expressions and movements.

[1741] Step 2:

[1742] The server processes the captured image data using OpenCV. The input is the image data, and the output is preprocessed image data. This preprocessing includes image resizing and noise removal.

[1743] Step 3:

[1744] The server inputs the preprocessed image data into EmotionRecognizer to recognize the user's emotions. The input is the preprocessed image data, and the output is the user's emotional information (e.g., surprise, joy, sadness, etc.). EmotionRecognizer extracts features from the image data and classifies the emotions.

[1745] Step 4:

[1746] The server sends the recognized emotion information to the MRGUI and displays it on the display of the smart glasses. The input is emotion information, and the output is an operation panel displayed on the MRGUI. The user visually checks this operation panel and performs operations.

[1747] Step 5:

[1748] The user operates the operation panel of the IoT device through the MRGUI. The input is the user's operation information, and the output is the operation command. The user changes the settings of the IoT device using the buttons and sliders on the operation panel.

[1749] Step 6:

[1750] The server controls IoT devices using IoTDeviceController based on the user's operation information and emotional information. The input is the operation command and emotional information, and the output is the behavior of the IoT device. For example, if the user shows surprise, the brightness of the lights is gently adjusted.

[1751] Step 7:

[1752] The server feeds back the operation results of the IoT device to the user. The input is the operation result of the IoT device, and the output is the feedback information. The user can check the current status of the IoT device through the display of the smart glasses.

[1753] The above are the specific processing steps of this system.

[1754] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1755] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1756] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1757] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1758] [Third embodiment]

[1759] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1760] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1761] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1762] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1763] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1764] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1765] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1766] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1767] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1768] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1769] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1770] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1771] "Example 1"

[1772] One embodiment of the present invention is a system that uses mixed reality (MR) technology to enable direct communication with IoT home appliances. Specifically, a user wears special goggles or a headset to visually recognize the IoT home appliance. At this time, MR technology displays a direct operation interface on the IoT home appliance. For example, in the case of an air conditioner, a temperature control slider and airflow control buttons are displayed on the air conditioner body using MR, and the user can change the air conditioner settings by directly operating these.

[1773] "Example 2"

[1774] Furthermore, one embodiment of the present invention is a system that uses generative AI to enable conversations between users and IoT appliances. Specifically, when a user speaks to an IoT appliance, the generative AI understands the instruction and transmits the appropriate operation to the IoT appliance. For example, if a user says, "Heat the room," the generative AI understands the instruction and instructs the air conditioner to operate in heating mode.

[1775] "Example 3"

[1776] Another embodiment of the present invention is a system that allows users to intuitively and freely operate IoT home appliances using an MR-GUI. Specifically, users operate IoT home appliances using a GUI displayed using MR technology. For example, an air conditioner's control panel displayed using MR displays a temperature control slider, buttons for adjusting airflow, and buttons for switching operation modes, allowing users to freely change the air conditioner's settings by directly operating these.

[1777] The processing flow of each embodiment will be described below.

[1778] "Example 1"

[1779] Step 1: The user puts on the MR-enabled goggles or headset.

[1780] Step 2: The user visually recognizes the IoT appliance. At this time, a direct operation interface is displayed on the IoT appliance using mixed reality technology.

[1781] Step 3: The user changes the settings of the IoT appliance by directly operating the operation interface displayed in MR.

[1782] "Example 2"

[1783] Step 1: The user talks to the IoT appliance.

[1784] Step 2: The generative AI understands the user's instructions.

[1785] Step 3: The generative AI communicates the understood instructions to the IoT appliance, which then performs the operation according to the instructions.

[1786] "Example 3"

[1787] Step 1: The user puts on the MR-enabled goggles or headset.

[1788] Step 2: The user operates the IoT appliance using the GUI displayed by MR technology.

[1789] Step 3: The user can freely change the settings of the IoT home appliance by directly operating the control panel displayed in MR.

[1790] Example 1

[1791] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1792] Conventional methods of operating IoT devices typically involve using smartphones or remote controls, which require unintuitive operation and complex settings. Furthermore, there are limited ways for users to directly communicate with IoT devices, creating issues with operability and convenience. Furthermore, the transmission and analysis of real-time operational information is insufficient, making rapid control difficult. To resolve these issues, a system that allows for more intuitive and flexible operation is needed.

[1793] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1794] In this invention, the server includes means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, means for enabling conversation between the user and the IoT device using generative AI, means for the user to intuitively and freely operate the IoT device using MR-GUI, means for visually recognizing the IoT device using a visual device worn by the user and displaying an operation interface, and means for transmitting user operation information to the server in real time, which then analyzes the operation information and transmits control signals to the corresponding IoT device. This allows the user to intuitively and freely operate the IoT device, achieving rapid control in real time.

[1795] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1796] An "IoT device" is an electronic device that can communicate with other devices and systems via the Internet and send, receive, and control data.

[1797] "Generative AI" is a type of artificial intelligence technology that refers to algorithms and models for interacting with users and generating data.

[1798] "MR-GUI" is a graphical user interface that uses Mixed Reality technology and is designed to allow users to operate it intuitively.

[1799] "Visual devices" are devices worn by users that provide visual information and blend the real world with the virtual world. Examples include goggles and headsets.

[1800] An "operation interface" is a visual control element that users use to operate IoT devices, including sliders and buttons.

[1801] A "server" is a computer system that processes and manages data on a network, and is responsible for analyzing user operation information and sending control signals to corresponding IoT devices.

[1802] A "control signal" is a signal sent from a server to an IoT device, and includes instructions for changing the device's operation or settings.

[1803] This invention is a system that enables direct communication with IoT devices using Mixed Reality (MR) technology. The user wears special visual devices (goggles or headset) and visually recognizes the IoT devices. The visual devices use MR technology to display the IoT devices in the user's field of vision and overlay an operation interface on top of them.

[1804] Hardware and software used

[1805] Visual equipment: goggles or headset

[1806] Image recognition software: OpenCV, etc.

[1807] MR development platform: Unity, etc.

[1808] Server-side programming languages: Python, Node.js, etc.

[1809] Communication technologies: Wi-Fi, Bluetooth, MQTT, HTTP, etc.

[1810] Explanation of program processing

[1811] When a user wears the vision device, the device recognizes objects in the user's field of vision and identifies IoT devices. For example, to recognize an air conditioner in the living room, the device uses the goggles' built-in camera and image recognition software (e.g., OpenCV). The device displays an operation interface overlaid on the recognized IoT device. In the case of an air conditioner, a slider for adjusting the temperature and buttons for adjusting the airflow are displayed. This interface is generated using an MR development platform such as Unity.

[1812] The user operates the device with their hands or eyes while looking at the operation interface displayed in their field of vision. For example, they change the temperature setting by dragging a temperature control slider with their finger. The device captures the user's operation information and transmits it to the server in real time. This communication uses wireless communication technologies such as Wi-Fi and Bluetooth.

[1813] The server analyzes the received operation information and generates an appropriate control signal for the corresponding IoT device. For example, if a user sets the temperature control slider to 25 degrees, the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. This process uses server-side programming languages ​​such as Python and Node.js. The IoT device receives the control signal sent from the server and changes the setting. In the case of an air conditioner, its internal control system changes the temperature setting to 25 degrees. Protocols such as MQTT and HTTP are used for this communication.

[1814] Specific examples

[1815] A user wants to operate the air conditioner in their living room. The user puts on MR goggles and visually recognizes the air conditioner in the living room. Using MR technology, a temperature control slider and airflow control buttons are displayed on the air conditioner itself. The user operates the temperature control slider to change the set temperature to 25 degrees. This operation information is sent from the goggles to the server, and the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. The air conditioner receives this signal, and its internal control system changes the temperature setting to 25 degrees.

[1816] Prompt Sentence Examples

[1817] "Describe a scenario in which a user puts on mixed reality goggles and operates the air conditioner in their living room. Please describe in detail the process by which the user operates the temperature control slider to change the set temperature to 25 degrees."

[1818] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1819] Step 1:

[1820] The user puts on the MR goggles.

[1821] Input: User puts on MR goggles and turns them on.

[1822] Output: The goggles are up and ready to capture the user's view.

[1823] How it works: The user puts on the MR goggles and presses the power button to turn them on. The camera built into the goggles begins capturing the user's field of vision.

[1824] Step 2:

[1825] The terminal displays IoT devices in the user's field of view.

[1826] Input: Video data captured by the goggles' built-in camera.

[1827] Output: The IoT device is displayed in the user's field of view.

[1828] Specific operation: The device analyzes the video data from the camera and identifies IoT devices using image recognition software (e.g., OpenCV). Identified IoT devices are displayed in the user's field of view.

[1829] Step 3:

[1830] The terminal displays the operation interface.

[1831] Input: Location and type of IoT device.

[1832] Output: The operation interface is displayed overlaid on the IoT device.

[1833] Specific operation: The terminal uses an MR development platform (e.g., Unity) to generate and display operation interfaces such as temperature control sliders and airflow control buttons on the IoT device.

[1834] Step 4:

[1835] The user operates the operation interface.

[1836] Input: User's hand and eye movements.

[1837] Output: Operation information is generated.

[1838] Specific actions: The user operates the interface displayed in the field of view with their hands or eyes. For example, they change the temperature setting by dragging the temperature control slider with their finger.

[1839] Step 5:

[1840] The terminal transmits operation information to the server.

[1841] Input: User operation information.

[1842] Output: Operation information is sent to the server.

[1843] How it works: The device captures user operation information and transmits it to the server in real time. This communication is carried out using wireless communication technologies such as Wi-Fi and Bluetooth.

[1844] Step 6:

[1845] The server analyzes the operation information and sends a control signal to the corresponding IoT device.

[1846] Input: Operation information sent from the device.

[1847] Output: Control signal for IoT devices.

[1848] How it works: The server analyzes the received operation information and generates the appropriate control signal for the corresponding IoT device. For example, if a user sets the temperature control slider to 25 degrees, the server sends a signal to the air conditioner to change the temperature setting to 25 degrees. This process is performed using server-side programming languages ​​such as Python and Node.js.

[1849] Step 7:

[1850] The IoT device receives the control signal and changes its settings.

[1851] Input: Control signal sent from the server.

[1852] Output: The settings of the IoT device are changed.

[1853] Specific operation: The IoT device receives a control signal sent from the server and changes the setting. In the case of an air conditioner, the internal control system changes the temperature setting to 25 degrees. Protocols such as MQTT and HTTP are used for this communication.

[1854] (Application example 1)

[1855] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1856] Conventional IoT home appliances are typically operated via smartphones or remote controls, which makes intuitive operation difficult. Furthermore, there are limited ways to efficiently obtain product and promotion information in physical stores, creating a need to improve the user shopping experience.

[1857] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1858] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT home appliances using generative AI, a means for the user to intuitively and freely operate the IoT home appliances using MR-GUI, and a means for visually displaying product information and promotion information in a physical store using smart glasses and providing an interface that the user can operate directly. This not only enables the user to intuitively operate the IoT home appliances, but also improves the shopping experience in the physical store.

[1859] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1860] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[1861] "Generative AI" is an artificial intelligence technology that generates appropriate responses and actions based on user input and environmental data.

[1862] "MR-GUI" is a graphical user interface displayed using MR technology, which allows users to operate it intuitively.

[1863] "Smart glasses" are wearable devices equipped with a display, camera, and sensors that allow users to experience visual information as augmented reality.

[1864] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[1865] "Product information" refers to information necessary for users to make purchasing decisions, such as product price, specifications, and stock status.

[1866] "Promotion information" refers to information aimed at promoting product sales, such as discounts and campaigns.

[1867] An "interface" is the means or method by which a user interacts with a system or device.

[1868] A system for implementing the present invention is configured as follows.

[1869] First, the server uses mixed reality (MR) technology to enable direct communication with IoT appliances. When a user puts on smart glasses and visually recognizes an IoT appliance, MR technology displays an operation interface on the appliance. For example, in the case of an air conditioner, the temperature control slider and airflow control buttons are displayed on the air conditioner body using MR, allowing the user to change the air conditioner settings by directly operating them.

[1870] Next, generative AI is used to enable conversations between users and IoT appliances. When a user gives a command via voice, the generative AI analyzes the command and executes the appropriate operation. For example, if the command is "Set the air conditioner temperature to 25 degrees," the generative AI will understand the command and set the temperature to 25 degrees.

[1871] Furthermore, MR-GUI allows users to intuitively and freely operate IoT home appliances. MR-GUI provides a visual interface that allows users to easily change various settings of home appliances.

[1872] The smart glasses also visually display product and promotion information from physical stores, providing an interface that users can operate directly. When a user looks at a specific product in a physical store, the camera in the smart glasses recognizes it and displays product and promotion information. Users can tap on this interface to check detailed information or complete the purchase process.

[1873] The following hardware and software are required to realize this system: For hardware, smart glasses (e.g., Microsoft HoloLens) and IoT home appliances are required. For software, OpenCV (image processing), TensorFlow / Keras (AI model for product recognition), and ARToolkit (MR interface generation) are used.

[1874] As a concrete example, imagine a user wearing smart glasses in a brick-and-mortar store. When the user looks at a particular product (e.g., a refrigerator), the smart glasses' camera recognizes the product and displays an MR interface. The interface displays the product's price, availability, and promotional information, and the user can tap to view more information or check out.

[1875] An example prompt is, "When you wear the smart glasses and look at a product in a physical store, detailed information and promotional information about the product will be displayed as an MR interface. Users can tap on this interface to view more information about the product or to complete the purchase."

[1876] In this way, users can not only intuitively operate IoT home appliances, but also improve their shopping experience in physical stores.

[1877] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1878] Step 1:

[1879] A user wears smart glasses and walks around a physical store. The camera in the smart glasses captures the surroundings in real time. The input is the camera image, and the output is the captured video data.

[1880] Step 2:

[1881] The device (smart glasses) inputs the captured video data into an AI model for product recognition. The AI ​​model uses TensorFlow / Keras to recognize products from the video data. The input is the video data, and the output is the ID of the recognized product.

[1882] Step 3:

[1883] The server receives the ID of the recognized product and retrieves the corresponding product information and promotion information from the database. The input is the product ID, and the output is the product information and promotion information.

[1884] Step 4:

[1885] Based on the product information and promotion information acquired by the server, an MR interface is generated using ARToolkit. The input is the product information and promotion information, and the output is the MR interface.

[1886] Step 5:

[1887] The device (smart glasses) displays the generated MR interface in the user's field of view. The input is the MR interface, and the output is the interface displayed in the user's field of view.

[1888] Step 6:

[1889] The user operates the MR interface to check detailed information and carry out purchase procedures. The input is the user's operation, and the output is the operation result.

[1890] Step 7:

[1891] The terminal (smart glasses) sends the results of the user's operations to the server, which then executes the necessary processing. For example, in the case of a purchase, the server works with an electronic payment system to complete the payment. The input is the result of the user's operations, and the output is the processing result.

[1892] Step 8:

[1893] The server sends the processing results to the device (smart glasses) and notifies the user. The input is the processing results, and the output is the notification to the user.

[1894] In this way, users can intuitively obtain and manipulate product information and promotional information in physical stores through smart glasses.

[1895] Example 2

[1896] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1897] Conventional methods of operating IoT home appliances require users to use dedicated remote controls or smartphone apps, which often require unintuitive operation. While systems using voice recognition technology exist, there are issues with the accuracy of interpreting voice commands and the speed of operation. This makes it difficult for users to comfortably operate IoT home appliances.

[1898] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1899] In this invention, the server includes a means for a user to issue a voice instruction, a means for a terminal to capture the voice and send it to the server, a means for the server to convert the voice data into text data, a means for the server to analyze the text data using a generative AI model and generate an operation command, a means for the server to send the operation command to the IoT home appliance, and a means for the IoT home appliance to operate in accordance with the operation command. This enables a user to intuitively and quickly operate the IoT home appliance through natural conversation.

[1900] "User" refers to a person who uses the system to operate IoT home appliances.

[1901] "Device" refers to a device that captures a user's voice and transmits it to a server, such as a smart speaker or smartphone.

[1902] "Server" refers to a computer system that converts voice data into text data, analyzes it using a generative AI model, generates operation commands, and sends them to IoT home appliances.

[1903] "Voice data" refers to data that digitally represents the voice uttered by the user to the terminal.

[1904] "Text data" refers to character string data obtained by analyzing voice data.

[1905] A "generative AI model" refers to an artificial intelligence model that analyzes text data and generates appropriate operation commands.

[1906] An "operation command" refers to an instruction to instruct an IoT home appliance to perform a specific action.

[1907] "IoT home appliances" refer to home appliances that are connected to the Internet and can be controlled remotely. Examples include air conditioners, lighting, and smart locks.

[1908] "Behavior" refers to the specific action that an IoT home appliance performs in accordance with an operation command.

[1909] This invention is a system that enables users to operate IoT home appliances through natural conversation. This system consists of a user, a terminal, a server, and IoT home appliances.

[1910] First, the user issues a voice command, for example, "heat the room." Next, the device (e.g., a smart speaker or smartphone) captures the user's voice and sends the voice data to a server. The server then uses voice recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data.

[1911] The server then analyzes the text data using a generative AI model (e.g., GPT-4) and generates an appropriate operation command. For example, from the text data "heat the room," it generates the operation command "set the air conditioner to heating mode." The server then uses an IoT platform (e.g., AWS IoT Core) to send the generated operation command to the IoT home appliance.

[1912] Finally, the IoT appliance (e.g., air conditioner) operates according to the received operation command. Specifically, the air conditioner starts operating in heating mode, raising the temperature in the living room.

[1913] As a concrete example, consider a scenario in which a user says, "Heat the room." In this case, the device captures the voice and sends it to the server. The server converts the voice into text, analyzes the instruction "Heat the room" using a generative AI model, and generates an operation command such as "Set the air conditioner to heating mode." The server then sends the operation command to the IoT appliance, and the air conditioner starts operating in heating mode.

[1914] An example of a prompt to input to a generative AI model is as follows:

[1915] When the user says "heat the room," generate an operation command to set the air conditioner to heating mode.

[1916] In this way, users can intuitively and quickly operate IoT home appliances through natural conversation.

[1917] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1918] Step 1:

[1919] The user issues a voice command.

[1920] A specific example of this behavior is when the user is in the living room and says, "Warm up the room."

[1921] Input: User's voice commands

[1922] Output: Audio data

[1923] Step 2:

[1924] The device captures the audio and sends it to the server.

[1925] Specifically, the smart speaker captures the user's voice and sends the voice data to a server via the Internet.

[1926] Input: Audio data

[1927] Output: Audio data sent to the server

[1928] Step 3:

[1929] The server converts the voice data into text data.

[1930] Specifically, the server uses voice recognition software (for example, Google Speech-to-Text API) to convert the received voice data into text data such as "Warm up the room."

[1931] Input: Audio data sent to the server

[1932] Output: Text data

[1933] Step 4:

[1934] The server analyzes the text data using a generative AI model and generates operation commands.

[1935] Specifically, the server uses a generative AI model (e.g., GPT-4) to analyze the text data "heat the room" and generate an operation command such as "set the air conditioner to heating mode."

[1936] Input: Text data

[1937] Output: Operation command

[1938] Step 5:

[1939] The server sends operation commands to the IoT appliances.

[1940] Specifically, the server uses an IoT platform (e.g., AWS IoT Core) to send the generated operation command to the air conditioner.

[1941] Input: Operation command

[1942] Output: Operation commands sent to IoT appliances

[1943] Step 6:

[1944] IoT home appliances operate according to operational commands.

[1945] Specifically, the air conditioner will start operating in heating mode in response to the received command, raising the temperature in the living room and providing a comfortable environment for the user.

[1946] Input: Operation command sent to IoT appliances

[1947] Output: The air conditioner starts operating in heating mode

[1948] (Application example 2)

[1949] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1950] In conventional brick-and-mortar stores, it was difficult for customers to intuitively and freely operate IoT devices in the store. Furthermore, customers could not directly interact with IoT devices, making operation cumbersome. Furthermore, there was a lack of a system for understanding and appropriately executing instructions for operating IoT devices, creating a need for an improved customer experience.

[1951] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1952] In this invention, the server includes means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, means for enabling conversation between the user and the IoT device using generative AI, means for the user to intuitively and freely operate the IoT device using MR-GUI, means for enabling customers in a physical store to talk to the IoT device using a smartphone, and means for the generative AI to understand the customer's instructions and transmit appropriate operations to the IoT device. This enables customers to intuitively and freely operate the IoT device in a physical store and talk to it directly through the generative AI.

[1953] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[1954] "IoT devices" are electronic devices that are interconnected via the Internet and are capable of sending, receiving, and controlling data.

[1955] "Generative AI" is an artificial intelligence technology that generates natural language based on user input and provides appropriate responses and instructions.

[1956] "MR-GUI" is a graphical user interface that uses MR technology and allows users to operate it intuitively.

[1957] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[1958] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use applications.

[1959] A "customer" is a consumer who visits a physical store to purchase a product or service.

[1960] "Communication" is the act of exchanging information and communicating intentions.

[1961] "Operation" is the act of controlling a device or system to perform a desired action.

[1962] "Instruction" refers to an order to perform a specific action or operation.

[1963] To implement the present invention, the following system configuration and program are required.

[1964] System Configuration

[1965] 1. Hardware:

[1966] Smartphone: A device that customers use in a physical store and that is capable of voice input and internet connectivity.

[1967] IoT devices: Electronic devices that can be connected to the internet, such as digital signage, smart lighting, and sound systems, installed in physical stores.

[1968] Server: Hosts the generative AI model and manages communication between smartphones and IoT devices.

[1969] 2. Software:

[1970] Speech Recognition Library: Use the speech_recognition library to capture audio from your smartphone's microphone and convert it into text.

[1971] Generative AI models: Use generative AI models, such as OpenAI's GPT-3, to understand customer instructions and generate appropriate responses.

[1972] HTTP Request Library: Use the requests library to send the generated instructions to the IoT device.

[1973] Program processing explanation

[1974] 1. Voice input and recognition:

[1975] When customers speak into their smartphone's microphone, a speech recognition library converts the speech into text.

[1976] For example, if a customer says, "Turn up the lights in this area," the speech is converted into text.

[1977] 2. Use of generative AI models:

[1978] The converted text is sent as a prompt to a generative AI model, which generates appropriate operating instructions for the IoT device.

[1979] Example prompt: "User instructions: Increase the lighting in this area and suggest appropriate IoT device actions."

[1980] 3. Sending instructions to IoT devices:

[1981] The generated instructions are sent to the IoT device as an HTTP request.

[1982] For example, a generative AI model generates an instruction to "brighten the lights" and sends it to a smart light.

[1983] Specific examples

[1984] When a customer uses their smartphone in a brick-and-mortar store to say, "Turn up the lights in this area," a speech recognition library converts that speech into text. A generative AI model then understands the text and generates the command "Turn up the lights." Finally, that command is sent as an HTTP request to the smart lighting, which then turns on the lights.

[1985] In this way, customers can intuitively and freely operate IoT devices in physical stores and can speak directly to them through generative AI.

[1986] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1987] Step 1:

[1988] The user speaks into the smartphone's microphone.

[1989] Input: User's voice

[1990] Output: Audio data

[1991] Specific action: The user says, "Turn up the lights in this area."

[1992] Step 2:

[1993] The device uses a speech recognition library to convert the voice data into text.

[1994] Input: Audio data

[1995] Output: Text data

[1996] Specific operation: The speech recognition library (speech_recognition) analyzes the voice data and converts it into text such as "Turn on the lights in this area."

[1997] Step 3:

[1998] The device sends text data as prompts to the generative AI model, which generates appropriate operating instructions for the IoT device.

[1999] Input: Text data

[2000] Output: Generated instructions

[2001] Specific operation: A generative AI model (OpenAI's GPT-3) receives text data and generates the instruction "Turn the lights on."

[2002] Example prompt: "User instructions: Increase the lighting in this area and suggest appropriate IoT device actions."

[2003] Step 4:

[2004] The server sends the generated instructions to the IoT device as an HTTP request.

[2005] Input: Generated instructions

[2006] Output: HTTP request

[2007] What happens: The server formats the generated instructions as an HTTP request and sends it to the smart lighting.

[2008] Step 5:

[2009] The IoT device performs an action based on the received instructions.

[2010] Input: HTTP request

[2011] Output: IoT device operation

[2012] Specific behavior: The smart lighting receives the HTTP request and performs the action of brightening the light.

[2013] In this way, users can intuitively and freely operate IoT devices using their smartphones in physical stores, and can speak directly to them through generative AI, improving the customer experience.

[2014] Example 3

[2015] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2016] Conventional methods for operating IoT devices often had complex and unintuitive user interfaces, making it difficult for users to operate them. Furthermore, interaction with and operation of IoT devices was limited, making it difficult to accurately reflect the user's intentions. Furthermore, the lack of an operating environment that combined the real world and the virtual world did not improve the user experience.

[2017] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for enabling direct communication with the IoT device using MR technology, means for enabling interaction between the user and the IoT device using a generation AI, means for the user to intuitively and freely operate the IoT device using an MR-GUI, and means for the user to wear the MR device, the server to generate the MR-GUI, the terminal to display the GUI, the user to operate the GUI, the terminal to send operation information to the server, and the server to control the IoT device. This allows the user to intuitively and freely operate the IoT device, significantly improving the user experience.

[2018] "MR technology" is a technology that provides visual information that combines the real world and the virtual world.

[2019] An "IoT device" is an electronic device that communicates via the Internet and can be remotely controlled and collect data.

[2020] "Generative AI" is an artificial intelligence technology that interacts with users and generates data.

[2021] "MR-GUI" is a graphical user interface displayed using MR technology.

[2022] A "server" is a computer system that processes data and provides services over a network.

[2023] A "terminal" is a device that a user can directly operate, including MR devices.

[2024] A "user" is a person who operates the system.

[2025] "Operation information" refers to the data and instructions entered by the user through the terminal.

[2026] "Communication" is the process of sending and receiving data or information.

[2027] "Dialogue" is the process by which a user and a system exchange information with each other.

[2028] "Control" is the process by which a system manages and coordinates the behavior of IoT devices.

[2029] This invention is a system that allows users to intuitively and freely operate IoT devices using mixed reality (MR) technology. Specifically, the user operates the IoT device using an MR-GUI displayed by MR technology. Detailed embodiments of this system are described below.

[2030] Hardware and software used

[2031] 1. MR device:

[2032] Example: Microsoft HoloLens

[2033] Function: Provides visual information that blends the real and virtual worlds.

[2034] 2. Server:

[2035] Function: Generates MR-GUI, processes operation information, and controls IoT devices.

[2036] 3. IoT devices:

[2037] Examples: air conditioners, lighting, smart locks, etc.

[2038] Function: Communicates via the Internet, allowing remote control and data collection.

[2039] 4. Generation AI:

[2040] Function: Interacts with the user and generates data.

[2041] System Operation

[2042] When a user puts on an MR device, the server acquires information about the user's environment and generates an MR-GUI based on that information. The MR device displays the GUI received from the server in the user's field of view. The user intuitively operates the GUI displayed through the MR device. For example, the user can change the air conditioner temperature by moving the temperature control slider, or adjust the air volume by pressing the air volume control button. The MR device transmits the user's operation information to the server in real time. The server controls the IoT devices based on the received operation information.

[2043] Specific examples

[2044] For example, if a user wants to set the air conditioner temperature to 25 degrees, they would follow these steps:

[2045] 1. The user wears the MR device.

[2046] 2. The server generates an MR-GUI that includes the air conditioner control panel.

[2047] 3. The MR device displays the operation panel in the user's field of view.

[2048] 4. The user sets the temperature slider to 25 degrees.

[2049] 5. The MR device sends this operation information to the server.

[2050] 6. The server sends a command to the air conditioner to set the temperature to 25 degrees.

[2051] Prompt Sentence Examples

[2052] A system description can be generated by inputting a prompt like the one below into the generative AI model.

[2053] "Please explain the system process in which users operate IoT devices using mixed reality technology. Please clearly state what kind of data processing and calculations are performed, including the names of specific hardware and software. Also, please provide specific examples."

[2054] By using this prompt sentence, the generative AI model can generate a detailed explanation of the system. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[2055] Step 1:

[2056] The user puts on the MR device.

[2057] Input: The user puts on a mixed reality device (e.g., Microsoft HoloLens).

[2058] Data processing: The MR device acquires the user's location information and surrounding environmental data.

[2059] Output: The MR device sends the user's location and environmental data to the server.

[2060] Specific Action: The user puts the HoloLens on their head and turns on the device.

[2061] Step 2:

[2062] The server generates the MR-GUI.

[2063] Input: User location and environmental data sent from the MR device.

[2064] Data processing: The server analyzes the data received and generates an MR-GUI that is suitable for the user's environment.

[2065] Output: Send the generated MR-GUI data to the MR device.

[2066] Specific operation: The server generates an air conditioner control panel based on the user's location information and surrounding environmental data.

[2067] Step 3:

[2068] The terminal displays the GUI.

[2069] Input: MR-GUI data sent from the server.

[2070] Data processing: The MR device renders the received GUI data to display in the user's field of view.

[2071] Output: The MR-GUI displayed in the user's field of view.

[2072] Specific operation: HoloLens displays the air conditioner control panel overlaid on the user's field of vision.

[2073] Step 4:

[2074] The user operates the GUI.

[2075] Input: The user operates the MR-GUI displayed in the field of view.

[2076] Data processing: The MR device acquires the user's operation information and analyzes the operation content.

[2077] Output: Sends the parsed operation information to the server.

[2078] What happens: The user moves their hand to set a slider displayed in the HoloLens' field of view to 25 degrees.

[2079] Step 5:

[2080] The terminal transmits operation information to the server.

[2081] Input: User operation information.

[2082] Data processing: The MR device packets the operation information and sends it to the server via the network.

[2083] Output: Operation information sent to the server.

[2084] Specific operation: HoloLens sends user operation information (e.g., changing the temperature setting to 25 degrees) to the server.

[2085] Step 6:

[2086] The server controls the IoT devices.

[2087] Input: Operation information sent from the MR device.

[2088] Data processing: The server analyzes the operation information and generates control commands for the corresponding IoT devices.

[2089] Output: Control commands sent to IoT devices.

[2090] Specific operation: The server sends a command to the air conditioner to "set the temperature to 25 degrees," and the air conditioner adjusts the temperature according to that command.

[2091] (Application example 3)

[2092] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2093] Conventional IoT home appliances are generally operated using remote controls or smartphone apps, which makes intuitive operation difficult. Furthermore, in brick-and-mortar stores, there is a lack of means to efficiently manage and operate multiple IoT devices, which makes it difficult for store staff to respond quickly. To solve these issues, a more intuitive and efficient operation method is needed.

[2094] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[2095] In this invention, the server includes a means for enabling direct communication with IoT home appliances using MR (Mixed Reality) technology, a means for enabling conversation between a user and IoT home appliances using generative AI, a means for a user to intuitively and freely operate IoT home appliances using MR-GUI, a means for intuitively operating IoT devices in a physical store using smart glasses, and a means for detecting hand movements to operate the MR-GUI, thereby enabling a user to intuitively operate IoT home appliances and IoT devices in a physical store.

[2096] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[2097] "IoT home appliances" are home appliances that are connected to the Internet and can be remotely controlled and used for data communication.

[2098] "Generative AI" is an artificial intelligence technology that generates natural conversations and responses based on user input.

[2099] "MR-GUI" is a graphical user interface displayed using Mixed Reality technology, and is an interface that users can operate intuitively.

[2100] "Smart glasses" are wearable devices that have built-in displays, cameras, and sensors, allowing users to visually check information.

[2101] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[2102] An "IoT device" is an electronic device that is connected to the Internet and is capable of data communication and remote control.

[2103] "Means for detecting hand movements" refers to technology that uses cameras and sensors to recognize the position and movement of the user's hands and perform operations based on that.

[2104] This invention relates to a system that uses mixed reality (MR) technology to intuitively operate IoT home appliances and IoT devices in brick-and-mortar stores. Specific embodiments for realizing this system are described below.

[2105] System Configuration

[2106] The system consists of the following major components:

[2107] 1. Server:

[2108] Generative AI: Generates natural-sounding conversations and responses based on user input.

[2109] Database: Stores configuration information for IoT home appliances and IoT devices, as well as user operation history.

[2110] 2. Terminal:

[2111] Smart glasses: Wearable devices with built-in displays, cameras, and sensors that allow users to visually view information.

[2112] Camera: Used to detect the user's hand movements.

[2113] MR-GUI: A graphical user interface displayed using Mixed Reality technology.

[2114] Program processing

[2115] The server uses generative AI to enable conversations between users and IoT appliances and devices. The user wears smart glasses and operates the MR-GUI through a camera that detects hand movements. Specifically, the following process is performed:

[2116] 1. Hand movement detection:

[2117] The camera recognizes the position and movement of the user's hands and detects hand landmarks.

[2118] The sliders and buttons of the MR-GUI are operated based on the detected hand movements.

[2119] 2. Display MR-GUI:

[2120] The smart glasses display shows the operation panels of IoT home appliances and devices.

[2121] For example, the air conditioner control panel displays a temperature control slider, a button to adjust the airflow, and a button to switch between operating modes.

[2122] 3. Generative AI Conversation:

[2123] When a user gives a command via voice, the generative AI analyzes the command and generates an appropriate response.

[2124] For example, in response to an instruction such as "Set the air conditioner temperature to 24 degrees," the generative AI will change the air conditioner temperature.

[2125] Specific examples

[2126] Consider a scenario in which a store clerk wears smart glasses and adjusts the temperature of the air conditioning in a physical store. The clerk moves his or her hand to operate the slider on the MR-GUI, changing the temperature from 22 degrees to 24 degrees. In this case, the generative AI analyzes the clerk's voice instructions and generates an appropriate response.

[2127] Prompt Sentence Examples

[2128] "Put on the smart glasses and use your hand to operate the slider on the MR-GUI to change the temperature of the store's air conditioning from 22 degrees to 24 degrees."

[2129] In this way, the present invention enables users to intuitively operate IoT home appliances and IoT devices in physical stores.

[2130] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[2131] Step 1:

[2132] The user puts on the smart glasses and starts the system.

[2133] Input: Smart glasses activation signal

[2134] Output: MR-GUI is displayed on the smart glasses display.

[2135] Specific operation: When the user turns on the smart glasses, the system starts up and the MR-GUI is displayed on the display, allowing the user to visually confirm the operation panel.

[2136] Step 2:

[2137] The device's camera detects the user's hand movements.

[2138] Input: Camera image

[2139] Output: Hand landmark data

[2140] How it works: The camera captures the position and movement of the user's hands in real time and detects hand landmarks, which allows the system to recognize the user's hand movements.

[2141] Step 3:

[2142] The device analyzes the hand landmark data and operates the MR-GUI.

[2143] Input: Hand landmark data

[2144] Output: MR-GUI operation instructions

[2145] Specific operation: Sliders and buttons in the MR-GUI are operated based on the hand landmark data. For example, when a user moves their hand to operate a slider, the slider's position changes accordingly.

[2146] Step 4:

[2147] The user issues a command by voice.

[2148] Input: Voice command

[2149] Output: Audio data

[2150] How it works: When a user gives a voice command such as "Set the air conditioner temperature to 24 degrees," the voice is captured by the microphone in the smart glasses.

[2151] Step 5:

[2152] The server uses generative AI to analyze voice commands and generate an appropriate response.

[2153] Input: Audio data

[2154] Output: Operation instruction data

[2155] Specific operation: The server analyzes the voice data and understands the user's instructions. The generative AI generates an appropriate response and generates operation instruction data to set the air conditioner temperature to 24 degrees.

[2156] Step 6:

[2157] The terminal operates IoT home appliances based on the operation instruction data.

[2158] Input: Operation instruction data

[2159] Output: Change the settings of IoT appliances

[2160] Specific operation: Based on the operation instruction data, the air conditioner temperature is set to 24 degrees. This executes the user's instruction.

[2161] Step 7:

[2162] The server provides feedback on the operation results to the user.

[2163] Input: IoT appliance settings change results

[2164] Output: Feedback message

[2165] Specific operation: The server confirms that the air conditioner temperature has been set to 24 degrees, generates a feedback message for the user such as "The air conditioner temperature has been set to 24 degrees," and displays it on the smart glasses display.

[2166] In this way, users can intuitively operate IoT home appliances and IoT devices in physical stores through smart glasses.

[2167] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2168] "Example 1"

[2169] One embodiment of the present invention is a system in which a user wears mixed reality (MR) goggles or a headset and communicates directly with IoT appliances. This system includes an emotion engine that recognizes the user's emotions. Specifically, when the user speaks to an IoT appliance, the system analyzes the user's emotions from the tone of voice and facial expressions, and reflects the results in the operation of the IoT appliance. For example, if the user expresses anger, the IoT appliance can quickly perform the appropriate operation, enabling it to operate in accordance with the user's emotions.

[2170] "Example 2"

[2171] Another example is a system that uses generative AI to enable conversations between users and IoT appliances. This system also includes an emotion engine that recognizes the user's emotions. Specifically, it analyzes the user's speech and reflects the results in the operation of the IoT appliance. For example, if the user expresses joy, the IoT appliance will operate in a joyful manner, enabling it to operate in accordance with the user's emotions.

[2172] "Example 3"

[2173] Another example is a system that uses MR-GUI to allow users to intuitively and freely operate IoT appliances. This system also includes an emotion engine that recognizes the user's emotions. Specifically, it analyzes the user's emotions from their operations and reflects the results in the operation of the IoT appliance. For example, if the user expresses surprise, the IoT appliance can operate in a gentle manner, allowing it to respond to the user's emotions.

[2174] The processing flow of each embodiment will be described below.

[2175] "Example 1"

[2176] Step 1: The user puts on MR (Mixed Reality) goggles or a headset.

[2177] Step 2: The user talks to the IoT appliance.

[2178] Step 3: The emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[2179] Step 4: Reflect the analysis results in the operation of IoT home appliances.

[2180] "Example 2"

[2181] Step 1: The user speaks to the IoT appliance via generative AI.

[2182] Step 2: The emotion engine analyzes the emotion from the user's speech.

[2183] Step 3: Reflect the analysis results in the operation of IoT home appliances.

[2184] "Example 3"

[2185] Step 1: The user operates the IoT appliance using the MR-GUI.

[2186] Step 2: The emotion engine analyzes the emotions from the user's operations.

[2187] Step 3: Reflect the analysis results in the operation of IoT home appliances.

[2188] Example 1

[2189] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2190] Conventional IoT devices require the use of dedicated remote controls or smartphone apps, making them difficult to operate intuitively. Furthermore, since operations cannot take into account the user's emotions, it is difficult to improve the user experience. Furthermore, communication with IoT devices is limited, and users cannot directly talk to them, limiting the degree of freedom in operation.

[2191] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2192] In this invention, the server includes a means for enabling direct communication with IoT devices using MR (Mixed Reality) technology, a means for enabling conversation between the user and the IoT device using generative AI, a means for the user to intuitively and freely operate the IoT device using MR-GUI, and a means for recognizing the user's emotions and optimizing the operation of the IoT device based on those emotions. This allows the user to intuitively and freely operate the IoT device, and further realizes optimal operation according to the user's emotions.

[2193] "MR (Mixed Reality) technology" is a technology that combines the real world and the virtual world, allowing users to experience both environments simultaneously.

[2194] "IoT devices" are home appliances and devices that are connected to each other via the Internet and can send, receive, and control data.

[2195] "Generative AI" is a system that uses artificial intelligence technology to generate natural conversations and interactions with users.

[2196] "MR-GUI" is a graphical user interface that uses Mixed Reality technology, providing an interface that users can operate intuitively.

[2197] "Emotion recognition" is a technology that analyzes emotions from the user's tone of voice, facial expressions, etc. and obtains the results.

[2198] An "operation interface" is a visual or physical means by which a user operates an IoT device.

[2199] "Optimization" refers to adjusting something to the most effective state for specific conditions or purposes.

[2200] This invention is a system that enables direct communication with IoT devices using MR (Mixed Reality) technology. This system starts when a user wears MR-compatible goggles or a headset and visually recognizes the IoT device. Specifically, the user wears special goggles or a headset and visually recognizes the IoT device. At this time, a direct operation interface is displayed on the IoT device using MR technology.

[2201] Hardware and software used

[2202] Hardware: Mixed reality goggles or headset (e.g., Microsoft HoloLens)

[2203] Software: Emotion recognition engine (e.g., Affectiva SDK), IoT device control software (e.g., Home Assistant)

[2204] Data processing and calculation

[2205] 1. User visual recognition:

[2206] The user puts on MR-compatible goggles or a headset.

[2207] The cameras in the goggles or headset recognize surrounding IoT devices and use MR technology to display an operating interface on the device.

[2208] Example: The air conditioner itself displays a temperature control slider and airflow control buttons.

[2209] 2. Emotion recognition:

[2210] The user speaks to the IoT device.

[2211] The emotion recognition engine analyzes the user's tone of voice and facial expressions to recognize their emotions.

[2212] Example: If a user expresses an angry emotion, an emotion recognition engine will analyze that information.

[2213] 3. Operation reflection:

[2214] IoT devices are operated based on the analysis results of the emotion recognition engine.

[2215] Example: If the user is showing signs of anger, quickly change the air conditioning settings.

[2216] Specific examples

[2217] Example 1: Air conditioner operation

[2218] Scenario: A user wants to lower the temperature in their room.

[2219] Operation steps:

[2220] 1. The user puts on the MR compatible goggles.

[2221] 2. The goggles will recognize the air conditioner and a temperature control slider will appear on the air conditioner itself.

[2222] 3. The user operates the slider to lower the temperature.

[2223] 4. The device analyzes the user's facial expressions to confirm satisfaction.

[2224] 5. The server changes the air conditioning settings, lowering the room temperature.

[2225] Example 2: Lighting control

[2226] Scenario: A user wants to dim the lights in their room.

[2227] Operation steps:

[2228] 1. The user puts on the mixed reality headset.

[2229] 2. The headset will recognize the lighting and a button to adjust the lighting brightness will appear.

[2230] 3. The user presses a button to decrease the brightness.

[2231] 4. The device analyzes the user's tone of voice to ensure they are relaxed.

[2232] 5. The server changes the lighting settings and the room becomes less bright.

[2233] Prompt Sentence Examples

[2234] Prompt 1: "I want to turn down the air conditioner. If you show any signs of anger, I will change the setting immediately."

[2235] Prompt 2: "I want the lights in the room to be dimmed. If I'm relaxing, I slowly lower the brightness."

[2236] The above is an embodiment of the present invention.

[2237] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2238] Step 1:

[2239] The user puts on MR-compatible goggles or a headset.

[2240] Input: User puts on mixed reality goggles or headset.

[2241] Data processing: The goggles or headset activates and displays the MR environment in the user's field of vision.

[2242] Output: The mixed reality environment is displayed in the user's field of view.

[2243] Specific operation: When the user puts on the goggles and turns them on, the MR environment is displayed on the goggles' display.

[2244] Step 2:

[2245] The device recognizes surrounding IoT devices.

[2246] Input: Cameras in goggles or headset scan the surrounding environment.

[2247] Data processing: The device recognizes IoT devices (e.g., air conditioners, lighting) and obtains their location information.

[2248] Output: Location information of recognized IoT devices.

[2249] How it works: The goggles' camera scans the room and recognizes the air conditioner and lights.

[2250] Step 3:

[2251] The device displays the operating interface using MR technology.

[2252] Input: Location information of the recognized IoT device.

[2253] Data processing: Using MR technology, the device displays an operation interface on the IoT device it recognizes.

[2254] Output: Operation interface displayed on the IoT device.

[2255] Specific operation: The goggles' display will show an air conditioning temperature control slider and lighting brightness adjustment buttons.

[2256] Step 4:

[2257] The user operates the IoT device using the operation interface.

[2258] Input: The user operates a visually displayed control interface.

[2259] Data processing: IoT device settings are changed based on user actions.

[2260] Output: The modified IoT device configuration.

[2261] Specific action: The user moves their hand to operate a slider on the goggles' display to set the air conditioner temperature.

[2262] Step 5:

[2263] The device re...

Claims

1. A means for displaying an operation interface of an IoT device identified by performing image recognition on video data captured by a camera together with a video of the IoT device; means for transmitting operation information for the operation interface to a server; A means for controlling the IoT device based on a control signal transmitted from the server based on the operation information; A system including:

2. A visual device is a device worn by a user to provide visual information and to fuse the real world with the virtual world, and the operation interface and the image of the IoT device are displayed on the visual device. The system of claim 1 .

3. A means for acquiring voice data including an operation instruction for an IoT home appliance; means for generating an operation command corresponding to the operation instruction based on the voice data; A means for transmitting the operation command to the IoT home appliance; A system including:

Images