Information processing system, device controller, and program

Abstract

Provided is an information processing system, a device controller, and a program that enable an improvement in convenience. An information processing system according to an embodiment has a control section. The control section displays one or both of information associated with a home electric appliance or an operation section for operating the home electric appliance by a virtual space technique.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to an information processing system, a device controller, and a program. BACKGROUND

[0002] In a vehicle-mounted system of an automobile, a display technique of providing image information of augmented reality to a user is known. An information processing system related to a home appliance device expects an improvement in convenience.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT DOCUMENTS

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-142792 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] The problem to be solved by the present application is to provide an information processing system, a device controller, and a program that enable an improvement in convenience.

[0008] MEANS FOR SOLVING THE PROBLEMS

[0009] The information processing system of the technical solution has a control section. The control section causes one or both of a display that displays information associated with a home appliance device or an operation section that operates the home appliance device to be displayed by virtual space technology.

[0010] EFFECTS OF THE INVENTION

[0011] According to the configuration of the present application, compared with a general device controller, the display size and arrangement of the buttons can be dynamically changed. Thus, a user interface that is easier for a user to use can be provided, and an improvement in convenience for the user can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a diagram showing an example of a use environment in which the smart glasses of the first embodiment are used.

[0013] Figure 2 is a perspective view showing the physical structure of the smart glasses of the first embodiment.

[0014] Figure 3 is a block diagram showing the functional structure of the smart glasses of the first embodiment.

[0015] Figure 4 is a diagram showing a control example of scenario 1 of the first embodiment.

[0016] Figure 5 is a diagram showing a control example of scenario 2 of the first embodiment.

[0017] Figure 6 Fig. 2 is a diagram showing a control example of scenario 2 of the first embodiment.

[0018] Figure 7 Fig. 3 is a diagram showing a control example of scenario 3 of the first embodiment.

[0019] Figure 8 Fig. 4 is a diagram showing a control example of scenario 4 of the first embodiment.

[0020] Figure 9 Fig. 5 is a diagram showing a control example of scenario 5 of the first embodiment.

[0021] Figure 10 Fig. 6 is a diagram showing a control example of scenario 5 of the first embodiment.

[0022] Figure 11 Fig. 7 is a diagram showing a variation of the control example of scenario 5 of the first embodiment.

[0023] Figure 12 Fig. 8 is a diagram showing a control example of scenario 6 of the first embodiment.

[0024] Figure 13 Fig. 9 is a diagram showing a variation of the control example of scenario 6 of the first embodiment.

[0025] Figure 14 Fig. 10 is a diagram showing a control example of scenario 7 of the first embodiment.

[0026] Figure 15 Fig. 11 is a diagram showing an example of simulation results of the flow of the air current of the first embodiment.

[0027] Figure 16 Fig. 12 is a diagram showing a set condition related to the movement of the particle when colliding against the wall of the first embodiment.

[0028] Figure 17 Fig. 13 is a diagram showing a first display example of the image of visualizing the air current of the first embodiment.

[0029] Figure 18 Fig. 14 is a diagram showing a second display example of the image of visualizing the air current of the first embodiment.

[0030] Figure 19 Fig. 15 is a diagram showing a coordinate system used in the first embodiment.

[0031] Figure 20 Fig. 16 is a diagram showing a control example of the second embodiment.

[0032] Figure 21 Fig. 17 is a diagram showing a terminal device of the third embodiment.

[0033] Figure 22 is a diagram that shows the information processing system of the fourth embodiment. DETAILED DESCRIPTION

[0034] Hereinafter, an information processing system, a device controller, and a program of an embodiment will be described with reference to the drawings. In the following description, the same or similar functions are assigned the same reference numerals. Also, there are cases in which repeated description of the structures is omitted. "Based on XX" means "based on at least XX", and can include cases in which other elements are based on in addition to XX. "Based on XX" is not limited to cases in which XX is used directly, and can include cases in which elements that have been operated or processed on XX are based on. "XX or YY" is not limited to cases in which either of XX and YY, and can include cases in which both of XX and YY. The same applies to cases in which the number of selected elements is three or more. "XX" and "YY" are arbitrary elements (for example, arbitrary information). In this specification, "image" is not limited to still images, and includes moving images (video).

[0035] (First Embodiment)

[0036] [1. Example of usage environment]

[0037] First, a usage environment in which the smart glasses 1 of the first embodiment are used will be described. The smart glasses 1 are an example of an "information processing apparatus", and are a device controller that operates a home appliance device. The smart glasses 1 of the present embodiment are also an example of an "information processing system".

[0038] Figure 1 is a diagram that shows an example of a usage environment in which the smart glasses 1 are used. The smart glasses 1 are used, for example, in a house of a user U. The house of the user U has, for example, an air conditioner 2, an electric fan 3, a window 4, and a ventilator 5. However, the electric fan 3, the window 4, and the ventilator 5 can not exist.

[0039] The air conditioner 2 includes, for example, a heat pump, and is a device that performs air conditioner operation such as cooling operation, heating operation, or dehumidifying operation. The air conditioner 2 is an example of each of an "air conditioner", a "home appliance device", and a "first home appliance device". The air conditioner 2 adjusts, for example, the temperature and humidity of the air in a room by blowing air streams from an air outlet of the air conditioner 2 to the room. In this specification, "action of a home appliance device" or "action state of a home appliance device" is not limited to the action of the home appliance device itself, and can include a result (for example, movement of air streams blown from an air conditioner) and the like caused by the action of the home appliance device.

[0040] The air conditioner 2 has, for example, a temperature sensor 2a, a humidity sensor 2b, and a communication module 2c. The temperature sensor 2a detects the temperature of a room in which the air conditioner 2 is installed. The humidity sensor 2b detects the humidity of the room in which the air conditioner 2 is installed. The communication module 2c communicates with a device controller (for example, the smart glasses 1) that operates the air conditioner 2. Note that the "air conditioner" is not limited to an air conditioner, but can be an air purifier or the like.

[0041] The electric fan 3 is an example of a home appliance device (a second home appliance device) that has an influence on the operation (for example, the airflow blown out from the air conditioner 2) of the air conditioner 2. The electric fan 3 changes the flow direction of at least a part of the airflow blown out from the air conditioner 2 by being operated. In the present embodiment, the electric fan 3 is an example of an element in which the smart glasses 1 can acquire information related to at least a part of the state (for example, the direction in which the electric fan 3 faces).

[0042] The window 4 and the ventilator 5 are each an example of an article that has an influence on the operation (for example, the airflow blown out from the air conditioner 2) of the air conditioner 2. Depending on the open / close state of the window 4 and the operation state of the ventilator 5, the flow direction of at least a part of the airflow blown out from the air conditioner 2 is changed. In the present embodiment, the window 4 and the ventilator 5 are each an example of an element in which the smart glasses 1 cannot acquire information related to the state (for example, the open / close state of the window 4, the operation state of the ventilator 5).

[0043] [2. Physical structure of smart glasses]

[0044] Next, the physical structure of the smart glasses 1 will be described.

[0045] Figure 2 is a perspective view that shows the physical structure of the smart glasses 1. The smart glasses 1 are a wearable information processing device that a user U wears on the head and uses. The smart glasses 1 have, for example, a frame 10, a lens 20, a projection section 30, a camera 40, a microphone 50, a communication module 60, and a control substrate 70.

[0046] The frame 10 is the same as the frame of a general pair of glasses, and includes a nose pad portion and an ear hook portion that the user U can wear. The lens 20 is mounted on the frame 10, and is located in front of the eyes of the user U. The lens 20 has, for example, a display surface 20a formed of a transparent material and on which an image can be projected by the projection section 30. The user U can view the scenery around via the lens 20.

[0047] The projection section 30 has, for example, a small projector that projects display information generated by the control board 70 as an image on the display surface 20a. Thereby, the user U can view the display represented by the virtual space technology superimposed on the surrounding scenery. The "virtual space technology" described in this specification refers to so-called XR, which refers to Augmented Reality (AR), Mixed Reality (MR), Virtual Reality (VR), or Substitutional Reality (SR), and the like. The projection section 30 of this embodiment outputs a display represented by Augmented Reality or Mixed Reality. An example of outputting a display by Virtual Reality or Substitutional Reality is described later.

[0048] The projection section 30 is an example of an "information output section". However, the information output section is not limited to the above example, and can be a retinal projection type projection section that projects an image directly to the eyes of the user U, or the like. In this case, the lens 20 is not needed.

[0049] The camera 40 is provided on the frame 10, for example, and photographs the surroundings (for example, the front) of the smart glasses 1. The microphone 50 is provided on the frame 10, for example, and acquires the sound of the surroundings of the smart glasses 1.

[0050] The communication module 60 is capable of communicating with the communication module 2c of the air conditioner 2. The communication module 60 is a communication module that communicates by Wi-Fi (registered trademark), Bluetooth (registered trademark), or infrared communication, for example, but is not limited to these. The communication module 60 is capable of acquiring the detection results of the temperature sensor 2a and the humidity sensor 2b of the air conditioner 2 from the air conditioner 2 by communicating with the communication module 2c of the air conditioner 2.

[0051] The control board 70 includes a circuit board and a plurality of electronic components mounted on the circuit board. The control board 70 collectively controls the smart glasses 1. The control board 70 controls the display by the projection section 30, the communication by the communication module 60, and the like. The control board 70 is described in detail later.

[0052] [3. Functional configuration of smart glasses]

[0053] Figure 3is a block diagram showing a functional configuration of the smart glasses 1. The control substrate 70 has, for example, a detection section 110, a reception section 120, a control section 130, and a storage section 140. The detection section 110, the reception section 120, and the control section 130 are realized by a hardware processor such as a CPU (Central Processing Unit) on the control substrate 70 executing a program (software). However, all or a part of these functional sections can be realized by hardware (circuit section; including a circuit) such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array), or can be realized by a combination of software and hardware. The storage section 140 is realized by, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

[0054] [3.1 Detection Section]

[0055] The detection section 110 performs the processes described below based on an image photographed by the camera 40, a sound received by the microphone 50, or information received by the communication module 60. The detection section 110 has, for example, an electric appliance detection section 111, a state detection section 112, an article detection section 113, a surrounding environment detection section 114, and an operation detection section 115.

[0056] The electric appliance detection section 111 determines what kind of electric appliance exists in the angle of view of the camera 40 based on an image photographed by the camera 40. For example, in the storage section 140, electric appliance information I1 in which characteristic amounts of electric appliances to be detected are registered is stored. The "characteristic amounts of electric appliances" are the size, shape, color, setting position (for example, the setting height seen by the user U), and the like of electric appliances. The electric appliance detection section 111 determines whether or not an electric appliance that is set in advance as a detection target (for example, an electric appliance that can be operated by the smart glasses 1, such as the air conditioner 2) exists in an image photographed by the camera 40 by comparing the image photographed by the camera 40 with the characteristic amounts of each electric appliance included in the electric appliance information I1. In addition, the electric appliances set as detection targets can be added or changed by the user U.

[0057] In this embodiment, the home appliance detection unit 111, based on the image captured by the camera 40, detects other home appliances (a second home appliance, such as an electric fan 3) that affect the operation of the target home appliance (a first home appliance, for example, air conditioner 2). The home appliance detection unit 111 compares the image of the second home appliance captured by the camera 40 with the feature values ​​of the home appliances included in the home appliance information I1, detecting not only the presence of the second home appliance but also its operating state. For example, regarding the operating state of the electric fan 3, the home appliance detection unit 111 detects whether the electric fan 3 is moving, its orientation, and whether it is oscillating.

[0058] The state detection unit 112 detects the surrounding state. "Surrounding state" refers to the state around the user U wearing the smart glasses 1, such as the state related to the room where the user U is located. In this embodiment, the state detection unit 112 detects the temperature and humidity of the room where the user U is located based on the detection results of the temperature sensor 2a and humidity sensor 2b of the air conditioner 2 received by the communication module 60. Alternatively, if the smart glasses 1 is equipped with a temperature sensor and / or humidity sensor, the state detection unit 112 may also detect the temperature and / or humidity of the room where the user U is located based on the detection results of the temperature sensor and / or humidity sensor installed on the smart glasses 1. Furthermore, the smart glasses 1 may obtain meteorological information via an external network (e.g., the Internet) and detect the external temperature, etc., of the area where the user U's residence is located based on the meteorological information.

[0059] Furthermore, the status detection unit 112 determines whether the room temperature and / or humidity detected as described above meet the specified conditions. For example, the storage unit 140 stores threshold information I2, which includes various thresholds. The threshold information I2 includes various thresholds used to determine whether the surrounding conditions meet the specified conditions.

[0060] For example, the status detection unit 112 determines whether the conditions for high-temperature precaution are met by comparing the detected room temperature with a threshold for high-temperature precaution (a threshold for heatstroke risk, such as 28°C) included in the threshold information I2. Similarly, the status detection unit 112 determines whether the conditions for low-temperature precaution are met by comparing the detected room temperature with a threshold for low-temperature precaution (such as 10°C) included in the threshold information I2. Furthermore, the status detection unit 112 may also determine whether the conditions for high-temperature precaution and / or low-temperature precaution are met based on the outside temperature detected when the air conditioner 2 is not operating. Additionally, the status detection unit 112 determines whether the conditions for high humidity precaution corresponding to a specific item are met by comparing the detected room humidity with a threshold for high humidity precaution (such as 70%) corresponding to a specific item described later. The status detection unit 112 may also determine whether the conditions for low humidity precaution are met by comparing the detected room humidity with a threshold for low humidity precaution (such as 30%). These various thresholds can also be added or changed by the user U.

[0061] The item detection unit 113 determines whether a specific item exists in the field of view of the camera 40 based on the image captured by the camera 40. For example, the specific item might be an item requiring temperature or humidity management, such as a camera or a musical instrument. For example, the storage unit 140 stores item information I3, which lists the characteristic quantities of each specific item. The characteristic quantities of a specific item include its representative size, shape, color, etc. The item detection unit 113 determines whether a specific item, as a detection candidate, exists in the image captured by the camera 40 by comparing the image captured by the camera 40 with the characteristic quantities of the specific item contained in the item information I3. Specific items can be added or changed by the user U. For example, the user U can register a specific item by taking a picture of it with the camera 40.

[0062] The surrounding environment detection unit 114 detects the surrounding environment of the user U based on the image captured by the camera 40. The surrounding environment of the user U includes, for example, the layout of the room where the user U is located and the items (e.g., furniture) arranged in the room. The information representing the surrounding environment of the user U detected by the surrounding environment detection unit 114 is stored in the storage unit 140 as surrounding environment information I4.

[0063] The operation detection unit 115 detects user U's operation on the target home appliance (e.g., air conditioner 2) based on the image captured by camera 40. In this embodiment, the display control unit 132, described later, displays the operation unit D2 (see reference 115) for operating the home appliance using virtual space technology. Figure 4The operation detection unit 115 detects the user U's operation on the target home appliance based on the positional relationship between the virtual operation unit D2 displayed to the user U (the position visible to the user U) and the position of the user U's fingertip contained in the image captured by the camera 40, as well as the movement of the user U's fingertip. Alternatively, or in addition, the operation detection unit 115 may detect the user U's operation on the target home appliance based on the user U's audio input to the microphone 50.

[0064] [3.2 Reception Department]

[0065] The receiving unit 120 accepts input from user U regarding elements for which information about their status cannot be obtained (e.g., window 4 or exhaust fan 5). For example, the display control unit 132, described later, displays a query to user U inquiring about the status of elements for which information cannot be obtained. Furthermore, the receiving unit 120 obtains the status of the element for which information cannot be obtained based on user U's input to the aforementioned query (user U's answer). User U's input to the aforementioned query can be, for example, an action detected by the operation detection unit 115 or audio input from user U to the microphone 50. In addition, the receiving unit 120 can also accept input from one or both user Us indicating the layout of a room or the arrangement of items in which the appliances to be operated are located.

[0066] [3.3 Control Department]

[0067] The control unit 130 performs the following processing based on the information detected by the detection unit 110 and the information received by the receiving unit 120. The control unit 130 includes, for example, a calculation unit 131, a display control unit 132, and an operation control unit 133.

[0068] The calculation unit 131 performs simulations of the operating states of the home appliances. For example, the calculation unit 131 performs simulations related to the airflow from the air conditioner 2. Specific examples of the calculations performed by the calculation unit 131 will be described in detail later.

[0069] The display control unit 132 uses virtual space technology to display display D1 (refer to) that represents information associated with home appliances. Figure 4 ) or the operating unit D2 used to operate household appliances (see reference) Figure 4One or both of them. For example, the display control unit 132 displays display D1 including the status of the home appliance to be operated. The "status of the home appliance" is, for example, one or more of the following: the ON / OFF status of the power supply of the home appliance, the operating mode being performed, the candidate of the selectable (changeable) operating mode, the status of the timer setting, etc. In this embodiment, the display control unit 132, as one of the home appliance statuses displayed on display D1, can display a visualization of the airflow from the air conditioner 2 simulated by the calculation unit 131, such as FI (see reference). Figure 5 Specific examples of displays performed by the display control unit 132 will be described in detail in the descriptions of the various scenarios described later.

[0070] Based on the operation of user U on operation unit D2 in virtual space detected by operation detection unit 115, operation control unit 133 selects or generates control commands for operating home appliances and sends the selected or generated control commands to the home appliances to be operated via communication module 60.

[0071] In this embodiment, when the operation control unit 133 sends a control command to the air conditioner 2, it does not send information indicating the target location of the airflow itself. Instead, similar to the control commands for a regular air conditioner controlled by an infrared remote control, it sends a control command to the air conditioner 2 to select a certain airflow direction from multiple selectable levels of airflow direction in the left-right direction, a certain airflow direction from multiple selectable levels of airflow direction in the up-down direction, or a certain airflow intensity from multiple selectable levels of airflow intensity. The control content that can be selected by the air conditioner 2 is pre-stored in the storage unit 140 as control command information I5. The control command information I5 can be obtained, for example, from the website of the air conditioner 2 manufacturer via an external network (e.g., the Internet). Specific examples of control performed by the operation control unit 133 will be described in detail in the descriptions of the various scenarios described later.

[0072] [3. Control Example]

[0073] The following describes several scenarios for controlling home appliances using smart glasses 1. The following example illustrates this using air conditioner 2 as the object of control (the first home appliance) and electric fan 3 as the home appliance that affects the object of control (the second home appliance).

[0074] [3.1 Scene 1]

[0075] Figure 4 This is a diagram illustrating a control example for scenario 1. Scenario 1 represents a situation where air conditioner 2 is not operating. Figure 4 (a) in the figure represents the instant when smart glasses 1 detects air conditioner 2.Figure 4 (b) in the diagram represents the state after the smart glasses 1 detects the air conditioner 2. In this embodiment, when the air conditioner 2 is not within the field of view of the camera 40, the display control unit 132 will not display display D1 and operation unit D2 (see reference). Figure 4 (a)). On the other hand, when the air conditioner 2 enters the camera's field of view (i.e., when the home appliance detection unit 111 detects that the air conditioner 2 has entered the image captured by the camera 40), the display control unit 132 causes the display D1 and the operation unit D2 to be superimposed on the surrounding scenery and displayed (see reference). Figure 4 (b) in the middle.

[0076] Display D1 includes information D1a that displays the type (air conditioner) and model of the household appliance included in the field of view of camera 40, the status of air conditioner 2, the status of the room where air conditioner 2 is installed, and information obtained from air conditioner 2 (room temperature and humidity). Information D1a is displayed, for example, as character information. On the other hand, the operation unit D2 includes a button B1a to start the cooling operation of air conditioner 2, a button B1b to start the automatic operation of air conditioner 2, a button B1c to start the heating operation of air conditioner 2, and a button B1d to change the timer setting of air conditioner 2. The term "button" used in this specification is a convenient term for explanation and is not limited to pressing actions, but also includes actions such as sliding or other actions. "Button" may also be referred to as "icon" or "operation element".

[0077] In this embodiment, when the surrounding conditions detected by the state detection unit 112 meet the first condition (for example, when the room temperature is high or the room temperature is above the first threshold), the display control unit 132 displays the operation unit D2 corresponding to the first condition (the operation unit D2 that emphasizes the cooling operation button B1a) (see reference). Figure 4 (b)). On the other hand, if the surrounding conditions detected by the state detection unit 112 meet a second condition that is different from the first condition (for example, if the room temperature is low or the room temperature is below the second threshold), the display control unit 132 displays the operation unit D2 corresponding to the second condition (the operation unit D2 that highlights the heating operation button B1c).

[0078] In this embodiment, the display control unit 132 arranges the multiple buttons B1a, B1b, and B1c from top to bottom according to the conditions met by the surrounding conditions detected by the state detection unit 112. For example, when the room temperature is high, the cooling operation button B1a is arranged at the top, the automatic operation button B1b is arranged in the middle, and the heating operation button B1c is arranged at the bottom. This display order is reversed when the room temperature is low. Furthermore, in this embodiment, as... Figure 4 As shown in (b), the buttons that are more likely to be operated by the user U (e.g., button B1a for cooling operation) are displayed larger than other buttons (e.g., buttons B1b, B1c, B1d). On the other hand, the buttons that are less likely to be operated by the user U (e.g., button B1c for heating operation) are displayed smaller than other buttons (e.g., buttons B1a, B1b, B1d).

[0079] [3.2 Scenario 2]

[0080] Figure 5 This is a diagram representing a control example for scenario 2. Scenario 2 represents a subsequent scenario after scenario 1. Figure 5 (a) in the figure represents the instant when user U operates the cooling operation button B1a. Figure 5 (b) in the diagram represents the state after user U operates button B1a to activate the cooling function. In this embodiment, if user U operates operation unit D2, the operation detection unit 115 detects the operation. Operation control unit 133 selects a control command from control command information I5 that corresponds to the operation detected by operation detection unit 115 and sends the selected control command to air conditioner 2. Thus, operation of air conditioner 2 begins.

[0081] The display control unit 132 simulates the airflow within the room (e.g., air volume and direction) based on the user U's operation detected by the operation detection unit 115 and the changed settings of the air conditioner 2. Furthermore, the display control unit 132 displays a display D1 including an image FI that visualizes the simulation results of the airflow. Specifically, the display control unit 132 overlays the image FI, which visualizes the simulation results of the airflow, onto the surrounding scenery.

[0082] like Figure 5As shown, the direction (wind direction) of the airflow in the virtual image FI is indicated, for example, by arrows representing the flow direction (vector) at various points in space. Furthermore, states without direction in space are indicated by the intensity of color. For example, the air volume in the virtual image FI is indicated by the intensity of color representing the magnitude (scalar) of the air volume at various points in space. For example, areas with higher air volume are displayed with darker colors, and areas with lower air volume are displayed with lighter colors. Alternatively, the virtual image FI can also be displayed by the flow of particles contained within the airflow. Examples of this case will be described in detail later.

[0083] In this embodiment, when the user U wearing smart glasses 1 changes position or orientation, the display control unit 132 displays a virtual image FI of the airflow corresponding to the new position or orientation. For example, when the user U wearing smart glasses 1 is facing the opposite side of the air conditioner 2, the display control unit 132 displays a display D1 that includes a virtual image FI of the airflow blown from the air conditioner 2 and flowing behind the user U.

[0084] In this embodiment, the display control unit 132 can switch between displaying simulation results showing the state of the home appliance immediately after the change and simulation results showing the state of the home appliance after a certain period of time since the change in the state of the home appliance. For example, regarding air conditioner 2, during periods when the room is hot, air conditioner 2 operates at a higher level, and the amount of air blown out by air conditioner 2 is greater. On the other hand, after the room cools down, air conditioner 2 only needs to operate at a lower level to maintain the room temperature, and the amount of air blown out by air conditioner 2 is relatively less.

[0085] Therefore, when the calculation unit 131 receives an operation from the user U regarding the operating state of the air conditioner 2 (for example, when the button B1a for cooling operation is pressed when the room is hot, or when the button B1c for heating operation is pressed when the room is cold), it outputs simulation results for the air conditioner 2 immediately after the operating state changes (e.g., immediately after operation begins) and simulation results for the air conditioner 2 after the room temperature has stabilized. Furthermore, the display control unit 132 displays images FI that can be switched according to the user U's operation: a virtual image FI visualizing the airflow after the simulation results of the air conditioner 2 immediately after the operating state changes (e.g., immediately after operation begins), and a virtual image FI visualizing the airflow after the simulation results of the air conditioner 2 after the room temperature has stabilized. With this structure, the user U can easily identify the airflow after the air conditioner 2's operating state changes and the airflow after the room temperature has stabilized.

[0086] Alternatively, the display control unit 132 may switch between displaying the simulation results based on airflow and the simulation results based on room temperature, based on the operation of the user U, as simulation results of the operating state of the air conditioner 2 from multiple perspectives.

[0087] Figure 6 This diagram illustrates a modified example of the control example for scenario 2. In this modified example, the display control unit 132 functions as display D1, displaying an overlay of a virtual image FI1 that visualizes the airflow after the simulation results of the air conditioner 2's operating state have just changed (e.g., immediately after operation begins) and an additional virtual image FI2 that visualizes the airflow after the simulation results of the air conditioner 2's operating state have stabilized in the room. With this structure, the user U can easily distinguish the airflow after the air conditioner 2's operating state has just changed and the airflow after the room temperature has stabilized.

[0088] [Scene 3.3]

[0089] Figure 7 This diagram illustrates a control example for scenario 3. Scenario 3 represents a scenario where the state detection unit 112 detects that the surrounding state meets predetermined conditions. In this embodiment, the display control unit 132, as described above, displays the operation unit D2 for operating the air conditioner 2 when the air conditioner 2 enters the field of view of the camera 40 (i.e., when the home appliance detection unit 111 detects that the air conditioner 2 has entered the image captured by the camera 40). On the other hand, when the air conditioner 2 is not in the field of view of the camera 40, the display control unit 132 sets the operation unit D2 for operating the air conditioner 2 to not be displayed.

[0090] However, in this embodiment, even when the air conditioner 2 is not within the field of view of the camera 40, if the state detection unit 112 detects that the surrounding conditions meet the prescribed conditions, the display D1 showing information associated with the air conditioner 2 and the operation unit D2 for operating the air conditioner 2 are also displayed. "The situation where the surrounding conditions meet the prescribed conditions" is, for example, the situation where the conditions for high temperature caution or low temperature caution described above are met according to the state detection unit 112.

[0091] When the status detection unit 112 detects that the conditions for high temperature alert are met, the display control unit 132, even if the air conditioner 2 is not in the field of view of the camera 40, will display the alert indicating that the room temperature is high (D3), the status display indicating the air conditioner 2 (D1), and the operation unit (D2) including the cooling operation button (B1a) and the non-operation button (B1e) (see reference). Figure 7On the other hand, when the status detection unit 112 detects that the conditions for low temperature attention are met, the display control unit 132 displays the attention alert D3 indicating that the room temperature is low, the display D1 indicating the status of the air conditioner 2, and the operation unit D2, including the heating operation button B1c and the non-operation button B1e, even when the air conditioner 2 is not in the field of view of the camera 40.

[0092] [Scene 4, 3.4]

[0093] Figure 8 This diagram illustrates a control example for scenario 4. Scenario 4 represents a situation where a specific item SI (e.g., a camera or musical instrument) requiring temperature or humidity management enters the field of view of camera 40. In this embodiment, as described above, the display control unit 132 uses the operation unit D2 for operating the air conditioner 2 to display when the air conditioner 2 enters the field of view of camera 40. On the other hand, when the air conditioner 2 is not in the field of view of camera 40, the display control unit 132 sets the operation unit D2 for operating the air conditioner 2 to not be displayed.

[0094] However, in this embodiment, when the air conditioner 2 is not within the field of view of the camera 40, but the surrounding conditions detected by the state detection unit 112 meet predetermined conditions (e.g., the room temperature is above a specific threshold corresponding to a specific item SI, or the room humidity is above a specific threshold corresponding to a specific item SI), and the specific item SI is within the field of view of the camera 40 (i.e., the specific item SI is detected by the item detection unit 113 to be within the image captured by the camera 40), the operation unit D2 for operating the air conditioner 2 is displayed. The correspondence between the type of specific item SI and the threshold of temperature or humidity corresponding to that specific item SI is stored in the storage unit 140 as part of the threshold information I2.

[0095] For example, if the status detection unit 112 detects that the room humidity is above a specific threshold corresponding to a specific item SI and the specific item enters the field of view of the camera 40, the display control unit 132 will, even if the air conditioner 2 is not in the field of view of the camera 40, display the attention alert D4 indicating that the room humidity is high for the specific item SI, display the status of the air conditioner 2 D1, and display the operation unit D2 including the button B1f for starting the dehumidification operation of the air conditioner 2 and the button B1e for selecting no operation (see reference). Figure 8On the other hand, when the status detection unit 112 detects that the room temperature is above a specific threshold corresponding to a specific item SI and the specific item SI enters the field of view of the camera 40, the display control unit 132 causes attention to be drawn to the higher room temperature for the specific item SI (D4), displays the status of the air conditioner 2 (D1) even when the air conditioner 2 is not in the field of view of the camera 40, and displays the operation unit (D2) including the button B1a for cooling operation of the air conditioner 2 and the button B1e for selecting no operation.

[0096] [Scene 5 of 3.5]

[0097] Figure 9 This diagram illustrates a control example for scenario 5, specifically showing the case where the first form of the operation unit D21 is displayed as operation unit D2. Scenario 5 represents the operation of the air conditioner 2. In this embodiment, when the display control unit 132 displays the first form of the operation unit D21 as operation unit D2, it does not cause the display of operation unit D2 to follow the movement or change of orientation of the user U wearing the smart glasses 1. On the other hand, when the display control unit 132 displays the second form of the operation unit D22, which is different from the first form, it causes the display of operation unit D2 to follow the movement or change of orientation of the user U wearing the smart glasses 1.

[0098] In detail, in this embodiment, when the air conditioner 2 is in operation and enters the field of view of the camera 40, the display control unit 132 uses the operation unit D21, which is used to operate the air conditioner 2 in its first form, as the operation unit D2. The operation unit D21 in its first form is an operation unit that cannot perform detailed settings for the airflow of the air conditioner 2. For example, the operation unit D21 in its first form includes a button B2a to stop the operation of the air conditioner 2, a button B2b to switch to the operation screen for setting the temperature of the air conditioner 2, and a button B2c to switch to the operation screen for setting the airflow of the air conditioner 2. Regarding the button B2c that displays the operation screen for setting the airflow of the air conditioner 2, detailed airflow settings cannot be performed simply by pressing this button B2c. Therefore, the operation unit D2, which includes the button B2c that switches to the operation screen for setting the airflow of the air conditioner 2, is an example of an "operation unit that cannot perform detailed settings for the airflow of the air conditioner 2".

[0099] When the display control unit 132 displays the first-mode operation unit D21 as the operation unit D2, it prevents the display of the operation unit D2 from following the movement or change of orientation of the user U. "Preventing the display of the operation unit D2 from following" means, for example, that the operation unit D2 is not displayed by the display control unit 132 when the air conditioner 2 deviates from the viewpoint of the camera 40 (the operation unit D2 is set to not be displayed). That is, when the user U wearing the smart glasses 1 is viewing in a direction different from that of the air conditioner 2, the display of the operation unit D2 also disappears when the air conditioner 2 disappears from the user U's field of vision.

[0100] Figure 10 This diagram shows a continuation of scenario 5, illustrating the second form of the operation unit D22 as operation unit D2. In this embodiment, by operating a specific button of the first form operation unit D21 (e.g., the airflow setting button B2c), the display control unit 132 displays the second form operation unit D22, which has been transferred from the first form operation unit D21. The second form operation unit D22 is an operation unit capable of detailed settings for the airflow of the air conditioner 2. For example, the second form operation unit D22 includes an operation member B3a for changing the airflow of the air conditioner 2, an operation member B3b for changing the left-right airflow direction of the air conditioner 2, an operation member B3c for changing the up-down airflow direction of the air conditioner 2, a cancel button B3d for stopping the airflow setting change, and a send button B3e for confirming the content of the airflow setting change and instructing the air conditioner 2.

[0101] In this embodiment, the operation of the air conditioner 2 is not changed only by the user U operating the aforementioned operating components B3a, B3b, and B3c. However, when the send button B3e is pressed, the content of the change is determined, and the operation of the air conditioner 2 is changed. In this embodiment, when the calculation unit 131 receives a specified operation from the user U related to the operating state of the air conditioner 2 (e.g., an operation on one of the operating components B3a, B3b, and B3c), it performs a simulation to predict the changed operating state of the air conditioner 2 (e.g., the changed airflow intensity and direction corresponding to the operation on operating components B3a, B3b, and B3c).

[0102] Furthermore, before changing the operating state of the air conditioner 2, the display control unit 132 overlays a display D1 on the surrounding scenery. The display D1 includes a virtual image FI of the airflow, which visualizes the simulation results of the changed operating state of the air conditioner 2 obtained by the calculation unit 131. In this embodiment, before the operating state of the air conditioner 2 is determined (for example, before the send button B3e is pressed), if one of the operation components B3a, B3b, and B3c has been operated multiple times, the calculation unit 131 performs the above simulation again each time the operation component B3a, B3b, or B3c is operated, and displays the display D1, which includes the virtual image FI of the airflow visualized by the simulation results, updated each time (i.e., the display is switched).

[0103] When the display control unit 132 displays the second form of the operation unit D2 as the operation unit D2, it causes the display of the operation unit D2 to follow the movement or change of orientation of the user U wearing the smart glasses 1. "Making the display of the operation unit D2 follow" means, for example, that even when the air conditioner 2 deviates from the viewpoint of the camera 40, the display control unit 132 continues to display the operation unit D22 within the user U's field of vision. That is, even when the user U wearing the smart glasses 1 is looking in a different direction than the air conditioner 2, the display of the operation unit D2 remains within the user U's field of vision.

[0104] As described above, in this embodiment, the display control unit 132 displays a virtual image FI of the airflow observed from a direction different from that of the air conditioner 2 when the user U wearing the smart glasses 1 is looking in that direction. That is, when the user U wearing the smart glasses 1 is facing the opposite side of the air conditioner 2, a virtual image FI of the airflow blowing from the air conditioner 2 and flowing towards the user U's rear is displayed. In this state, if the user U operates the operation components B3a, B3b, and B3c, the display control unit 132 changes the virtual image FI of the airflow flowing towards the user U's rear according to the operation of the operation components B3a, B3b, and B3c. Thus, the user U can check, for example, the virtual image FI of the airflow behind and to their side while adjusting the airflow settings by operating the operation components B3a, B3b, and B3c.

[0105] Figure 11This diagram illustrates a modified example of the control example in scenario 5. In this modified example, when the display control unit 132 receives a specified operation from the user U regarding the operating state of the air conditioner 2 (for example, when the user U operates one or more of the operation components B3a, B3b, and B3c), it displays, as display D1, a virtual image FIA ​​of the airflow after visualizing the simulation results of the air conditioner 2 before the change in its operating state and a virtual image FIB of the airflow after visualizing the simulation results of the air conditioner 2 after the change in its operating state. With this structure, the user U can easily compare the state before and after the change in its operating state. This further improves the convenience for the user U.

[0106] [Scene 6, 3.6]

[0107] Figure 12 This is a diagram representing scenario 6. In this embodiment, the operation control unit 133, based on the position of the object OB (object) configured by the user U in the virtual space, derives one or more motion parameters of the home appliance and sends the derived motion parameters to the home appliance.

[0108] For example, the display control unit 132 can configure the object OB in the virtual space instead of the operation unit D2, or configure the object OB in the virtual space in addition to the operation unit D2. The object OB itself is a virtual object generated by the display control unit 132. The user U can change the position of the object OB in the virtual space, for example, through fingertip operation. Figure 12 The example shown illustrates a scenario where the operating state of air conditioner 2 is changed based on the position of object OB to cause airflow from air conditioner 2 toward object OB.

[0109] Here, when the operation control unit 133 changes the operation state of the air conditioner 2 according to the position of the object OB, it does not send information indicating the position of the object OB to the air conditioner 2. In this embodiment, the appropriate operation parameters of the air conditioner 2 corresponding to the position of the object OB are calculated by the calculation unit 131. The operation parameters of the air conditioner 2 are, for example, similar to general control commands for an air conditioner operated by an infrared remote controller, parameters (control commands) such as selecting a certain air supply direction among multiple selectable levels of air supply direction in the left-right direction, selecting a certain air supply direction among multiple selectable levels of air supply direction in the up-down direction, or selecting a certain air supply intensity among multiple selectable levels of air supply intensity, are selected from the control command information I5.

[0110] Figure 13 This is a diagram representing a variation of scenario 6. Figure 13The example shown illustrates how changing the operating state of air conditioner 2 based on the location of object OB can prevent airflow from flowing from air conditioner 2 towards object OB. For instance, by positioning object OB in a location where the airflow is not desired (such as a bed or a place where ornamental plants are placed), user U can adjust the room temperature or humidity so that the airflow is directed away from specific areas within the room.

[0111] [Scene 7, Part 3.7]

[0112] Figure 14 This is a diagram representing scenario 7. In this embodiment, when a different home appliance (a second home appliance) exists compared to the target home appliance (the first home appliance), the calculation unit 131 performs a simulation reflecting the influence of the second home appliance. For example, when the home appliance detection unit 111 detects the electric fan 3, the calculation unit 131 performs a simulation reflecting the airflow from the air conditioner 2, reflecting the influence of the electric fan 3.

[0113] In this embodiment, the home appliance detection unit 111 determines the orientation of the head of the electric fan 3 (the airflow direction of the electric fan 3) based on the image captured by the camera 40. Furthermore, the calculation unit 131 simulates the airflow from the air conditioner 2 in accordance with the determined airflow direction of the electric fan 3. Finally, the display control unit 132 displays a display D1 that includes a virtual image FI of the airflow, which visualizes the simulation results of the airflow from the air conditioner 2 reflecting the airflow direction of the electric fan 3.

[0114] Additionally, the home appliance detection unit 111 can also detect communication between the smart glasses 1 and the electric fan 3, obtaining information representing the airflow of the electric fan 3 based on images of the multi-bladed electric fan captured at time intervals during which the rotation angle of the impeller can be detected by the camera 40. In this case, the control unit 130 can also simulate the airflow from the air conditioner 2 in response to the specific airflow direction and airflow of the electric fan 3, and display the virtual image FI, which visualizes the airflow results of the simulation, on the display D1.

[0115] In this embodiment, when the receiving unit 120 receives input from a user U representing the state of an element (e.g., window 4 or ventilation fan 5), the calculation unit 131 simulates the state of the element (e.g., the open / closed state of window 4, the operating state of ventilation fan 5) based on the input received by the receiving unit 120. Information about the state of this element cannot be obtained. Furthermore, the display control unit 132 displays a display D1 containing a virtual image FI of the airflow, which visualizes the simulation results of the airflow from the air conditioner 2 reflecting the state of the element. Alternatively, the state of elements (e.g., window 4 or ventilation fan 5) for which no state information can be obtained can also be acquired by the camera 40.

[0116] [4. Simulation of airflow]

[0117] Next, an example of airflow simulation will be explained.

[0118] [4.1 Obtaining information about the surrounding environment]

[0119] In this embodiment, the computing unit 131 identifies one or both of the room layout where home appliances are located and the arrangement of items (e.g., furniture) within the room (i.e., detected by the surrounding environment detection unit 114) based on the image captured by the camera 40, and stores the identified room layout and item arrangement as surrounding environment information I4 in the storage unit 140. Furthermore, the computing unit 131 simulates the airflow by using the walls and surfaces of items within the room corresponding to the identified room layout as constraints on the airflow from the air conditioner 2.

[0120] Alternatively, the control unit 130 may replace the acquisition of ambient information by the camera 40, or, in addition to this, the receiving unit 120 may receive input from one or both users U representing the layout of a room equipped with home appliances or the arrangement of items within the room. In this case, the receiving unit 120 stores all or part of the ambient information I4 based on the user U's input into the storage unit 140. Furthermore, the calculation unit 131 simulates the airflow by using the surfaces of the walls and items within the room corresponding to the room layout input from the user U as constraints on the airflow from the air conditioner 2.

[0121] [4.2 Approximation using particles]

[0122] Figure 15 This is a diagram illustrating an example of the simulation results of airflow. In this embodiment, the calculation unit 131 reduces the computational load by calculating the motion of multiple particles P contained within the airflow, instead of completely calculating the airflow itself. Furthermore, instead of completely calculating the motion of multiple particles P, the calculation unit 131 in this embodiment uses pre-prepared simulation results (the motion of multiple particles P) corresponding to one or more operating parameters of the air conditioner 2 (the airflow volume and direction set in stages), or modifies them to fit the room layout and the positions of objects within the room, thereby deriving the desired simulation results. The display control unit 132 then displays a virtual image FI visualizing the airflow after the derived simulation results are visualized. Specifically, the following conditions are applied to the particles P to simulate the airflow.

[0123] (1) The initial state of particle P (the state immediately after being blown out of air conditioner 2).

[0124] (1-1) The airflow volume is approximated by averaging the initial velocities of multiple particles P. For multiple particles P, an initial velocity following a certain distribution is assigned.

[0125] (1-2) The direction of the airflow is approximated by averaging the directions of the initial velocities of multiple particles P. For multiple particles P, a direction is assigned to the initial velocity following a certain distribution.

[0126] (1-3) Assume that the initial position of particle P occurs in the vicinity of air conditioner 2 in a distributed manner.

[0127] (1-4) The greater the initial velocity of particle P, the greater the number of particles P generated. Therefore, when the velocity of particle P is relatively high, it is possible to suppress the appearance of dispersion. In addition, by increasing the number of particles P generated by the greater the initial velocity of particle P, the airflow volume can be represented.

[0128] (2) The subsequent actions of particle P (its state after a certain period of time since it was blown out of air conditioner 2).

[0129] (2-1) The speed of particle P decreases over time.

[0130] (2-2) In a collision with the walls or surfaces of objects in the room (hereinafter referred to collectively as "walls"), the velocity of particle P does not decrease significantly, and the angle of reflection relative to the wall is set to be larger than the angle of incidence. This will be described in detail later.

[0131] (2-3) The particle P experiences an upward acceleration after a certain period of time. Furthermore, the acceleration is set to different values ​​for cooling and heating operations.

[0132] (2-4) The condition for the annihilation of particle P is that a certain amount of time has elapsed. However, the more collisions it makes with the wall, the faster it is annihilated.

[0133] Figure 16 This diagram illustrates the settings for the motion of particle P upon collision with wall W. Generally, it is known that when an air particle P collides with wall W, the angle of incidence α and the angle of reflection β1 are equal. However, in this embodiment, to reduce the computational burden of the simulation, it is assumed that collisions do not occur between the particle P reflected from wall W and subsequent particles P moving towards wall W. Therefore, in this embodiment, the reflection angle β2 is set such that when particle P collides with wall W, the angle of reflection β2 is larger than the angle of incidence α (i.e., particle P flows towards the vicinity of wall W). Thus, it is possible to approximate the actual motion of particle P colliding with wall W without calculating collisions with subsequent particles P.

[0134] [4.3 Encounter Detection]

[0135] In this embodiment, the calculation unit 131 performs a process called "encounter determination." "Encounter determination" is a process that counts the number of particles P blown out of the air conditioner 2 that encounter the user U wearing smart glasses 1 per unit time during the simulation of determining the motion of particle P. The number of particles P that encounter the user U is related to the intensity of the airflow felt by the user U. That is, the fewer particles P that encounter the user U, the weaker the wind felt by the user U. On the other hand, the more particles P that encounter the user U, the stronger the wind felt by the user U. Therefore, by calculating the number of particles P that encounter the user U, the intensity of the wind felt by the user U can be numerically quantified.

[0136] Here, when the cooling operation is in progress, the calculation unit 131 positions the area for collision determination at a location corresponding to the actual height of the user U's body, and counts the number of collisions (passage counts) of particles P with respect to that area. On the other hand, when the heating operation is in progress, the calculation unit 131 positions the area for collision determination at a location lower than the actual height of the user U's body (for example, at a height near the user U's feet), and counts the number of collisions (passage counts) of particles P with respect to that area. With this structure, the user U's sensations during heating operation can be more accurately reflected.

[0137] [4.4 Example of displaying a virtual image of airflow]

[0138] like Figure 15 As shown, the display control unit 132 displays the motion of multiple particles P as a virtual image FI of the airflow after visualizing the simulation results. That is, it displays the motion of multiple particles P blown out from near the outlet of the air conditioner 2. This motion of particles P is easier for the user U to intuitively understand compared to representing the airflow with lines and arrows.

[0139] In this embodiment, the display control unit 132 displays a display D1 including an index D1b that represents the "intensity of the wind felt by user U," which is numerically represented based on the result of the "contact determination" described above. In this embodiment, the index D1b is displayed in the form of a bar. That is, the state where the wind does not contact user U is set to "0," and the state where the wind contacts user U at a certain level or more is set to "100," indicating the level at which the wind contacts user U. Alternatively, the index D1b may be represented instead of the above example by a colored bar that varies in color according to the level, a character such as "light breeze" or "strong wind," or a combination thereof.

[0140] Figure 17This is a diagram illustrating a first display example, such as FI, of the airflow visualization. In this modified example, among the multiple particles P blown out from air conditioner 2, particles PA that will not touch user U are displayed in the first color (e.g., blue). On the other hand, among the multiple particles P blown out from air conditioner 2, particles PB that may touch user U are displayed in the second color (e.g., red). Through this first display example, it is also possible to easily convey to user U the level at which the wind touches user U.

[0141] Figure 18 This is a diagram illustrating a second display example, similar to FI, of the airflow visualization. In this variation, among the multiple particles P blown out from air conditioner 2, the particle PC that most recently encountered user U is displayed in the first color (e.g., yellow). Among the multiple particles P blown out from air conditioner 2, the particle PD that previously encountered user U is displayed in the second color (e.g., red). Among the multiple particles P blown out from air conditioner 2, the particle PE, which is other than particles PB and PC, is displayed in the third color (e.g., blue). This second display example allows user U to easily understand and grasp the level at which the wind encountered them.

[0142] [4.5 Setting up the coordinate system]

[0143] Figure 19 This is a diagram showing the coordinate systems used in this embodiment. Three coordinate systems are used in this embodiment.

[0144] (1) A coordinate system based on the position of air conditioner 2 (first coordinate system S1)

[0145] (2) A coordinate system (second coordinate system S2) with the position of user U (smart glasses 1) as the reference.

[0146] (3) The coordinate system (third coordinate system S3) represents the coordinates in the real space of the room equipped with air conditioner 2.

[0147] In this embodiment, the calculation unit 131 performs a simulation of the operating state of the air conditioner 2 (e.g., a simulation to determine the motion of particle P) in the first coordinate system S1. However, the calculation unit 131 does not calculate the collision of particle P with the wall (the wall of the room or the surface of an object) and its subsequent flow in the first coordinate system S1. Furthermore, the display control unit 132 transforms the simulation results performed in the first coordinate system S1 into the content of the third coordinate system S3 and maintains it. That is, the display control unit 132 establishes and maintains the correlation between the simulation results performed in the first coordinate system S1 and the coordinates in the real space of the room in the third coordinate system S3. The third coordinate system S3 is an example of a coordinate system whose reference position is different from that of the first coordinate system S1.

[0148] On the other hand, the calculation unit 131 performs the aforementioned "collision determination" processing in the second coordinate system S2. For example, the calculation unit 131 transforms the simulation results used to calculate the motion of particle P obtained in the first coordinate system S1 to the second coordinate system S2, and counts the number of particles P that collide with the area set in the second coordinate system S2 for "collision determination". Furthermore, the display control unit 132 determines the display positions of display D1 and operation unit D2 in the second coordinate system S2.

[0149] The calculation unit 131 uses the simulation results, which have been transformed into the third coordinate system S3, to calculate the collision of particle P with the wall W (the wall of the room or the surface of an object) and the subsequent flow in the third coordinate system S3, correcting the simulation results calculated in the first coordinate system S1. In the third coordinate system S3, the layout of the room equipped with the air conditioner 2 and the objects (e.g., furniture) are set.

[0150] Furthermore, the display control unit 132 generates an image, such as FI (motion of multiple particles P), that visualizes the airflow based on the simulation results corrected in the third coordinate system S3, and displays the generated image as part of the display D1. In this case, the display control unit 132 resets the positional relationship between the air conditioner 2 and the third coordinate system S3 whenever the air conditioner 2 enters the field of view of the camera 40. Thus, even if the initial positional relationship between the air conditioner 2 and the third coordinate system S3 is not correctly set, the positional relationship between the air conditioner 2 and the third coordinate system S3 is corrected.

[0151] Based on this coordinate system, processing is less prone to delays even when the smart glasses 1 is moving violently (camera 40 is moving violently). Furthermore, if the simulation calculations are performed in either the first coordinate system S1 or the third coordinate system S3, even if the smart glasses 1's own coordinates are incorrect, it is less likely to be affected. Moreover, considering the rotation caused by changes in wind direction, the first coordinate system S1 makes it easier to establish calculation algorithms and formulas for the wind blowing from the air conditioner 2. Furthermore, since the wind (approximately particle P) is a simulation result, without the need to obtain the relative position of particle P and air conditioner 2, the calculation of the air conditioner 2 reference becomes easier.

[0152] As a first variation regarding the coordinate system, the calculation unit 131 can also perform a complete simulation of airflow, including the collision of particle P with the wall (the wall of the room or the surface of an object), within the first coordinate system S1, and display the simulation results held by the first coordinate system S1 as display D1. According to this first variation, although there may be processing delays when the smart glasses 1 moves violently (the camera 40 moves violently), the impact can be reduced even if there is a deviation in the current coordinates recognized by the smart glasses 1.

[0153] As a second variation regarding the coordinate system, the calculation unit 131 can also perform a complete simulation of airflow, including the collision of particle P with the wall (the wall of the room or the surface of an object), within the second coordinate system S2, and display the simulation results held by the second coordinate system S2 as display D1. According to this second variation, although there may be processing delays when the smart glasses 1 moves violently (the camera 40 moves violently), calculations and displays can be performed without problems in other situations.

[0154] As a third variation regarding the coordinate system, the calculation unit 131 can also perform a complete simulation of the airflow in the third coordinate system S3 (i.e., simulate the operating state of the air conditioner 2 by establishing a connection with the coordinates within the actual space of the room), and the simulation results maintained by the third coordinate system S3 are displayed as display D1. That is, the coordinates used in the calculation will not be affected by the movement of the smart glasses 1, and the calculation results are established to correspond with the actual coordinates. According to this third variation, if the coordinates of the smart glasses 1 itself are incorrect, the result will also be incorrect, but since the recalculation of the airflow is suppressed even if the smart glasses 1 moves, the impact can be reduced even if the air conditioner 2 is missed or the camera 40 moves violently.

[0155] [5. Advantages]

[0156] In this embodiment, the smart glasses 1 have a control unit 130 that displays one or both of a display D1 or an operation unit D2 using virtual space technology. The display D1 represents information associated with the air conditioner 2, and the operation unit D2 is used to operate the air conditioner 2. With this structure, the display size and configuration of the buttons can be dynamically changed compared to a conventional device controller. This provides a more user-friendly interface for the user U, improving the user's convenience.

[0157] In this embodiment, when the surrounding conditions detected by the detection unit 110 meet a first condition, the control unit 130 displays the operation unit D2 corresponding to the first condition; when the surrounding conditions detected by the detection unit 110 meet a second condition different from the first condition, the control unit 130 displays the operation unit D2 corresponding to the second condition. With this structure, the display of the operation unit D2 is changed according to the situation. This further improves the convenience for the user U.

[0158] In this embodiment, the control unit 130 displays the operation unit D2 when the air conditioner 2 is within the image captured by the camera 40, and also displays the operation unit D2 when the surrounding conditions detected by the detection unit 110 meet predetermined conditions, even when the air conditioner 2 is not within the image captured by the camera 40. With this structure, suggestions can be made to the user U at any time, regardless of whether the situation is hot, cold, or humid. This further enhances the convenience for the user U.

[0159] In this embodiment, the control unit 130 activates the operation unit D2 when the surrounding conditions meet predetermined requirements and a specific item SI is detected. With this structure, even when there are items with low tolerance to heat or humidity, suggestions can be made to the user U at any given time. This further enhances the convenience for the user U.

[0160] In typical infrared remote controls, when changing the airflow direction, it's unclear where the actual airflow will be directed. Furthermore, it's difficult to accurately assess the overall airflow settings for the room. However, in this embodiment, a virtual image of the airflow (FI) is displayed. Therefore, when setting the airflow of the air conditioner 2, it's easy to know, for example, that adjusting the airflow a few levels to the right will direct the airflow towards the user, and adjustments are prevented if the image is incorrect. This further enhances the convenience for the user. Moreover, since three-dimensional information is used, it's possible to view the overall airflow from other angles, making it easier to accurately assess the overall airflow settings for the room.

[0161] (Second Implementation)

[0162] Next, the second embodiment will be described. The smart glasses 1 of the second embodiment differ from the first embodiment in that it displays information associated with home appliances (D1) but does not display the operation unit (D2). The structure described below, except for the parts described below, is the same as that of the first embodiment.

[0163] Figure 20 This diagram illustrates a control example of the second embodiment. In this embodiment, the display control unit 132 displays information associated with the air conditioner 2 (display D1), but does not display the operation unit D2. The user U checks the status of the air conditioner 2 and the virtual image FI of the airflow while operating the air conditioner 2 using a conventional infrared remote control RC.

[0164] With this structure, user U can operate air conditioner 2 while simultaneously checking its status and airflow via a virtual space-based display D1 (similar to FI). Therefore, user U's convenience is improved.

[0165] (Third Implementation)

[0166] Next, the third embodiment will be described. The third embodiment differs from the first embodiment in that the information processing device operated by the user U is not smart glasses 1, but a terminal device 1A such as a smartphone or tablet computer. The structure described below, except for the smart glasses 1, is the same as that of the first embodiment.

[0167] Figure 21 This diagram illustrates the terminal device 1A according to the third embodiment. Like the smart glasses 1 of the first embodiment, the terminal device 1A includes a camera 40, a microphone 50, a communication module 60, and a control board 70. The terminal device 1A includes a display device 30A, which is a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 30A includes an electrostatic capacitive contact sensor disposed overlaid on the display screen.

[0168] The terminal device 1A is equipped with an application program that implements the detection unit 110, the receiving unit 120, and the control unit 130 described above. The terminal device 1A is an example of an "information processing device," which is a device controller for operating home appliances. The terminal device 1A in this embodiment is an example of an "information processing system."

[0169] In this embodiment, the display control unit 132 causes the display screen of the display device 30A to display the surrounding scenery captured in real time by the camera 40. Furthermore, the display control unit 132 causes the display screen of the display device 30A to display a virtual space-based display D1 and / or an operation unit D2. That is, the display control unit 132 displays the virtual space-based display D1 and / or the operation unit D2 on the display device 30A, and overlays the surrounding scenery captured in real time by the camera 40 onto the display screen. Display D1 includes a display D1a indicating the status of home appliances and / or an image FI visualizing the airflow from the air conditioner 2. The operation detection unit 115 accepts the user U's operation based on the user U's operation on the display screen of the display device 30A.

[0170] Based on this structure, the application can be launched when needed without importing new devices, thus enabling the use of an augmented reality remote control. Therefore, user convenience is improved.

[0171] The terminal device 1A of this embodiment can implement all the control examples described regarding the smart glasses 1 of the first embodiment. For example, when the user U faces the terminal device 1A towards the side opposite to the air conditioner 2, the display control unit 132 causes the display D1 to display a virtual image FI including the airflow blowing from the air conditioner 2 and flowing behind the user U. Regarding the description of this embodiment, it is sufficient to refer to "smart glasses 1" as "terminal device 1A" and "user U wearing smart glasses 1" as "user U holding terminal device 1A" in the description of the first embodiment. In this embodiment, "the case where user U changes orientation" can include "the case where user U changes orientation of terminal device 1A".

[0172] (Fourth implementation)

[0173] Next, the fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that a portion of the functions of the smart glasses 1 in the first embodiment are provided in other devices. The structure other than that described below is the same as that of the first embodiment.

[0174] Figure 22 This diagram illustrates the information processing system 200 according to the fourth embodiment. The information processing system 200 includes a host device 210 and one or more smart glasses 1 (or terminal devices 1A). The host device 210 has some or all of the functions of the detection unit 110, the receiving unit 120, and the control unit 130 described above. The host device 210 is capable of communicating with the smart glasses 1 (or terminal devices 1A). The host device 210 may be, for example, a computer installed in a home, but it may also be a server device on an external network.

[0175] With this structure, similar to the first embodiment, the convenience for user U is improved. According to this embodiment, multiple users U can share the calculation results of the host device 210, and multiple users U can view the same content. Therefore, the convenience for user U is further improved.

[0176] The above describes several implementation methods and variations, but the implementation methods and variations are not limited to the examples described above. For example, the display of display D1 and operation unit D2 is not limited to augmented reality (AR) and mixed reality (MR), but may also be virtual reality (VR) or alternative reality (SR), etc. For example, the surrounding scenery superimposed on display D1 can also be displayed through virtual reality. In addition, the past operating status of home appliances can also be displayed through alternative reality.

[0177] Furthermore, the smart glasses 1 and terminal device 1A are not limited to devices for operating the air conditioner 2. For example, the smart glasses 1 can also display a recipe by overlaying it with the surrounding scenery. The recipe is an example of "information associated with home appliances." In this case, the user U can cook while viewing the recipe and can operate the device without washing hands, thus eliminating situations where the screen is too small to see or press.

[0178] According to at least one embodiment described above, the information processing system can improve convenience by having one or both of a display that uses virtual space technology to display information associated with home appliances or an operation unit for operating home appliances.

[0179] The following is a brief description of several information processing systems.

[0180] [1]: An information processing system comprising a control unit, which uses virtual space technology to display information associated with home appliances or to display one or both of the operation units used to operate the home appliances.

[0181] [2]: In the information processing system described in [1], when the control unit receives the user's specified operation related to the operating state of the aforementioned home appliance, it displays the image of the simulation result of the operating state of the aforementioned home appliance immediately after the change and the image of the simulation result of the operating state after a certain period of time since the change of the aforementioned home appliance overlay or switchably.

[0182] [3]: In the information processing system described in [1] or [2], when the control unit displays the operation unit in the first form, it does not make the display of the operation unit follow the user's movement or change of orientation. When the operation unit is displayed in the second form, which is different from the first form, it makes the display of the operation unit follow the user's movement or change of orientation.

[0183] [4]: In the information processing system described in [3], the aforementioned home appliance is an air conditioner; the operation unit of the first form is an operation unit that cannot perform detailed settings of the airflow of the air conditioner; the operation unit of the second form is an operation unit that can perform detailed settings of the airflow of the air conditioner.

[0184] [5]: In any of the information processing systems described in [1] to [4], the control unit simulates the operating state of the home appliance in a coordinate system based on the position of the home appliance, and displays the display including the simulation results.

[0185] [6]: In any of the information processing systems described in [1] to [4], the control unit establishes a connection with the coordinates in the real space of the room where the home appliances are installed and performs a simulation of the operating state of the home appliances, so that the display including the simulation results is displayed.

[0186] [7]: In any of the information processing systems described in [1] to [4], the control unit simulates the operating state of the home appliance in a coordinate system based on the position of the home appliance, transforms the simulation result into content that is associated with the coordinates in the real space of the room where the home appliance is located, and displays the display including the transformed simulation result.

[0187] Several embodiments of the present invention have been described, but these embodiments are merely illustrative and not intended to limit the scope of the invention. These embodiments can be implemented in a wide variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope or spirit of the invention, and are included in the scope of the invention as described in the claims and its equivalents.

[0188] Label Explanation

[0189] 1…Smart glasses; 2…Air conditioner (home appliance, first home appliance); 3…Electric fan (second home appliance); 40…Camera; 70…Control board; 110…Detection department; 120…Acceptance department; 130…Control department; 200…Information processing system.

Claims

1. An information processing system, wherein, have: The control unit, through virtual space technology, displays one or both of the following displays or the following operation units, wherein the displays represent information associated with the home appliance as an air conditioner, and the operation units are used to operate the home appliance. camera; as well as The detection department is capable of detecting the ambient temperature or humidity, which are indicators to be adjusted. The aforementioned testing department determines whether the detected indicators meet the prescribed conditions. When the aforementioned home appliance is included in the image captured by the aforementioned camera, the aforementioned control unit causes the aforementioned operation unit to display that, if the display conditions, including the aforementioned indicator detected by the aforementioned detection unit meeting the aforementioned predetermined conditions, are met, the aforementioned operation unit will also display that, even if the aforementioned home appliance is not included in the image captured by the aforementioned camera, the aforementioned predetermined conditions are at least one of the following: the ambient temperature of the aforementioned indicator detected by the aforementioned detection unit is above a threshold set for a specific item, or the ambient humidity of the aforementioned indicator detected by the aforementioned detection unit is above a threshold set for the aforementioned specific item.

2. The information processing system as described in claim 1, wherein, The above-mentioned conditions are at least one of the following: the ambient temperature of the above-mentioned indicator detected by the above-mentioned detection unit is above the threshold corresponding to high temperature, or the ambient temperature of the above-mentioned indicator detected by the above-mentioned detection unit is below the threshold corresponding to low temperature.

3. The information processing system as described in claim 1, wherein, The aforementioned detection unit is capable of detecting the specific item mentioned above, as an item present in the surrounding environment. The above-mentioned display conditions are based on the fact that the above-mentioned detection department detected the above-mentioned specific item and met the above-mentioned specified conditions.

4. The information processing system as described in claim 1, wherein, The specific items mentioned above are cameras or musical instruments.

5. The information processing system as described in claim 3, wherein, The specific items mentioned above are cameras or musical instruments.

6. The information processing system as described in claim 1 or 2, wherein, When the state of the surroundings detected by the detection unit meets the first condition, the control unit displays the operation unit corresponding to the first condition; when the state of the surroundings detected by the detection unit meets the second condition, which is different from the first condition, the control unit displays the operation unit corresponding to the second condition.

7. The information processing system as described in claim 1 or 2, wherein, When the control unit receives a user's requested operation related to the operating state of the aforementioned home appliance, before changing the operating state of the aforementioned home appliance, it displays the aforementioned image, which includes a visualization of the simulation results of the changed operating state of the aforementioned home appliance, overlaid with the surrounding scenery.

8. A device controller, wherein, have: The control unit, through virtual space technology, displays one or both of the following displays or the following operation units, wherein the displays represent information associated with the home appliance as an air conditioner, and the operation units are used to operate the home appliance. camera; as well as The testing department is capable of detecting the ambient temperature or humidity, which are indicators that need to be adjusted. The aforementioned testing department determines whether the detected indicators meet the prescribed conditions. When the aforementioned home appliance is included in the image captured by the aforementioned camera, the aforementioned control unit causes the aforementioned operation unit to display that, if the display conditions, including the aforementioned indicator detected by the aforementioned detection unit meeting the aforementioned predetermined conditions, are met, the aforementioned operation unit will also display that, even if the aforementioned home appliance is not included in the image captured by the aforementioned camera, the aforementioned predetermined conditions are at least one of the following: the ambient temperature of the aforementioned indicator detected by the aforementioned detection unit is above a threshold set for a specific item, or the ambient humidity of the aforementioned indicator detected by the aforementioned detection unit is above a threshold set for the aforementioned specific item.

9. A computer program product comprising a computer program, wherein, The above computer program causes the computer to execute: The display step involves using virtual space technology to display one or both of the following displays or the following operating units, wherein the displays represent information associated with the home appliance as an air conditioner, and the operating units are used to operate the home appliance. as well as The testing procedure involves measuring the ambient temperature or humidity, which is the indicator to be adjusted. The above detection steps include determining whether the detected indicators meet the specified conditions. The above-described display step causes the operation unit to display when the aforementioned home appliance is included in the image captured by the camera. Even when the aforementioned home appliance is not included in the image captured by the camera, the operation unit is still displayed when the display conditions, including the aforementioned indicator detected in the above-described detection step, meet the aforementioned specified conditions. The aforementioned specified conditions are at least one of the following: the ambient temperature of the aforementioned indicator detected in the above-described detection step is above a threshold set for a specific item, or the ambient humidity of the aforementioned indicator detected in the above-described detection step is above a threshold set for the aforementioned specific item.

Images