Environmental characteristic visualization
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-10
- Publication Date
- 2026-08-11
Smart Images
Figure CN122547441A_ABST
Abstract
Description
Cross-reference to related applications
[0001] This application claims priority to U.S. Provisional Application No. 63 / 756,652, filed on February 10, 2025, which is incorporated herein by reference in its entirety. Technical Field
[0002] This disclosure relates in general to systems, methods, and apparatus for visualizing the environmental characteristics of a physical environment.
[0003] summary
[0004] In various specific implementations, the equipment displays a visualization of the physical environment, such as a floor plan or a dollhouse. Furthermore, the locations of physical devices within the physical environment (such as lights, locks, or blinds) can be represented by their corresponding locations within the visualization of the physical environment. Attached Figure Description
[0005] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.
[0006] Figure 1 It is a block diagram based on some specific implementation examples of operating environments.
[0007] Figures 2A to 2L Examples of XR environments during various time periods based on some specific implementations are shown.
[0008] Figure 3 It is a flowchart representation of a method based on the environmental characteristics of a visualized physical environment in some specific implementations.
[0009] Figure 4 It is a block diagram based on some specific implementation examples of controllers.
[0010] Figure 5 It is a block diagram based on some specific implementation examples of electronic devices.
[0011] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Summary of the Invention
[0012] The various embodiments disclosed herein include devices, systems, and methods for displaying status indicators. In various embodiments, the method is performed by a device having a display, one or more processors, and non-transitory memory. The method includes displaying a virtual representation of a physical environment. The method includes obtaining status information indicating the status of a physical device from a physical device at a physical location within the physical environment. The method includes displaying a virtual representation of the physical device at a virtual device location corresponding to the physical location of the device in the physical environment within the virtual representation of the physical environment. The method includes determining multiple values of environmental characteristics of the physical environment at multiple environmental physical locations within the physical environment, wherein the multiple values are at least partially based on the status information. The method includes displaying indications of the multiple values of the environmental characteristics at multiple virtual environmental locations corresponding to the physical location of the environmental location in the virtual representation of the physical environment.
[0013] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Detailed Implementation
[0014] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.
[0015] As described above, in various specific implementations, electronic devices can display a virtual representation of the physical environment, wherein virtual representations of physical devices within the physical environment are displayed at corresponding locations within the virtual representation. Furthermore, these virtual representations of physical devices can indicate the state of the physical devices. For example, a lamp's virtual representation can be bright, with the indicator light in an "on" state, or dim, with the indicator light in an "off" state. Moreover, these virtual representations of physical devices can be interactive, acting as indications of changes in the state of the physical devices. For example, when a lamp's virtual representation is activated, the electronic device can send a command to the lamp to change its state. In response to receiving such a command, the lamp can change its state from "on" to "off," and vice versa.
[0016] In addition to displaying a virtual representation of the physical devices, in various specific implementations, the virtual representation of the physical environment is also displayed together with a visualization of environmental characteristics affected by one or more physical devices. For example, a visualization of lighting in the physical environment can be displayed based on the state of one or more lights in the physical environment (displayed together with the virtual representation of the physical environment).
[0017] Figure 1 This is a block diagram of an example operating environment 100 according to some specific implementations. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring more relevant aspects of the example implementations disclosed herein. Therefore, as a non-limiting example, operating environment 100 includes a controller 110 and electronic devices 120.
[0018] In some implementations, controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, controller 110 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 4 The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to a physical environment 105. For example, the controller 110 is a local server located within the physical environment 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some embodiments, the controller 110 is communicatively coupled to the electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within a housing of the electronic device 120. In some embodiments, the functionality of the controller 110 is provided by and / or combined with the electronic device 120.
[0019] In some embodiments, electronic device 120 is configured to provide an XR experience to a user. In some embodiments, electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some embodiments, electronic device 120 presents XR content to the user via display 122 when the user is virtually or physically present within a physical environment 105, which includes a table 107 within the field of view 111 of electronic device 120. Thus, in some embodiments, the user holds electronic device 120 in one or both of his / her hands. In some embodiments, when providing XR content, electronic device 120 is configured to display XR objects (e.g., XR cylinder 109) and implement video pass-through on display 122 to the physical environment 105 (e.g., a representation 117 including table 107). The following is relative to Figure 5 The electronic device 120 is described in more detail.
[0020] In some embodiments, the user wears the electronic device 120 on his / her head. For example, in some embodiments, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted enclosure (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various embodiments, the electronic device 120 surrounds the user's field of view. In some embodiments, the electronic device 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user no longer wears the electronic device 120 but holds it in their hand, with the display facing the user's field of view and the camera facing the physical environment 105. In some embodiments, the handheld device may be placed within an enclosure that can be worn on the user's head. In some embodiments, the electronic device 120 is replaced by an XR pod, enclosure, or chamber configured to present XR content, in which the user no longer wears or holds the electronic device 120.
[0021] Figures 2A to 2L An XR environment 200 based on the physical environment of a bedroom is illustrated, which is located from the user's perspective of an electronic device and is displayed at least partially by the device's displays. In various specific embodiments, the electronic device includes multiple displays (e.g., a left display positioned in front of the user's left eye and a right display positioned in front of the user's right eye) configured to provide a stereoscopic view of the XR environment 200. For ease of illustration, Figures 2A to 2L An XR environment 200 is illustrated on a single display among the multiple displays.
[0022] In various embodiments, the user's perspective originates from the position of the image sensor of the electronic device. For example, in various embodiments, the electronic device is a handheld electronic device, and the user's perspective originates from the position of the image sensor of the handheld electronic device facing the physical environment. In various embodiments, the user's perspective originates from the user's position on the electronic device. For example, in various embodiments, the electronic device is a head-mounted electronic device, and the user's perspective originates from the user's position facing the physical environment, which, in the absence of a head-mounted electronic device, typically approximates the user's field of view. In various embodiments, the user's perspective originates from the position of the user's avatar. For example, in various embodiments, the XR environment 200 is a virtual environment, and the user's angle originates from the position of the user's avatar or other representation facing the virtual environment.
[0023] Figures 2A to 2L An XR environment 200 is illustrated during a series of time periods. In various specific implementations, each time period can be an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.
[0024] The XR environment 200 includes multiple objects, including one or more real objects (e.g., bed 211, dressing table 212, ceiling furniture 213, window 214, floor lamp 215, vent 216, thermostat 217, robotic vacuum cleaner 218, and hand 292) and one or more virtual objects (e.g., virtual clock 221 and virtual audiobook player 222). Ceiling furniture 213 includes a ceiling fan 231 and a chandelier 232. Window 214 includes blinds 241 and window panels 242.
[0025] In various implementations, certain objects (such as real objects and virtual audiobook players 222) are presented at a location within the XR environment 200, for example, at a location defined by three coordinates in a three-dimensional (3D) XR coordinate system. Therefore, as the electronic device moves within the XR environment 200 (e.g., changes position and / or orientation), the object moves on the display of the electronic device but maintains its (potentially time-dependent) position within the XR environment 200. Such virtual objects that move on the display in response to movement of the electronic device but maintain their position within the XR environment 200 are referred to as world-locked objects. In various implementations, certain virtual objects (such as a virtual clock 221) are displayed at a location on the display such that the object remains stationary on the display of the electronic device as the electronic device moves within the XR environment 200. Such virtual objects that maintain their position on the display in response to movement of the electronic device are referred to as head-locked objects or display-locked objects.
[0026] Figures 2A to 2LAn example is shown: gaze position indicator 291, which indicates the user's gaze position, for example, the position the user is looking at within the XR environment 200. Although in Figures 2A to 2L The example illustrates a gaze position indicator 291, but in various specific implementations, the gaze position indicator 291 is not displayed by the electronic device.
[0027] Figure 2A An XR environment 200 is illustrated during a first time period. During this first time period, the user is looking at a neutral position (e.g., the floor as indicated by gaze position indicator 291), and hand 292 is in a neutral position. During this first time period, the user requests an electronic device to display a virtual representation of the physical environment. In various embodiments, the user verbally requests such a display. In various embodiments, the user requests such a display by opening an application using the electronic device.
[0028] Figure 2B An XR environment 200 is illustrated during a second time period following the first time period. In response to a user request, the electronic device displays a visualization window 250, which includes a two-dimensional virtual representation of the physical environment as a floor plan 290. Specifically, the floor plan 290 includes a scaled view of the physical environment viewed from above, where lines correspond to walls, doors, and windows of the physical environment. Although Figure 2B The virtual representation of the physical environment in the XR environment 200 includes only a single room (bedroom), but it should be understood that the virtual representation of the physical environment may include multiple rooms. In various specific implementations, users can change the size and / or position of the virtual representation by changing the size and / or position of windows in the XR environment 200.
[0029] although Figure 2B The virtual representation of the physical environment in the XR is a plan view 290 displayed in a window, but in various implementations, the virtual representation of the physical environment is a three-dimensional virtual representation in the form of a stereoscopic model. Specifically, the stereoscopic model is a scaled model of the physical environment viewed from a specific angle, where each surface corresponds to the walls, doors, and windows of the physical environment. When the virtual representation of the physical environment is a stereoscopic model, the virtual representation can be displayed as a world-locked virtual object in the XR environment without a window. In various implementations, the user can change the size and / or position of the virtual representation just as with any other world-locked virtual object.
[0030] In addition to the floor plan 290, the visualization window 250 also includes virtual representations of the physical devices in the physical environment displayed at positions in the floor plan 290 corresponding to the positions of the physical devices in the physical environment. For example, the visualization window 250 includes a virtual representation 251 of a thermostat 217, a virtual representation 252 of a vent 216, a virtual representation 253 of a floor lamp 215, a virtual representation 254 of a chandelier 232, a virtual representation 255 of a ceiling fan 231, a virtual representation 256 of a louver 241, a virtual representation 257 of a window panel 242, and a virtual representation 258 of a robotic vacuum cleaner 218.
[0031] In various specific implementations, the virtual representation indicates the state of the corresponding physical device. For example, in Figure 2B In the above examples, floor lamp virtual representation 253 includes a bright bulb to indicate that floor lamp 215 is on, while chandelier virtual representation 254 includes a dark bulb to indicate that chandelier 232 is off. As another example, ceiling fan virtual representation 255 shows static blades to indicate that ceiling fan 231 is off, but it could also show moving blades to indicate that ceiling fan 231 is on. As another example, venetian blind virtual representation 256 shows a half-filled square to indicate that venetian blind 241 is half-lowered, but it could also show an empty square to indicate that venetian blind 241 is fully raised, or show a fully filled square to indicate that venetian blind 241 is fully lowered.
[0032] The visualization window 250 also includes multiple environmental characteristic displays 261-263, which, when selected, overlay a visualization of the environmental characteristics of the physical environment onto the plan layout 290. For example, the visualization window 250 includes a lighting display 261, which, when selected, overlays a visualization of the lighting of the physical environment. The visualization window 250 includes a temperature display 262, which, when selected, overlays a visualization of the temperature of the physical environment. The visualization window 250 includes a cleanliness display 263, which, when selected, overlays a visualization of the cleanliness of the physical environment. Although... Figure 2B The visualization window 250 shows only three environmental characteristic indicators 261-263, but it should be understood that the visualization window may have any number of environmental characteristic indicators, including those that show sound loudness, wireless signal strength, air quality, humidity, etc.
[0033] During the second time period, the user activates the illumination indicator 261 (e.g., by looking at the illumination indicator 261 as indicated by the gaze position indicator 291 and performing a gesture with the hand 292).
[0034] Figure 2CAn XR environment is illustrated during a third time period following the second time period. During the third time period, in response to the detection of activation of the lighting indicator 261, the visualization window 250 includes a visualization of the lighting of the physical environment overlaid on the plan layout diagram 290, which is presented as a contour map including a first set of contour lines 271a-271d. Although Figure 2C The visualization of lighting in the diagram is exemplified as a contour map, but in various specific implementations, the visualization of lighting is a heat map or any other visualization indicating various values at corresponding locations in plan layout 290. Figure 2C In order to indicate that the selected lighting display 261 is displaying a visualization of the lighting, in conjunction with... Figure 2B The lighting indicator is displayed in different ways 261. Specifically, in Figure 2B In the middle, the background of the illumination indicator 261 is white, while... Figure 2C In the image, the background of the illumination indicator 261 is gray.
[0035] The visualization of lighting is based on the state of certain physical devices with corresponding virtual representations. Specifically, the visualization of lighting is based on the state of floor lamp 215, chandelier 232, and blinds 241. Therefore, in conjunction with... Figure 2B The virtual representations of the floor lamp (253), chandelier (254), and blinds (256) are displayed in different ways. Specifically, in Figure 2B In the middle, these representations have thin line widths, while... Figure 2C In this representation, these lines have a thick line width. Therefore, users can quickly determine which physical devices are affecting the lighting in the physical environment (and the corresponding visualization). For example, because floor lamp 215 is on and chandelier 232 is off, the first contour line 271a of the first set of contour lines 271a-271d surrounds the floor lamp virtual representation 253. When light enters through window 214 and partially lowered blinds 241, the first contour line 271a stretches towards the blind virtual representation 256.
[0036] During the third time period, the user activates the floor lamp virtual representation 253 (e.g., by looking at the floor lamp virtual representation 253 as indicated by the gaze position indicator 291 and performing a gesture with the hand 292).
[0037] Figure 2DAn XR environment 200 is illustrated during a fourth time period following a third time period. During the fourth time period, in response to the detection of a selection of the virtual representation 253 of the floor lamp, the visualization window 250 includes a floor lamp window 281. The floor lamp window 281 includes an on / off indicator 282 indicating the on / off state of the floor lamp 215 (and facilitating its change) and a color indicator 283 indicating the color state of the floor lamp 215 (and facilitating its change). Furthermore, in response to the detection of a selection of the virtual representation 253 of the floor lamp, [the following text is incomplete and likely refers to a different context:] ...to... Figure 2D The floor lamp is displayed virtually in different ways, representing 253. Specifically, in Figure 2C In the image, the floor lamp virtually represents a white background for number 253, while... Figure 2D In the image, the floor lamp, which is virtually represented by the number 253, has a gray background.
[0038] During the fourth time period, the user activates the on / off power indicator 282 (e.g., by looking at the on / off power indicator 282 as indicated by the gaze position indicator 291, and performing a gesture with the hand 292).
[0039] although Figure 2C and Figure 2D Examples include a first user input activating the floor lamp virtual representation 253 to open the floor lamp window 281 and a second user input activating the on / off display representation 282 to change the state of the floor lamp 215. However, in various implementations, the state of the physical device is changed only through a single interaction with the corresponding virtual representation. For example, in various implementations, the state of the floor lamp 215 is changed only through a single interaction with the floor lamp virtual representation 253. This can be particularly useful for devices with only two states (such as on / off, open / closed, or locked / unlocked).
[0040] Figure 2E An example of an XR environment 200 during a fifth time period following a fourth time period is illustrated. During the fifth time period, in response to the detection of activation of the on / off indicator 282, the electronic device sends a command to the floor lamp 215 to change from an "on" state to an "off" state. In response to receiving such a command, the floor lamp 215 changes its state. Therefore, in Figure 2E In this setting, floor lamp 215 is off (as seen in the physical environment and as shown in the virtual representation 253 of the floor lamp). Furthermore, the visualization of lighting in the physical environment changes from the first set of contour lines to the second set of contour lines 272a-272c. Specifically, the first contour line 272a in the second set of contour lines 272a-272c partially surrounds window 214, which is the sole light source.
[0041] During the fifth time period, the user activates the temperature indicator 262 (e.g., by looking at the temperature indicator 262 as indicated by the gaze position indicator 291 and performing a gesture with the hand 292).
[0042] Figure 2F An example of the XR environment 200 during the sixth time period following the fifth time period is illustrated. During the sixth time period, in response to the detection of activation of the temperature indicator 262, the visualization of the physical environment's lighting is replaced with a visualization of the physical environment's temperature in the form of a heatmap 275. Figure 2F In the middle, to indicate that temperature display 262 has been selected and is displaying a visualization of the temperature, in conjunction with... Figure 2D The temperature indicator 262 is displayed in a different way (and the lighting indicator 261 is restored to its original display mode).
[0043] Temperature visualization is based on the state of certain physical devices with corresponding virtual representations. Specifically, temperature visualization is based on the state of thermostat 217, vent 216, ceiling fan 231, and window panel 242. Therefore, in conjunction with... Figure 2D The virtual representations of the thermostat 251, vent 252, ceiling fan 255, and window panel 257 are displayed in different ways (while other representations are restored to their original display). Therefore, users can quickly determine which physical devices are affecting the temperature in the physical environment (and the corresponding visualization). For example, because vent 216 is open and window panel 242 is partially open, the hottest spot in the physical environment is represented by the lower left corner of the heat map 275.
[0044] During the sixth time period, the user activates heatmap 275 (e.g., by looking at heatmap 275 as indicated by gaze position indicator 291 and performing a gesture with hand 292) and requests an increase in temperature. Therefore, in Figure 2D In this context, users manipulate individual devices via a power display to alter environmental characteristics in the physical environment. Figure 2F In this system, users can change the environmental characteristics of the physical environment by manipulating the visualization of environmental characteristics.
[0045] Figure 2G An example of the XR environment 200 during the seventh time period following the sixth time period is shown. In response to detecting that a user has activated thermal map 275 and requested an increase in temperature, the electronic device sends a command to thermostat 217 to change the temperature state from "70" to "72", and sends a command to window panel 242 to change the open state from "partially open" to "closed". Therefore, during the seventh time period, thermostat 217 is set to 72 and window panel 242 is closed.
[0046] During the seventh time period, the user activates the chandelier virtual representation 254 (e.g., by looking at the chandelier virtual representation 254 as indicated by the gaze position indicator 291 and performing a gesture with the hand 292).
[0047] Figure 2H An example of an XR environment 200 during an eighth time period following the seventh time period is illustrated. During the eighth time period, in response to the detection of activation of the chandelier virtual representation 254, the temperature indicator 262 is not selected and the lighting indicator 261 is selected. Therefore, the visualization of temperature in the form of a heat map 275 is replaced by the visualization of lighting in the form of a second set of contour lines 272a-272c. Furthermore, during the eighth time period, in response to the detection of selection of the chandelier virtual representation 254, the visualization window 250 includes a chandelier window 284. The chandelier window 284 includes an on / off indicator 285 indicating the on / off state of the chandelier 232 (and facilitating its change) and a color indicator 286 indicating the color state of the chandelier 232 (and facilitating its change). Additionally, in response to the detection of selection of the chandelier virtual representation 254, [the following text is incomplete and requires further context: "to..."] Figure 2G The chandelier is displayed in different ways, with a virtual representation of 254.
[0048] During the eighth time period, the user activates the on / off display 285 (e.g., by looking at the on / off display 285 as indicated by the gaze position indicator 291 and performing a gesture with the hand 292).
[0049] Figure 2I An example of an XR environment 200 during a ninth time period following the eighth time period is illustrated. During the ninth time period, in response to detecting the activation of the on / off indicator 285, the electronic device sends a command to the chandelier 232 to change from an "off" state to an "on" state. In response to receiving such a command, the chandelier 232 changes its state. Therefore, in Figure 2I In this configuration, chandelier 232 is turned on (as seen in the physical environment and as shown in the chandelier virtual representation 254). Furthermore, the visualization of lighting in the physical environment changes from the second set of contour lines to the third set of contour lines 273a-273c. Specifically, the first contour line 273a in the third set of contour lines 273a-273c surrounds the chandelier virtual representation 254. When light enters through window 214 and the partially lowered louvers 241, the first contour line 271a stretches towards the louver virtual representation 256.
[0050] During the ninth time period, the user activates the cleanliness indicator 263 (e.g., by looking at the cleanliness indicator 263 as indicated by the gaze position indicator 291 and performing a gesture with the hand 292).
[0051] Figure 2JAn example of the XR environment 200 during the tenth time period following the ninth time period is illustrated. During the tenth time period, in response to the detection of activation of the cleanliness indicator 263, the visualization of the lighting of the physical environment is replaced with a visualization of the cleanliness of the physical environment in the form of a binary map 277, where areas recently cleaned by the robotic vacuum cleaner 218 are shown in gray, while areas that have not yet been cleaned are shown in white. Figure 2J In order to indicate the selected cleanliness level, display indicator 263 is shown, and a visualization of the cleanliness level is being displayed, in conjunction with... Figure 2I The cleanliness indicator 263 is displayed in different ways (and the lighting indicator 261 is restored to its original display mode).
[0052] During the tenth time period, the user activates binary map 277 (e.g., by looking at binary map 277 as indicated by gaze position indicator 291 and performing a gesture with hand 292) and requests cleaning of a certain area of the physical environment (e.g., an area behind a door that the robotic vacuum cleaner previously found inaccessible) by indicating the corresponding area of the floor plan 290 (or binary map 277 overlaid thereon).
[0053] Figure 2K An example of an XR environment 200 during the eleventh time period following the tenth time period is illustrated. In response to detecting that a user has activated binary image 277 and requested cleaning of a certain area of the physical environment, the electronic device sends a command to the robotic vacuum cleaner 218 to clean that area of the physical environment. Therefore, during the eleventh time period, the robotic vacuum cleaner 218 is in that area of the physical environment, and binary image 277 shows that the area has been partially cleaned. In various embodiments, the robotic vacuum cleaner virtual representation 258 representing the robotic vacuum cleaner 218 is moved to the corresponding area of the plan view 290. In various embodiments, such as Figure 2K As illustrated, the robotic vacuum cleaner virtual representation 258 is stationary, indicating the location of the robotic vacuum cleaner 218's home or docking station.
[0054] Figure 2L An example of an XR environment 200 is shown in the twelfth time period following the eleventh time period. In response to the passage of time, the lighting conditions in the physical environment change because light no longer passes through window 214. In various specific implementations, a visualization window 250 is displayed in response to the change in conditions (rather than user input). For example, in... Figure 2LIn response to changes in lighting conditions, a visualization window 250 is displayed. Furthermore, the visualization window 250 includes a recommendation window 287, which includes a recommendation to improve lighting conditions by turning on the floor lamp 215. The recommendation window 287 includes a "yes" indication 288 and a "no" indication 289, where the "yes" indication indicates that the floor lamp 215 should be turned on when activated, and the "no" indication indicates that the recommendation window 287 should be closed when activated (and in various specific embodiments, the visualization window 250 is closed).
[0055] In various implementations, electronic devices may recommend moving physical objects. For example, an electronic device may recommend moving a router to achieve better wireless network coverage. Similarly, in various implementations, electronic devices may recommend adding physical objects to the physical environment. For example, an electronic device may recommend installing light fixtures to achieve better lighting conditions.
[0056] Figure 3 This is a flowchart illustrating a method 300 for displaying a virtual representation of a physical environment, based on some specific implementations. In various specific implementations, method 300 is performed by electronic devices (such as...) Figure 1 The method is executed by an electronic device 120. In various embodiments, method 300 is executed by a device having a display, one or more processors, and non-transitory memory. In some embodiments, method 300 is executed by processing logic components, including hardware, firmware, software, or combinations thereof. In some embodiments, method 300 is executed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).
[0057] Method 300 begins in box 310, where the device displays a virtual representation of the physical environment. For example, in Figures 2B to 2L In this embodiment, the electronic device displays a floor plan 290 in a visualization window 250. In various embodiments, displaying a virtual representation of the physical environment includes displaying a scaled view or model of that physical environment. For example, in various embodiments, displaying a virtual representation of the physical environment includes displaying a two-dimensional floor plan. In various embodiments, displaying a virtual representation of the physical environment includes displaying a three-dimensional model.
[0058] In various specific implementations, displaying a virtual representation of the physical environment includes displaying a virtual representation of the physical environment as a world-locked virtual object in association with the physical environment. For example, in Figures 2B to 2LIn this embodiment, the electronic device displays the visualization window 250 as a world-locked virtual object associated with the bedroom. In various embodiments, displaying a virtual representation of the physical environment in association with the physical environment includes composited rendering of the virtual representation of the physical environment with an image of the physical environment and displaying the composite image on an opaque display. In various embodiments, displaying a virtual representation of the physical environment in association with the physical environment includes rendering a virtual representation of the physical environment on a transparent display when the user is in the physical environment.
[0059] In various specific implementations, world-locked virtual objects are displayed with orientations that match the physical environment. For example, when the virtual representation of the physical environment is a 3D model, the portion of the model representing a wall is displayed parallel to and closest to that wall. Similarly, the portion of the model representing a wall opposite to that wall is displayed parallel to and closest to that wall. As another example, when the virtual representation of the physical environment is a 2D floor plan, the floor plan is displayed as a horizontal plane parallel to the floor, and the portion of the floor plan representing a wall is displayed parallel to and closest to that wall. Similarly, the portion of the floor plan representing a wall opposite to that wall is displayed parallel to and closest to that wall. As another example, when the virtual representation of the physical environment is a 2D floor plan, the floor plan is displayed as a vertical plane perpendicular to the floor, and the portion of the floor plan representing a wall is displayed parallel to and closest to that wall. Similarly, the portion of the floor plan representing an adjacent wall is displayed at the top or bottom of the floor plan.
[0060] although Figures 2A to 2L Examples illustrate the display of a virtual representation of a physical environment in association with a physical environment; however, in various embodiments, the virtual representation of the physical environment is not displayed in association with a physical environment. In various embodiments, the virtual representation of the physical environment is displayed when the device is far from the physical environment (e.g., in a different physical environment). In various embodiments, the virtual representation of the physical environment is displayed in association with a different physical environment. In various embodiments, the virtual representation of the physical environment is displayed without association with any physical environment, for example, as an application on a telephone or tablet.
[0061] Method 300 continues in block 320, wherein the device obtains status information indicating the state of the physical device from a physical device at its physical location in the physical environment. In various specific implementations, obtaining the status information includes sending a query to the physical device and receiving the status information in response to the query. For example, in Figure 2B In the process, the electronic device obtains status information indicating that the floor lamp 215 is turned on from the floor lamp 215.
[0062] Method 300 continues in box 330, wherein a virtual representation of the physical device is displayed at a virtual location in the virtual representation of the physical environment corresponding to the physical location of the device in the physical environment. For example, in Figure 2B In this embodiment, the electronic device displays a virtual representation 253 of the floor lamp 215 in the bedroom at a position corresponding to the floor lamp 215 in the floor plan 290. In various specific implementations, the virtual representation of the physical device is displayed based at least in part on state information. For example, in… Figure 2B In the diagram, because floor lamp 215 is on, the virtual representation 253 of the floor lamp includes a bright bulb. Conversely, because chandelier 232 is off, the virtual representation 254 of the chandelier includes a dark bulb.
[0063] In various specific implementations, such as Figures 2B to 2L As illustrated, the virtual representation of a physical device is an icon. In various specific embodiments, the virtual representation of a physical device is a scaled drawing or model of that physical device. In various specific embodiments, the virtual representation of a physical device is a scaled drawing or model of a physical device of the same type as that physical device. For example, in various specific embodiments, the virtual representation of floor lamp 215 could be a drawing of a small lamp, but not necessarily... Figures 2A to 2L The example shows a three-bulb lamp.
[0064] Method 300 continues in block 340, wherein the device determines multiple values of environmental characteristics of the physical environment at multiple environmental physical locations in the physical environment, wherein the multiple values are at least partially based on state information.
[0065] In various specific implementations, determining multiple values for environmental characteristics includes determining multiple lighting values for the physical environment at multiple physical locations within the physical environment. For example, in Figure 2C In this context, electronic devices determine multiple lighting values of the physical environment at multiple locations in the bedroom based in part on the state of floor lamp 215 (e.g., "on"), chandelier 232 (e.g., "off"), and blinds 241 (e.g., "partially open"), and display contour maps based on these values.
[0066] In various specific implementations, determining multiple values of environmental characteristics includes determining multiple temperature values of the physical environment at multiple physical locations within the physical environment. For example, in Figure 2F In this process, the electronic device determines multiple temperature values of the physical environment at multiple locations in the bedroom based in part on the state of the thermostat 217 (e.g., "70"), the state of the vent 216 (e.g., "on"), the state of the ceiling fan (e.g., "off"), and the state of the window panel 242 (e.g., "partially open"), and displays a thermal map based on these values.
[0067] In various implementations, environmental characteristics include lighting characteristics, temperature characteristics, sound characteristics, wireless signal characteristics, air quality characteristics, and humidity characteristics. In various implementations, lighting characteristics include brightness and / or color. Typically, in various implementations, determining multiple values for environmental characteristics involves simulating or modeling the physical environment based on state information and, in various implementations, on other available information.
[0068] In various specific implementations, the determination of multiple values for environmental characteristics is also based on models of the physical environment. For example, in various specific implementations, when the environmental characteristic is the sound loudness at multiple physical locations in the environment, the device models the sound propagation throughout the physical environment based on wall textures, floor textures, wall materials (e.g., wood or concrete), etc.
[0069] In various specific implementations, the determination of multiple values for environmental characteristics is also based on the time of day. For example, in Figure 2C In this context, the illumination value is based on the light passing through the partially opened blinds 241. The amount of light passing through the partially opened blinds 241 is based on the time of day.
[0070] In various specific implementations, the determination of multiple values for environmental characteristics is also based on weather information. For example, as mentioned above, in Figure 2C In this context, the illumination value is based on the light passing through the partially opened veneer 241. The amount of light passing through the partially opened veneer 241 is based on the weather. Similarly, in... Figure 2F In this context, the temperature value is based on the heat passing through the partially opened window panel 242. The heat passing through the partially opened window panel 242 is based on the weather.
[0071] In various specific implementations, the determination of multiple values for environmental characteristics is also based on sensor data. For example, in Figure 2F In various specific implementations, the vent 216 includes a thermometer. Therefore, the temperature value is based on temperature information obtained from the thermometer by the vent 216 and sent to the electronic device.
[0072] Method 300 continues in block 350, wherein the device displays indications of multiple values for environmental characteristics at multiple virtual locations in a virtual representation of the physical environment corresponding to physical locations within the physical environment. In various embodiments, displaying indications of multiple values for environmental characteristics includes displaying contour plots. For example, in Figure 2C In this context, electronic devices display contour maps of illumination values. In various specific implementations, displaying indications of multiple values for environmental characteristics includes displaying heat maps. For example, in... Figure 2F In the image, an electronic device displays a heatmap showing temperature values.
[0073] In various specific implementations, the virtual representation of a physical device is an indication of the state of the physical device. For example, in Figure 2EIn response to user input pointing to the virtual representation 253 of the floor lamp, the state of the floor lamp 215 changes from "on" to "off". Therefore, in various specific implementations, method 300 includes receiving user input pointing to the virtual representation of the physical device, and in response to receiving the user input, sending a command to the physical device to change the state of the physical device.
[0074] In various specific implementations, the virtual representation of environmental characteristics is also an indication of changes in the state of physical devices. For example, in Figure 2G In response to user input pointing to heatmap 275, the state of thermostat 217 changes from "70" to "72", and the state of window panel 242 changes from "partially open" to "closed". Therefore, in various specific implementations, method 300 includes receiving user input indicating multiple values for environmental characteristics, and in response to receiving the user input, sending a command to the physical device to change the state of the physical device.
[0075] In various specific implementations, changing the state of physical devices alters the visualization of the environmental characteristics of the physical environment. For example, in Figure 2E In response to the floor lamp 215 changing from an "on" state to an "off" state, the first set of contour lines 271a-271d is replaced by the second set of contour lines 272a-272c. Therefore, in various specific embodiments, method 300 includes obtaining update status information indicating the updated status of the physical device from the physical device. Furthermore, method 300 includes determining multiple update values of environmental characteristics of the physical environment at multiple environmental physical locations in the physical environment, wherein these multiple update values are at least partially based on the update status information. Additionally, method 300 includes displaying indications of the multiple update values of the environmental characteristics at multiple virtual environmental locations in a virtual representation of the physical environment.
[0076] In various specific implementations, multiple values of the environmental characteristics of the physical environment are based on the state of more than one physical device. For example, in Figure 2C In this example, electronic devices determine multiple lighting values of the physical environment at multiple locations in the bedroom, partly based on the state of floor lamp 215 (e.g., "on"), chandelier 232 (e.g., "off"), and blinds 241 (e.g., "partially open"), and display contour maps based on these values. As another example, in Figure 2F In this process, the electronic device determines multiple temperature values of the physical environment at multiple locations in the bedroom based in part on the state of the thermostat 217 (e.g., "70"), the state of the vent 216 (e.g., "on"), the state of the ceiling fan (e.g., "off"), and the state of the window panel 242 (e.g., "partially open"), and displays a thermal map based on these values.
[0077] Therefore, in various specific implementations, method 300 further includes obtaining additional state information from the additional physical device at the physical location of the additional device in the physical environment, and displaying an additional virtual representation of the additional physical device at the virtual location of the additional physical device corresponding to the physical location of the additional device in the physical environment in a virtual representation of the physical environment. Furthermore, in block 350, multiple values for determining the environmental characteristics of the physical environment are also based on the additional state information.
[0078] In various specific implementations, the device can display virtual representations of multiple environmental characteristics. In various specific implementations, the virtual representations are displayed individually and sequentially, such as in... Figure 2C and Figure 2F However, in various specific implementations, visualizations of multiple environmental characteristics are displayed simultaneously. For example, a virtual representation of lighting on a first plan view may be displayed next to a virtual representation of temperature on a second plan view. In various specific implementations, virtual representations of multiple environmental characteristics are displayed simultaneously on the same plan view. For example, a virtual representation of lighting may be displayed as a contour map, and a virtual representation of temperature may be displayed as a heat map. As another example, a virtual representation of temperature may be displayed as a blue heat map, and a virtual representation of humidity may be displayed as a red heat map. Thus, when both temperature and humidity are high, the plan view is purple; when temperature is high and humidity is low, the plan view is blue; when temperature is low and humidity is high, the plan view is red; and when both temperature and humidity are low, the plan view is in the default color (e.g., black or white).
[0079] Therefore, in various specific implementations, method 300 further includes obtaining additional state information indicating the state of the additional physical device from the additional physical device at the additional physical device's physical location in the physical environment. Method 300 includes displaying an additional virtual representation of the additional physical device at the additional physical device's virtual location corresponding to the additional physical device's physical location in the physical environment in a virtual representation of the physical environment. Method 300 includes determining additional multiple values of additional environmental characteristics of the physical environment at multiple environmental physical locations in the physical environment, wherein these additional multiple values are at least partially based on the additional state information. Method 300 includes displaying indications of the additional multiple values of the additional environmental characteristics at multiple environmental virtual locations in the virtual representation of the physical environment.
[0080] Figure 4This is a block diagram of an example controller 110 according to some specific implementations. Although certain specific features are illustrated, those skilled in the art will understand from this disclosure that various other features are not illustrated for the sake of brevity and to avoid obscuring more relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, controller 110 includes one or more processing units 402 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, and / or similar processing units), one or more input / output (I / O) devices 406, one or more communication interfaces 408 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 410, memory 420, and one or more communication buses 404 for interconnecting these components and various other components.
[0081] In some embodiments, one or more communication buses 404 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices 406 include at least one of the following: keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.
[0082] Memory 420 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 420 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 420 optionally includes one or more storage devices remotely located to one or more processing units 402. Memory 420 includes a non-transitory computer-readable storage medium. In some embodiments, memory 420 or the non-transitory computer-readable storage medium of memory 420 stores programs, modules, and data structures, or subsets thereof, including optional operating system 430 and XR experience module 440.
[0083] Operating system 430 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, XR experience module 440 is configured to manage and coordinate single or multiple XR experiences for one or more users (e.g., single XR experiences for one or more users, or multiple XR experiences for corresponding groups of one or more users). To this end, in various implementations, XR experience module 440 includes a data acquisition unit 442, a tracking unit 444, a coordination unit 446, and a data transmission unit 448.
[0084] In some specific implementations, the data acquisition unit 442 is configured to acquire data from at least... Figure 1 The electronic device 120 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific embodiments, the data acquisition unit 442 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0085] In some specific implementations, the tracking unit 444 is configured to map the physical environment 105 and at least track the electronic device 120 relative to it. Figure 1 The location / position of the physical environment 105. To this end, in various specific implementations, the tracking unit 444 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0086] In some implementations, coordination unit 446 is configured to manage and coordinate the XR experience presented to the user by electronic device 120. To this end, in various implementations, coordination unit 446 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0087] In some implementations, the data transmission unit 448 is configured to transmit data (e.g., presentation data, location data, etc.) to the electronic device 120. To this end, in various implementations, the data transmission unit 448 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0088] Although the data acquisition unit 442, tracking unit 444, coordination unit 446 and data transmission unit 448 are shown residing on a single device (e.g., controller 110), it should be understood that in other implementations, any combination of the data acquisition unit 442, tracking unit 444, coordination unit 446 and data transmission unit 448 may reside in a separate computing device.
[0089] also, Figure 4This is used more as a functional description of various features that may exist in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, individually shown items can be combined, and some items can be separated. For example, Figure 4 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.
[0090] Figure 5 This is a block diagram of an example of an electronic device 120 according to some specific embodiments. Although certain specific features are illustrated, those skilled in the art will understand from this disclosure that various other features are not illustrated for the sake of brevity and to avoid obscuring more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, electronic device 120 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 506, one or more communication interfaces 508 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, one or more XR displays 512, one or more optional internal and / or external image sensors 514, memory 520, and one or more communication buses 504 for interconnecting these components and various other components.
[0091] In some embodiments, one or more communication buses 504 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 506 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), etc.
[0092] In some embodiments, one or more XR displays 512 are configured to provide an XR experience to a user. In some embodiments, one or more XR displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more XR displays 512 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, electronic device 120 includes a single XR display. In another example, the electronic device includes an XR display for each of the user's eyes. In some embodiments, one or more XR displays 512 are capable of presenting MR and VR content.
[0093] In some embodiments, one or more image sensors 514 are configured to acquire image data corresponding to at least a portion of a user's face (including the user's eyes) (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 514 are configured to face forward to acquire image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). One or more optional image sensors 514 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, etc.
[0094] Memory 520 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 520 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 520 may optionally include one or more storage devices remotely located to one or more processing units 502. Memory 520 includes a non-transitory computer-readable storage medium. In some embodiments, memory 520 or its non-transitory computer-readable storage medium stores programs, modules, and data structures, or subsets thereof, including optional operating system 530 and XR rendering module 540.
[0095] Operating system 530 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, XR presentation module 540 is configured to present XR content to a user via one or more XR displays 512. Therefore, in various implementations, XR presentation module 540 includes a data acquisition unit 542, a value determination unit 544, an XR presentation unit 546, and a data transmission unit 548.
[0096] In some specific implementations, the data acquisition unit 542 is configured to acquire data from at least... Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific implementations, the data acquisition unit 542 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0097] In some implementations, the value determination unit 544 is configured to determine multiple values of environmental characteristics of the physical environment. To this end, in various implementations, the value determination unit 544 includes instructions and / or logic components for these instructions, as well as heuristics and metadata for the heuristics.
[0098] In some implementations, the XR rendering unit 546 is configured to display indications of multiple values on a virtual representation of the physical environment via one or more XR displays 512. To this end, in various implementations, the XR rendering unit 546 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0099] In some embodiments, the data sending unit 548 is configured to send at least data (e.g., presentation data, location data, etc.) to the controller 110. In some embodiments, the data sending unit 548 is configured to send authentication credentials to an electronic device. Therefore, in various embodiments, the data sending unit 548 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.
[0100] Although the data acquisition unit 542, value determination unit 544, XR presentation unit 546 and data transmission unit 548 are shown residing on a single device (e.g., electronic device 120), it should be understood that in other embodiments, any combination of the data acquisition unit 542, value determination unit 544, XR presentation unit 546 and data transmission unit 548 may reside in a separate computing device.
[0101] also, Figure 5This is used more as a functional description of various features that may exist in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, individually shown items can be combined, and some items can be separated. For example, Figure 5 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.
[0102] While various aspects of specific embodiments within the scope of the appended claims have been described above, it should be apparent that the various features of the above-described embodiments can be embodied in a wide variety of forms, and any particular structure and / or function described above are merely illustrative. Based on this disclosure, those skilled in the art will understand that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or practice a method. Furthermore, such an apparatus and / or practice a method can be implemented using other structures and / or functionalities besides or different from one or more aspects set forth herein.
[0103] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.
[0104] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” when used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0105] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.
Claims
1. A method comprising: In a device that includes a display, non-transitory memory, and one or more processors: Displays a virtual representation of the physical environment; Obtain status information indicating the status of the physical device from the physical device at the physical location of the device in the physical environment; In the virtual representation of the physical environment, a virtual representation of the physical device is displayed at the virtual location of the device corresponding to the physical location of the device in the physical environment; Determine multiple values of environmental characteristics of the physical environment at multiple environmental physical locations within the physical environment, wherein the multiple values are at least partially based on the state information; as well as In the virtual representation of the physical environment, indications of the multiple values of the environmental characteristics are displayed at multiple virtual locations corresponding to the physical locations of the environment in the physical environment.
2. The method of claim 1, wherein displaying the virtual representation of the physical environment includes displaying a scaled copy of the physical environment.
3. The method of claim 1, wherein displaying the virtual representation of the physical environment includes displaying a two-dimensional planar layout.
4. The method of claim 1, wherein displaying the virtual representation of the physical environment includes displaying a three-dimensional model.
5. The method of claim 1, wherein displaying the virtual representation of the physical environment comprises displaying the virtual representation of the physical environment as a world-locked virtual object in association with the physical environment.
6. The method of claim 5, wherein the world-locked virtual object is displayed in an orientation that matches the physical environment.
7. The method of claim 1, wherein obtaining the status information comprises sending a query to the physical device and receiving the status information in response to the query.
8. The method of claim 1, wherein the virtual representation of the physical device is displayed based at least in part on the state information.
9. The method of claim 1, wherein determining the plurality of values of the environmental characteristics includes determining a plurality of lighting values of the physical environment at the plurality of environmental physical locations in the physical environment.
10. The method of claim 1, wherein determining the plurality of values of the environmental characteristics includes determining a plurality of temperature values of the physical environment at the plurality of environmental physical locations in the physical environment.
11. The method of claim 1, wherein determining the plurality of values of the environmental characteristics is further based on a model of the physical environment.
12. The method of claim 1, wherein the determination of the plurality of values of the environmental characteristics is further based on sensor data.
13. The method of claim 1, wherein the indication displaying the plurality of values of the environmental characteristics includes displaying a heatmap.
14. The method according to claim 1, further comprising: Receive user input pointing to the virtual representation of the physical device; as well as In response to receiving the user input, a command is sent to the physical device to change the state of the physical device.
15. The method according to claim 1, further comprising: Receive user input indicating the plurality of values of the environmental characteristics; as well as In response to receiving the user input, a command is sent to the physical device to change the state of the physical device.
16. The method according to claim 1, further comprising: Obtain update status information indicating the update status of the physical device; Determine multiple updated values of the environmental characteristics of the physical environment at the multiple environmental physical locations in the physical environment, wherein the multiple updated values are at least partially based on the updated state information; as well as Indications of the multiple updated values of the environmental characteristics are displayed at the multiple virtual locations of the environment in the virtual representation of the physical environment.
17. The method according to claim 1, further comprising: Additional status information is obtained from the additional physical device at the physical location of the additional device in the physical environment; as well as An additional virtual representation of the additional physical device is displayed at the virtual location of the additional physical device, corresponding to the physical location of the additional device in the physical environment. The plurality of values for determining the environmental characteristics of the physical environment are also based on the additional state information.
18. The method according to claim 1, further comprising: Additional status information indicating the status of the additional physical device is obtained from the additional physical device at the physical location of the additional device in the physical environment; An additional virtual representation of the additional physical device is displayed at the virtual location of the additional physical device corresponding to the physical location of the additional device in the physical environment; Determine additional values of additional environmental characteristics of the physical environment at the multiple environmental physical locations in the physical environment, wherein the additional multiple values are at least partially based on the additional state information; as well as Indications of the additional multiple values of the additional environmental characteristics are displayed at the multiple virtual locations of the additional environmental characteristics in the virtual representation of the physical environment.
19. An apparatus comprising: monitor; Non-transitory memory; and One or more processors, said one or more processors being used for: Displays a virtual representation of the physical environment; Obtain status information indicating the status of the physical device from the physical device at the physical location of the device in the physical environment; In the virtual representation of the physical environment, a virtual representation of the physical device is displayed at the virtual location of the device corresponding to the physical location of the device in the physical environment; Determine multiple values of environmental characteristics of the physical environment at multiple environmental physical locations within the physical environment, wherein the multiple values are at least partially based on the state information; as well as In the virtual representation of the physical environment, indications of the multiple values of the environmental characteristics are displayed at multiple virtual locations corresponding to the physical locations of the environment in the physical environment.
20. A non-transitory memory storing one or more programs, said one or more programs, when executed by one or more processors of a device including a display, causing the device to: Displays a virtual representation of the physical environment; Obtain status information indicating the status of the physical device from the physical device at the physical location of the device in the physical environment; In the virtual representation of the physical environment, a virtual representation of the physical device is displayed at the virtual location of the device corresponding to the physical location of the device in the physical environment; Determine multiple values of environmental characteristics of the physical environment at multiple environmental physical locations within the physical environment, wherein the multiple values are at least partially based on the state information; as well as In the virtual representation of the physical environment, indications of the multiple values of the environmental characteristics are displayed at multiple virtual locations corresponding to the physical locations of the environment in the physical environment.