Computationally efficient model selection
By selecting the appropriate model according to the time step in the synthetic reality environment, dynamically determining the environment state is solved, and the problem of high computing burden in the prior art is improved.
Patent Information
- Application Number
- CN201980062386.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-28
- Filing Date
- 2019-09-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2039-09-24
AI Technical Summary
The prior art has a high computational burden when generating synthetic real environments, resulting in poor user experience, especially when multiple models need to be processed, making it difficult to efficiently determine the environment state.
By obtaining the initial environmental state of the SR environment on a device with a processor and non-transitory memory, and selecting a suitable model from multiple models according to the time step, the environment state is dynamically determined, reducing the computational burden.
It improves the generation efficiency of the SR environment, reduces the computing burden, and improves the user experience, and can effectively manage the asset status especially when dealing with long-term environmental changes.
Smart Images

Figure CN112740280B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to generating synthetic reality (SR) environments, and in particular, to systems, methods, and apparatus for determining environment states for SR environments in a computationally efficient manner. Background Art
[0002] A physical set refers to a world that an individual can perceive and / or interact with without the aid of an electronic system. A physical set (e.g., a physical forest) includes physical elements (e.g., physical trees, physical structures, and physical animals). An individual can directly interact with and / or perceive a physical set, such as by touch, sight, smell, hearing, and taste.
[0003] In contrast, a synthetic reality (SR) set refers to a set that is fully or partially computer-created that an individual can perceive and / or that the individual can interact with via an electronic system. In SR, a subset of the individual's movements is monitored, and in response thereto, one or more properties of one or more virtual objects in the SR set are changed in a manner that complies with one or more laws of physics. For example, the SR system may detect that the individual has taken a few steps forward, and in response thereto, adjust the graphics and audio presented to the individual in a manner similar to how such scenes and sounds would change in a physical set. Modifications to one or more properties of one or more virtual objects in the SR set may also be made in response to a representation of movement (e.g., an audio command).
[0004] The individual may interact with and / or perceive SR objects using any of his senses, including touch, smell, sight, taste, and sound. For example, the individual may interact with and / or perceive auditory objects that create a multi-dimensional (e.g., three-dimensional) or spatial auditory scenery and / or implement auditory transparency. The multi-dimensional or spatial auditory scenery provides the individual with the perception of discrete auditory sources in a multi-dimensional space. Auditory transparency selectively combines sounds from the physical scenery with or without computer-created audio. In some SR scenery, the individual may interact with and / or perceive only auditory objects.
[0005] An example of SR is virtual reality (VR). A VR set refers to a simulated set designed to include only computer-created sensory input for at least one sense. The VR set includes a plurality of virtual objects that an individual can interact with and / or perceive. The individual can interact with and / or perceive virtual objects in the VR set by simulating a subset of the individual's actions within the computer-created set and / or by simulating the individual or his / her presence within the computer-created set.
[0006] Another example of SR is mixed reality (MR). An MR set refers to a simulated set designed to integrate computer-created sensory input (e.g., virtual objects) with sensory input from a physical set, or representation thereof. On the reality spectrum, a mixed reality set is between VR sets at one end and fully physical sets at the other end, and does not include these sets.
[0007] In some MR sets, computer-created sensory input can adapt to changes in sensory input from the physical set. In addition, some electronic systems used to render MR sets can monitor orientation and / or position relative to the physical set to enable virtual objects to interact with real objects (i.e., physical elements from the physical set or representations thereof). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.
[0008] An example of mixed reality is augmented reality (AR). AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor, the imaging sensor being used to capture images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects may be projected, for example, on a physical surface or as a hologram, such that an individual using the system observes the virtual objects superimposed on the physical scenery.
[0009] An augmented reality set may also refer to a simulated set in which a representation of a physical set is altered by computer-created sensory information. For example, a portion of the representation of a physical set may be graphically altered (e.g., enlarged) such that the altered portion may still represent one or more originally captured images but is not a faithfully reproduced version. For another example, when providing video pass-through, the system may alter at least one of the sensor images to impose a particular viewpoint that is different from the viewpoint captured by one or more image sensors. For another example, the representation of a physical set may be altered by graphically blurring or eliminating portions thereof.
[0010] Another example of mixed reality is augmented virtuality (AV). An AV set refers to a simulated set in which a computer-created set or a virtual set incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may present the color of a physical element captured by one or more imaging sensors. As another example, a virtual object may present features consistent with actual weather conditions in a physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images taken of physical animals.
[0011] Many electronic systems enable individuals to interact with and / or perceive various SR scenery. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smart phone). The head-mounted system may have one or more imaging sensors and / or microphones for taking images / video of the physical scenery and / or capturing audio of the physical scenery, respectively. The head-mounted system may also have a transparent or translucent display. The transparent or translucent display may be combined with a substrate through which light representing the image is directed to the individual's eyes. The display may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one embodiment, the transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. As another example, the electronic system may be a projection-based system. A projection-based system may project an image onto the retina of an individual using retinal projection. Alternatively, the projection system may also project the virtual object into a physical setting (e.g., onto a physical surface or as a hologram). Other examples of SR systems include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, headphones or earbuds, a speaker arrangement, an input mechanism (e.g., a controller with or without tactile feedback), a tablet, a smartphone, and a desktop or laptop computer.
[0012] In order to render an SR environment including assets that evolve over time according to one or more models, a large amount of computation may be required. Therefore, in order to improve the SR experience of users, various embodiments disclosed herein select a model from multiple models in order to reduce the computational burden. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.
[0014] Figure 1A is a block diagram of an exemplary operational architecture according to some specific implementations.
[0015] Figure 1B is a block diagram of an exemplary operational architecture according to some specific implementations.
[0016] Figure 2 is a block diagram of an example controller according to some implementations.
[0017] Figure 3 is a block diagram of an exemplary HMD according to some implementations.
[0018] Figure 4 A scene is illustrated along with an electronic device that surveys the scene.
[0019] FIG. 5A to FIG. 5H Illustrated Figure 4 A portion of a display of an electronic device that displays an image including a representation of a scene of an SR environment.
[0020] Figure 6 Environmental states according to some specific implementations are shown.
[0021] Figure 7 A block diagram of an environment state generator according to some specific implementations is shown.
[0022] Figure 8 is a flowchart representation of a method of generating an environment state of an SR environment according to some specific implementations.
[0023] Fig. 9 is a flowchart representation of a method of generating an environment state of an SR environment according to some specific implementations.
[0024] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention
[0025] Various specific implementations disclosed herein include devices, systems, and methods for generating environmental states of SR environments. In various specific implementations, the method is performed at a device having one or more processors and non-volatile memory. The method includes obtaining a first environmental state of the SR environment associated with a first environmental time, wherein the first environmental state indicates that one or more assets are included in the SR environment, and also indicates one or more states of the one or more assets. The method includes determining a second environmental state associated with a second environmental time according to a first model and based on the first environmental state. The method includes receiving an input indicating a time step from the second environmental time to a third environmental time, wherein the time step is different from the difference between the first environmental time and the second environmental time. The method includes determining a third environmental state associated with a third environmental time according to a second model different from the first model and based on the second environmental state.
[0026] Various specific implementations disclosed herein include devices, systems, and methods for generating an environmental state of an SR environment based on a selected model. In various specific implementations, the method is performed at a device having one or more processors and non-volatile memory. The method includes obtaining a first environmental state of the SR environment associated with a first environmental time, wherein the first environmental state indicates that one or more assets are included in the SR environment, and also indicates one or more states of the one or more assets. The method includes determining a time step from the first environmental time to a second environmental time. The method includes selecting a model from a plurality of different models based on the time step. The method includes determining a second environmental state of the SR environment associated with a second environmental time based on the selected model.
[0027] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION
[0028] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.
[0029] In various implementations, the device surveys a scene and presents an SR environment within the scene, the SR environment including one or more assets that evolve over time (e.g., change position or age). Modeling such an SR environment at various time scales may be computationally intensive, especially when the SR environment is modeled over a larger time scale. Therefore, in various implementations, different models are used to determine the environmental state of the SR environment when the SR environment is presented at different time scales.
[0030] Figure 1A 1 is a block diagram of an exemplary operational architecture 100A according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operational architecture 100A includes an electronic device 120A.
[0031] In some implementations, the electronic device 120A is configured to present SR content to the user. In some implementations, the electronic device 120A includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120A presents the SR content to the user via a display 122 when the user is physically present in a physical environment 103, which includes a table 107 located within a field of view 111 of the electronic device 120A. Therefore, in some implementations, the user holds the electronic device 120A in his / her hand. In some implementations, when providing augmented reality (AR) content, the electronic device 120A is configured to display an AR object (e.g., an AR cube 109) and implement video pass-through of the physical environment 103 (e.g., including a representation 117 of the table 107) on the display 122.
[0032] Figure 1B 1 is a block diagram of an exemplary operating environment 100B according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100B includes a controller 110 and a head mounted device (HMD) 120B.
[0033] In some implementations, the controller 110 is configured to manage and coordinate SR presentation to users. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 2 The controller 110 is described in more detail. In some implementations, the controller 110 is a computing device that is located locally or remotely relative to the scene 105. For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some implementations, the controller 110 is communicatively coupled to the HMD 120B via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included in the housing of the HMD 120B.
[0034] In some implementations, the HMD 120B is configured to present SR content to the user. In some implementations, the HMD 120B includes a suitable combination of software, firmware, and / or hardware. Figure 3 HMD 120B is described in greater detail. In some implementations, the functionality of controller 110 is provided by and / or combined with HMD 120B.
[0035] According to some implementations, HMD 120B presents SR content to the user while the user is virtually and / or physically present within scene 105 .
[0036] In some implementations, the user wears the HMD 120B on his / her head. Thus, the HMD 120B includes one or more SR displays provided for displaying SR content. For example, in various implementations, the HMD 120B surrounds the user's field of view. In some implementations, such as in Figure 1A In some implementations, a handheld device (such as a smartphone or tablet) configured to present SR content is used in place of HMD 120B, and the user no longer wears HMD 120B but holds the device with the display facing the user's field of view and the camera facing the scene 105. In some implementations, the handheld device can be placed in a housing that can be worn on the user's head. In some implementations, HMD 120B is replaced with an SR pod, housing, or chamber configured to present SR content, wherein the user no longer wears or holds HMD 120B.
[0037] Figure 2is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.
[0038] In some implementations, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.
[0039] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and an SR experience module 240.
[0040] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the SR experience module 240 is configured to manage and coordinate single or multiple SR experiences of one or more users (e.g., a single SR experience of one or more users, or multiple SR experiences of corresponding groups of one or more users). To this end, in various implementations, the SR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.
[0041] In some specific implementations, the data acquisition unit 242 is configured to at least Figure 1B The HMD 120B obtains data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various specific implementations, the data acquisition unit 242 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.
[0042] In some implementations, the tracking unit 244 is configured to map the scene 105 and track at least the HMD 120B relative to the scene 105. Figure 1B The location / position of the scene 105. To this end, in various specific implementations, the tracking unit 244 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0043] In some implementations, the coordination unit 246 is configured to manage and coordinate the SR experience presented to the user by the HMD 120B. To this end, in various implementations, the coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.
[0044] In some implementations, the data transfer unit 248 is configured to transfer data (e.g., presentation data, position data, etc.) to at least the HMD 120B. To this end, in various implementations, the data transfer unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.
[0045] Although the data acquisition unit 242, tracking unit 244, coordination unit 246, and data transfer unit 248 are illustrated as residing on a single device (e.g., controller 110), it should be understood that in other implementations, any combination of the data acquisition unit 242, tracking unit 244, coordination unit 246, and data transfer unit 248 may be located in separate computing devices.
[0046] also, Figure 2It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.
[0047] Figure 3 1 is a block diagram of an example of an HMD 120B according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some implementations, the HMD 120B includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE802.3x, IEEE802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more SR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.
[0048] In some implementations, one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.
[0049] In some implementations, one or more SR displays 312 are configured to provide a SR experience to the user. In some implementations, one or more SR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more SR displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, HMD 120B includes a single SR display. In another example, HMD 120B includes an SR display for each eye of the user. In some implementations, one or more SR displays 312 can present MR and VR content.
[0050] In some implementations, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and thus may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to the scene that the user would see when the HMD 120B is not present (and thus may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.
[0051] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or a subset thereof, including an optional operating system 330 and an SR presentation module 340.
[0052] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the SR presentation module 340 is configured to present SR content to a user via one or more SR displays 312. To this end, in various implementations, the SR presentation module 340 includes a data acquisition unit 342, an SR presentation unit 344, an SR environment unit 346, and a data transfer unit 348.
[0053] In some implementations, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of FIG1 . To this end, in various implementations, the data acquisition unit 342 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.
[0054] In some implementations, the SR presentation unit 344 is configured to present the SR content via the one or more SR displays 312. To this end, in various implementations, the SR presentation unit 344 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.
[0055] In some implementations, the SR environment unit 346 is configured to generate one or more environment states of the SR environment. To this end, in various implementations, the SR environment unit 346 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0056] In some implementations, the data transfer unit 348 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the controller 110. To this end, in various implementations, the data transfer unit 348 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.
[0057] Although the data acquisition unit 342, the SR presentation unit 344, the SR environment unit 346, and the data transfer unit 348 are shown as residing on a single device (e.g., Figure 1B 120B), but it should be understood that in other specific implementations, any combination of the data acquisition unit 342, the SR presentation unit 344, the SR environment unit 346 and the data transmission unit 348 may be located in separate computing devices.
[0058] also, Figure 3 More as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic diagram of the structure of the specific implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.
[0059] Figure 4 A scene 405 is illustrated along with an electronic device 410 surveying the scene 405. The scene 405 includes a table 408 and a wall 407.
[0060] Electronic device 410 displays a representation of scene 415 on a display, including a representation of table 418 and a representation of wall 417. In various implementations, the representation of scene 415 is generated based on an image of the scene captured with a scene camera of electronic device 410, the scene camera having a field of view toward scene 405. The representation of scene 415 also includes SR environment 409 displayed on the representation of table 418.
[0061] As electronic device 410 moves around scene 405, the representation of scene 415 changes in accordance with the change in viewing angle of electronic device 410. Additionally, SR environment 409 changes accordingly in accordance with the change in viewing angle of electronic device 410. Thus, as electronic device 410 moves, SR environment 409 appears in a fixed relationship to the representation of table 418.
[0062] Figure 5A A portion of a display of an electronic device 410 is illustrated, the display showing a first image 500A including a representation of a scene 415 of an SR environment 409. Figure 5A In the embodiment, the SR environment 409 is defined by a first environment state and is associated with a first environment time (e.g., 1). The first environment state indicates that one or more assets are included in the SR environment 409, and further indicates one or more states of the one or more assets. In various specific implementations, the environment state is a data object, such as an XML file.
[0063] Thus, the SR environment 409 displayed in the first image 500A includes a plurality of assets defined by a first environment state. Figure 5A , SR environment 409 includes a first tree 511 having a first height 591 and a second tree 512 having a second height 592. SR environment 409 includes a first squirrel 521 at a first position 581 and a second squirrel 522 at a second position 582. SR environment includes a first acorn 531.
[0064] The first environmental state indication includes a first tree 511 and defines one or more states of the first tree 511. For example, the first environmental state indicates a first age of the first tree 511 and a first position of the first tree 511. The first environmental state indication includes a second tree 512 and defines one or more states of the second tree 512. For example, the first environmental state indicates a first age of the second tree 512 and a first position of the second tree 512.
[0065] The first environmental state indication includes a first squirrel 521 and defines one or more states of the first squirrel 521. For example, the first environmental state indicates a first age of the first squirrel 521, a first position of the first squirrel 521, and a first motion vector of the first squirrel 521, the first motion vector indicating that the first squirrel 521 is moving toward the first acorn 531. The first environmental state indication includes a second squirrel 522 and defines one or more states of the second squirrel 522. For example, the first environmental state indicates a first age of the second squirrel 522, a first position of the second squirrel 522, and a first motion vector of the second squirrel 522, the first motion vector indicating that the second squirrel 522 is moving toward the second tree 512.
[0066] The first environmental state indication includes the first acorn 531 and defines one or more states of the first acorn 531. For example, the first environmental state indicates a first position of the first acorn 531 and a first holding state of the first acorn 531, the first holding state indicating that the first acorn 531 is not held by a squirrel.
[0067] The first image 500A also includes a time indicator 540 and a plurality of time markers 551-554. The plurality of time markers 551-554 include a pause enable indication 551, a play enable indication 552, a fast play enable indication 553, and a faster play enable indication 554. Figure 5A , time indicator 540 indicates that the current time of SR environment 409 is 1. In addition, pause enable indication 551 is currently selected (as indicated by the different display modes).
[0068] Figure 5B A portion of a display of electronic device 410 is shown displaying second image 500B of a representation of scene 415 including SR environment 409 in response to a user selection to play affordance 552 and after a frame time (e.g., an inverse of the display frame rate). Figure 5B , the time indicator 540 indicates that the current time of the SR environment 409 is 2 (e.g., Figure 5A By comparison, the first time step is 1). Figure 5B In the example, the play indicator 552 is currently selected (as shown in different display modes).
[0069] exist Figure 5B In the example, SR environment 409 is defined by a second environment state and is associated with a second environment time (eg, 2). In various implementations, the second environment state is generated according to the first model and based on the first environment state.
[0070] In various embodiments, determining the second environmental state according to the first model includes determining a second age of the first tree 511 by adding a first time step (e.g., 1) to the first age of the first tree 511, determining a second age of the second tree 512 by adding the first time step to the first age of the second tree 512, determining a second age of the first squirrel 521 by adding the first time step to the first age of the first squirrel 521, and determining a second age of the second squirrel 522 by adding the first time step to the first age of the second squirrel 522.
[0071] In various implementations, determining the second environment state according to the first model includes determining the second position of the first tree 511 by replicating the first position of the first tree 511, and determining the second position of the second tree 512 by replicating the first position of the second tree 512. Therefore, the first model indicates that the first tree 511 and the second tree 512 (e.g., assets with asset type "tree") do not change positions.
[0072] In various implementations, determining the second environmental state according to the first model includes determining the second position of the first squirrel 521 by adjusting the first position of the first squirrel 521 according to the first motion vector of the first squirrel 521, and determining the second position of the second squirrel 522 by adjusting the first position of the second squirrel 522 according to the first motion vector of the second squirrel 522. Thus, the first model indicates that the first squirrel 521 and the second squirrel 522 (e.g., assets with asset type “animal”) change positions according to the motion vectors.
[0073] In various implementations, determining the second environmental state according to the first model includes determining a second motion vector for the first squirrel 521 based on proximity of other assets to the second location of the first squirrel 521, and determining a second motion vector for the second squirrel 522 based on proximity of other assets to the second location of the second squirrel 522. For example, the first model indicates that the first squirrel 521 and the second squirrel 522 (e.g., assets with asset type “animal” and asset subtype “squirrel”) have motion vectors that lead them to nearby assets (e.g., trees, acorns, or other squirrels).
[0074] In various implementations, determining the second environmental state includes determining the second position of the first acorn 531 based on the first position of the first acorn 531 and the first holding state of the first acorn 531. For example, the first model indicates that the first acorn 531 (e.g., an asset with an asset type of "acorn") does not change position when the holding state indicates that the first acorn 531 is not held, but changes according to a change in position of an asset (e.g., a squirrel) that is holding the first acorn 531.
[0075] In various implementations, determining the second environmental state includes determining a second holding state of the first acorn 531 based on the second position of the first acorn 531, the second position of the first squirrel 521, and the second position of the second squirrel 522. For example, the first model indicates that the first acorn 531 (e.g., an asset with an asset type of “acorn”) changes its holding state when a specific asset with an asset type of “animal” and an asset subtype of “squirrel” is located in the same position as the first acorn 531 to indicate that it is being held by the specific asset.
[0076] Therefore, in Figure 5B In, with Figure 5A In comparison, the first squirrel 521 has moved to the third position 583 which is closer to the first acorn 531 , and the second squirrel 522 has moved to a position closer to the second tree 512 .
[0077] Figure 5C A portion of a display of an electronic device 410 is illustrated, the display showing a third image 500C of a representation of a scene 415 including an SR environment 409 after another frame time. Figure 5C , the time indicator 540 indicates that the current time of the SR environment 409 is 3 (e.g., Figure 5B By comparison, the first time step is 1). Figure 5C , the play indicator 552 remains selected (as shown in the different display modes).
[0078] exist Figure 5C In the embodiment of the present invention, the SR environment 409 is defined by a third environment state and is associated with a third environment time. In various implementations, the third environment state is generated according to the first model and based on the second environment state.
[0079] In various embodiments, determining the third environmental state according to the first model includes determining a third age of the first tree 511 by adding the first time step (e.g., 1) to the second age of the first tree 511, determining the third age of the second tree 512 by adding the first time step to the second age of the second tree 512, determining the third age of the first squirrel 521 by adding the first time step to the second age of the first squirrel 521, and determining the third age of the second squirrel 522 by adding the first time step to the second age of the second squirrel 522.
[0080] In various specific implementations, determining the third environment state according to the first model includes determining the second position of the first tree 511 by copying the second position of the first tree 511, and determining the third position of the second tree 512 by copying the second position of the second tree 512. Therefore, the first model indicates that the first tree 511 and the second tree 512 (e.g., assets with asset type "tree") do not change position.
[0081] In various implementations, determining the third environmental state according to the first model includes determining a third position of the first squirrel 521 by adjusting the second position of the first squirrel 521 according to the second motion vector of the first squirrel 521, and determining a third position of the second squirrel 522 by adjusting the second position of the second squirrel 522 according to the second motion vector of the second squirrel 522. Thus, the first model indicates that the first squirrel 521 and the second squirrel 522 (e.g., assets with asset type “animal”) change positions according to the motion vectors.
[0082] In various implementations, determining the third environmental state according to the first model includes determining a third motion vector for the first squirrel 521 based on proximity of other assets to the third location of the first squirrel 521, and determining a third motion vector for the second squirrel 522 based on proximity of other assets to the third location of the second squirrel 522. For example, the first model indicates that the first squirrel 521 and the second squirrel 522 (e.g., assets with asset type “animal” and asset subtype “squirrel”) have motion vectors that lead them to nearby assets (e.g., trees, acorns, or other squirrels).
[0083] In various implementations, determining the third environmental state includes determining a third position of the first acorn 531 based on the second position of the first acorn 531 and the second holding state of the first acorn 531. For example, the first model indicates that the first acorn 531 (e.g., an asset with an asset type of "acorn") does not change position when the holding state indicates that the first acorn 531 is not held, but changes according to a change in position of an asset (e.g., a squirrel) that is holding the first acorn 531.
[0084] In various implementations, determining the third environmental state includes determining a third holding state of the first acorn 531 based on the third position of the first acorn 531 and the third position of the first squirrel 521 and the third position of the second squirrel 522. Thus, the first model indicates that the first acorn 531 (e.g., an asset with asset type “acorn”) changes its holding state when a specific asset with asset type “animal” and asset subtype “squirrel” is in the same position as the first acorn 531 to indicate that it is being held by the specific asset.
[0085] In various implementations, determining the third environmental state includes determining that the asset generates a new asset. For example, in various implementations, the first model indicates that an asset with asset type "tree" has a probability of generating an asset with asset type "acorn" (which may be based on the current age of the asset).
[0086] In various implementations, determining the third environment state includes determining asset expiration. For example, in various implementations, the first model indicates that an asset with an asset type of "squirrel" expires when the age of the asset reaches a threshold. For another example, in various implementations, the first model indicates that an asset with an asset type of "tree" has a probability of expiration (which may be based on the current age of the asset).
[0087] exist Figure 5C In, with Figure 5B In contrast, the first squirrel 521 has moved position to the fifth position 585 of the first acorn 531, and the second squirrel 522 has moved to the sixth position even closer to the second tree 512. In addition, the SR environment 409 includes a new asset generated by the second tree 512, namely the second acorn 532.
[0088] Figure 5D A portion of a display of the electronic device 410 is illustrated, the display showing a fourth image 500D of a representation of the scene 415 including the SR environment 409 after another frame time. Figure 5D , the time indicator 540 indicates that the current time of the SR environment 409 is 4 (e.g., Figure 5A In comparison, the time step is 1). Figure 5D , the play indicator 552 remains selected (as shown in the different display modes).
[0089] exist Figure 5D In the embodiment of the present invention, the SR environment 409 is defined by a fourth environment state and is associated with a fourth environment time. In various specific implementations, the fourth environment state is generated according to the first model and based on the third environment state. In various specific implementations, determining the fourth environment state according to the first model and based on the third environment state is performed as described above with respect to determining the third environment state according to the first model and based on the second environment state.
[0090] exist Figure 5D In, with Figure 5C In comparison, the first squirrel 521 (holding the first acorn 531 ) has moved a position farther from the first tree 511 , and the second squirrel 522 has moved to the seventh position 587 closer to the second acorn 532 .
[0091] Figure 5E A portion of a display of electronic device 410 is shown responsive to a user selection of fast play affordance 553 and displaying fifth image 500E of a representation of scene 415 including SR environment 409 after a frame time. Figure 5E , time indicator 540 indicates that the current time of SR environment 409 is 1,000,004 (e.g., Figure 5D In contrast, the second time step is 1,000,000). Figure 5E In the example, the fast play indicator indicates that 553 is currently selected (as shown in different display modes).
[0092] exist Figure 5E , the SR environment 409 is defined by a fifth environment state and is associated with a fifth environment time. In various implementations, the fifth environment state is generated according to a second model different from the first model and based on the fourth environment state because a second time step between the fourth environment time and the fifth environment time is much longer than, for example, a first time step between the first environment time and the second environment time.
[0093] Specifically, rather than determining the fifth environmental state based on applying the first model one million times, the fifth environmental state is generated based on applying the second model once.
[0094] In various specific implementations, the second model is more computationally efficient than the first model. After such a long period of time, certain states of certain assets in the fourth environmental state may not be available for determining corresponding states of the fifth environmental state. For example, in the first model, the position of the first squirrel 521 in the first environmental state is used to determine the position of the first squirrel 521 in the second environmental state, while the position of the first squirrel 521 in the fourth environmental state is not used to determine the position of the first squirrel 521 in the fifth environmental state, so the second model is more computationally efficient than the first model.
[0095] In various embodiments, determining the fifth environmental state according to the second model includes determining a fifth age of the first tree 511 by adding the second time step (e.g., 1,000,000) to the fourth age of the first tree 511, determining the fifth age of the second tree 512 by adding the second time step to the fourth age of the second tree 512, determining the fifth age of the first squirrel 521 by adding the second time step to the fourth age of the first squirrel 521, and determining the fifth age of the second squirrel 522 by adding the second time step to the fourth age of the second squirrel 522.
[0096] In various specific implementations, determining the fifth environment state according to the second model includes determining the fifth position of the first tree 511 by copying the fourth position of the first tree 511, and determining the fifth position of the second tree 512 by copying the fourth position of the second tree 512. Therefore, the second model indicates that the first tree 511 and the second tree 512 (e.g., assets with asset type "tree") do not change positions.
[0097] In various specific implementations, determining the fifth environmental state according to the second model includes determining the fifth position of the first squirrel 521 and determining the fifth position of the second squirrel 522. However, according to the first model, the fifth position of the first squirrel 521 and the fifth position of the second squirrel 522 are determined based on their respective fourth positions and fourth motion vectors, while in various specific implementations, the fifth position of the first squirrel 521 and the fifth position of the second squirrel 522 are determined independently of their respective fourth positions and fourth motion vectors. For example, in various specific implementations, determining the fifth position of the first squirrel 521 and determining the fifth position of the second squirrel 522 include randomly selecting a position of the SR environment. In some embodiments, the corresponding fifth position is selected based on a uniform distribution on the SR environment. In some embodiments, the corresponding fifth position is selected based on a probability distribution (which may be based on the presence of other assets). For example, in some embodiments, the first squirrel 521 and the second squirrel 522 are more likely to be positioned closer to other assets.
[0098] As described above, in various specific implementations, determining the environmental state according to the first model includes determining the motion vectors of the first squirrel 521 and the second squirrel 522. However, as also described above, in various specific implementations, the positions of the first squirrel 521 and the second squirrel 522 are determined independently of their respective motion vectors. Therefore, in various specific implementations, determining the fifth environmental state does not include determining the motion vectors of the first squirrel 521 and the second squirrel 522. In various specific implementations, determining the fifth environmental state includes determining the motion vector as a dummy value or a default value. Therefore, in various specific implementations, determining the fifth environmental state includes determining the motion vector independently of the proximity of other assets to the positions of the squirrels.
[0099] As described above, in various implementations, determining the state of the environment according to the first model includes determining the position and / or holding state of assets having asset type “acorn.” In contrast, in various implementations, determining the fifth state of the environment according to the second model includes removing assets of asset type “acorn” and potentially replacing them with new assets of asset type “acorn” based on the number of assets of asset type “tree” (and their respective probabilities of generating assets of asset type “acorn”).
[0100] Therefore, in Figure 5E In, with Figure 5D In comparison, the first squirrel 521 and the second squirrel 522 have completely changed positions, the second tree 512 has grown higher to a third height 593, the first acorn 531 and the second acorn 532 have disappeared, and the third acorn 533 has appeared.
[0101] Fig. 5F A portion of a display of an electronic device 410 is illustrated, the display showing a sixth image 500F of a representation of a scene 415 including an SR environment 409 after another frame time. Fig. 5F , time indicator 540 indicates that the current time of SR environment 409 is 2,000,004 (e.g., Figure 5E In comparison, the time step is 1,000,000). Fig. 5F In the example, the fast play indicator 553 remains selected (as shown in the different display modes).
[0102] exist Fig. 5F In the embodiment of the present invention, the SR environment 409 is defined by a sixth environment state and is associated with a sixth environment time. In various implementations, the sixth environment state is generated according to the second model and based on the fifth environment state. In various implementations, determining the sixth environment state according to the second model and based on the fifth environment state is performed as described above with respect to determining the fifth environment state according to the second model and based on the fourth environment state.
[0103] exist Fig. 5F In, with Figure 5E In comparison, the first squirrel 521 and the second squirrel have completely changed positions again, the second tree 512 has grown higher again to a fourth height 594, and the third acorn 533 has disappeared.
[0104] Figure 5G A portion of a display of electronic device 410 is shown displaying seventh image 500G of a representation of scene 415 including SR environment 409 after a frame time in response to a user selection of faster play affordance 553. Figure 5G, time indicator 540 indicates that the current time of SR environment 409 is 10,002,000,004 (e.g., Fig. 5F In comparison, the third time step is 10,000,000,000). Figure 5G In the example, the faster play indicator 554 is currently selected (as shown in the different display modes).
[0105] exist Figure 5G , the SR environment 409 is defined by a seventh environment state and is associated with a seventh environment time. In various implementations, the seventh environment state is generated based on the sixth environment state according to a third model different from the second model and because the third time step between the sixth environment time and the seventh environment time is much longer than, for example, the second time step between the fourth environment time and the fifth environment time.
[0106] Specifically, instead of determining the seventh environmental state based on applying the first model 10 billion times (or applying the second model 10,000 times), the fifth environmental state is generated based on applying the third model once.
[0107] In various implementations, the third model is computationally more efficient than the second model. After such a long period of time, some states of some assets in the sixth environmental state may not be available for determining the fifth environmental state. For example, in the second model, the age of the second tree 512 in the fourth environmental state is used to determine the age of the second tree 512 in the fifth environmental state, while the third model does not use the age and location of all assets. Instead, based on the number of trees, acorns, and squirrels, the third model generates new assets based on the expected number of trees, acorns, and squirrels at the seventh environmental time, and randomly distributes them in location and age.
[0108] exist Figure 5G In, with Fig. 5F In comparison, the first squirrel 521 and the second squirrel 522 have been replaced by the third squirrel 523 and the fourth squirrel 524; the first tree 511 and the second tree 512 have been replaced by the third tree 513, the fourth tree 514, and the fifth tree 515; and the fourth acorn 534 has appeared.
[0109] Figure 5H A portion of a display of an electronic device 410 is illustrated, the display showing an eighth image 500H of a representation of a scene 415 including an SR environment 409 after another frame time. Figure 5H , time indicator 540 indicates that the current time of SR environment 409 is 20,002,000,004 (e.g., Figure 5G In comparison, the third time step is 10,000,000,000). Figure 5H , the faster play indicator 554 remains selected (as shown in the different displays).
[0110] exist Figure 5H In the embodiment of the present invention, the SR environment 409 is defined by the eighth environment state and is associated with the eighth environment time. In various specific implementations, the eighth environment state is generated according to the third model and based on the seventh environment state. In various specific implementations, determining the eighth environment state according to the third model and based on the seventh environment state is performed as described above with respect to determining the seventh environment state according to the third model and based on the sixth environment state.
[0111] exist Figure 5H In, with Figure 5G In comparison, the third squirrel 523 and the fourth squirrel 524 have been replaced by the fifth squirrel 525, the sixth squirrel 526, and the seventh squirrel 527; the third tree 513, the fourth tree 514 and the fifth tree 515 have been replaced by the sixth tree 516, the seventh tree 517, the eighth tree 518 and the ninth tree 519; and the fourth acorn 534 has been replaced by the fifth acorn 535 and the sixth acorn 536.
[0112] In various implementations, based on the number of trees, acorns, and squirrels, the third model generates an asset distribution indicating the likelihood that an asset is present at each location at an environment time, rather than generating new assets based on the expected number of trees, acorns, and squirrels at the environment time. Displaying the asset distribution, for example, as a cloud, indicates the cluster density of asset types at the environment time.
[0113] FIG5I illustrates a portion of a display of the electronic device 410 showing a ninth image 500I of a representation of the scene 415 including the SR environment 409 with an asset distribution. In FIG5I , the time indicator 540 indicates that the current time of the SR environment 409 is 20,002,000,004 (e.g., Figure 5G In contrast, the third time step is 10,000,000,000). In FIG. 5I , faster play enable indication 554 remains selected (as shown in the different display modes).
[0114] In Figure 5I, Figure 5H In contrast, the fifth squirrel 525, the sixth squirrel 526, and the seventh squirrel 527 are not displayed. Instead, the squirrel distribution 561 is displayed, indicating the possibility of squirrels existing at the corresponding location at the eighth ambient time.
[0115] Figure 6 An environment state 600 according to some implementations is shown. In various implementations, the environment state 600 is a data object, such as an XML file. The environment state 600 indicates that one or more assets are included in the SR environment, and also indicates one or more states of the one or more assets.
[0116] The environmental state 600 includes a time field 610 indicating an environmental time associated with the environmental state.
[0117] The environment state 600 includes an asset field 620 that includes a plurality of individual asset fields 630 and 640 associated with respective assets of the SR environment. Figure 6 Only two assets are shown, but it should be understood that asset field 620 may include any number of asset fields.
[0118] The asset field 620 includes a first asset field 630. The first asset field 630 includes a first asset identifier field 631, which includes an asset identifier of the first asset. In various implementations, the asset identifier includes a unique number. In various implementations, the asset identifier includes a name of the asset.
[0119] The first asset field 630 includes a first asset type field 632 including data indicating an asset type of the first asset. The first asset field 630 includes an optional asset subtype field 633 including data indicating an asset subtype of the asset type of the first asset.
[0120] The first asset field 630 includes a first asset status field 634 having a plurality of first asset status fields 635A and 635B. In various specific implementations, the asset status field 634 is based on the asset type and / or asset subtype of the first asset. For example, when the asset type is "tree", the asset status field 634 includes an asset location field 635A and an asset age field 635B, the asset location field including data indicating the location of the asset in the SR environment, and the asset age field including data indicating the age of the asset. For another example, when the asset type is "animal", the asset status field 634 includes an asset motion vector field, the asset motion vector field including data indicating the motion vector of the asset. For another example, when the asset type is "acorn", the asset status field 634 includes an asset holding status field, the asset holding status field including data indicating which other asset (if any) is holding the asset. For another example, when the asset type is "weather", the asset status field 634 includes an asset temperature field, an asset humidity field, and / or an asset rainfall field, wherein the asset temperature field includes data indicating the temperature of the SR environment, the asset humidity field includes data indicating the humidity of the SR environment, and the asset rainfall field includes data indicating the rainfall conditions of the SR environment.
[0121] The asset field 620 includes a second asset field 640. The second asset field 640 includes a second asset identifier field 640 that includes an asset identifier of the second asset. The second asset field 630 includes a second asset type field 642 that includes data indicating an asset type of the second asset. The second asset field 642 includes an optional asset subtype field 643 that includes data indicating an asset subtype of the asset type of the second asset.
[0122] The second asset field 640 includes a second asset status field 643, which includes a plurality of second asset status fields 645A and 645B. In various implementations, the asset status field 644 is based on the asset type and / or asset subtype of the second asset.
[0123] Figure 7 A block diagram of an environment state generator 750 according to some implementations is shown. The environment state generator 750 receives as input a first environment state 710 associated with a first environment time and a time step 715. The environment state generator 750 generates as output a second environment state 720 associated with a second environment time that is equal to the first environment time incremented by the time step 715.
[0124] The environment state generator 750 includes a plurality of models 752A-752C including a first model 752A, a second model 752B, and a third model 752C connected in parallel between a multiplexer 751 and a demultiplexer 753. The environment state generator 750 includes a model selector 755 that controls the multiplexer 751 and the demultiplexer 753 based on the time step 715.
[0125] When the time step 715 is a first value (or within a first value range), the model selector 755 controls the multiplexer 751 and the demultiplexer 753 to feed the first environment state 710 and the time step 715 to the first model 752A to generate the second environment state 720. When the time step 715 is a second value (or within a second value range), the model selector 755 controls the multiplexer 751 and the demultiplexer 753 to feed the first environment state 710 and the time step 715 to the second model 752B to generate the second environment state 720. When the time step 715 is a third value (or within a third value range), the model selector 755 controls the multiplexer 751 and the demultiplexer 753 to feed the first environment state 710 and the time step 715 to the third model 752C to generate the second environment state 720.
[0126] Thus, each model 752A-752C receives the first environment state 710 and the time step 715 and generates the second environment state 720 when selected by the model selector 755. In various implementations, each model 752A-752C includes corresponding heuristics and metadata for generating the second environment state 720.
[0127] Figure 8 8 is a flowchart representation of a method 800 for generating an environment state of an SR environment according to some specific implementations. In various specific implementations, the method 800 is performed by a device (e.g., Figure 3 HMD 120B or Figure 4 In some implementations, the method 800 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof) (e.g., Figure 7 In some implementations, the method 800 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory). In brief, in some cases, the method 800 includes obtaining a first environment state and a time step associated with a first environment time of the SR environment, selecting a model from a plurality of models based on the time step, and applying the selected model to the first environment state to generate a second environment state associated with a second environment time.
[0128] The method 800 begins at block 810, where the device obtains a first environment state of the SR environment associated with a first environment time, where the first environment state indicates that one or more assets are included in the SR environment and further indicates one or more states of the one or more assets. In various implementations, the method 800 includes displaying the SR environment having the first environment state at a first time (e.g., a first real time).
[0129] In various implementations, the environment state is a data object, such as an XML file. In various implementations, the first environment state is manually programmed. In various implementations, the first environment state is generated by applying a model to a previous environment state.
[0130] At block 820, method 800 continues with the device obtaining a time step from the first ambient time to the second ambient time. In various implementations, the time step is manually programmed. In various implementations, the time step is determined based on a user's interaction with one or more time markers that are respectively associated with the one or more time steps.
[0131] The method 800 continues at block 830, where the device selects a model from a plurality of different models based on the time step. In various implementations, in response to obtaining a time step having a first value (or within a first value range), the device selects a first model, and in response to obtaining a time step having a second value (or within a second value range), the device selects a second model different from the first model. In various implementations, the second model is more computationally efficient than the first model.
[0132] The method 800 continues at block 840 where the device determines a second environment state of the SR environment associated with a second environment time based on the selected model. In various implementations, the method 800 includes displaying the SR environment with the second environment state at a second time (eg, a second real time).
[0133] In various implementations, determining the second environmental state according to the selected model (at block 840) includes determining a second value of the asset state of the second environmental state based on the first value of the asset state of the first environmental state according to the selection of the first model (at block 830). Figure 5C As described, determining the third environmental state includes determining the third position of the first squirrel 521 based on the second position of the first squirrel 521 and the second motion vector of the first squirrel 521. In various specific implementations, determining the second environmental state according to the selected model (at block 840) includes determining the second value of the asset state of the second environmental state independently of the first value of the asset state of the first environmental state according to the selection of the second model (at block 830). For example, as described above with respect to Figure 5E As described above, determining the fifth environmental state includes determining the fifth position of the first squirrel 521 independently of the fourth position of the first squirrel 521 and the fourth motion vector of the first squirrel 521 .
[0134] In various implementations, determining (at block 840) the second environmental state according to the selected model includes determining a second value of the asset state of the second environmental state based on the first environmental state according to the selection of the first model (at block 830). Figure 5C As described, determining the third environmental state includes determining a third motion vector of the first squirrel 521 based on the proximity of the third position of the first squirrel 521 to the third position of the other asset (which is based on the second position of its second environmental state). In various specific implementations, determining the second environmental state according to the selected model (at block 840) includes discarding the second value of the asset state for determining the second environmental state based on the first environmental state according to the selection of the second model (at block 830). For example, as described above with respect to Figure 5E As described above, determining the fifth environmental state does not include determining the fifth motion vector of the first squirrel 521 .
[0135] In various implementations, determining the second environmental state according to the selected model (at block 840) includes determining a second value of the asset state of the second environmental state based on the first value of the asset state of the first environmental state according to the selection of the first model (at block 830). Figure 5E As described above, determining the fifth environment state includes determining the fifth position of the first tree 511 based on the fourth position of the first tree 511. In various specific implementations, determining the second environment state according to the selected model (at block 840) includes excluding the asset having the asset state from the second environment state according to the selection of the second model (at block 830). Figure 5G As described above, determining the seventh environmental state includes removing the first tree 511 .
[0136] In various implementations, determining (at block 840) the second environment state according to the selected model includes including one or more new assets of the same asset type as the assets of the asset state based on the number of assets of the corresponding asset type in the first environment state according to the selection of the second model (at block 830). Figure 5G As described above, determining the seventh environmental state includes adding the third tree 513, the fourth tree 514, and the fifth tree 515 based on the number of trees (and the number of squirrels) in the sixth environmental state. Figure 5E As described, determining the fifth environmental state includes adding a third acorn 533 based on the number of trees in the fifth environmental state.
[0137] Fig. 9 900 is a flowchart representation of a method 900 for generating an environment state of an SR environment according to some specific implementations. In various specific implementations, the method 900 is performed by a device (e.g., Figure 3 HMD 120B or Figure 4 In some implementations, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof) (e.g., Figure 7 In some implementations, the method 900 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory). In brief, in some cases, the method 900 includes obtaining a first environment state and a time step associated with a first environment time of the SR environment, selecting a model from a plurality of models based on the time step, and applying the selected model to the first environment state to generate a second environment state associated with a second environment time.
[0138] The method 900 begins at block 910, where the device obtains a first environment state of the SR environment associated with a first environment time, where the first environment state indicates that one or more assets are included in the SR environment and further indicates one or more states of the one or more assets. For example, in various implementations, the first environment state includes data indicating that a first asset of the one or more assets is included, a type of the first asset, a corresponding location of the first asset in the SR environment, and a corresponding age of the first asset. In various implementations, the method 900 includes displaying the SR environment having the first environment state at the first time.
[0139] In various implementations, the environment state is a data object, such as an XML file. In various implementations, the first environment state is manually programmed. In various implementations, the first environment state is generated by applying a model to a previous environment state.
[0140] The method 900 continues at block 920, where the device determines, according to the first model and based on the first environment state, a second environment state associated with a second environment time. In various implementations, the second environment state includes data indicating a first asset among the one or more assets, a type of the first asset, a corresponding location of the first asset in the SR environment, and a corresponding age of the first asset. In various implementations, the method 900 includes displaying the SR environment with the second environment state at a second time (e.g., a frame time later than the first time).
[0141] The method 900 continues at block 930, where the device receives input indicating a time step from the second ambient time to a third ambient time, where the time step is different from a difference between the first ambient time and the second ambient time. In various implementations, receiving input indicating the time step includes receiving user input indicating a selection of one of a plurality of time-marked representations respectively associated with the plurality of time steps. For example, Figure 5E , the user has selected the fast play indication 553, and Figure 5D and Figure 5E The time step between Figure 5C and Figure 5D The time step between .
[0142] The method 900 continues at block 940, where the device determines a third environmental state associated with a third environmental time according to a second model different from the first model and based on the second environmental state. In various implementations, the second model is computationally more efficient than the first model. In various implementations, the third environmental state includes data indicating a first asset among the one or more assets, a type of the first asset, a corresponding location of the first asset in the SR environment, and a corresponding age of the first asset. In various implementations, the method 900 includes displaying the SR environment with the third environmental state at a third time (e.g., a frame time later than the third time).
[0143] In various implementations, determining (at block 920) the second environmental state according to the first model includes determining a second value of the asset state for the second environmental state based on the first value of the asset state for the first environmental state. Figure 5C As described, determining the third environmental state includes determining a third position of the first squirrel 521 based on the second position of the first squirrel 521 and the second motion vector of the first squirrel 521. In various specific implementations, (at block 940) determining the third environmental state according to the second model includes determining a third value of the asset state of the third environmental state independently of the second value of the asset state of the second environmental state. For example, as described above with respect to Figure 5E As described above, determining the fifth environmental state includes determining the fifth position of the first squirrel 521 independently of the fourth position of the first squirrel 521 and the fourth motion vector of the first squirrel 521 .
[0144] In various implementations, determining (at block 920) the second environmental state according to the first model includes determining a second value of the asset state of the second environmental state based on the first environmental state. Figure 5C As described, determining the third environmental state includes determining a third motion vector of the first squirrel 521 based on the proximity of the third position of the first squirrel 521 to the third position of the other asset (which is based on the second position of its second environmental state). In various specific implementations, (at block 940) determining the second environmental state according to the second model includes discarding the third value of the asset state that determines the third environmental state based on the second environmental state. For example, as described above with respect to Figure 5E As described above, determining the fifth environmental state does not include determining the fifth motion vector of the first squirrel 521 .
[0145] In various implementations, determining (at block 920) the second environmental state according to the first model includes determining a second value of the asset state for the second environmental state based on the first value of the asset state for the first environmental state. Figure 5EAs described above, determining the fifth environmental state includes determining the fifth position of the first tree 511 based on the fourth position of the first tree 511. In various specific implementations, (at block 940) determining the third environmental state according to the second model includes excluding the asset having the asset state from the third environmental state. Figure 5G As described above, determining the seventh environmental state includes removing the first tree 511 .
[0146] In various implementations, determining (at block 940) the third environmental state according to the second model includes including one or more new assets of the same asset type as the asset having the asset state based on the number of assets of the corresponding asset type in the second environmental state. Figure 5G As described above, determining the seventh environmental state includes adding the third tree 513, the fourth tree 514, and the fifth tree 515 based on the number of trees (and the number of squirrels) in the sixth environmental state. Figure 5E As described, determining the fifth environmental state includes adding a third acorn 533 based on the number of trees in the fifth environmental state.
[0147] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.
[0148] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0149] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.
[0150] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
Claims
1. A method for determining an environment state for a synthetic reality environment, comprising: At an electronic device comprising a processor and a non-transitory memory: Obtaining a first environment state of an environment associated with a first environment time, wherein the first environment state indicates that one or more assets are included in the environment and further indicates one or more states of the one or more assets; determining a second environmental state associated with a second environmental time according to the first model and based on the first environmental state; receiving an input indicating a time step from the second ambient time to a third ambient time, wherein the time step is different than a difference between the first ambient time and the second ambient time; A third environmental state associated with the third environmental time is determined according to a second model different from the first model and based on the second environmental state. The method of claim 1 , wherein the first environment state comprises an XML document.
3. The method of claim 1 or 2, wherein the first environmental state, the second environmental state, and the third environmental state include data indicating: including a first asset among the one or more assets; the type of the first asset; The respective location of the first asset in the environment; and The corresponding age of the first asset.
4. The method according to claim 1 or 2, further comprising: displaying the environment having the first environmental state at a first time; displaying the environment having the second environment state at a second time that is one frame time later than the first time; as well as The environment having the third environment state is displayed at a third time later than the second time by one of the frame times.
5. The method according to claim 1 or 2, wherein receiving an input indicating the time step comprises receiving a user input indicating a selection of one of a plurality of time-stamping indications respectively associated with a plurality of time steps.
6. The method of claim 1 or 2, wherein the second model is more computationally efficient than the first model.
7. The method according to claim 1 or 2, wherein: Determining the second environmental state according to the first model comprises determining a second value of the asset state of the second environmental state based on a first value of the asset state of the first environmental state; and Determining the third environmental state according to the second model includes determining a third value of the asset state for the third environmental state independent of the second value of the asset state for the second environmental state.
8. The method according to claim 1 or 2, wherein: Determining the second environmental state according to the first model includes determining a second value of the asset state of the second environmental state based on the first environmental state; as well as Determining the third environmental state according to the second model includes discarding a third value of the asset state in determining the third environmental state based on the second environmental state.
9. The method according to claim 1 or 2, wherein: Determining the second environmental state according to the first model comprises determining a second value of the asset state of the second environmental state based on a first value of the asset state of the first environmental state; and Determining the third environmental state according to the second model includes excluding the asset having the asset state from the third environmental state.
10. The method of claim 9, wherein determining the third environmental state according to the second model comprises including one or more new assets having the same asset type as the assets having the asset state based on a number of assets having the corresponding asset type in the second environmental state.
11. The method according to any one of claims 1 to 2 and 10, wherein determining the third environmental state according to the second model comprises determining a third value of the asset state of the third environmental state by adding the time step to the second value of the asset state of the second environmental state.
12. An electronic device comprising: one or more processors; non-transitory memory; and One or more programs stored in the non-volatile memory, which, when executed by the one or more processors, cause the electronic device to perform any of the methods described in claims 1 to 11.
13. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device, cause the device to perform any of the methods of claims 1 to 11.
14. A computer program product comprising instructions, which, when executed by a processor, cause a device comprising the processor to perform any of the methods according to claims 1-11.
