Content creation platform for XR devices

Abstract

A content creation system for an augmented reality (XR) system. The content creation system receives motion data of the XR device and generates trajectory data for a trajectory within a 3D environmental model of the real-world scene based on the motion data, where the trajectory simulates motion of the XR device within the real-world scene. The content creation system receives user interaction event data and generates analog sensor data based on the trajectory data, the 3D environment model, and the user interaction event data. The content creation system generates simulated tracking data based on the simulated sensor data and, when generating the simulated tracking data, determines simulated power consumption data and thermal condition data based on operation of the computer vision component. The content creation system generates a display and simulated power consumption and thermal data from a user perspective of the 3D environment model.

Description

[0001] Priority Claim

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 065,180, filed on December 13, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to user interfaces, and more particularly, to user interfaces for augmented reality or virtual reality. Background Art

[0004] A head-wearable XR device can be implemented with a transparent or semi-transparent display through which a user of the head-wearable XR device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or semi-transparent display and also enables the user to see objects (e.g., virtual objects such as renderings of 2D or 3D graphical models, images, videos, text, etc.) generated for display as part of and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". A head-wearable XR device can also completely occlude the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". In a hybrid form, a view of the surrounding environment is captured using a camera device and then the view is displayed to the user together with augmentations on a display that occludes the user's eyes. As used herein, unless the context otherwise indicates, the term extended reality (XR) refers to augmented reality, virtual reality, and any hybrid of these technologies.

[0005] An XR system having a head-wearable XR device may have limited battery capacity to perform operations associated with providing an XR experience to the user. Additionally, some long-duration operations generate heat that is difficult to dissipate in the head-wearable XR device, causing the head-wearable XR device to become too hot for the user to wear comfortably. Brief Description of the Drawings

[0006] In the drawings (which are not necessarily drawn to scale), like reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the most significant digits in the reference numeral refers to the figure number in which the element was first introduced. Some non-limiting examples are shown in the figures of the drawings, in which:

[0007] Figure 1 A content creation computing environment is shown in accordance with some examples.

[0008] Figure 2is a pictorial representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein.

[0009] Figure 3A is a process flow diagram of a content creation method according to some examples.

[0010] Figure 3B is a diagram of a content creation system according to some examples.

[0011] Figure 3C shows a user interface for inputting simulation parameters and displaying simulation results according to some examples.

[0012] Figure 3D shows a 3D environment according to some examples.

[0013] Figure 3E shows a user interface for displaying simulation results according to some examples.

[0014] Figure 4 is a block diagram showing a software architecture according to some examples.

[0015] Figure 5A is a perspective front view of a head-worn XR device according to some examples, and Figure 5B is a view of the head-worn XR device from the user's perspective.

[0016] Figure 6 is a pictorial representation of another machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein.

[0017] Figure 7 is a block diagram showing another software architecture according to some examples. Detailed Description

[0018] Developing immersive XR system applications is different from developing mobile applications or autonomous vehicle applications in terms of spatial considerations, user interaction methods, and resource constraints. To develop immersive XR applications for XR devices, content creators need to consider space, user interaction, and resource constraints simultaneously. Spatial changes, such as moving from indoors to outdoors or from outdoors to indoors, trigger underlying algorithm responses and hardware configuration changes, thus affecting user interaction and resource usage. User interaction (e.g., gesture recognition, touchpad events, etc.) is affected by the adaptation of various hardware (e.g., multiple camera device sensors), and thus results in different power usage over time. As an embedded system, XR devices are resource-constrained devices. When applications communicate, interact, and operate in different ways, the battery usage and heat distribution can vary.

[0019] Ideally, content creators can treat the underlying computer vision (CV) algorithms and hardware / operating system (HW / OS) configurations as black boxes and utilize all virtual event simulations to test or run XR applications. However, in reality, when XR devices are moved by users under limited power and thermal thresholds, the CV algorithm performance and HW conditions change significantly, and it may be beneficial to adjust the functionality of XR applications based on the real-world environment. Additionally, services supporting XR applications may face managing space-related use cases, such as XR navigation that switches between indoor and outdoor users. Therefore, it is desirable to formulate relevant sensor simulations and OS emulations into a structured format within an application development platform dedicated to the development of applications for XR wearable devices, consistent with various space-related events provided as presets to content creators. A content creation platform supported by comprehensive hardware and OS simulations can provide developers with a compact solution to address spatial changes, user interaction, and resource constraints for developing, testing, benchmarking, and debugging XR applications.

[0020] In some examples, the content creation platform receives motion data of the movement of an XR device and generates trajectory data of a trajectory within a virtual 3D environment represented by a 3D environmental model of the real-world scene, where the trajectory simulates the movement of the XR device within the real-world scene. The content creation platform also receives user interaction event data. The data simulation platform generates simulated sensor data based on the trajectory data, the 3D environmental model, and the user interaction event data. A computer vision component within the operating system emulator generates simulated tracking data based on the simulated sensor data. When generating the simulated tracking data, the operating system emulator determines simulated power consumption data and simulated thermal condition data based on the operation of the computer vision component. The content creation platform uses a user interface to provide the simulated power consumption data and the simulated thermal condition data to the user.

[0021] In some examples, receiving motion data of an XR device includes capturing data of the motion of the XR device as the user moves the XR device through a real-world scenario.

[0022] In some examples, receiving motion data of an XR device includes selecting motion data from a list of stored data.

[0023] In some examples, user interaction events are associated with time events.

[0024] In some examples, user interaction events are associated with a location within a real-world scenario.

[0025] In some examples, user interaction events are poses made by the user.

[0026] In some examples, the trajectory data for generating a trajectory within a 3D environment model further includes changing lighting parameters of the 3D environment model.

[0027] In some examples, the simulated sensor data includes camera device data, inertial motion unit data, and global positioning sensor data.

[0028] In some examples, the simulated tracking data includes device pose data and hand recognition result data.

[0029] In some examples, generating simulated tracking data based on simulated sensor data further includes generating simulated operating system events that affect the operation of computer vision components based on changes in the simulated environment over time.

[0030] In some examples, the simulated operating system events include thermal throttling events, network condition change events, ambient noise events, and display adjustment events.

[0031] Based on the following figures, description, and claims, other technical features may be readily apparent to those skilled in the art.

[0032] Computing environment

[0033] Figure 1FIG. 0 shows a content creation computing environment 100 according to some examples. The content creation computing environment 100 includes a content creation system 102 that simulates the operation of a head-worn XR device 108 as a user wearing the head-worn XR device 108 moves through a 3D environment 106 and interacts with an XR experience provided by an XR application executing on the head-worn XR device 108. A content creator 104 interacts with a user interface 112 provided by the content creation system 102 to input content creation platform configuration data 110 that configures various components of the content creation system 102 that simulate the operation of the head-worn XR device 108 in order to determine the power consumption and thermal conditions of the head-worn XR device 108.

[0034] Machine architecture

[0035] Figure 2 is a pictorial representation of a machine 200 within which instructions 202 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 200 to perform any one or more of the methods discussed herein. For example, the instructions 202 can cause the machine 200 to perform any one or more of the methods described herein. The instructions 202 transform a general, unprogrammed machine 200 into a particular machine 200 programmed to perform the described and illustrated functions in the described manner. The machine 200 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 200 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 200 can include, but is not limited to: a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 202 specifying actions to be taken by the machine 200. Further, while a single machine 200 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 202 to perform any one or more of the methods discussed herein. In some examples, the machine 200 can also include both a client system and a server system, where certain operations of a particular method or algorithm are executed on the server side and where certain operations of a particular method or algorithm are executed on the client side.

[0036] Machine 200 may include a processor 204, a memory 206, and input / output (I / O) components 208, and the processor 204, the memory 206, and the input / output (I / O) components 208 may be configured to communicate with each other via a bus 210. In an example, the processor 204 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 212 and a processor 214 that execute instructions 202. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 2 multiple processors 204 are shown, machine 200 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0037] The memory 206 includes a main memory 216, a static memory 232, and a storage unit 218, and the main memory 216, the static memory 232, and the storage unit 218 are all accessible by the processor 204 via the bus 210. The main memory 206, the static memory 232, and the storage unit 218 store instructions 202 that implement any one or more of the methods or functions described herein. During execution of the instructions 202 by the machine 200, the instructions 202 may also reside, wholly or partially, within the main memory 216, within the static memory 232, within the machine-readable medium 220 within the storage unit 218, within at least one of the processors 204 (e.g., within the cache memory of the processor), or within any suitable combination thereof.

[0038] The I / O components 208 may include a variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, and the like. The specific I / O components 208 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I / O components 208 may include Figure 2Many other components not shown. In various examples, I / O component 208 can include user output component 222 and user input component 224. User output component 222 can include visual components (e.g., a display such as a plasma display panel (PDP), light emitting diode (LED) display, liquid crystal display (LCD), projector, or cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. User input component 224 can include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or another pointing instrument), haptic input components (e.g., physical buttons, a touchscreen that provides the location and force of a touch or touch gesture, or other haptic input components), audio input components (e.g., a microphone), and the like.

[0039] A variety of techniques can be used to implement communication. I / O component 208 also includes communication component 226, which is operable to couple machine 200 to network 228 or device 230 via respective couplings or connections. For example, communication component 226 can include a network interface component for docking with network 228 or another suitable device. In additional examples, communication component 226 can include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power), components, and other communication components that provide communication via other modalities. Device 230 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0040] In addition, communication component 226 can detect identifiers, or include components operable to detect identifiers. For example, communication component 226 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting optical sensors for the following: one-dimensional barcodes, such as universal product code (UPC) barcodes; multi-dimensional barcodes, such as quick response (QR) codes, Aztec codes, data matrix, Data Glyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying audio signals of a tag). Additionally, various information can be derived via communication component 226, such as a location via Internet protocol (IP) geolocation, via Positions determined by signal triangulation, positions of NFC beacon signals that can indicate a specific location via detection, etc.

[0041] Various memories (e.g., main memory 216, static memory 232, and the memory of processor 204) and storage unit 218 can store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. When executed by processor 204, these instructions (e.g., instruction 202) cause various operations to implement the disclosed examples.

[0042] Instructions 202 can be transmitted or received over network 228 via a network interface device (e.g., the network interface component included in communication component 226) using a transmission medium and using any one of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 202 can be transmitted or received to device 230 via an interface (e.g., a peer-to-peer interface) using a transmission medium.

[0043] System Operations

[0044] Figure 3A is a process flow diagram of content creation method 300, Figure 3B is a diagram of content creation system 316, Figure 3C shows a user interface for inputting simulation parameters and displaying simulation results, Figure 3D shows a virtual 3D environment 360, and Figure 3E shows a user interface for displaying simulation results according to some examples. According to some examples, when a user interacts with an XR experience within a 3D environment represented by virtual 3D environment 360, content creator 324 uses content creation system 316 to simulate the operation of a head-worn XR device worn by the user.

[0045] In operation 302, content creation platform 318 receives a selection of motion data 338 for one or more motions of a head-worn XR device from content creator 324. When the user interacts with the XR experience provided by the head-worn XR device, motion data 338 describes the path or trajectory of the head-worn XR device when worn by the user. The XR experience is provided to the user as the user moves through the 3D environment represented by virtual 3D environment 360. For example, the motion can correspond to one or more movements or motions of the user, such as but not limited to walking, jumping, riding a scooter, riding a bicycle, running, etc.

[0046] In some examples, the motion data is motion data captured by a head-worn XR device worn by a user when the user makes one or more motions. In some examples, the motion data is simulated motion data based on a computational response of a mathematical model of a head-worn XR device subjected to simulated motion. In some examples, the motion data is a synthesis of actual motion data and simulated motion data captured by the head-worn XR device.

[0047] In some examples, the content creation platform 318 provides a user interface 336 including a motion preset 352 to the content creator 324, and the content creator 324 uses the user interface 336 to select a preset motion of the motion data 338.

[0048] In operation 304, the content creation platform 318 generates trajectory data 342 of a trajectory 362 within a virtual 3D environment 360 represented by the 3D environment model 340 based on motion data and position data of a set of one or more positions in the 3D environment model 340. The trajectory 362 simulates the motion of the XR device within the 3D environment represented by the virtual 3D environment 360. For example, the content creation platform 318 uses the velocity component of the motion data and the position data to determine vector data of one or more vectors describing an instantaneous position, orientation, or pose, and the ( Figure 3D of) field of view 364 of one or more imaging devices of the head-worn XR device to determine the position of the head-worn XR device in the 3D environment model at a certain moment. In some examples, the 3D environment model 340 includes one or more environmental regions having different environmental conditions (such as, but not limited to, lighting, ambient lighting level, temperature, ambient acoustic noise level, etc.). Figure 3D In operation 306, the content creation platform 318 receives user interaction event data 344. For example, the user interaction event data 344 includes one or more events related to a user's interaction with the XR experience. The event includes a timing indicator and / or a position indicator and one or more user interactions. The user interaction can be an interaction of the user with the XR experience, which is detected by hand tracking or pose recognition, such as, but not limited to, a pose made by the user, an interaction with a virtual object in the XR experience, an interaction with a physical object represented in the 3D environment model 340, etc. The user interaction represents a simulated user interaction, which can occur in the XR experience provided by the XR application 328, but does not physically or actually occur at the current time. They are selected by the XR application developer to represent specific types of user interactions that may occur to view possible results and impacts.

[0049]

[0050] ​In some examples, a user interface 336 provides a content creator 104 with a list 358 of possible user interactions. The list of possible user interactions is based on a set of user interactions identified by components in a service component 346 provided by an operating system emulator 320, where the components in the service component 346 include, by way of example and not limitation, a computer vision component 330. In some examples, the list of possible user interactions includes user interactions provided by an XR application. In some examples, a content creation platform 318 provides a user interface 336 to a content creator 324, and the content creator 324 uses the user interface 336 to select a user interaction and input a timing indicator and / or a location indicator of user interaction event data. The timing indicator and the location indicator represent the travel time and location of the XR device along a trajectory 362 within a virtual 3D environment 360.

[0051] In operation 308, a data simulation platform generates simulated sensor data 332 and simulated battery data 334 based on trajectory data, a 3D environment model, and user interaction event data. For example, a data simulation platform 322 generates simulated first-person perspective image data along a movement trajectory in a virtual 3D environment represented by a 3D environment model 340 based on a human body model simulation of user interactions of the 3D environment model 340, trajectory data 342, and user interaction event data 344. The simulated first-person perspective image data is used to generate the simulated sensor data 332 to generate simulated sensor data for various sensors of a head-mounted XR device. The simulated sensor data 332 includes operating parameters and output data of sensors of the head-mounted XR device as if the head-mounted XR device were operating in the 3D environment and as if the user wearing the head-mounted XR device were moving through the 3D environment represented by the 3D environment model 340 along the trajectory represented by the trajectory data 342.

[0052] The simulated sensor data 332 may include output data and operating parameter data of one or more sensors of the head-mounted XR device (such as, by way of example and not limitation, a camera device, an inertial motion unit (IMU), a global positioning system (GPS) sensor, etc.). The output data of the simulated sensor data 332 may include, but is not limited to, simulated image data, simulated orientation or pose data, simulated location data, etc. The operating parameter data may include operating parameter data of one sensor among a plurality of sensors, including but not limited to, automatic exposure settings or changes, camera device frame rates or changes, the number of camera devices affected by any changes or adjustments in response to changes in environmental variables of the 3D environment model 340, automatic changes in the IMU for capturing the orientation or pose of the head-mounted XR device, automatic changes in the GPS sensor for sensing the location of the head-mounted XR device, operating modes of one or more light sources used by the head-mounted XR device, etc.

[0053] In operation 310, the computer vision component 330 in the service component 346 of the operating system emulator 320 generates simulated tracking data 348 based on simulated sensor data. For example, the computer vision component 330 uses the simulated image data and simulated orientation or pose data of the simulated sensor data 332 to generate the simulated tracking data. The simulated tracking data includes the user interactions included in the user interaction event data 344. When the computer vision component 330 generates the simulated tracking data, the operating system emulator 320 uses simulated operating system events to impose simulated operating conditions on the computer vision component 330 when the computer vision component 330 generates the simulated tracking data. The simulated operating system events include, but are not limited to, thermal throttling of the processor execution speed, display changes of the display provided to the user during the XR experience, brightness adjustment in the ambient area, resolution changes of the operating parameters of the computer vision component 330 or one or more camera devices of the head-mounted XR device, refresh rate of the simulated sensor data 332, low power mode imposed by the operating system emulator 320 that affects one or more services provided by the service component 346, network conditions such as conditions for wireless communication, cellular phone connection, GPS sensor communication, impact of ambient noise on one or more services provided by 346, one or more impacts on one or more services of the operating system emulator 320 caused by other applications 326, etc.

[0054] In some examples, the computer vision component 330 generates the simulated tracking data 348 by classifying the simulated sensor data 332 based on a tracking model generated using artificial intelligence methods and previously using machine learning methods. In some examples, the tracking model includes, but is not limited to, neural networks, learning vector quantization networks, logistic regression models, support vector machines, random decision forests, naive Bayes models, linear discriminant analysis models, K-nearest neighbor models, etc. In some examples, the machine learning methods may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, anomaly detection, etc.

[0055] In some examples, the computer vision component 330 extracts the simulated tracking data 348 from the simulated sensor data 332 using computer vision methods, which include, but are not limited to, Harris corner detection, Shi-Tomasi corner detection, scale-invariant feature transform (SIFT), speeded-up robust features (SURF), features from accelerated segment test (FAST), oriented FAST and rotated BRIEF (ORB), etc.

[0056] In operation 312, when the computer vision component 330 generates simulated tracking data 348, the operating system emulator 320 determines simulated thermal condition data 350 and simulated power consumption data 366 based on the operation of the computer vision component 330. For example, when the computer vision component 330 generates simulated tracking data 348, the operating system emulator 320 determines the use of the computing device (e.g., processor, memory storage device, graphics processing unit (GPU), etc.) and sensors by the computer vision component 330, and determines the simulated energy consumption of each of those devices. The operating system emulator 320 uses the energy consumption of the computing device and sensors, and calculates the heat load on the head-mounted XR device and the resulting thermal conditions, such as the temperature of the portion of the head-mounted XR device associated with the computing device and sensors.

[0057] In operation 314, the content creation platform 318 provides the simulated thermal condition data 350 to a user (e.g., content creator 324). For example, the content creation platform 318 uses the user interface 336 to provide the simulated thermal condition data 350 in the thermal condition display 354 and the simulated power consumption data 366 in the power consumption display 356 to the content creator 324. The content creator 324 uses the displayed simulated thermal condition data 350 and simulated power consumption data 366 to evaluate different settings or configurations of the components of the head-mounted XR device (e.g., but not limited to the computer vision component 330). The simulated power consumption data may represent the total power consumed by the XR application and / or may represent the individual power consumption generated by any one or combination of the above motions or interactions.

[0058] In some examples, the content creation platform 318 generates one or more thermal control suggestions and one or more power control suggestions based on the simulated thermal condition data 350 and the simulated power consumption data 366. These suggestions can be accessed through the thermal control suggestion 368 menu item and the power control suggestion 370 menu item of the user interface 336. For example, the content creation platform 318 may suggest timing adjustments to how quickly the simulated system should respond to changes in environmental conditions. In some examples, the content creation platform 318 may suggest changing parameters of the software components of the computer vision component 330 to affect power consumption.

[0059] In some examples, content creation platform 318 generates a user perspective user interface 376 to provide simulated thermal condition data 350 and simulated power consumption data 366 to content creator 324. For example, user perspective user interface 376 includes one or more power usage meters 374 and one or more thermal condition meters 372. User perspective user interface 376 also includes a synchronized view 378 of virtual 3D environment 360 from the perspective of a user wearing head-mounted XR device 108, as if the user is moving along trajectory 362 in virtual 3D environment 360 and viewing virtual 3D environment 360 within the field of view 364 specified in trajectory data 342. When head-mounted XR device 108 moves virtually along trajectory 362 and virtually in response to user interaction event data 344, the values displayed in one or more thermal condition meters 372 and one or more power usage meters 374 correspond to the simulated operation of head-mounted XR device 108.

[0060] Software architecture

[0061] Figure 4 is a block diagram 400 showing a software architecture 402 that can be installed on any one or more of the devices described herein. Software architecture 402 is supported by hardware such as a machine 404 including a processor 406, a memory 408, and I / O components 410. In this example, software architecture 402 can be conceptually thought of as a stack of layers, where each layer provides a specific function. Software architecture 402 includes layers such as an operating system 412, libraries 414, frameworks 416, and applications 418. In operation, application 418 activates API calls 420 through the software stack and receives messages 422 in response to API calls 420.

[0062] Operating system 412 manages hardware resources and provides common services. Operating system 412 includes, for example, a kernel 424, services 426, and drivers 428. Kernel 424 serves as an abstraction layer between the hardware and other software layers. For example, kernel 424 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functions. Services 426 can provide other common services to other software layers. Drivers 428 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 428 can include a display driver, a camera device driver, or a low power driver, a flash driver, a serial communication driver (e.g., a USB driver), drivers, an audio driver, a power management driver, etc.

[0063] The library 414 provides a common low-level infrastructure used by the application 418. The library 414 may include a system library 430 (e.g., the C standard library), which provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the library 414 may include an API library 432, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), High Efficiency Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., the OpenGL framework for presenting in two-dimensional (2D) and three-dimensional (3D) in graphical content on a display), a database library (e.g., SQLite that provides various relational database functions), a web library (e.g., WebKit that provides web browsing functions), etc. The library 414 may also include a variety of other libraries 434 to provide many other APIs to the application 418.

[0064] The framework 416 provides a common high-level infrastructure used by the application 418. For example, the framework 416 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 416 may provide a wide range of other APIs that can be used by the application 418, some of which may be specific to a particular operating system or platform.

[0065] In some examples, the application 418 includes a content creation platform 436, a data simulation platform 440, an operating system emulator 438, and various other applications such as third-party applications. The application 418 is a program that executes functions defined in the program. One or more of the applications 418 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., the C language or assembly language).

[0066] Figure 5A is a perspective view of a head-wearable XR device of an XR system according to some examples. The head-wearable XR device (e.g., Figure 5AThe glasses 500) include a frame 502 made of any suitable material (such as plastic or metal), and the any suitable material includes any suitable shape memory alloy. In one or more examples, the frame 502 includes a first optical element holder or left optical element holder 504 (such as a display or lens holder) and a second optical element holder or right optical element holder 506 connected by a bridge 512. A first optical element or left optical element 508 and a second optical element or right optical element 510 can be disposed within the corresponding left optical element holder 504 and right optical element holder 506 respectively. The right optical element 510 and the left optical element 508 can be lenses, displays, display components, or a combination of the foregoing. Any suitable display component can be provided in the glasses 500.

[0067] The frame 502 further includes a left arm piece or right temple piece 522 and a right arm piece or left temple piece 524. In some examples, the frame 502 can be formed of a single piece of material so as to have an integrated or unitary construction.

[0068] The glasses 500 can include a computing system such as a machine 520, which can be of any suitable type to be carried by the frame 502, and in one or more examples, the computing system can be of a suitable size and shape to be partially disposed within one of the right temple piece 522 or the left temple piece 524. The machine 520 can include multiple processors, a memory, and various communication components that share a common power supply. As discussed below, various components of the machine 520 can include low-power circuitry, high-speed circuitry, and a display processor. Various other examples can include these elements integrated together in different configurations or in different ways. Additional details of aspects of the machine 520 can be implemented as shown by the machine 600 discussed below.

[0069] The machine 520 further includes a battery 518 or other suitable portable power supply. In some examples, the battery 518 is disposed within the left and right temple pieces 522 and is electrically coupled to the machine 520 disposed within the right and left temple pieces 524. The glasses 500 can include a connector or port (not shown) suitable for charging the battery 518, a wireless receiver, a transmitter, or a transceiver (not shown), or a combination of such devices.

[0070] The glasses 500 include a first imaging device or a left imaging device 514 and a second imaging device or a right imaging device 516. Although two imaging devices are depicted, other examples contemplate the use of a single or additional (i.e., more than two) imaging devices. In one or more examples, in addition to the left imaging device 514 and the right imaging device 516, the glasses 500 further include any number of input sensors or other input / output devices. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, and the like.

[0071] In some examples, the left imaging device 514 and the right imaging device 516 provide video frame data for the glasses 500 to extract 3D information from a real-world scene environment scene.

[0072] The glasses 500 may further include a touchpad 526 mounted to or integrated with one or both of the left temple piece 522 and the right temple piece 524. The touchpad 526 is typically arranged vertically, and in some examples, the touchpad 526 is approximately parallel to the user's temple. As used herein, being typically vertically aligned means that the touchpad is more vertical than horizontal, although potentially more vertical than said vertical. Additional user input may be provided by one or more buttons 528, which in the illustrated example are disposed on the outer upper edges of the left optical element holder 504 and the right optical element holder 506. The one or more touchpads 526 and buttons 528 provide a means by which the glasses 500 can receive input from a user of the glasses 500.

[0073] In some examples, the glasses 500 have a projector 530 in a forward position mounted on the frame 502 of the glasses 500. The projector may be used by the adaptive 3D sensing system of the glasses 500 to project a focused beam, enabling the adaptive 3D sensing system to perform adaptive 3D sensing.

[0074] Figure 5B The glasses 500 are shown from the user's perspective. For clarity, a number of elements shown in Figure 5A are omitted. As Figure 5A described in Figure 5B the glasses 500 shown in

[0075] The glasses 500 include a forward optical assembly 532 that includes a right projector 534 and a right near-eye display 538; and a forward optical assembly 546 that includes a left projector 550 and a left near-eye display 554.

[0076] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or optical elements such as mirrors, lenses, or prisms). The light 542 emitted by the projector 534 encounters the diffractive structure of the waveguide of the near-eye display 538, which guides the light towards the user's right eye to provide an image that superimposes a view of the real-world scene environment seen by the user on or in the right optical element 548. Similarly, the light 552 emitted by the projector 550 encounters the diffractive structure of the waveguide of the near-eye display 554, which guides the light towards the user's left eye to provide an image that superimposes a view of the real-world scene environment seen by the user on or in the left optical element 544. The combination of the GPU, the forward optical assembly 532, the left optical element 544, and the right optical element 548 provides the optical engine of the glasses 500. The glasses 500 use this optical engine to generate a superimposition of the view of the user's real-world scene environment, including displaying a user interface to the user of the glasses 500.

[0077] However, it should be understood that other display technologies or configurations can be utilized within the optical engine to display images to the user within the user's field of view. For example, instead of the projector 534 and the waveguide, an LCD, an LED, or other display panel or surface can be provided.

[0078] In use, information, content, and various user interfaces can be presented to the user of the glasses 500 on the near-eye display. As described in more detail herein, the user can then interact with the glasses 500 using the touchpad 526 and / or the buttons 528, voice input or touch input on an associated device (e.g., a mobile computing system), and / or hand movements, positions, and orientations detected by the glasses 500.

[0079] In some examples, the glasses 500 include a stand-alone XR system that provides an XR experience to the user of the glasses 500. In some examples, the glasses 500 are a component of an XR system that includes one or more other devices that provide additional computing resources and / or additional user input and output resources. The other devices can include a smartphone, a general-purpose computer, etc.

[0080] Machine architecture

[0081] Figure 6is an illustrative representation of a machine 600 within which instructions 602 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 600 to perform any one or more of the methods discussed herein. For example, the instructions 602 can cause the machine 600 to perform any one or more of the methods described herein. The instructions 602 transform a general, unprogrammed machine 600 into a particular machine 600 programmed to perform the described and illustrated functions in the described manner. The machine 600 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 600 can include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular telephones, smart phones, mobile devices, wearable devices (e.g., smart watches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing the instructions 602 specifying the actions to be taken by the machine 600. Further, while a single machine 600 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 602 to perform any one or more of the methods discussed herein. In some examples, the machine 600 can also include both a client system and a server system, where certain operations of a particular method or algorithm are executed on the server side and where certain operations of a particular method or algorithm are executed on the client side.

[0082] The machine 600 can include a processor 604, a memory 606, and an input / output I / O component 608, which can be configured to communicate with each other via a bus 610. In an example, the processor 604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 612 and a processor 614 that execute the instructions 602. The term "processor" is intended to include multi-core processors that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 6A number of processors 604 are shown, but machine 600 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0083] Memory 606 includes main memory 616, static memory 640, and storage unit 618, all of which are accessible by processor 604 via bus 610. Main memory 606, static memory 640, and storage unit 618 store instructions 602 that implement any one or more of the methods or functions described herein. During execution of instructions 602 by machine 600, instructions 602 may also reside completely or partially within main memory 616, within static memory 640, within machine-readable medium 620 within storage unit 618, within at least one of processors 604 (e.g., within a cache memory of the processor), or within any suitable combination thereof.

[0084] I / O components 608 can include a variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O components 608 included in a particular machine will depend on the type of machine. For example, a portable machine (e.g., a mobile phone) may include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 608 can include Figure 6 many other components not shown. In various examples, I / O components 608 can include user output components 622 and user input components 624. User output components 622 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), tactile components (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. User input components 624 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), etc.

[0085] In additional examples, I / O component 608 may include: a biometric component 626, a motion component 628, an environmental component 630, or a positioning component 632, as well as a variety of other components. For example, biometric component 626 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. Motion component 628 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotational sensor component (e.g., a gyroscope).

[0086] For example, environmental component 630 includes one or more camera devices (with still image / photo and video capabilities), a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases for safety or for measuring pollutants in the atmosphere), a depth or distance sensor (e.g., a sensor for determining the distance to an object or the depth of an object's features in a 3D coordinate system), or other components that can provide an indication, measurement result, or signal corresponding to the surrounding physical environment.

[0087] Positioning component 632 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0088] Various techniques can be used to implement communication. I / O component 608 also includes a communication component 634, which is operable to couple machine 600 to network 636 or device 638 via a corresponding coupling or connection. For example, communication component 634 may include a network interface component for docking with network 636 or another suitable device. In additional examples, communication component 634 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power consumption), components, and other communication components that provide communication via other modalities. Device 638 can be another machine or any peripheral device among a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0089] In addition, the communication component 634 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 634 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcode and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of the tag). In addition, various information can be obtained via the communication component 634, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0090] Various memories (e.g., main memory 616, static memory 640, and memory of processor 604) and storage unit 618 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. When executed by processor 604, these instructions (e.g., instructions 602) cause various operations to implement the disclosed examples.

[0091] The instructions 602 may be transmitted or received over the network 636 via a network interface device (e.g., a network interface component included in the communication component 634) using a transmission medium and using any of a number of well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 602 may be transmitted or received to the device 638 via a coupling (e.g., a peer-to-peer coupling) using a transmission medium.

[0092] Software Architecture

[0093] Figure 7 700 is a software architecture 702 that illustrates a software architecture 702 that may be installed on any one or more of the devices described herein. The software architecture 702 is supported by hardware such as a machine 704 that includes a processor 706, a memory 708, and an I / O component 710. In this example, the software architecture 702 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 702 includes layers such as an operating system 712, a library 714, a framework 716, and an application 718. In operation, the application 718 invokes an API call 720 through the software stack and receives a message 722 in response to the API call 720.

[0094] The operating system 712 manages hardware resources and provides common services. The operating system 712 includes, for example, a kernel 724, services 726, and drivers 728. The kernel 724 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 724 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The services 726 can provide other common services for other software layers. The drivers 728 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 728 can include a display driver, a camera device driver, or a low-power driver, a flash driver, a serial communication driver (e.g., a USB driver), a driver, an audio driver, a power management driver, etc.

[0095] The library 714 provides common low-level infrastructure used by the applications 718. The library 714 can include a system library 730 (e.g., a C standard library), which provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the library 714 can include an API library 732, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, the various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., the OpenGL framework for presenting graphical content in two-dimensional (2D) and three-dimensional (3D) on a display), a database library (e.g., SQLite that provides various relational database functions), a web library (e.g., WebKit that provides web browsing functions), etc. The library 714 can also include a variety of other libraries 734 to provide many other APIs to the applications 718.

[0096] The framework 716 provides common high-level infrastructure used by the applications 718. For example, the framework 716 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 716 can provide a wide range of other APIs that can be used by the applications 718, some of which can be specific to a particular operating system or platform.

[0097] In an example, the applications 718 may include a home application 736, a contacts application 738, a browser application 740, a book reader application 742, a location application 744, a media application 746, a messaging application 748, a gaming application 750, and various other applications such as third-party applications 752. The applications 718 are programs that execute functions defined in a program. One or more of the applications 718 can be created using various programming languages that are structured in various ways, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C language or assembly language). In a particular example, the third-party application 752 (e.g., an application developed using an ANDROID TM or IOS TM software development kit (SDK) by an entity other than the vendor of a particular platform) can be mobile software that runs on a mobile operating system such as IOS TM , ANDROID TM , Phone or another mobile operating system. In this example, the third-party application 752 can activate API calls 720 provided by the operating system 712 to facilitate the functions described herein.

[0098] Conclusion

[0099] Changes and modifications can be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

[0100] Glossary

[0101] "Carrier signal" means any non-tangible medium that can store, encode, or carry instructions executable by a machine and includes digital or analog communication signals or other non-tangible media to facilitate the communication of such instructions. Instructions can be transmitted or received via a network using a transmission medium through a network interface device.

[0102] "Communication network" means one or more portions of a network, which can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, a network or a part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling may implement any of a variety of data transmission technologies, such as Single-Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data Rate for GSM Evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, Fourth Generation Wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0103] "Component" refers to a device, physical entity, or logic having the following boundaries: the boundaries are defined by functions or subroutine calls, branch points, APIs, or other techniques provided for partitioning or modularizing a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed to be used with other components and can be part of a program that typically performs a particular function among related functions. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in a physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or a part of an application) to operate to perform certain operations as described herein as a hardware component. A hardware component can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) that is uniquely customized to perform the configured function and is no longer a general-purpose processor. It should be understood that a decision can be made based on cost and time considerations as to whether to implement a hardware component mechanically in dedicated and permanently configured circuitry or in circuitry that is temporarily configured (e.g., configured by software). Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hard-wired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component at any given time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a dedicated processor, the general-purpose processor can be configured separately as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more particular processors to, for example, constitute a particular hardware component at one moment and different hardware components at different moments. Hardware components can provide information to other hardware components and receive information from other hardware components.Accordingly, the described hardware components can be considered to be communicatively coupled. In cases where multiple hardware components are present simultaneously, communication can be achieved through signal transmission between or among two or more of the hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving the information from the memory structure. For example, one hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware component can access the memory device at a subsequent time to retrieve the stored output and process it. Hardware components can also initiate communication with input devices or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors temporarily configured (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily configured or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least in part processor-implemented, where a particular one or more processors are examples of hardware. For example, at least some of the operations of the method can be performed by one or more processors or processor-implemented components. Additionally, one or more processors can also operate to support the execution of relevant operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as an example of machines including processors), where the operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of certain operations can be distributed among processors, not residing solely within a single machine but deployed across multiple machines. In some examples, the processor or processor-implemented components can be located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processor or processor-implemented components can be distributed across multiple geographical locations.

[0104] "Machine-readable storage medium" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "computer-readable medium", "machine-readable medium", and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0105] "Non-transitory machine-readable storage medium" means a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Accordingly, the term should be regarded as including, but not limited to, solid-state memory as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium".

[0106] "Signal medium" means any intangible medium that can store, encode, or carry instructions executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be regarded as including any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose one or more characteristics are set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0107] Changes and modifications may be made to the disclosed examples without departing from the scope of the disclosure. These and other changes or modifications are intended to be included within the scope of the disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method, comprising: Receiving, by a content creation platform, motion data of the motion of an XR device; Generating, by the content creation platform, trajectory data of a trajectory within a 3D environment model of a real-world scene based on the motion data, the trajectory simulating the motion of the XR device within the real-world scene; Receiving, by the content creation platform, user interaction event data; Generating, by a data simulation platform, simulated sensor data based on the trajectory data, the 3D environment model, and the user interaction event data; Generating, by a computer vision component within an operating system emulator, simulated tracking data based on the simulated sensor data; When generating the simulated tracking data, determining, by the operating system emulator, simulated power consumption data and simulated thermal condition data based on the operation of the computer vision component; And Providing, by the content creation platform, the simulated power consumption data and the simulated thermal condition data to a user.

2. The computer-implemented method according to claim 1, wherein, Receiving the motion data of the XR device includes: Capturing data of the motion of the XR device when a user moves the XR device through a real-world scene.

3. The computer-implemented method according to claim 1, wherein, Receiving the motion data of the XR device includes: Selecting the motion data from a list of stored data.

4. The computer-implemented method according to claim 1, wherein, The user interaction event is associated with a time event.

5. The computer-implemented method according to claim 1, wherein, The user interaction event is associated with a location within the real-world scene.

6. The computer-implemented method according to claim 1, wherein, The user interaction event is a gesture made by a user.

7. The computer-implemented method according to claim 1, wherein, Generating the trajectory data of the trajectory within the 3D environment model further includes: Changing illumination parameters of the 3D environment model.

8. The computer-implemented method according to claim 1, wherein, The simulated sensor data includes camera device data, inertial motion unit data, and global positioning sensor data.

9. The computer-implemented method according to claim 1, wherein, The simulated tracking data includes device pose data and hand recognition result data.

10. The computer-implemented method according to claim 1, wherein, Generating, by the computer vision component within the operating system emulator, the simulated tracking data based on the simulated sensor data further includes: Generating, by the operating system emulator, simulated operating system events that affect the operation of the computer vision component based on changes in the simulated environment over time.

11. The computer-implemented method according to claim 10, wherein, The simulated operating system events include thermal throttling events, network condition change events, environmental noise events, and display adjustment events.

12. A machine, comprising: One or more processors; And A memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations, the operations including: Receiving, by a content creation platform, motion data of the motion of an XR device; Generating, by the content creation platform, trajectory data of a trajectory within a 3D environment model of a real-world scene based on the motion data, the trajectory simulating the motion of the XR device within the real-world scene; Receiving, by the content creation platform, user interaction event data; Generating, by a data simulation platform, simulated sensor data based on the trajectory data, the 3D environment model, and the user interaction event data; Generating, by a computer vision component within an operating system emulator, simulated tracking data based on the simulated sensor data; When generating the simulated tracking data, determining, by the operating system emulator, simulated power consumption data and simulated thermal condition data based on the operation of the computer vision component; and The content creation platform provides the user with the simulated power consumption data and the simulated thermal condition data.

13. The machine according to claim 12, wherein, Receiving data on the movement of the XR device includes: Capturing data on the movement of the XR device when the user moves the XR device through the real-world scenario.

14. The machine according to claim 12, wherein, Receiving data on the movement of the XR device includes: Selecting the movement data from a list of stored data.

15. The machine according to claim 12, wherein The user interaction event is associated with a time event.

16. The machine according to claim 12, wherein The user interaction event is associated with a location within the real-world scenario.

17. The machine according to claim 12, wherein The user interaction event is a gesture made by the user.

18. The machine according to claim 12, wherein, Generating the trajectory data for the trajectory within the 3D environment model further includes: Changing the lighting parameters of the 3D environment model.

19. The machine according to claim 12, wherein, The simulated sensor data includes camera device data, inertial motion unit data, and global positioning sensor data.

20. A non-transitory machine-readable storage medium, the machine-readable storage medium includes instructions that, when executed by a computer, cause the computer to perform operations, the operations including: Receiving, by a content creation platform, movement data on the movement of an XR device; Generating, by the content creation platform, trajectory data for a trajectory within a 3D environment model of a real-world scenario, the trajectory simulating the movement of the XR device within the real-world scenario, based on the movement data; Receiving, by the content creation platform, user interaction event data; Generating, by a data simulation platform, simulated sensor data based on the trajectory data, the 3D environment model, and the user interaction event data; Generating, by a computer vision component within an operating system emulator, simulated tracking data based on the simulated sensor data; When generating the simulated tracking data, determining, by the operating system emulator, simulated power consumption data and simulated thermal condition data based on the operation of the computer vision component; And Providing, by the content creation platform, the simulated power consumption data and the simulated thermal condition data to the user.

Images