Artificial reality scene synthesizer

Abstract

Aspects of the disclosure relate to synthesizing an editable artificial reality scene. Embodiments include a synthesizer tool for generating an editable artificial reality scene. For example, a scene creator may generate an artificial reality scene and publish the scene such that other users may access and / or edit the scene. The artificial reality scene may include scene components such as an avatar, a virtual object, motion dynamics, a background, a camera viewing angle, and the like. Implementations of a permission manager may manage access and / or edit permissions of stored artificial reality scenes. For example, a scene creator may define a range of users where an original scene may be accessed and / or edited, and / or define editable scene components. In some embodiments, multiple users may edit an original artificial reality scene multiple times, thereby generating multiple edited versions of the original scene.

Description

Technical Field

[0001] The present disclosure relates to synthesizing editable artificial reality scenarios. Background Art

[0002] Artificial reality systems are becoming increasingly popular among users, and this growth is expected to accelerate. Artificial reality also provides new technologies for social interactions. For example, modern Internet users share and consume a large amount of social media content, such as images and videos. However, traditional systems lack intuitive and practical mechanisms to create, edit, and / or share content that utilizes artificial reality environments. A more intuitive and realistic experience of artificial reality can enhance the user experience of such content. Summary of the Invention

[0003] According to a first aspect of the present disclosure, there is provided a method for editing a stored three-dimensional (3D) artificial reality (XR) scene, the method comprising: accessing a stored 3D XR scene having scene components, the scene components including at least an avatar, avatar movement dynamics, and an XR environment, wherein executing the 3D XR scene generates a three-dimensional depiction of the avatar moving according to the avatar movement dynamics defined in the XR environment; receiving one or more edits to the scene components of the 3D XR scene, wherein the one or more edits include at least replacing the avatar with a replacement avatar; and storing the edited 3D XR scene, wherein executing the edited 3D XR scene generates a three-dimensional depiction of the replacement avatar moving within the XR environment according to the avatar movement dynamics.

[0004] In some embodiments, the scene components of the 3D XR scene further include one or more virtual objects and one or more virtual object movement dynamic parameters, and the three-dimensional depiction generated by executing the 3D XR scene includes the one or more virtual objects moving within the XR environment according to the one or more virtual object movement dynamic parameters.

[0005] In some embodiments, editing the scene components of the 3D XR scene further includes replacing one or more virtual objects with one or more replacement virtual objects, and the three-dimensional depiction generated by executing the edited 3D XR scene includes the one or more replacement virtual objects moving within the XR environment according to the one or more virtual object movement dynamic parameters.

[0006] In some embodiments, editing the scene components of the 3D XR scene further includes editing the avatar movement dynamics, and the three-dimensional depiction generated by executing the edited 3D XR scene includes the replacement avatar moving within the XR environment according to the edited avatar movement dynamics.

[0007] In some embodiments, editing avatar motion dynamics further includes: capturing user motion via one or more sensors, where at least one of the one or more sensors includes a camera; and converting the user motion into avatar motion dynamics.

[0008] In some embodiments, editing the scene components of a 3D XR scene further includes editing avatar motion dynamics, where editing avatar motion dynamics includes: capturing user motion via one or more sensors, where at least one of the one or more sensors includes a camera; and converting the user motion into avatar motion dynamics.

[0009] In some embodiments, editing the scene components of a 3D XR scene further includes adding one or more additional scene components, and the additional scene components include one or more of additional avatars, additional virtual objects, or any combination thereof.

[0010] In some embodiments, the stored 3D XR scene is created by a creator user who defines editing permissions for the stored 3D XR scene, and accessing the stored 3D XR scene further includes: receiving a request from another user different from the creator user to edit the stored 3D XR scene; and verifying that the other user has editing permissions for the stored 3D XR scene, where the stored 3D XR scene is accessed in response to the verification.

[0011] In some embodiments, the editing permissions include a user scope related to a social graph, and when the other user is included in the user scope related to the social graph, the editing permissions of the other user are verified.

[0012] In some embodiments, the stored 3D XR scene is created by a creator user who defines at least one editing parameter for at least one scene component of the stored 3D XR scene, and where a first value of the editing parameter allows editing, replacing, or deleting the at least one scene component, and a second value of the editing parameter restricts editing, replacing, or deleting the at least one scene component.

[0013] In some embodiments, the at least one scene component includes an avatar, and receiving one or more edits to the scene components of the 3D XR scene further includes: verifying that the editing parameter of the avatar allows the avatar to be replaced with a replacement avatar.

[0014] In some embodiments, the method further includes publishing the edited 3D XR scene on a social media platform, where the publishing allows multiple social media platform users to execute the edited 3D XR scene.

[0015] In some embodiments, the avatar motion dynamics at least include an avatar pose and an avatar expression.

[0016] According to a second aspect of the present disclosure, there is provided a computing system for editing a stored artificial reality (XR) scene. The computing system includes: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process that includes: storing an XR scene having scene components that at least include an avatar, avatar motion dynamics, and an XR environment, where the XR scene is created by a creator user who defines editing permissions, and executing the XR scene generates a three-dimensional depiction of the avatar moving within the XR environment according to the avatar motion dynamics; receiving a request from a different other user of the creator user to edit the stored XR scene; verifying that the other user includes editing permissions for the stored XR scene; in response to the verification, accessing the stored XR scene; receiving one or more edits to the scene components of the XR scene, where the one or more edits at least include replacing the avatar with a replacement avatar; and storing the edited XR scene, where executing the edited XR scene generates a three-dimensional depiction of the replacement avatar moving within the XR environment according to the avatar motion dynamics.

[0017] In some embodiments, the scene components of the XR scene further include one or more virtual objects and one or more virtual object motion dynamic parameters, and the three-dimensional depiction generated by executing the XR scene includes the one or more virtual objects moving within the XR environment according to the one or more virtual object motion dynamic parameters.

[0018] In some embodiments, editing the scene components of the XR scene further includes replacing the one or more virtual objects with one or more replacement virtual objects, and the three-dimensional depiction generated by executing the edited XR scene includes the one or more replacement virtual objects moving within the XR environment according to the one or more virtual object motion dynamic parameters.

[0019] In some embodiments, editing the scene components of the XR scene further includes editing the avatar motion dynamics, and the three-dimensional depiction generated by executing the edited XR scene includes the replacement avatar moving within the XR environment according to the edited avatar motion dynamics.

[0020] In some embodiments, editing the scene components of the XR scene further includes editing the avatar motion dynamics, where editing the avatar motion dynamics further includes: capturing user motion via one or more sensors, where at least one of the one or more sensors includes a camera; and converting the user motion into avatar motion dynamics.

[0021] In some embodiments, the editing privileges include a user scope related to the social graph, and when other users are included in the user scope related to the social graph, the editing privileges of such other users are verified.

[0022] In some embodiments, the creator user defines at least one editing parameter for at least one scene component of the stored XR scene, and wherein a first value of the editing parameter allows editing, replacing, or deleting the at least one scene component, and a second value of the editing parameter restricts editing, replacing, or deleting the at least one scene component.

[0023] According to a third aspect of the present disclosure, there is provided a computer-readable storage medium storing instructions which, when executed by a computing system, cause the computing system to perform a process for editing a stored artificial reality (XR) scene, the process comprising: storing an XR scene having scene components, the scene components including at least an avatar, avatar movement dynamics, and an XR environment, wherein the XR scene is created by a creator user who defines editing privileges, and executing the XR scene generates a three-dimensional depiction of the avatar moving within the XR environment according to the avatar movement dynamics; receiving a request from another user different from the creator user to edit the stored XR scene; verifying that the other user includes editing privileges for editing the stored XR scene; accessing the stored XR scene in response to the verification; receiving one or more edits to the scene components of the XR scene, wherein the one or more edits include at least replacing the avatar with a replacement avatar; and storing the edited XR scene, wherein executing the edited XR scene generates a three-dimensional depiction of the replacement avatar moving within the XR environment according to the avatar movement dynamics.

[0024] According to a fourth aspect of the present disclosure, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the method of the first aspect of the present disclosure.

[0025] It will be appreciated that any features described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure are intended to be generalizable across any and all aspects and embodiments of the present disclosure. Those skilled in the art can understand other aspects of the present disclosure based on the specification, claims, and drawings of the present disclosure. The above general description and the following detailed description are merely exemplary and illustrative, and are not limiting of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram showing an overview of multiple devices on which some embodiments of the present technology may operate.

[0027] Figure 2Ais a line diagram showing a virtual reality headset that can be used in some embodiments of the present technology.

[0028] Figure 2B is a line diagram showing a mixed reality headset that can be used in some embodiments of the present technology.

[0029] Figure 2C is a line diagram showing a plurality of controllers, which in some embodiments, a user can hold with one or both hands to interact with an artificial reality environment.

[0030] Figure 3 is a block diagram showing an overview of an environment in which some embodiments of the present technology can operate.

[0031] Figure 4 is a block diagram showing a plurality of components that can be used in a system employing the disclosed technology in some embodiments.

[0032] Figure 5 is an example artificial reality environment with multiple user avatars.

[0033] Figure 6A is an example artificial reality scene with a user avatar and virtual objects.

[0034] Figure 6B is an example tool for synthesizing an editable artificial reality environment scene.

[0035] Figure 7 is a system diagram for synthesizing and editing an artificial reality scene.

[0036] Figure 8 is a flowchart showing a process for synthesizing an editable artificial reality scene in some embodiments of the present technology.

[0037] Figure 9 is a flowchart showing a process for editing a stored artificial reality scene in some embodiments of the present technology.

[0038] The various technologies introduced herein can be better understood by reference to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals refer to the same or functionally similar elements. Detailed Description

[0039] Aspects of the present disclosure relate to synthesizing editable artificial reality scenes. An artificial reality scene can include components such as one or more avatars, one or more virtual objects, backgrounds, audio, lighting, camera / viewpoint, the motion dynamics of the one or more avatars and / or the one or more virtual objects, and any other suitable scene components. An artificial reality scene can include animation (such as movement, expressions, dance, and other suitable animations) of the one or more avatars and / or the one or more virtual objects from a camera / viewpoint within a defined background. In some examples, the artificial reality scene can also include a textual representation of audio or language played during the animation.

[0040] An artificial reality scene can be stored as a data structure, a data blob, or in any other suitable manner. Executing the stored artificial reality scene can generate a two-dimensional or three-dimensional display of avatar / virtual object animation from a camera / viewpoint within a background, which can include an audio output. For example, an artificial reality system can provide a three-dimensional display (and output audio) to a user. In other examples, other suitable client devices (such as a laptop computer, a smartphone, a device with a display, etc.) can provide a two-dimensional display, a three-dimensional display, and / or audio to one or more users.

[0041] An example artificial reality scene is a user's avatar playing a virtual instrument in a three-dimensional artificial reality environment that includes multiple virtual objects. The scene components can include: the user avatar (such as pose, movement, expression, etc.), the virtual instrument (such as state, animation), one or more camera positions and lighting, audio, the three-dimensional environment (such as a room, background, outdoor environment, etc.), and multiple virtual objects. Example avatars of an artificial reality scene include three-dimensional avatars, two-dimensional avatars, or any other suitable virtual user representation. Example virtual objects include three-dimensional object models, two-dimensional images or panels, alphanumeric text, video, or other suitable virtual objects. Example backgrounds include two-dimensional images, three-dimensional models, backgrounds with animation, and any other suitable backgrounds.

[0042] Various embodiments include a composer tool for creating and editing artificial reality scenes. For example, a library of scene components (such as avatars, virtual objects, backgrounds, three-dimensional environments, audio, motion dynamics, etc.) can be predefined to support user customization of artificial reality scenes. In some embodiments, an artificial reality environment including one or more virtual objects, one or more avatars, a background, and / or audio can be recorded, and artificial reality scene components can be identified from the recorded artificial reality environment.

[0043] Each embodiment may store artificial reality scenes generated and / or edited using a synthesizer tool. For example, components of an artificial reality scene may be serialized to generate one or more data structures that support storage, sharing, and execution of virtual reality scenes. In some examples, the stored artificial reality scenes may be published to, for example, a social application or any other suitable digital location that allows multiple users to access the published scenes. Users may discover, share, access, and execute artificial reality scenes via publication.

[0044] In some examples, the user who generates an original artificial reality scene may be a creator user who sets access and / or edit permissions for the original scene. For example, the creator user may share the artificial reality scene (e.g., by publishing to a social application) and / or allow one or more other users to edit the scene. Example access restrictions and / or edit restrictions include: access permissions that define which users may view / execute the stored artificial reality scene; edit permissions that define which users may edit the stored artificial reality scene; a limit on the number of iterative edits, which defines how many consecutive edits may be performed on the stored artificial reality scene; a limit on the total number of different edited versions of the stored artificial reality scene; restrictions on individual components of the stored artificial reality scene, which define whether those individual components are editable; or any combination thereof.

[0045] In some embodiments, once the creator user shares an artificial reality scene, an editing user with edit permissions may load and edit the scene (e.g., via a synthesizer tool). For example, when permitted by the creator user, the editing user may use the synthesizer tool to perform one or more of the following edits: A) replace the creator user's avatar with the editing user's avatar; B) add the editing user's avatar to the composition adjacent to the creator user's avatar; C) replace or delete any of the plurality of virtual objects; D) change, replace, or edit the three-dimensional environment, such as room visuals, background, lighting, etc.; E) change, replace, or edit the audio; F) change or edit the camera position; or G) any combination thereof.

[0046] In some embodiments, the scene creator may share portions of the scene, such as lighting conditions, entities used, or camera positions. Other users may then obtain these scene portions (e.g., from an online scene library, a social media source, etc.) and incorporate them into their own scenes.

[0047] In some embodiments, the editing user can then store and publish the edited scene. For example, the edited scene can be published to a social application, a digital location accessible by multiple users, etc. Then, based on the publication, other users can discover, share, access, and execute the edited artificial reality scene. In some embodiments, another user can edit the edited artificial reality scene itself to generate another edited version of the artificial reality scene. Correspondingly, the original artificial reality scene can undergo multiple iterative edits by multiple editing users.

[0048] Each embodiment allows for social interaction through editable and publishable artificial reality scenes. The creator user can control the level of social interaction of the original artificial reality scene through access and / or editing permissions. When users generate multiple edited versions of the original artificial reality scene through multiple edits / iterative edits, the multiple versions can share some common threads (e.g., background, virtual objects, avatars, motion dynamics, audio, etc.). Given this common thread, users who edit, publish, and share the multiple versions can achieve a certain level of connectivity through the similarities between their versions of the artificial reality scene and convey individuality through the differences between their versions of the artificial reality scene.

[0049] Each embodiment of the disclosed technology may include or be implemented in conjunction with an artificial reality system. Artificial reality or extended reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., a photograph of the real world). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that provides a three-dimensional effect to a viewer). Additionally, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, such as for creating content in artificial reality and / or using in artificial reality (e.g., performing an activity in artificial reality). An artificial reality system that provides artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0050] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet computer having a camera on the back can capture real-world images and then display the images on a screen on the side of the tablet computer opposite the camera. The tablet computer can process and adjust or "augment" the images as they pass through the system (e.g., by adding virtual objects). "Mixed reality" or "MR" refers to a system in which the light entering a user's eyes is partially generated by a computing system and partially composed of light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped like a pair of glasses having a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects that are mixed with real objects visible to the user. As used herein, "artificial reality", "hyperreality", or "XR" refers to any one of the following: VR, AR, MR, or any combination or mixture thereof.

[0051] Although some conventional software allows customization of shareable media such as images and / or videos, the conventional software cannot incorporate artificial reality elements into such customization. In addition, the conventional software also cannot provide access and edit control to an owner or originating user. Thus, a user's shareable media can typically be manipulated without a user-defined control mechanism.

[0052] Each embodiment includes a synthesizer tool for generating an editable artificial reality scene. For example, an originating user or creator user can generate an original artificial reality scene and publish the scene such that other users can access and / or edit the scene. The artificial reality scene can include three-dimensional components such as avatars, virtual objects, immersive backgrounds, camera perspectives, etc. Thus, the synthesizer tool supports editable scenes with artificial reality content.

[0053] Implementations of a permissions manager can manage access and / or edit permissions for an artificial reality scene. For example, a creator user of an artificial reality scene can define a scope of users who can access and / or edit the user's original scene, and / or define scene components that can be edited. Thus, each implementation allows an owner / originating user to control how the original artificial reality scene is accessed and edited. As used herein, a "creator user" can be a user who initially created the scene or scene component, or a user who has obtained ownership of the scene or scene component from the original creator.

[0054] Several implementations are discussed in more detail below with reference to the accompanying drawings. Figure 1 is a block diagram showing an overview of multiple devices on which some implementations of the disclosed technology can run. These devices can include the following hardware components of a computing system 100: These hardware components synthesize an editable artificial reality scene. In various implementations, the computing system 100 can include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103), which communicate via a wired channel or a wireless channel to distribute processing and share input data. In some implementations, the computing system 100 can include a standalone head-mounted viewer that can provide a computer-generated or enhanced experience to a user without the need for external processing or external sensors. In other implementations, the computing system 100 can include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), where some processing operations are performed on the head-mounted viewer and other processing operations are transferred to the core processing component. The following is described in conjunction with Figure 2A and Figure 2B an example head-mounted viewer. In some implementations, position data and environmental data can be collected only by sensors incorporated in the head-mounted viewer device, while in other implementations, one or more of the multiple non-head-mounted viewer computing devices can include sensor components that can track environmental data or position data.

[0055] The computing system 100 can include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 can be a single processing unit or multiple processing units, which are located in one device or distributed across multiple devices (e.g., distributed across two or more of the computing devices 101 to 103).

[0056] The computing system 100 may include one or more input devices 120 that provide input to the processor 110 to notify the processors of actions. These actions may be relayed by a hardware controller that interprets signals received from the input devices and conveys information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, keyboard, touch screen, touchpad, wearable input device (e.g., haptic gloves, bracelets, rings, earrings, necklaces, watches, etc.), camera (or other light-based input device, such as an infrared sensor), microphone, or other user input device.

[0057] The processor 110 may be coupled to other hardware devices, for example, by using an internal or external bus, or a wireless connection, such as a Peripheral Component Interconnect Standard (PCI) bus, Small Computer System Interface (SCSI) bus. The processor 110 may communicate with the hardware controllers for various devices (e.g., the display 130). The display 130 may be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or equipped with an eye movement direction monitoring system, as part of the display. In some embodiments, the display is separate from the input device. Examples of display devices include liquid crystal display (LCD) screens, light emitting diode (LED) screens, projection displays, holographic displays, or augmented reality displays (e.g., head-up display devices or head-mounted devices), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as network chips or network cards, video chips or video cards, audio chips or audio cards, Universal Serial Bus (USB), FireWire, or other external devices, cameras, printers, speakers, compact disc read-only memory (CD-ROM) drives, digital versatile disc (DVD) drives, disk drives, etc.

[0058] In some embodiments, inputs from I / O device 140 (such as a camera, depth sensor, inertial motion unit (IMU) sensor, GPS unit, lidar (LiDAR), or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map a user's physical environment while tracking the user's position within that environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topological, grid, etc.) of an area (which can be a room, building, outdoor space, etc.), and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track the user within the area based on factors such as GPS data, match the identified objects and structures with the mapped objects and structures, monitor acceleration and other position changes, etc.

[0059] Computing system 100 can include a communication device that is capable of communicating wirelessly or based on a wired connection with other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using the Transmission Control Protocol / Internet Protocol (TCP / IP protocol). Computing system 100 can use the communication device to distribute operations across multiple network devices.

[0060] The processor 110 can access the memory 150, which can be included on one of the multiple computing devices of the computing system 100, or can be distributed across multiple computing devices of the computing system 100, or multiple other external devices. The memory includes one or more hardware devices for volatile or non-volatile storage and can include both read-only memory and writable memory. For example, the memory can include one or more of the following: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory such as flash memory, hard disk drives, floppy disks, optical discs (CDs), DVDs, magnetic storage devices, and tape drives. The memory is not a propagated signal detached from the underlying hardware; thus, the memory is non-transitory. The memory 150 can include a program memory 160 that stores programs and software such as an operating system 162, a scene synthesizer 164, and other application programs 166. The memory 150 can also include a data memory 170 that can include, for example, avatar data (such as avatar structures, poses, states, etc.), virtual object data (such as virtual object structures, movements, etc.), images, audio, and / or video files, 3D data models, configuration data, settings, user options or preferences, etc., that can be provided to the program memory 160 or any element of the computing system 100.

[0061] Some embodiments can operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with this technology include, but are not limited to: XR head-mounted viewers, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments including any of the above multiple systems or devices.

[0062] Figure 2AFIG. 0 is a line diagram of a virtual reality head-mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a strap 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. The position sensors 220, the IMU 215, and the computing units 230 may be located inside the HMD 200 and may be invisible to the user. In various embodiments, the IMU 215, the position sensors 220, and the locator 225 may track the movement and position of the HMD 200 in the real world and in an artificial reality environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locator 225 may emit infrared light beams that create light spots on real objects around the HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect the light spots. The computing unit 230 in the HMD 200 may use the detected light spots to infer the position and movement of the HMD 200, as well as to identify the shape and position of real objects around the HMD 200.

[0063] The electronic display 245 may be integrated with the front rigid body 205 and may provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 may be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., a micro LED, a LASER, etc.), some other display, or some combination thereof.

[0064] In some embodiments, the HMD 200 may be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC may use the HMD in combination with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0065] Figure 2B is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 may communicate via a wireless connection (e.g., a 60 gigahertz (GHz) link) as indicated by the link 256. In other embodiments, the mixed reality system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 may house various electronic components (not shown), such as optical projectors (e.g., lasers, LEDs, etc.), cameras, eye-tracking sensors, microelectromechanical system (MEMS) components, network components, etc.

[0066] The projector may be coupled to the see-through display 258 via, for example, optical elements to display media to the user. These optical elements may include one or more waveguide assemblies, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for guiding light from the projector to the user's eyes. Image data may be transmitted from the core processing component 254 to the HMD 252 via the link 256. A controller in the HMD 252 may convert the image data into a plurality of light pulses from the projector, and these light pulses may be transmitted to the user's eyes as output light via the optical elements. The output light may be mixed with the light passing through the display 258, thereby allowing the output light to present the following virtual objects: These virtual objects appear as if they exist in the real world.

[0067] Similar to the HMD 200, the HMD system 250 may also include a motion and position tracking unit, a camera, a light source, etc., which allow the HMD system 250 to track itself, for example, in 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), render virtual objects to appear stationary as the HMD 252 moves, and make virtual objects respond to gestures and other real-world objects.

[0068] Figure 2C Shows a plurality of controllers 270 (including controllers 276A and 276B). In some embodiments, a user may hold the controllers with one or both hands to interact with an artificial reality environment presented by HMD 200 and / or HMD 250. The controllers 270 may communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controllers may have their own IMU units, position sensors, and / or may emit light points that travel farther. Sensors in HMD 200 or HMD 250, external sensors, or the controllers may track these controller light points to determine the position and / or orientation of the controllers (e.g., tracking the controllers in 3DoF or 6DoF). The computing unit 230 in HMD 200 or the core processing component 254 may use this tracking to monitor the position and movement of the user's hand by combining the IMU output and the position output. The controllers may also include various buttons (e.g., buttons 272A to 272F) and / or joysticks (e.g., joysticks 274A to 274B) that the user may actuate to provide input and interact with objects.

[0069] In various embodiments, HMD 200 or HMD 250 may also include additional subsystems (e.g., an eye tracking unit, an audio system, various network components, etc.) that are used to monitor indications of user interaction and intent. For example, in some embodiments, instead of or in addition to the controllers, one or more cameras included in HMD 200 or HMD 250 or from a plurality of external cameras may monitor the position and posture of the user's hand to determine the posture and other hand and body movements. As another example, one or more light sources may illuminate one or both of the user's eyes, and HMD 200 or HMD 250 may use an eye-facing camera to capture the reflection of the light to determine the eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes, and determine the gaze direction.

[0070] Figure 3 Is a block diagram showing an overview of an environment 300 in which some embodiments of the disclosed technology may operate. The environment 300 may include one or more client computing devices 305A to 305D, examples of which may include the computing system 100. In some embodiments, some of the plurality of client computing devices (e.g., client computing device 305B) may be HMD 200 or HMD system 250. The plurality of client computing devices 305 may operate in a network environment using a logical connection over a network 330 to one or more remote computers (e.g., server computing devices).

[0071] In some embodiments, server 310 can be an edge server that receives multiple client requests and coordinates the implementation of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 can include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, these server computing devices can each be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0072] Client computing device 305, and server computing devices 310 and 320 can each act as a server or a client to one or more other server / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to a corresponding database 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can each be a distributed computing environment that includes multiple computing devices, which can be located within their corresponding servers, or at the same physical location or at geographically different physical locations.

[0073] Network 330 can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or some other wired or wireless network. Network 330 can be the Internet or some other public or private network. Client computing device 305 can be connected to network 330 via a network interface, such as through wired communication or wireless communication. Although the connection between server 310 and multiple servers 320 is shown as a separate connection, these connections can be any type of local area network, wide area network, wired network, or wireless network that includes network 330 or a separate public or private network.

[0074] Figure 4FIG. 0 is a block diagram showing various components 400 that may be used in a system employing the disclosed technology in some embodiments. Each component 400 may be included in one device of the computing system 100 or may be distributed across multiple devices of the computing system 100. The components 400 include hardware 410, middleware 420, and a plurality of specialized components 430. As discussed above, systems implementing the disclosed technology may use a variety of hardware, including a processing unit 412, working memory 414, input and output devices 416 (e.g., cameras, displays, IMU units, network connections, etc.), and storage memory 418. In various embodiments, the storage memory 418 may be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 may be one or more hard disk drives or flash drives accessible via a system bus, or may be a cloud storage provider (e.g., in storage devices 315 or 325) or other network storage device accessible via one or more communication networks. In various embodiments, the component 400 may be implemented in a client computing device (e.g., client computing device 305) or on a server computing device (e.g., server computing devices 310 or 320).

[0075] The middleware 420 may include components that transfer resources between the hardware 410 and the plurality of specialized components 430. For example, the middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuitry, or other hardware or software systems.

[0076] The specialized components 430 may include software or hardware configured to perform operations for synthesizing editable artificial reality scenes. The specialized components 430 may include: a synthesizer tool 430; one or more stored scenes 436; a permissions manager 438; a publisher 440; and components and APIs (such as interface 432) that may be used to provide a user interface, transfer data, and control the specialized components. In some embodiments, each component 400 may be in a computing system distributed across multiple computing devices or may be an interface to a server-based application that executes one or more of the specialized components 430. Although the specialized components 430 are depicted as separate components, these specialized components may be functional logical distinctions or other non-physical distinctions and / or may be sub-modules or code blocks of one or more applications.

[0077] The synthesizer tool 434 is a tool for synthesizing XR scenes. For example, the synthesizer tool 434 can provide the following interface: at this interface, the user interacts with XR scene components to define XR scenes, and these XR scene components are, for example, one or more avatars, one or more virtual objects, a background, audio, one or more camera positions, and lighting, etc. The user can use the synthesizer tool 434 to define the scene components of the XR scene and store the generated XR scene as one or more stored scenes 436. In some embodiments, the synthesizer tool 434 can load one or more of the one or more stored scenes 436 for the user to edit, and the edited scene can be stored as one of the one or more stored scenes 436. The following regarding Figure 6B 、 Figure 7 、 Figure 8 in boxes 802, 804, 806, 808, and 810, and Figure 9 in boxes 908, 910, 912, 914, and 916 in provide additional details about the synthesizer tool 434.

[0078] One or more stored scenes 436 can include a data structure (e.g., a serialized data structure) that stores definitions of the scene components that make up the XR scene. These definitions can include avatar definitions (e.g., structure, pose, etc.), virtual object definitions (e.g., structure, display state, etc.), movement definitions (e.g., avatar movement, virtual object movement, etc.), background definitions (e.g., 3D models, images, etc.), audio and / or video files, and other suitable scene component definitions. The scene synthesizer 436 can generate one or more stored scenes 436 and load one or more stored scenes 436 for editing. The following regarding Figure 6B 、 Figure 7 、 Figure 8 in box 808, and Figure 9 in boxes 902, 908, and 914 in provide additional details about one or more stored scenes 436.

[0079] The permission manager 438 can manage access, editing, or other appropriate permissions for one or more stored scenarios 436. For example, a user (such as the owner or initiator of one or more stored scenarios 436) can define permissions for accessing and / or editing an XR scenario. The access permissions can define which users are allowed to execute the XR scenario. The editing permissions can include scenario editing permissions and / or scenario component editing permissions. The scenario editing permissions can define which users are allowed to edit one or more scenario components of the XR scenario. The scenario component editing permissions can define editing parameters for one or more of the individual scenario components that make up the XR scenario, where the editing parameter indicates whether the individual component can be edited by another user (e.g., a non-owner and / or non-initiator). The following regarding Figure 7 and Figure 9 the boxes 902, 904, 906, and 908 in

[0080] provide additional details about the permission manager 438. Figure 7 , Figure 8 the box 810 in Figure 9 and the box 916 in

[0081] Each implementation synthesizes one or more XR scenarios that, when executed, generate an XR environment display. Figure 5 is an example artificial reality environment with multiple user avatars. The XR environment 500 includes avatars 502 and 504 displayed in the XR environment. The XR environment 500 can be a three-dimensional immersive environment displayed to a user via an XR system. The avatars 502 and 504 can move within the XR environment, which is a multi-row chair in the XR environment 500. In some examples, the avatars 502 and 504 can be full-body avatars or any other appropriate virtual representation of the user.

[0082] In some embodiments, the elements of the XR environment 500 can be components of an XR scene. For example, the avatars 502 and 504 and their corresponding movements can be components of the XR scene. In some embodiments, the background (e.g., multiple rows of chairs) can also be a scene component of the XR scene. When executed, the XR scene can display the movements of the avatars 502 and 504 relative to the background (e.g., in a three-dimensional environment from a given camera view / perspective). Some embodiments of the XR scene include additional scene components, such as virtual objects, audio, lighting, and other suitable scene components.

[0083] Figure 6A is an example artificial reality scene with user avatars and virtual objects. The XR scene 600 can include an avatar 602, virtual objects 604 and 606, and a background 608. In some embodiments, one or more of the avatar 602, virtual object 604, virtual object 606, or any combination thereof can include motion dynamics (e.g., avatar motion dynamics, virtual object motion dynamics, etc.). For example, an XR scene recorder can record the XR scene 600 of the XR environment presented to the user. Embodiments of the XR scene recorder can identify the components of the XR environment as scene components of the XR scene, such as the avatar 602, the motion dynamics of the avatar 602, the virtual object 604, the motion dynamics of the virtual object 604, the virtual object 606, the motion dynamics of the virtual object 606, the lighting of the XR environment, the audio output to the user during the XR environment, and other suitable scene components.

[0084] In some embodiments, synthesis tools for XR scenes can be used to generate and / or edit the XR scene 600. For example, a scene synthesizer tool can load the recorded XR environment for editing. In another example, the scene synthesizer tool can generate (e.g., from scratch) an XR scene without a recorded XR environment.

[0085] Figure 6BAn example tool for synthesizing an editable artificial reality environment scene. The scene synthesizer tool 610 includes a scene 612 and a tool 614. The scene 612 includes an avatar 602, virtual objects 604 and 606, and a background 608. For example, the scene 612 can be a record of a loaded XR environment. In some embodiments, the synthesizer tool 610 can automatically identify the avatar 602, virtual objects 604 and 606, and the background 608 as XR scene components of the recorded XR environment. In another example, an embodiment of an XR scene recorder can automatically identify the components of the scene 612, separate data blocks for each XR scene component can be stored in association with the recorded XR scene, and the scene synthesizer tool 610 can load the components of the scene 612. The tool 614 can include predefined XR scene components for generating and / or editing the scene 612. For example, the tool 614 can be a set of elements shared from a scene created by others and shared to a cloud-based library.

[0086] For example, the scene 612 can be generated using the tool 614 without an XR environment record. The tool 614 includes an avatar element 616, a virtual object element 618, a background element 620, and an audio element 622. The avatar element 616 can include one or more predefined avatar models, which can be included in the scene 612. For example, the avatar 602 can be included in the scene 612 via the avatar element 616. The predefined avatars can include one or more avatars predefined for a given user (or group of users), one or more default avatars, one or more avatars corresponding to a given group (such as avatars from a game application, a movie, etc.), or any other suitable predefined avatars. In some embodiments, the user interaction with the scene synthesizer tool 610 can be a drag-and-drop interaction, where the user selects a predefined avatar from the avatar element 616, drags the selected avatar to position it (e.g., two-dimensional positioning, three-dimensional positioning, etc.) within the scene 612, and places the selected avatar at that position. Any other suitable user interaction can add one or more predefined avatars from the avatar element 616 to the scene 612.

[0087] The virtual object element 618 can be similar to the avatar element 616, but the virtual object element 618 can include predefined virtual objects. The predefined virtual objects can include one or more virtual objects predefined by the user, one or more default virtual objects, one or more virtual objects corresponding to a given set (e.g., virtual objects from a game application, a movie, etc.), or any other suitable predefined virtual objects. In some embodiments, the user interaction with the scene composer tool 610 can be a drag-and-drop interaction, where the user selects a predefined virtual object from the virtual object element 618, drags the selected virtual object to position it within the scene 612, and places the selected virtual object at that location or on another element (e.g., applying a predefined motion profile to an avatar that has been added to the scene). The predefined virtual objects can include two-dimensional structures or three-dimensional structure models. Any other suitable user interaction can add one or more predefined virtual objects from the virtual object element 618 to the scene 612.

[0088] The background element 620 can provide a background component for the scene 612. For example, the user can select the background 608 of the scene 612 from one or more predefined backgrounds of the background element 620. The predefined backgrounds can include one or more user-defined backgrounds (e.g., backgrounds uploaded by the user to the scene composer tool 610, backgrounds designed by the user, etc.), one or more default backgrounds, one or more backgrounds corresponding to a given set (e.g., backgrounds from a game application, a movie, etc.), or any other suitable predefined backgrounds. The predefined backgrounds can be two-dimensional (e.g., an image), three-dimensional (e.g., a 3D image / model), can include animation (e.g., an animated image / model), or can be any other suitable background. The user can select a predefined background from the background element 620 to add the background 608 to the scene 612 or replace an existing background of the scene 612.

[0089] The audio element 622 can provide an audio component for the scene 612. For example, a user can select the audio for the scene 612 from one or more predefined audio recordings of the audio element 622. The predefined audio can include user-defined audio (e.g., audio uploaded by the user to the scene synthesizer tool 610, audio recorded by the user, etc.), default audio, audio corresponding to a given group (e.g., audio from a game application, a movie, etc.), or any other suitable predefined audio. The user can select the predefined audio from the audio element 622 to add audio to the scene 612 or replace the existing audio of the scene 612. Implementations of the audio element 622 can include a recording element to record live audio via a client device operated by the user (e.g., an XR system, a laptop computer, a smartphone, etc.). For example, one or more microphones of the user's client device can be configured to capture live audio, and the audio element 622 can add / replace the live audio to the scene 612.

[0090] In some implementations, the scene synthesizer tool 610 can define the avatar movement dynamics of the avatar 602. The avatar movement dynamics can include the avatar pose over time, the avatar facial expression over time, or any other suitable movement dynamics. The user can generate avatar movement dynamics via the scene synthesizer tool 610 by selecting the avatar 602 and manipulating the avatar to move over time. Implementations of the scene synthesizer tool 610 can include a movement dynamics recording element that records the movement of the avatar 602 caused by the user's operation. In another example, a client device operated by the user (e.g., an XR system, a laptop computer, etc.) can capture the user's movement / motion via one or more cameras, and the scene synthesizer tool 610 can convert the captured user movement into avatar movement. For example, one or more machine learning models (e.g., a computer vision model) can isolate the user from the background in the captured user image, detect the movement, motion, facial expression, etc. of the isolated user, and convert these detected user movement, motion, and facial expression into avatar movement dynamics (e.g., avatar pose, avatar facial expression, avatar movement (such as walking, dancing, running, singing, etc.)).

[0091] In some implementations, a client device operated by the user can capture user audio via one or more microphones (e.g., similar to the recording function of the audio element 622), and the scene synthesizer tool 610 can convert the captured audio into avatar movement, such as a facial expression that lip-syncs to the captured audio (e.g., captured speech or sound). In another example, the scene synthesizer tool 610 can generate avatar movement (e.g., facial expression, such as lip-syncing) based on the stored audio (e.g., audio loaded via the audio element 622).

[0092] In another example, the scene synthesizer tool 610 can provide predefined avatar motion dynamics (e.g., avatar waving, smiling, running, dancing, etc.), and the user can select one or a series of predefined avatar motion dynamics for the avatar 602. For example, the predefined motion dynamics can include the individual dance steps of the avatar, and the user can select a series of individual dance steps to generate a dance series for the avatar 602. In some embodiments, the scene 612 is displayed based on the recorded XR environment, and the motion dynamics of the avatar 602 can be part of the recorded XR environment display.

[0093] In some embodiments, the virtual objects 604 and / or 606 include virtual object motion dynamics. The virtual object motion dynamics can include the movement of the virtual object model over time, the change of the virtual object display state over time, the movement of the virtual object in the background over time, or any other suitable motion dynamics. The user can generate virtual object motion dynamics via the scene synthesizer tool 610 by selecting the virtual object 604 and manipulating the movement of the virtual object over time. Embodiments of the scene synthesizer tool 610 can include a motion dynamics recording element that records the movement of the virtual object 604 caused by the user's operation.

[0094] In another example, the scene synthesizer tool 610 can provide predefined virtual object motion dynamics (e.g., virtual object display state changes, such transitions or other suitable animations, virtual object motion patterns, virtual object dances, etc.), and the user can select one or a series of predefined virtual object motion dynamics for the virtual object 604. For example, the predefined motion dynamics can include individual camera movements of the virtual camera, and the user can select a series of individual camera movements to generate a scene capture sequence around the virtual object 604. In some embodiments, the scene 612 is displayed based on the recorded XR environment, and the motion dynamics of the virtual object 604 and / or the avatar 606 can be part of the recorded XR environment display.

[0095] In some embodiments, the scene synthesizer tool 610 can include an effect element that supports the user in adding and / or editing XR effects to the XR scene. The XR effects can include animations, sound effects, visual effects (e.g., blur, lighting effects, etc.), filters, and any other suitable effects. In some embodiments, XR effects related to other components of the XR scene, such as avatars and / or virtual objects, can be added, replaced, and / or deleted. For example, the visual effects overlaying the avatar can be edited, replaced, or deleted.

[0096] In some embodiments, the scene synthesizer tool 610 may include lighting elements that support a user in adjusting and / or editing the lighting of an XR scene. For example, a user may adjust (e.g., increase or decrease) the light of an XR scene. In some embodiments, a user may select / define a region or volume of the XR scene and adjust the lighting of that individual region or volume.

[0097] Embodiments of the scene synthesizer tool 610 may generate a scene 612 from scratch, load a previously stored XR scene, and / or load a recorded XR environment as an XR scene. The scene 612 may be generated / edited using an avatar element 616, a virtual object element 618, a background element 620, and / or an audio element 622. For example, one or more avatars, one or more virtual objects, one or more backgrounds, audio, and / or motion dynamics may be added to the scene 612, or existing components of the scene 612 (e.g., one or more existing avatars, one or more virtual objects, one or more backgrounds, audio, and / or motion dynamics) may be replaced.

[0098] When a previously stored XR scene is loaded as the scene 612, the scene synthesizer tool 610 provides replacements, substitutions, and / or additions to the components of the previously stored XR scene. For example, one or more replacement avatars, one or more virtual objects, one or more backgrounds, audio, and / or motion dynamics may create an edited XR scene that is then stored for later execution. Different users may then iteratively edit the edited XR scene multiple times. The multiple iterative edits create multiple different versions of the XR scene that may share common threads such as avatar movement, background, audio, virtual objects, etc. In some embodiments, the users making these edits are connected via a social platform. Thus, the multiple different versions may serve as a mechanism for social interaction between users.

[0099] In some embodiments, a user may remove or delete components from the scene 612 (e.g., a recorded scene, a stored scene, etc.) via the scene synthesizer tool 610. For example, components of the scene 612 (such as one or more avatars, one or more virtual objects, backgrounds, audio, one or more motion dynamics, etc.) may be selected and deleted once they are loaded. The user selection / deletion may be performed by individual scene components displayed by the scene 612 (e.g., selection and deletion based on a cursor), scene components listed by elements of the tool 614 (e.g., avatar element 616, virtual object element 618, background element 620, and audio element 622), or any other suitable technique.

[0100] In some embodiments, the stored XR scene may include a three-dimensional display (e.g., a three-dimensional model with avatar / object movement, audio, etc.), a two-dimensional display (e.g., animated images, videos, audio, etc.), or any other suitable display. For example, the stored XR scene may be executed via the user's client system (e.g., an XR system, a laptop computer, a smartphone, etc.) and displayed to the user as a three-dimensional display and / or a two-dimensional display. In some embodiments, a two-dimensional display of the three-dimensional XR scene may be generated based on the position of a camera selected or defined by the owner user / editor user within the three-dimensional XR scene. The generated two-dimensional display may be from the perspective of the selected / defined camera position.

[0101] Figure 7 is a system diagram for synthesizing and editing an artificial reality scene. System 700 includes XR system 704, user 706, server 702, laptop computer 708, and computing device 710. Server 702 includes Figure 4 a synthesizer tool 434, one or more stored scenes 436, a permissions manager 438, and a publisher 440. A user (e.g., an owner, an editor, etc.) may interact with the synthesizer tool 434 to generate / edit one or more stored scenes 436. The synthesizer tool 434 may include interface components (e.g., a front end) displayed to the user on any suitable device. For example, user 706 may view the synthesizer tool 434 via XR system 704. In other examples, the user may interact with the synthesizer tool 434 via laptop computer 708 or computing device 710 (e.g., a smartphone, a desktop computer, a smart home device, an Internet of Things device, etc.). The synthesizer tool 434 (or portions of the tool) may be executed on server 702, XR system 704, laptop computer 708, computing device 710, or any combination thereof.

[0102] The permission manager 438 may manage access, editing, or any other suitable permissions for one or more stored scenarios 436. For example, a creator user may initiate a stored scenario among one or more stored scenarios 436 and define access permissions and / or editing permissions for the stored scenario. Access permissions may define which users can view / execute the stored XR scenario. Example access permissions may be public (e.g., any user of a social application), friends (e.g., connections to the creator user on a social graph), user-related portions of the social graph (e.g., friends of friends, distance from the creator user on the social graph, etc.), group access (e.g., access for a group of users who are members of a user group such as a game team, social group, employment group, etc.), allow list (e.g., an explicit list of users permitted access), block list (e.g., an explicit list of users not permitted access), and any combination thereof. Example editing permissions may be public (e.g., any user of a social application), friends (e.g., connections to the creator user on a social graph), user-related portions of the social graph (e.g., friends of friends, distance from the creator user on the social graph, etc.), group permissions (e.g., editing permissions for a group of users who are members of a user group such as a game team, social group, employment group, etc.), allow list (e.g., an explicit list of users permitted to edit), block list (e.g., an explicit list of users not permitted to edit), and any combination thereof.

[0103] In some embodiments, the permission manager 438 may enforce editing of parameters of the XR scenario itself, such as limits on the number of editing iterations, limits on the number of different versions of the XR scenario, and limits on individual components of the XR scenario that can be edited. The user creator (or any other suitable owner) may define: limits on the number of editing iterations (e.g., the number of edits that can be performed in sequence), limits on the number of different edited versions of the XR scenario (e.g., the total number of edits stored using the original XR scenario), and limits on individual components of the XR scenario that can be edited (e.g., editing restrictions on individual scenario components such as avatars, virtual objects, motion dynamics, backgrounds, audio, lighting, camera angle / viewpoint, etc.).

[0104] For example, the limit on the number of editing iterations may control how many sequential / chain edits can be made on the XR scenario. The stored XR scenario may be edited by a first editing user, and then the resulting edited XR scenario may be edited by a second editing user. This example shows two editing iterations in a chain / sequence. The creator user may define a threshold number of sequential editing iterations for the original XR scenario. The permission manager 438 may reject requests to edit an edited XR scenario that exceeds the threshold.

[0105] In another example, a limit on the number of different versions of an XR scene can control how many different edited versions of the XR scene can be generated. For example, an original XR scene can be edited twice by two different users, resulting in two edited XR scenes. Then one of the two edited XR scenes can be edited in sequence by another user, resulting in a third edited XR scene. The creator user can define a threshold number of total edited versions for the original XR scene. The permission manager 438 can reject requests to edit the original XR scene or an edited XR scene that exceeds the threshold.

[0106] In another example, limits on editing individual components of an XR scene can be defined by the creator user. For example, a given scene component (such as an avatar, a virtual object, the motion dynamics of the avatar and / or virtual object, audio, background, camera perspective, and lighting, etc.) can include an edit parameter that defines whether other users (e.g., non-owners or non-originators) can edit the given scene component. A first value of the edit parameter can lock the scene component so that it cannot be edited by other users. A second value of the edit parameter can allow other users to edit the scene component. The permission manager 438 and / or the compositor tool 434 can manage the editing limits for the individual scene components of the XR scene.

[0107] In some embodiments, the permission manager 438 verifies / confirms the editing permissions for an XR scene before the compositor tool 434 loads the XR scene for editing. For example, the editing permissions of a user requesting to edit a stored XR scene can be verified (e.g., relative to the creator user of the stored XR scene on the social graph, etc.). When the permission manager 438 verifies / confirms the editing permissions of the requesting user, the compositor tool 434 can load the XR scene. In some embodiments, the permission manager 438 can also provide the editing limits defined for the individual scene components of the XR scene to the compositor tool 434. The compositor tool 434 can lock the individual scene components that cannot be edited in response to receiving the editing limits for the scene components, while enabling the editing of other scene components.

[0108] Publisher 440 can publish an XR scene or an edited XR scene to a publicly accessible application, repository, social platform, or Publisher 440 can perform any other digital publication as follows: These digital publications allow users to access and / or edit the published XR scene. In some embodiments, Publisher 440 can publish an XR scene to users whose access restrictions to the XR scene are defined by Permission Manager 438. A user (e.g., a user permitted to access) can access the published XR scene, for example, by clicking on a uniform resource locator (URL) or other suitable XR scene identifier. In response to the access, the XR scene can be executed (e.g., by server 702, XR system 704, laptop 708, computing device 710, or any combination thereof) to display a two-dimensional or three-dimensional display of the XR scene to the user and, in some examples, output audio to the user.

[0109] Those skilled in the art will understand that the Figures 1 to 5 , Figure 6A , Figure 6B and Figure 7 shown in, and each of the flowcharts shown in the flowcharts discussed below can be changed in various ways. For example, the order of the logic can be rearranged, sub-steps can be performed in parallel, the shown logic can be omitted, other logic can be included, etc. In some embodiments, one or more of the above components can perform one or more of the processes described below.

[0110] Figure 8 is a flowchart showing a process for synthesizing an editable extended reality (XR) scene in some embodiments of the present technology. In some embodiments, process 800 can be executed on a server, XR system, or any other suitable computing system. Embodiments of process 800 can be triggered when initializing an XR scene synthesizer tool.

[0111] In block 802, process 800 can define an avatar for the XR scene. For example, the avatar can be defined by recording an XR environment including the avatar, loading a predefined avatar, or in any other suitable way. In some embodiments, a scene synthesizer tool including a predefined avatar can be used to define the avatar. In some embodiments, the avatar can be defined when recording the XR environment and, for example, automatically identified by an XR recording tool and / or a scene synthesizer tool in the recorded XR environment.

[0112] At block 804, process 800 can define the avatar motion dynamics of an avatar. For example, a user can manipulate the avatar via a scene synthesizer tool to define the avatar motion dynamics. In another example, the user's client system (e.g., a laptop computer, an XR system, a smartphone, etc.) can capture the user via one or more cameras, isolate the user in the captured image, and detect the user's motion, movement, and / or expression. The detected user motion, movement, and / or expression can be converted into avatar motion, movement, and / or expression (e.g., avatar pose, avatar expression, etc.).

[0113] In some embodiments, the scene synthesizer tool can include a plurality of predefined avatar motion dynamics, and the user can select one or more (e.g., a sequence) of the plurality of predefined avatar motion dynamics. In another example, the recorded XR environment can include avatar movement and expressions that are detected as avatar motion dynamics.

[0114] At block 806, process 800 can define one or more additional scene components for the XR scene. For example, the additional scene components can include one or more virtual objects, virtual object motion dynamics, backgrounds, audio, lighting, or any other suitable scene components. The additional scene components can be defined via a synthesizer tool, identified in the recorded XR environment, or defined in any other suitable manner.

[0115] At block 808, process 800 can store the XR scene with the defined components. For example, a data structure can store data blocks, models, audio files, video files, images, and any other suitable data elements for the XR scene components. The stored XR scene can be configured such that executing the XR scene generates a two-dimensional or three-dimensional display of the XR scene for the user, which can include audio in some examples.

[0116] At block 810, process 800 can publish the stored XR scene. For example, the stored XR scene can be published to a social platform or any other suitable application accessible by multiple users. The published XR scene can be viewed and selected for editing by one or more of the multiple social platform users.

[0117] Figure 9 is a flowchart showing a process for editing a stored artificial reality scene in some embodiments of the present technology. In some embodiments, process 900 can be executed on a server, an XR system, or any other suitable computing system. When an edit request for the stored XR scene is received, an embodiment of process 900 can be triggered.

[0118] At block 902, process 900 may receive a request to edit a stored XR scene. For example, the request may be received from a user via the user's client device. In some embodiments, the stored XR scene may be published to a social platform, and a user may view the published XR scene and submit an edit request by clicking on a uniform resource locator (URL) or other suitable XR scene identifier.

[0119] At block 904, process 900 may determine whether the request includes edit permissions for the XR scene. For example, the edit permissions of the requesting user may be verified or denied. In some embodiments, the XR scene is created by a creator user (such as the user who initiated the XR scene). The edit permissions for the XR scene may be defined by the creator user. Example edit permissions for an XR scene include public (e.g., any user of the social platform), friends (e.g., connections on the social graph to the creator user), a relevant portion of the social graph related to the user (e.g., friends of friends, distance on the social graph from the creator user, etc.), group access (e.g., access by a group of users who are members of a user group such as a gaming team, social group, employment group, etc.), allow list (e.g., an explicit list of users allowed access), block list (e.g., an explicit list of users not allowed access), and any combination thereof.

[0120] In some embodiments, the requesting user may be compared with the edit permissions of the XR scene (e.g., comparing the requesting user with the creator user on the social graph, allow list, block list, group members, etc.) to verify or deny the request. When the request includes edit permissions to edit the requested XR scene, process 900 may proceed to block 908. When the request does not include edit permissions to edit the requested XR scene, process 900 may proceed to block 906, where the request is denied. For example, a message describing the lack of edit permissions may be sent to the requesting client device.

[0121] At block 908, process 900 may load a stored XR scene. For example, a scene compositor tool may load scene components such as one or more avatars, one or more virtual objects, the motion dynamics of one or more avatars and / or one or more virtual objects, a background, a camera / viewpoint, audio, lighting, and other suitable scene components. At block 910, process 900 may receive edits to the scene components of the stored XR scene. For example, one or more of the following may be edited or replaced via user interaction with the scene compositor tool: one or more avatars, one or more virtual objects, the motion dynamics of one or more avatars and / or one or more virtual objects, a background, a camera / viewing perspective, audio, and / or lighting. In some embodiments, one or more edits to the scene components may include removal or deletion. For example, one or more of the following may be removed or deleted from the XR scene: one or more avatars, one or more virtual objects, a background, audio, or any combination thereof.

[0122] At block 912, process 900 may receive one or more additional scene components. For example, one or more of the following may be added via user interaction with the scene compositor tool: one or more additional avatars, one or more additional virtual objects, or the motion dynamics of the one or more avatars and / or the one or more virtual objects.

[0123] At block 914, process 900 may store the edited XR scene having the edited or added components. For example, a data structure that stores data blocks, models, audio files, video files, images, and any other suitable data elements of the XR scene components may be modified to store the edits to the scene components and / or additional scene components. The edited XR scene may be stored such that execution of the edited XR scene will generate a two-dimensional or three-dimensional display of the edited XR scene to the user, which may include audio in some examples.

[0124] At block 916, process 900 may publish the edited XR scene. For example, the edited XR scene may be posted to a social platform or any other suitable application accessible by multiple users. One or more of the multiple social platform users may view and select the posted edited XR scene for editing.

[0125] In some embodiments, other users may select the edited XR scene for editing. For example, process 900 may be repeated through several iterations such that multiple different edited versions of the original XR scene may be generated. These different versions may be shared among users via posting to a social platform, messaging, or any other suitable sharing mechanism.

[0126] References to "multiple embodiments" (e.g., "some embodiments", "various embodiments", "one embodiment", "an embodiment", etc.) in this specification mean that the particular features, structures, or characteristics described in connection with that embodiment are included in at least one embodiment of the present disclosure. These expressions that appear in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate or alternative embodiments that are mutually exclusive of other embodiments. Additionally, various features are described that may be rendered by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0127] As used herein, above a threshold means that the value of the comparison item is higher than a specified other value, the comparison item is among a specified number of items having the maximum value, or the comparison item has a value within the specified highest percentage value. As used herein, below a threshold means that the value of the comparison item is lower than a specified other value, the comparison item is among a specified number of items having the minimum value, or the comparison item has a value within the specified lowest percentage value. As used herein, within a threshold means that the value of the comparison item is between two specified other values, the comparison item is among the specified number of items in the middle, or the comparison item has a value within the specified middle percentage range. When not otherwise limited, relative terms such as high or insignificant can be understood to assign a value and determine how that value compares to an established threshold. For example, the phrase "select a fast connection" can be understood to mean selecting a connection having a value assigned to its connection speed that is above the threshold.

[0128] As used herein, the word "or" refers to any possible permutation of a set of items. For example, the phrase "A, B, or C" refers to at least one of the following: A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiple of any item (e.g., A and A; B, B, and C; A, A, B, C, and C, etc.).

[0129] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims need not be limited to the specific features or acts described above. Specific embodiments and implementations have been described herein for purposes of illustration, but various modifications can be made without departing from the scope of these embodiments and implementations. The above specific features and acts are disclosed as example forms for implementing the appended claims. Thus, the embodiments and implementations are not limited except as by the appended claims.

Claims

1. A method for editing a stored 3D artificial reality (XR) scene, the method comprising: Accessing a stored 3D XR scene having scene components, the scene components including at least an avatar, avatar motion dynamics, and an XR environment, wherein executing the 3D XR scene generates a three-dimensional depiction of the avatar moving according to the avatar motion dynamics defined in the XR environment; Receiving one or more edits to the scene components of the 3D XR scene, wherein the one or more edits include at least replacing the avatar with a replacement avatar; and Storing the edited 3D XR scene, wherein executing the edited 3D XR scene generates a three-dimensional depiction of the replacement avatar moving within the XR environment according to the avatar motion dynamics.

2. The method according to claim 1, wherein, The scene components of the 3D XR scene further include one or more virtual objects and one or more virtual object motion dynamic parameters, and the three-dimensional depiction generated by executing the 3D XR scene includes the one or more virtual objects moving within the XR environment according to the one or more virtual object motion dynamic parameters; And preferably, wherein editing the scene components of the 3D XR scene further includes replacing the one or more virtual objects with one or more replacement virtual objects, and the three-dimensional depiction generated by executing the edited 3D XR scene includes the one or more replacement virtual objects moving within the XR environment according to the one or more virtual object motion dynamic parameters.

3. The method according to claim 1 or 2, wherein Editing the scene components of the 3D XR scene further includes editing the avatar motion dynamics, and wherein the three-dimensional depiction generated by executing the edited 3D XR scene includes the replacement avatar moving within the XR environment according to the edited avatar motion dynamics.

4. The method according to any one of claims 1 to 3, wherein, Editing the avatar motion dynamics further includes: Capturing user motion via one or more sensors, wherein at least one of the one or more sensors includes a camera; and Converting the user motion into avatar motion dynamics.

5. The method according to any one of claims 1 to 4, wherein Editing the scene components of the 3D XR scene further includes adding one or more additional scene components, and the additional scene components include one or more of an additional avatar, additional virtual objects, or any combination thereof.

6. The method according to any one of claims 1 to 5, wherein The stored 3D XR scene is created by a creator user who defines editing permissions for the stored 3D XR scene, and accessing the stored 3D XR scene further includes: Receiving a request from another user different from the creator user to edit the stored 3D XR scene; and Verifying that the other user has editing permissions to edit the stored 3D XR scene, wherein the stored 3D XR scene is accessed in response to the verification; And preferably, wherein the editing permission includes a user scope related to the social graph, and when the other user is included in the user scope related to the social graph, the editing permission of the other user is verified.

7. The method according to any one of claims 1 to 6, wherein The stored 3D XR scene is created by a creator user, the creator user defines at least one editing parameter for at least one scene component of the stored 3D XR scene, and wherein a first value of the editing parameter allows editing, replacing or deleting the at least one scene component, and a second value of the editing parameter restricts editing, replacing or deleting the at least one scene component.

8. The method according to claim 7, wherein The at least one scene component includes the avatar, and receiving one or more edits to the scene component of the 3D XR scene further includes: Verifying that the editing parameter of the avatar allows the avatar to be replaced with the alternative avatar.

9. The method according to any one of claims 1 to 8, the method further comprising: Publishing the edited 3D XR scene on a social media platform, wherein the publishing allows multiple social media platform users to execute the edited 3D XR scene.

10. The method according to any one of claims 1 to 9, wherein, The avatar motion dynamics at least include avatar pose and avatar expression.

11. A computing system for editing a stored artificial reality (XR) scene, the computing system comprising: One or more processors; And One or more memories storing instructions which, when executed by the one or more processors, cause the computing system to perform a process, the process including: Storing an XR scene having scene components, the scene components at least including an avatar, avatar motion dynamics, and an XR environment, wherein the XR scene is created by a creator user who defines editing permissions, and executing the XR scene generates a three-dimensional depiction of the avatar moving within the XR environment according to the avatar motion dynamics; Receiving a request from another user different from the creator user to edit the stored XR scene; Verifying that the other user includes an editing permission to edit the stored XR scene; Accessing the stored XR scene in response to the verification; Receiving one or more edits to the scene components of the XR scene, wherein the one or more edits at least include replacing the avatar with an alternative avatar; and Storing the edited XR scene, wherein executing the edited XR scene generates a three-dimensional depiction of the alternative avatar moving within the XR environment according to the avatar motion dynamics.

12. The computing system according to claim 11, the computing system further comprising at least one of the following features: (i) Among them, The scene components of the XR scene further include one or more virtual objects and one or more virtual object motion dynamic parameters, and the three-dimensional depiction generated by executing the XR scene includes the one or more virtual objects moving within the XR environment according to the one or more virtual object motion dynamic parameters; and preferably, wherein editing the scene components of the XR scene further includes replacing the one or more virtual objects with one or more alternative virtual objects, and the three-dimensional depiction generated by executing the edited XR scene includes the one or more alternative virtual objects moving within the XR environment according to the one or more virtual object motion dynamic parameters; (ii) wherein editing the scene components of the XR scene further includes editing the avatar motion dynamics, and wherein the three-dimensional depiction generated by executing the edited XR scene includes the alternative avatar moving within the XR environment according to the edited avatar motion dynamics; and / or (iii) wherein editing the avatar motion dynamics further includes: capturing user motion via one or more sensors, wherein at least one of the one or more sensors includes a camera; and converting the user motion into avatar motion dynamics.

13. The computing system according to claim 11 or 12, the computing system further comprising at least one of the following features: Among them, The editing permission includes a user scope related to the social graph, and when the other user is included in the user scope related to the social graph, the editing permission of the other user is verified; and / or wherein the creator user defines at least one editing parameter for at least one scene component of the stored XR scene, and wherein a first value of the editing parameter allows editing, replacing, or deleting the at least one scene component, and a second value of the editing parameter restricts editing, replacing, or deleting the at least one scene component.

14. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for editing a stored extended reality (XR) scene, the process comprising: storing an XR scene having scene components, the scene components including at least an avatar, avatar motion dynamics, and an XR environment, wherein the XR scene is created by a creator user who defines editing permissions, and executing the XR scene generates a three-dimensional depiction of the avatar moving within the XR environment according to the avatar motion dynamics; receiving a request from another user different from the creator user to edit the stored XR scene; verifying that the other user includes editing permissions to edit the stored XR scene; responding to the verification by accessing the stored XR scene; receiving one or more edits to the scene components of the XR scene, wherein the one or more edits include at least replacing the avatar with an alternative avatar; and Store the edited XR scene, wherein executing the edited XR scene generates a three-dimensional depiction of the alternative avatar moving within the XR environment according to the avatar motion dynamics.

15. A computer program comprising instructions that, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

Images