Method and apparatus for switching between viewing vectors in a synthetic reality scene

By using non-transitory memory and processor to detect user input in synthetic reality set devices and updating view vectors, the problem of poor visual and position matching in the prior art is solved, and the user experience is improved.

CN112189203BActive Publication Date: 2024-11-01APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980034829.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-03-11
Filing Date
2019-05-28
Publication Date
2024-11-01
Estimated Expiration
2039-05-28

AI Technical Summary

Technical Problem

Existing synthetic reality set devices are difficult to effectively update and present when users choose different time points and perspectives, resulting in poor visual and position matching and affecting the user's interactive experience.

Method used

Using a device, including non-transitory memory and processor, ensures content at different points in time by presenting synthetic reality sets associated with predefined view vectors, detecting user inputs, and updating the sets to match different view vectors. Display correctly.

Benefits of technology

It realizes smooth visual and position updates in synthetic reality sets, improving user interaction experience and content matching.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112189203B_ABST
    Figure CN112189203B_ABST
Patent Text Reader

Abstract

In some embodiments, a method includes: presenting a synthetic reality (SR) scene associated with predetermined content based on a first view vector at a first time; presenting a navigation timeline for the predetermined content, the navigation timeline including a plurality of view vector affordances at corresponding times on the navigation timeline, wherein each view vector represents how to display the SR content at the corresponding time; detecting an input selecting a corresponding view vector affordance on the navigation timeline associated with a second time within the predetermined content; and in response to detecting the input selecting the corresponding view vector affordance on the navigation timeline, updating the SR scene to present the SR scene based on a second view vector different from the first view vector, wherein the second view vector represents display of the predetermined content at the second time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to presenting synthetic reality (SR) settings, and more particularly to switching between view vectors in SR settings. Background Art

[0002] Some devices are capable of generating and presenting synthetic reality sets. Some synthetic reality sets include virtual sets that are simulated replacements for physical sets. Some synthetic reality sets include augmented sets that are modified versions of physical sets. Some devices that present synthetic reality sets include mobile communication devices, such as smart phones, head-mounted displays (HMDs), glasses, heads-up displays (HUDs), head-mounted housings, and optical projection systems. Most previously available devices that present synthetic reality sets are ineffective at presenting representations of certain objects. For example, some previously available devices that present synthetic reality sets are not suitable for presenting representations of objects associated with actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0004] Figure 1A and Figure 1B is an illustration of an example operating environment according to some implementations.

[0005] Figure 2 is a block diagram of an example system according to some implementations.

[0006] Figure 3A is a block diagram of an example goal achiever engine according to some specific implementations.

[0007] Figure 3B is a block diagram of an exemplary neural network according to some specific implementations.

[0008] Figure 4 is a flowchart representation of a method of presenting a target implementer according to some specific implementations.

[0009] Figure 5 is a block diagram of an electronic device according to some specific implementations.

[0010] Fig. 6A is a block diagram of conditionally related synthetic reality (SR) content threads according to some specific implementations.

[0011] Figure 6B and Figure 6C A timeline associated with an object enabler (OE) package according to some implementations is shown.

[0012] 7A to 7C An exemplary SR scene according to some implementations is shown.

[0013] FIG. 8A to FIG. 8C is a block diagram of an emerging content architecture based on some specific implementations.

[0014] 9A to 9F A number of exemplary SR scenes according to different viewing vectors are shown according to some implementations.

[0015] Fig.10 is a flowchart representation of a method of changing a look vector within an SR scene according to some specific implementations.

[0016] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0017] Various embodiments disclosed herein include devices, systems, and methods for changing a view vector within a synthetic reality (SR) scene. In various embodiments, a device includes a non-transitory memory and one or more processors coupled to the non-transitory memory. In some embodiments, a method includes: presenting an SR scene associated with predetermined content according to a first view vector at a first time, wherein the first view vector represents a display of the predetermined content at the first time; presenting a navigation timeline of the predetermined content, the navigation timeline including a current time indicator located at the first time and a plurality of view vector affordances, each view vector affordance being associated with a different view vector at a corresponding time on the navigation timeline, wherein each view vector represents how the SR content is displayed at the corresponding time; detecting an input selecting a corresponding view vector affordance on the navigation timeline associated with a second time within the predetermined content; and in response to detecting an input selecting a corresponding view vector affordance on the navigation timeline, updating the SR scene to present the SR scene according to a second view vector different from the first view vector, wherein the second view vector represents a display of the predetermined content at the second time.

[0018] According to some specific implementations, the device includes one or more processors, non-transitory memory, and one or more programs. In some specific implementations, one or more programs are stored in non-transitory memory and executed by one or more processors. In some specific implementations, one or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of a device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-transitory memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0019] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0020] A physical set refers to a world that an individual can perceive and / or interact with without the aid of an electronic system. A physical set (e.g., a physical forest) includes physical elements (e.g., physical trees, physical structures, and physical animals). An individual can directly interact with and / or perceive a physical set, such as by touch, sight, smell, hearing, and taste.

[0021] In contrast, a synthetic reality (SR) set refers to a set that is fully or partially computer-created that an individual can perceive and / or that the individual can interact with via an electronic system. In SR, a subset of the individual's movements is monitored, and in response thereto, one or more properties of one or more virtual objects in the SR set are changed in a manner that complies with one or more laws of physics. For example, the SR system may detect that the individual has taken a few steps forward, and in response thereto, adjust the graphics and audio presented to the individual in a manner similar to how such scenes and sounds would change in a physical set. Modifications to one or more properties of one or more virtual objects in the SR set may also be made in response to a representation of movement (e.g., an audio command).

[0022] The individual may interact with and / or perceive SR objects using any of his senses, including touch, smell, sight, taste, and sound. For example, the individual may interact with and / or perceive auditory objects that create a multi-dimensional (e.g., three-dimensional) or spatial auditory scenery and / or implement auditory transparency. The multi-dimensional or spatial auditory scenery provides the individual with the perception of discrete auditory sources in a multi-dimensional space. Auditory transparency selectively combines sounds from the physical scenery with or without computer-created audio. In some SR scenery, the individual may interact with and / or perceive only auditory objects.

[0023] An example of SR is virtual reality (VR). A VR set refers to a simulated set designed to include only computer-created sensory input for at least one sense. The VR set includes a plurality of virtual objects that an individual can interact with and / or perceive. The individual can interact with and / or perceive virtual objects in the VR set by simulating a subset of the individual's actions within the computer-created set and / or by simulating the individual or his / her presence within the computer-created set.

[0024] Another example of SR is mixed reality (MR). An MR set refers to a simulated set designed to integrate computer-created sensory input (e.g., virtual objects) with sensory input from a physical set, or representation thereof. On the reality spectrum, a mixed reality set is between VR sets at one end and fully physical sets at the other end, and does not include these sets.

[0025] In some MR sets, computer-created sensory input can adapt to changes in sensory input from the physical set. In addition, some electronic systems used to render MR sets can monitor orientation and / or position relative to the physical set to enable virtual objects to interact with real objects (i.e., physical elements from the physical set or representations thereof). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0026] An example of mixed reality is augmented reality (AR). AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor, the imaging sensor being used to capture images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects may be projected, for example, on a physical surface or as a hologram, such that an individual using the system observes the virtual objects superimposed on the physical scenery.

[0027] An augmented reality set may also refer to a simulated set in which a representation of a physical set is altered by computer-created sensory information. For example, a portion of the representation of a physical set may be graphically altered (e.g., enlarged) such that the altered portion may still represent one or more originally captured images but is not a faithfully reproduced version. For another example, when providing video pass-through, the system may alter at least one of the sensor images to impose a particular viewpoint that is different from the viewpoint captured by one or more image sensors. For another example, the representation of a physical set may be altered by graphically blurring or eliminating portions thereof.

[0028] Another example of mixed reality is augmented virtuality (AV). An AV set refers to a simulated set in which a computer-created set or a virtual set incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may present the color of a physical element captured by one or more imaging sensors. As another example, a virtual object may present features consistent with actual weather conditions in a physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images taken of physical animals.

[0029] Many electronic systems enable individuals to interact with and / or perceive various SR scenery. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smartphone). The head-mounted system may have one or more imaging sensors and / or microphones for taking images / video of the physical scenery and / or capturing audio of the physical scenery, respectively. The head-mounted system may also have a transparent or translucent display. The transparent or translucent display may be combined with a substrate through which light representing the image is directed to the individual's eyes. The display may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one specific implementation, the transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. As another example, the electronic system may be a projection-based system. A projection-based system may project an image onto the retina of an individual using retinal projection. Alternatively, the projection system may also project the virtual object into a physical setting (e.g., onto a physical surface or as a hologram). Other examples of SR systems include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, headphones or earbuds, a speaker arrangement, an input mechanism (e.g., a controller with or without tactile feedback), a tablet, a smartphone, and a desktop or laptop computer.

[0030] When experiencing a synthetic reality (SR) set that includes predetermined content (e.g., a narrative or story frame, a movie, a historical event, a fictional story, etc.) spanning a period of time, a user is able to move within the SR set and select different perspectives of the predetermined content (e.g., an omniscient point of view (POV) within a historical battle, a general's POV, an infantry unit's POV, an individual soldier's POV, etc.) from which to experience portions of the predetermined content. However, if the user chooses to jump between different and discrete portions of the predetermined content in time, SR characteristics characterizing a viewing vector (e.g., field of view (FOV), pose / rotation coordinates, translation coordinates in SR, viewing angle, etc.) associated with a first time may not match the predetermined content well at a second time.

[0031] Thus, in various implementations, when presenting the SR scenery using a first view vector, a navigation timeline is displayed for predetermined content (e.g., overlaid on the SR scenery, presented as part of a separate control interface, etc.), with corresponding multiple times on the navigation timeline having multiple view vector affordances, wherein each view vector is characterized by a FOV, a pose, a set of SR coordinates, a viewing angle, etc. In some implementations, the view vectors associated with the multiple view vector affordances are pre-selected by a director, a historian, etc. In some implementations, the view vectors associated with the multiple view vector affordances are selected based on one or more filters determined by a user (e.g., a view vector associated with a machine for a mechanical filter, a view vector associated with naval training for a naval trainee, a view vector associated with weather for a meteorologist, etc.).

[0032] Figure 1A 1 is a block diagram of an exemplary operating environment 100A according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100A includes a controller 102 and an electronic device 103. In Figure 1A In the example of , the electronic device 103 is being held by the user 10. In some implementations, the electronic device 103 includes a smart phone, a tablet computer, a laptop computer, etc.

[0033] like Figure 1A As shown, the electronic device 103 presents a synthetic reality set 106. In some implementations, the synthetic reality set 106 is generated by the controller 102 and / or the electronic device 103. In some implementations, the synthetic reality set 106 includes a virtual set that is a simulated replacement of a physical set. In other words, in some implementations, the synthetic reality set 106 is simulated by the controller 102 and / or the electronic device 103. In such implementations, the synthetic reality set 106 is different from the physical set where the electronic device 103 is located. In some implementations, the synthetic reality set 106 includes an augmented set that is a modified version of the physical set. For example, in some implementations, the controller 102 and / or the electronic device 103 modifies (e.g., augments) the physical set where the electronic device 103 is located in order to generate the synthetic reality set 106. In some implementations, the controller 102 and / or the electronic device 103 generates the synthetic reality set 106 by simulating a copy of the physical set where the electronic device 103 is located. In some implementations, the controller 102 and / or the electronic device 103 generates the synthetic reality set 106 by removing and / or adding items to a simulated copy of the physical set in which the electronic device 103 is located.

[0034] In some implementations, the synthetic reality set 106 includes various SR representations of goal achievers, such as a boy action figure representation 108a, a girl action figure representation 108b, a robot representation 108c, and a drone representation 108d. In some implementations, the goal achievers represent characters from fictional materials such as movies, video games, comics, and novels. For example, the boy action figure representation 108a represents a "boy action figure" character from a fictional comic, and the girl action figure representation 108b represents a "girl action figure" character from a fictional video game. In some implementations, the synthetic reality set 106 includes goal achievers representing characters from different fictional materials (e.g., from different movies / games / comics / novels). In various implementations, the goal achievers represent things (e.g., tangible objects). For example, in some implementations, the goal achievers represent equipment (e.g., machines such as airplanes, tanks, robots, cars, etc.). Figure 1A In the example of , robot representation 108 c represents a robot, and drone representation 108 d represents a drone. In some implementations, goal achievers represent things (e.g., equipment) from fictional materials. In some implementations, goal achievers represent things from a physical set, including things located inside and / or outside of synthetic reality set 106.

[0035] In various implementations, the goal achiever performs one or more actions in order to achieve (e.g., complete / satisfy / achieve) one or more goals. In some implementations, the goal achiever performs a series of actions. In some implementations, the controller 102 and / or the electronic device 103 determines the actions to be performed by the goal achiever. In some implementations, the actions of the goal achiever are within a certain degree of similarity to the actions performed by the corresponding character / thing in the fictional material. Figure 1A In the example of , the girl action figure representation 108b is performing a flying action (e.g., because the corresponding "girl action figure" character is capable of flying and / or "girl action figure" characters often fly in fictional materials). Figure 1A In the example of FIG. 1 , the drone representation 108 d is performing a hovering action (e.g., because the drone is capable of hovering in the physical scene). In some implementations, the controller 102 and / or the electronic device 103 obtains the actions for the goal achiever. For example, in some implementations, the controller 102 and / or the electronic device 103 receives the actions for the goal achiever from a remote server that determines (e.g., selects) the actions.

[0036] In various implementations, a goal achiever performs actions in order to meet (e.g., complete or achieve) a goal. In some implementations, a goal achiever is associated with a specific goal, and the goal achiever performs actions that improve the likelihood of meeting the specific goal. In some implementations, the SR representation of a goal achiever is called an object representation, for example, because the SR representation of a goal achiever represents various objects (e.g., real objects or fictional objects). In some implementations, a goal achiever representing a role is called a role goal achiever. In some implementations, a role goal achiever performs actions to achieve a role goal. In some implementations, a goal achiever representing an equipment is called an equipment goal achiever. In some implementations, an equipment goal achiever performs actions to achieve an equipment goal. In some implementations, a goal achiever representing an environment is called an environment goal achiever. In some implementations, an environment goal achiever performs environment actions to achieve an environment goal.

[0037] In some implementations, the synthetic reality set 106 is generated based on input from the user 10. For example, in some implementations, the electronic device 103 receives input indicating terrain for the synthetic reality set 106. In such implementations, the controller 102 and / or the electronic device 103 configures the synthetic reality set 106 so that the synthetic reality set 106 includes the terrain indicated via the input. In some implementations, the input indicates an environmental condition for the synthetic reality set 106. In such implementations, the controller 102 and / or the electronic device 103 configures the synthetic reality set 106 to have the environmental condition indicated by the input. In some implementations, the environmental condition includes one or more of temperature, humidity, pressure, visibility, ambient light level, ambient sound level, time of day (e.g., morning, afternoon, evening, or night), and precipitation (e.g., cloudy, rainy, or snowy).

[0038] In some implementations, actions of the goal achiever are determined (e.g., generated) based on input from user 10. For example, in some implementations, electronic device 103 receives input indicating placement of an SR representation of a goal achiever. In such implementations, controller 102 and / or electronic device 103 positions the SR representation of the goal achiever according to the placement indicated by the input. In some implementations, the input indicates a specific action that the goal achiever is allowed to perform. In such implementations, controller 102 and / or electronic device 103 selects an action of the goal achiever from the specific actions indicated by the input. In some implementations, controller 102 and / or electronic device 103 discards actions that are not among the specific actions indicated by the input. In some implementations, controller 102 and / or electronic device 103 includes FIG. 8A to FIG. 8C At least part of the emerging content architecture in .

[0039] Figure 1B 1 is a block diagram of an exemplary operating environment 100B according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100B includes a controller 102 and a head mountable device (HMD) 104. In Figure 1B In the example of FIG. 1 , HMD 104 is worn by user 10. In various implementations, HMD 104 is worn with Figure 1A In some implementations, the HMD 104 performs the same operation as the electronic device 103 shown in FIG. Figure 1A The HMD 104 may be configured to operate substantially the same as the electronic device 103 shown. In some implementations, the HMD 104 includes a head-mountable housing. In various implementations, the head-mountable housing is shaped to receive an electronic device having a display (e.g., Figure 1A For example, in some implementations, Figure 1A The electronic device 103 is shown to slide into the HMD 104. In some implementations, the HMD 104 includes an integrated display for presenting a synthetic reality experience to the user 10. In some implementations, the controller 102 and / or the HMD 104 includes FIG. 8A to FIG. 8C At least part of the emerging content architecture in .

[0040] Figure 2 is a block diagram of an exemplary system 200 for generating goals for various goal achievers in an SR setting. For example, the system 200 is Figure 1A and 1B The boy action figure representation 108a, girl action figure representation 108b, robot representation 108c, and / or drone representation 108d are shown generating targets. Figure 2 In the example of , the system 200 includes a boy action figure engine 208a, a girl action figure engine 208b, a robot equipment engine 208c, and a drone equipment engine 208d, which generate actions 210 for the boy action figure representation 108a, the girl action figure representation 108b, the robot representation 108c, and the drone representation 108d, respectively. In some specific implementations, the system 200 also includes an environment engine 208e, an emerging content engine 250, and a display engine 260.

[0041] In various implementations, the emerging content engine 250 generates corresponding goals 254 for a goal achiever in the SR scene and / or the environment of the SR scene. Figure 2In the example of , the emerging content engine 250 generates a boy action figure target 254a for the boy action figure representation 108a, a girl action figure target 254b for the girl action figure representation 108b, a robot target 254c for the robot representation 208c, a drone target 254d for the drone representation 108d, and / or an environmental target 254e (e.g., an environmental condition) for the environment of the SR set 106. Figure 2 As shown, the emerging content engine 250 provides the target 254 to the corresponding character / equipment / environment engine. Figure 2 In the example of , emerging content engine 250 provides boy action figure target 254a to boy action figure engine 208a, girl action figure target 254b to girl action figure engine 208b, robot target 254c to robot equipment engine 208c, drone target 254d to drone equipment engine 208d, and environment target 254e to environment engine 208e.

[0042] In various implementations, the emerging content engine 250 generates the goal 254 based on possible goals 252 (e.g., a set of predefined goals), context information 258 characterizing the SR scenery, and a function of the action 210 provided by the character / equipment / environment engine. For example, in some implementations, the emerging content engine 250 generates the goal 254 by selecting the goal 254 from the possible goals 252 based on the context information 258 and / or the action 210. In some implementations, the possible goals 252 are stored in a data repository. In some implementations, the possible goals 252 are obtained from corresponding fictional source materials (e.g., by scrubbing video games, movies, novels, and / or comics). For example, in some implementations, the possible goals 252 of the girl action figure representation 108b include saving lives, saving pets, fighting crime, etc.

[0043] In some implementations, the emerging content engine 250 generates the goal 254 based on the action 210 provided by the character / equipment / environment engine. In some implementations, the emerging content engine 250 generates the goal 254 so that the probability of completing the goal 254 meets a threshold (e.g., the probability is greater than a threshold, e.g., the probability is greater than 80%) given the action 210. In some implementations, the emerging content engine 250 generates the goal 254 with a high probability of being completed with the action 210.

[0044] In some implementations, the emerging content engine 250 ranks the possible goals 252 based on the actions 210. In some implementations, the rank of a particular possible goal 252 indicates the likelihood of accomplishing the particular possible goal 252 given the actions 210. In such implementations, the emerging content engine 250 generates the goals 254 by selecting the top N ranked possible goals 252, where N is a predefined integer (e.g., 1, 3, 5, 10, etc.).

[0045] In some implementations, the emerging content engine 250 establishes an initial / end state 256 of the SR set based on the goal 254. In some implementations, the initial / end state 256 indicates the placement (e.g., location) of various character / equipment representations within the SR set. In some implementations, the SR set is associated with a duration (e.g., seconds, minutes, hours, or days). For example, the SR set is scheduled to last for the duration. In such implementations, the initial / end state 256 indicates the placement of various character / equipment representations at / towards the beginning of the duration and / or at / towards the end of the duration. In some implementations, the initial / end state 256 indicates the environmental conditions of the SR set at / towards the beginning / end of the duration associated with the SR set.

[0046] In some implementations, in addition to the character / equipment / environment engine, the emerging content engine 250 provides the goal 254 to the display engine 260. In some implementations, the display engine 260 determines whether the action 210 provided by the character / equipment / environment engine is consistent with the goal 254 provided by the emerging content engine 250. For example, the display engine 260 determines whether the action 210 satisfies the goal 254. In other words, in some implementations, the display engine 260 determines whether the action 210 improves the likelihood of completing / achieving the goal 254. In some implementations, if the action 210 satisfies the goal 254, the display engine 260 modifies the SR scenery according to the action 210. In some implementations, if the action 210 does not satisfy the goal 254, the display engine 260 forgoes modifying the SR scenery according to the action 210.

[0047] Figure 3A is a block diagram of an exemplary emerging content engine 300 according to some implementations. In some implementations, the emerging content engine 300 implements Figure 2 In various implementations, the emerging content engine 300 generates goals 254 for various goal achievers instantiated in the SR scene (e.g., character / equipment representations, such as Figure 1A and Figure 1BIn some implementations, at least some of the targets 254 are used to affect an environment engine (e.g., Figure 2 The environment engine 208e shown).

[0048] In various implementations, the emerging content engine 300 includes a neural network system 310 (hereinafter referred to as "neural network 310" for simplicity), a neural network training system 330 (hereinafter referred to as "training module 330" for simplicity) that trains (e.g., configures) the neural network 310, and an eraser 350 that provides possible targets 360 to the neural network 310. In various implementations, the neural network 310 generates a target 254 (e.g., Figure 2 Target 254a of boy action figure representation 108a, target 254b of girl action figure representation 108b, target 254c of robot representation 108c, target 254d of drone representation 108d, and / or environment target 254e) are shown.

[0049] In some implementations, neural network 310 includes a long short-term memory (LSTM) recurrent neural network (RNN). In various implementations, neural network 310 generates target 254 based on a function of possible targets 360. For example, in some implementations, neural network 310 generates target 254 by selecting a portion of possible targets 360. In some implementations, neural network 310 generates target 254 so that target 254 is within a certain similarity to possible targets 360.

[0050] In various implementations, the neural network 310 generates the target 254 based on the context information 258 characterizing the SR scene. Figure 3A As shown, in some implementations, context information 258 indicates instantiation of equipment representation 340, instantiation of character representation 342, user-specific scene / environment information 344, and / or instantiation of actions 210 from the goal achiever engine.

[0051] In some implementations, the neural network 310 generates the target 254 based on the instantiated equipment representation 340. In some implementations, the instantiated equipment representation 340 refers to the equipment representation located in the SR scene. Figure 1A and Figure 1B , the instantiated equipment representation 340 includes the robot representation 108c and the drone representation 108d in the SR scene 106. In some implementations, the goal 254 includes interacting with one or more of the instantiated equipment representations 340. For example, see Figure 1A and 1BIn some implementations, one of the goals 254a for the boy action figure representation 108a includes destroying the robot representation 108c, and one of the goals 254b for the girl action figure representation 108b includes protecting the robot representation 108c.

[0052] In some implementations, neural network 310 generates goals 254 for each character representation based on instantiated equipment representation 340. For example, see Figure 1A and 1B If the SR set 106 includes the robot representation 108 c , one of the goals 254 a of the boy action figure representation 108 a includes destroying the robot representation 108 c . However, if the SR set 106 does not include the robot representation 108 c , one of the goals 254 a of the boy action figure representation 108 a includes maintaining peace within the SR set 106 .

[0053] In some implementations, the neural network 310 generates the target 254 for each equipment representation based on other equipment representations instantiated in the SR scene. Figure 1A and 1B If the SR scenery 106 includes the robot representation 108 c , one of the objectives 254 d of the drone representation 108 d includes protecting the robot representation 108 c . However, if the SR scenery 106 does not include the robot representation 108 c , the objective 254 d of the drone representation 108 d includes hovering at the center of the SR scenery 106 .

[0054] In some implementations, the neural network 310 generates the target 254 based on the instantiated character representation 342. In some implementations, the instantiated character representation 342 refers to a character representation in the SR scene. Figure 1A and 1B , the instantiated character representations 342 include the boy action figure representation 108a and the girl action figure representation 108b in the SR scenery 106. In some implementations, the goal 254 includes interacting with one or more of the instantiated character representations 342. For example, see Figure 1A and 1B In some implementations, one of the goals 254d of the drone representation 108d includes following the girl action figure representation 108b. Similarly, in some implementations, one of the goals 254c of the robot representation 108c includes avoiding the boy action figure representation 108a.

[0055] In some implementations, the neural network 310 generates the target 254 for each character representation based on other character representations instantiated in the SR scene. Figure 1A and 1BIf the SR set 106 includes the boy action figure representation 108 a , then the goal 254 b of the girl action figure representation 108 b includes catching the boy action figure representation 108 a . However, if the SR set 106 does not include the boy action figure representation 108 a , then the goal 254 b of the girl action figure representation 108 b includes flying around the SR set 106 .

[0056] In some implementations, the neural network 310 generates the target 254 for each equipment representation based on the character representation instantiated in the SR scene. Figure 1A and Figure 1B If the SR set 106 includes the girl action figure representation 108 b , one of the objectives 254 d of the drone representation 108 d includes following the girl action figure representation 108 b . However, if the SR set 106 does not include the girl action figure representation 108 b , the objective 254 d of the drone representation 108 d includes hovering at the center of the SR set 106 .

[0057] In some implementations, the neural network 310 generates the goal 254 based on the user-specific scene / environment information 344. In some implementations, the user-specific scene / environment information 344 indicates the boundaries of the SR set. In such implementations, the neural network 310 generates the goal 254 so that the goal 254 can be satisfied (e.g., achieved) within the boundaries of the SR set. In some implementations, the neural network 310 generates the goal 254 by selecting a portion of the possible goals 252 that is more appropriate for the environment indicated by the user-specific scene / environment information 344. For example, when the user-specific scene / environment information 344 indicates that the sky within the SR set is clear, the neural network 310 sets one of the goals 254d of the drone representation 108d to hover above the boy action figure representation 108a. In some implementations, the neural network 310 abandons selecting a portion of the possible goals 252 that is not appropriate for the environment indicated by the user-specific scene / environment information 344. For example, when the user-specific scene / environment information 344 indicates high winds within the SR scenery, the neural network 310 abandons the hovering goal of the drone representation 108d.

[0058] In some implementations, the neural network 310 generates the goals 254 based on the actions 210 provided by the various goal achiever engines. In some implementations, the neural network 310 generates the goals 254 so that the goals 254 can be satisfied (e.g., achieved) given the actions 210 provided by the goal achiever engines. In some implementations, the neural network 310 evaluates possible goals 360 relative to the actions 210. In such implementations, the neural network 310 generates the goals 360 by selecting possible goals 360 that can be satisfied by the actions 210 and deselecting possible goals 360 that cannot be satisfied by the actions 210.

[0059] In various implementations, the training module 330 trains the neural network 310. In some implementations, the training module 330 provides the neural network 310 with neural network (NN) parameters 312. In some implementations, the neural network 310 includes a neuron model, and the neural network parameters 312 represent weights of the model. In some implementations, the training module 330 generates (e.g., initializes or initiates) the neural network parameters 312, and refines (e.g., adjusts) the neural network parameters 312 based on the target 254 generated by the neural network 310.

[0060] In some implementations, the training module 330 includes a reward function 332 that utilizes reinforcement learning to train the neural network 310. In some implementations, the reward function 332 assigns positive rewards to desired targets 254 and assigns negative rewards to undesired targets 254. In some implementations, during the training phase, the training module 330 compares the target 254 to the validation data including the verified targets. In such implementations, if the target 254 is within a certain similarity to the verified target, the training module 330 stops training the neural network 310. However, if the target 254 is not within a certain similarity to the verified target, the training module 330 continues training the neural network 310. In various implementations, the training module 330 updates the neural network parameters 312 during / after training.

[0061] In various implementations, the eraser 350 erases content 352 to identify possible targets 360. In some implementations, the content 352 includes movies, video games, comics, novels, and fan-created content such as blogs and reviews. In some implementations, the eraser 350 utilizes various methods, systems, and / or devices associated with content erasure to erase the content 352. For example, in some implementations, the eraser 350 utilizes one or more of text pattern matching, HTML (Hypertext Markup Language) parsing, DOM (Document Object Model) parsing, image processing, and audio analysis to erase the content 352 and identify possible targets 360.

[0062] In some implementations, the goal achiever is associated with a representation type 362, and the neural network 310 generates the goal 254 based on the representation type 362 associated with the goal achiever. In some implementations, the representation type 362 indicates physical characteristics of the goal achiever (e.g., color, material type, texture, etc.). In such implementations, the neural network 310 generates the goal 254 based on the physical characteristics of the goal achiever. In some implementations, the representation type 362 indicates behavioral characteristics of the goal achiever (e.g., aggressiveness, friendliness, etc.). In such implementations, the neural network 310 generates the goal 254 based on the behavioral characteristics of the goal achiever. For example, the neural network 310 generates a destructive goal for the boy action figure representation 108a in response to the behavioral characteristics including aggressiveness. In some implementations, the representation type 362 indicates functional and / or performance characteristics of the goal achiever (e.g., strength, speed, flexibility, etc.). In such implementations, the neural network 310 generates the goal 254 based on the functional characteristics of the goal achiever. For example, the neural network 310 generates a always moving target for the girl action figure representation 108b in response to the behavioral characteristics including speed. In some implementations, the representation type 362 is determined based on user input. In some implementations, the representation type 362 is determined based on a combination of rules.

[0063] In some implementations, neural network 310 generates goal 254 based on specified goal 364. In some implementations, specified goal 364 is provided by an entity that controls (e.g., owns or creates) the fictional material from which the character / equipment is derived. For example, in some implementations, specified goal 364 is provided by a movie producer, a video game producer, a novelist, etc. In some implementations, possible goals 360 include specified goal 364. Thus, in some implementations, neural network 310 generates goal 254 by selecting a portion of specified goal 364.

[0064] In some implementations, possible goals 360 for the goal achiever are limited by a limiter 370. In some implementations, limiter 370 limits neural network 310 from selecting a portion of possible goals 360. In some implementations, limiter 370 is controlled by an entity that owns (e.g., controls) the fictional material from which the character / equipment is derived. For example, in some implementations, limiter 370 is controlled by a movie producer, a video game producer, a novelist, etc. In some implementations, limiter 370 and neural network 310 are controlled / operated by different entities. In some implementations, limiter 370 limits neural network 310 from generating goals that violate standards defined by the entity controlling the fictional material.

[0065] Figure 3B is a block diagram of a neural network 310 according to some specific implementations. Figure 3B In the example of , the neural network 310 includes an input layer 320, a first hidden layer 322, a second hidden layer 324, a classification layer 326, and a target selection module 328. Although the neural network 310 includes two hidden layers as an example, a person of ordinary skill in the art will understand from this disclosure that in various specific implementations, there are one or more additional hidden layers. Adding additional hidden layers increases computational complexity and memory requirements, but can improve performance for certain applications.

[0066] In various implementations, the input layer 320 receives various inputs. In some implementations, the input layer 320 receives the context information 258 as input. Figure 3B In the example of , the input layer 320 receives input indicating that the equipment 340 is instantiated, the role 342, the user-specific scene / environment information 344, and / or the action 210 from the goal achiever engine is instantiated. In some specific implementations, the neural network 310 includes a feature extraction module (not shown) that generates a feature stream (e.g., a feature vector) based on the instantiated equipment 340, the instantiated role 342, the user-specific scene / environment information 344, and / or the action 210. In such embodiments, the feature extraction module provides the feature stream to the input layer 320. Therefore, in some specific implementations, the input layer 320 receives a feature stream as a function of the instantiated equipment 340, the instantiated role 342, the user-specific scene / environment information 344, and the action 210. In various specific implementations, the input layer 320 includes a plurality of LSTM logic elements 320a, which are also referred to as neurons or neuron models by those of ordinary skill in the art. In some such specific implementations, the input matrix from the feature to the LSTM logic element 320a includes a rectangular matrix. The size of this matrix is ​​a function of the number of features contained in the feature stream.

[0067] In some implementations, the first hidden layer 322 includes a plurality of LSTM logic elements 322a. Figure 3B As shown in the example of , the first hidden layer 322 receives its input from the input layer 320.

[0068] In some implementations, the second hidden layer 324 includes a plurality of LSTM logic elements 324a. In some implementations, the number of LSTM logic elements 324a is the same as or similar to the number of LSTM logic elements 320a in the input layer 320 or the number of LSTM logic elements 322a in the first hidden layer 322. Figure 3B As shown in the example of , the second hidden layer 324 receives its input from the first hidden layer 322. Additionally or alternatively, in some embodiments, the second hidden layer 324 receives its input from the input layer 320.

[0069] In some implementations, the classification layer 326 includes a plurality of LSTM logic elements 326a. In some implementations, the number of LSTM logic elements 326a is the same or similar to the number of LSTM logic elements 320a in the input layer 320, the number of LSTM logic elements 322a in the first hidden layer 322, or the number of LSTM logic elements 324a in the second hidden layer 324. In some implementations, the classification layer 326 includes an implementation of a polynomial logic function (e.g., a flexible maximum function) that produces a number of outputs approximately equal to the number of possible actions 360. In some implementations, each output includes a probability or confidence measure of a corresponding goal being satisfied by the action 210. In some implementations, the output does not include goals that have been excluded by the operation of the limiter 370.

[0070] In some implementations, target selection module 328 generates target 254 by selecting the top N target candidates provided by classification layer 326. In some implementations, the top N target candidates may be satisfied by action 210. In some implementations, target selection module 328 provides target 254 to the rendering and display pipeline (e.g., Figure 2 In some implementations, the goal selection module 328 provides the goal 254 to one or more goal achiever engines (e.g., Figure 2 8 (shown as boy action figure engine 208a, girl action figure engine 208b, robot equipment engine 208c, drone equipment engine 208d, and / or environment engine 208e).

[0071] Figure 4 4 is a flowchart representation of a method 400 for generating and presenting a goal achiever in a synthetic reality (SR) scene. In various implementations, the method 400 is performed by a processor having a non-transitory memory and one or more processors (e.g., Figure 1A The controller 102, the electronic device 103 and / or Figure 1B The method 400 is performed by a device (such as the HMD 104 shown). In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 400 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory). In short, in some implementations, the method 400 includes instantiating a goal achiever into an SR scene, obtaining context information of the SR scene, generating a goal for the goal achiever, setting an environmental condition of the SR scene, establishing an initial condition for the goal achiever based on the goal, and modifying the goal achiever based on the goal.

[0072] As shown in block 410, in various implementations, method 400 includes instantiating a goal achiever into the SR scenery (e.g., instantiating the boy action figure representation 108a, the girl action figure representation 108b, the robot representation 108c, and / or the drone representation 108d into the Figure 1A and 1B In some implementations, the goal achiever consists of a set of predefined goals (e.g., Figure 3A The possible target 360 shown is characterized by a set of visual rendering properties.

[0073] As indicated at block 420, in various implementations, method 400 includes obtaining context information characterizing the SR scene (e.g., Figures 2 to 3B Context information 258 is shown. In some implementations, method 400 includes receiving context information (eg, from a user).

[0074] As shown in block 430, in various implementations, method 400 includes generating a goal for a goal achiever based on a function of the set of predefined goals, context information, and a set of predefined actions for the goal achiever. Figure 2 , method 400 includes generating a goal 254 based on possible goals 252 , context information 258 , and actions 210 .

[0075] As shown in block 440, in various implementations, method 400 includes setting environmental conditions for the SR scene based on the goal of the goal achiever. Figure 2 , method 400 includes generating environmental targets 254e (eg, environmental conditions).

[0076] As shown in block 450, in various implementations, method 400 includes establishing an initial condition and a set of current actions for a goal achiever based on the goal of the goal achiever. Figure 2 , method 400 includes establishing initial / end states 256 of various goal achievers (eg, character representations, equipment representations, and / or environments).

[0077] As shown in block 460, in various implementations, method 400 includes modifying the goal achiever based on the goal. Figure 2 In some implementations, method 400 includes providing goal 254 to display engine 260 and / or one or more goal achiever engines.

[0078] Figure 5 is based on some specific implementation (for example, using Figure 1A The controller 102 and / or the electronic device 103 and / or Figure 1B104 is a block diagram of an electronic device 500 (enable) with one or more components of the HMD 104 shown. Although some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the electronic device 500 includes one or more processing units (CPUs) 501, a network interface 502, a programming interface 503, a memory 504, and one or more communication buses 505 for interconnecting these and various other components.

[0079] In some specific implementations, the network interface 502 is configured to communicate with one or more local devices (e.g., via near field communication or a local network) and / or one or more remote devices (e.g., WiFi, Ethernet, etc.). In some embodiments, one or more communication buses 505 include circuits for interconnecting and controlling communication between system components. The memory 504 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 504 optionally includes one or more storage devices remotely located from one or more CPUs 501. The memory 504 includes a non-transitory computer-readable storage medium.

[0080] In some implementations, the memory 504 or the non-transitory computer-readable storage medium of the memory 504 stores the following programs, modules, and data structures, or a subset thereof, including the optional operating system 506, the neural network 310, the training module 330, the eraser 350, and the possible goals 360. As described herein, the neural network 310 is associated with the neural network parameters 312. As described herein, the training module 330 includes a reward function 332 for training (e.g., configuring) the neural network 310 (e.g., by determining the neural network parameters 312). As described herein, the neural network 310 determines the goal of the goal achiever in the SR scene and / or the goal of the environment of the SR scene (e.g., Figures 2 to 3B In some implementations, the memory 504 includes FIG. 8A to FIG. 8C At least part of the emerging content architecture in .

[0081] In some implementations, the electronic device 500 optionally includes one or more input devices, such as an eye tracker, a touch-sensitive surface, a keypad or keyboard, an accelerometer, a gyroscope, an inertial measurement unit (IMU), a grip sensor, one or more microphones, one or more buttons, one or more internally facing image sensors, one or more externally facing image sensors, one or more depth sensors, one or more physiological sensors (e.g., a heartbeat sensor, a glucose level detector, etc.), one or more environmental sensors (e.g., a barometer, a humidity sensor, a thermometer, an ambient light detector, etc.), etc. In some implementations, the electronic device 500 optionally includes one or more output / feedback devices, such as a haptic engine, a skin shear engine, one or more displays, one or more speakers, etc.

[0082] Fig. 6A 6 is a block diagram of a conditionally relevant SR content thread 600 according to some implementations. Although some specific features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the conditionally relevant SR content thread 600 corresponds to a relevant content thread (e.g., a story, narrative, etc.) associated with a movie, television series, novel, stage play, or event such as a sporting event, ceremony, coronation, opening ceremony, concert, opera, theater performance, war, or other large event.

[0083] According to some implementations, conditionally-related SR content thread 600 includes multiple content threads (sometimes also referred to herein as "stories" or "story nodes" for simplicity) that are linked together in a tree, grid, or network of interrelated stories. In some implementations, each of the multiple stories within conditionally-related SR content thread 600 corresponds to a particular viewpoint of an event. In some implementations, each of the multiple stories within conditionally-related SR content thread 600 is based on source assets / materials, including, for example, plans of an event (such as battle plans or battle plans), ground truth of an event (such as battle conditions and outcomes), historical records and books, movies, video games, novels, and the like.

[0084] According to some implementations, the root of the conditionally related SR content thread 600 includes a super macro story 610, which in turn is associated with one or more macro stories 620a, 620b, ... 620n (sometimes collectively referred to herein as macro stories 620). Fig. 6AAs shown, in some implementations, each macro story 620 is associated with one or more sub-macro stories. For example, macro story 620a is associated with sub-macro stories 632a, 632b, ..., 632n (sometimes collectively referred to herein as sub-macro stories 632), and macro story 620n is associated with sub-macro stories 634a, 634b, ..., 634n (sometimes collectively referred to herein as sub-macro stories 634).

[0085] like Fig. 6A As shown, in some specific implementations, each sub-macro story is associated with one or more micro stories. For example, sub-macro story 634a is associated with micro story 640a ... 640n (sometimes collectively referred to as micro story 640 in this article). In some specific implementations, each of micro story 640 is associated with one or more sub-micro stories, and each of the sub-micro stories is then associated with one or more super micro stories. For example, as shown in Figure 6, micro story 640a is associated with sub-micro story 650, and sub-micro story 650 is associated with super micro story 660.

[0086] For example, super macro story 610 corresponds to an overall story associated with a particular battle in a war in the historical record. Continuing with this example, macro story 620 corresponds to the various military forces of a country involved in the particular battle. Thus, in this example, macro story 620n corresponds to naval forces. Continuing with this example, sub-macro story 634 corresponds to various ships. Thus, in this example, sub-macro story 634a corresponds to a particular transport ship.

[0087] like Fig. 6A As shown, there is a relationship 635 between the sub-macro story 632n corresponding to the aircraft squadron and the sub-macro story 634a. In this example, the aircraft squadron provides air support and defense for a particular transport ship (as well as other transport ships).

[0088] Continuing with this example, micro-story 640 corresponds to a plurality of amphibious landing craft transported by a particular transport ship. Continuing with this example, sub-micro-story 650 corresponds to a platoon of infantry or marines assigned to a corresponding amphibious landing craft among the plurality of amphibious landing craft transported by the particular transport ship. Continuing with this example, super-micro-story 660 corresponds to a particular infantry or marine assigned to the corresponding amphibious landing craft.

[0089] In some implementations, the user is presented with SR content associated with a third-person omniscient view of the super macro story 610 (e.g., a particular campaign). The user can switch to another viewpoint within the conditionally relevant SR content thread 600 to view SR content associated with a selected viewpoint within the conditionally relevant SR content thread 600, such as the perspective of a naval unit associated with the macro story 620n, the perspective of a particular transport ship associated with the sub-macro story 634a, the perspective of a corresponding amphibious landing craft 640a, the perspective of an infantry platoon or marines associated with the sub-micro story 650, or the perspective of a particular infantryman or marine assigned to a corresponding amphibious landing craft associated with the super micro story 660.

[0090] According to some implementations, a user may remove a node of the conditionally relevant SR content thread 600 in order to view a simulation of events excluding the particular node and associated child nodes. According to some implementations, a user may add a node to the conditionally relevant SR content thread 600 in order to view a simulation of events including the particular node and associated child nodes.

[0091] Figure 6B 675 is a block diagram of a timeline 675 associated with an object enabler (OE) package according to some implementations. Although some specific features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, Figure 6B As shown, timeline 675 illustrates the state of an OE package from the perspective of the lowest level OE 684 at multiple time periods or points in time T0, T1, T2, and T3.

[0092] In some implementations, an OE package includes multiple conditional, associated, related, associated, or related OEs that are packaged or nested based on contextual information. In some implementations, each OE corresponds to a role within a synthetic reality (SR) environment. For example, an OE package corresponds to a set of related roles or entities, such as a transport ship that includes N amphibious landing craft, each with M marines. In this example, the lowest level OE is a single marine that is packaged / nested within its company, which in turn is packaged / nested within an amphibious landing craft. Continuing with this example, the amphibious landing craft is packaged / nested within the transport ship.

[0093] In some implementations, the first OE is associated with the second OE when the first OE is encapsulated within the second OE. In some implementations, the first OE is related to the second OE when the first OE is encapsulated within the second OE. In some implementations, the first OE is associated with the second OE when the first OE is encapsulated within the second OE. In some implementations, the first OE is related to the second OE when the first OE is encapsulated within the second OE. In other words, because the first OE is encapsulated within the second OE, the first OE is set with a target and performs actions consistent with the context and attributes of the second OE.

[0094] like Figure 6B As shown, at time T0, OE package 685a includes OE 684 packaged / nested within OE 682, which in turn is packaged / nested within OE 680. For example, at time T0, while approaching a beach for an amphibious landing, OE package 685a includes OE 684 (e.g., a single Marine) packaged / nested within OE 682 (e.g., a company of Marines), which in turn is packaged / nested within OE 680 (e.g., an amphibious landing craft).

[0095] In some implementations, when a first OE is encapsulated / nested within a second OE, the first OE is set with a goal based on the context of the second OE in which it is encapsulated, and the first OE then performs actions consistent with that context. For example, a Marine on a transport ship is not given a reconnaissance goal of performing a reconnaissance patrol or a fire suppression goal of attacking a target with a mortar while on the transport ship. In some implementations, a lower-level OE performs actions consistent with the higher-level OE in its encapsulation. In some implementations, one or more other lower-level OEs are encapsulated within the first OE. In some implementations, a second OE is encapsulated within one or more other higher-level OEs.

[0096] In some embodiments, the system determines a set of OE packages for each time period of the event. Therefore, the OE package may change as the event progresses, such as by stripping or adding OEs (layers). For example, in transit (T0), a particular campaign package begins with a transport ship -> amphibious assault vehicle -> Marine company -> single Marine. Continuing with the example, when approaching the beach (T1), the OE package changes to amphibious assault vehicle -> Marine company -> single Marine. Continuing with the example, when landing on the beach (T2), the OE package changes again to Marine company -> single Marine. Continuing with the example, when the Marines may find a vehicle and enter the vehicle after arriving at the beach (T3), the OE package changes again, which changes the package to a vehicle -> a subset of the Marine company -> a single Marine.

[0097] like Figure 6B As shown, at time T1, OE package 685b includes OE 684 packaged / nested within OE 682. For example, at time T1, after landing on the beach, OE package 685b includes OE 684 (e.g., a single Marine) packaged / nested within OE 682 (e.g., a company of Marines).

[0098] like Figure 6B As shown, at time T2, OE package 685c includes OE 684. For example, at time T2, after landing on the beach and completing the mission, OE package 685c includes OE 684 (e.g., a single Marine) detached from OE 682.

[0099] like Figure 6B As shown, at time T3, OE package 685d includes OE 684 packaged / nested within OE 686. For example, at time T3, after landing on the beach and arriving at the assembly point, OE package 685d includes OE 684 (e.g., a single Marine) packaged / nested within OE 686 (e.g., an evacuation helicopter or truck).

[0100] Figure 6C 695 is a block diagram of a timeline 695 associated with an object enabler (OE) package according to some implementations. Although some specific features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, timeline 695 is shown with respect to Figure 6B 695 shows the parallel operation of these OEs even after the OE packages have changed through these time periods or time points T0, T1, T2 and T3. In other words, Figure 6C OE 680 and OE 682 stripped from OE packages 685a and 685b are shown continuing to operate in parallel with OE 684 over time.

[0101] 7A to 7C Exemplary SR scenes 700a and 700b are shown according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein.

[0102] Fig. 7A An exemplary SR scene 700a is shown. Fig. 7AAs shown, physical set 706 includes table 702. In this example, a portion of physical set 706 is within field of view 704 of electronic device 103, where field of view 704 is associated with an outward-facing image sensor of electronic device 103 (e.g., a tablet or mobile phone). In other words, the user is viewing table 702 from the side or through the viewing orientation of electronic device 103. As such, the portion of physical set 706 that includes table 702 is displayed on display 705 of electronic device 103 (e.g., a real-time video stream or video pass-through of physical set 706). Fig. 7A As shown, the electronic device 103 displays the SR scene 106 on the display 705, and the SR scene includes SR content superimposed on the table 702 and the wall or composited with the table and the wall.

[0103] like Fig. 7A As shown, for example, the SR content (e.g., associated with a particular campaign) overlaid or superimposed on the table 702 includes a first SR content layer 710a (e.g., an underwater layer) having SR content elements 712 (e.g., a submarine, a submersible amphibious landing craft, a mine, etc.), a second SR content layer 710b (e.g., a surface layer) having SR content elements 714 (e.g., an amphibious landing craft with an infantry platoon or marines, a transport ship, etc.), and a third SR content layer 710c (e.g., an aerial layer) having SR content elements 716 (e.g., an aircraft, a spacecraft, a missile, etc.). For example, as shown in FIG. Fig. 7A As shown, the SR content 720 covered on the wall includes SR content elements 722 (e.g., defensive positions on land, fixed machine guns, etc.). Those skilled in the art will understand that Fig. 7A The number, structure, size, and placement of SR content layers and associated SR content elements in are arbitrary and may vary in various other implementations.

[0104] Figure 7B and Figure 7C An exemplary SR scene 700b is shown. Figure 7B and Figure 7C As shown, for example, user 10 wears an HMD 104 (e.g., AR-enabled glasses) on his / her head that has an optical perspective of a physical scene 706 (e.g., the user's living room). Figure 7B and Figure 7C As another example, user 10 wears an HMD 104 (eg, an SR-enabled headset) on his / her head with video pass-through of a physical setting 706 (eg, the user's living room).

[0105] like Figure 7BAs shown, in state 725 (e.g., associated with time period T1), the HMD 104 overlays or overlays SR content on the table 702 and the wall of the physical scenery 706. In this example, referring to state 725, the SR content overlaid on the table 702 (e.g., associated with a particular campaign) includes a first SR content layer 710a (e.g., an underwater layer) having SR content elements 712 (e.g., a submarine, a submarine amphibious landing craft, a mine, etc.), a second SR content layer 710b (e.g., a surface layer) having SR content elements 714 (e.g., an amphibious landing craft with an infantry platoon or a Marine Corps, a transport ship, etc.), and a third SR content layer 710c (e.g., an aerial layer) having SR content elements 716 (e.g., an aircraft, a spacecraft, a missile, etc.). Figure 7B As shown, with reference to state 725, SR content element 720 covering the front wall includes SR content element 722 (e.g., a defensive position on land, a fixed machine gun, etc.), and SR content element 730 covering the side wall includes surrounding or environmental details. A person of ordinary skill in the art will understand that Figure 7B The number, structure, size, and placement of SR content layers and associated SR content elements in are arbitrary and may vary in various other implementations.

[0106] like Figure 7B As shown, state 725 corresponds to a first SR view of the event, such as a third-person omniscient view of the overall event (e.g., a particular battle). In response to receiving input (e.g., a voice command, a gesture, etc.) from the user 10 selecting the third AR content layer 710c (e.g., the aerial layer), the HMD 104 presents a second SR view of the event associated with the third SR content layer 710c (e.g., the aerial layer).

[0107] like Figure 7C As shown, in state 735 (eg, associated with time period T2), the HMD 104 updates the SR content superimposed or overlaid on the table 702 in response to the selection of the third AR content layer 710c. Figure 7C As shown, in state 735, the HMD 104 superimposes or overlays SR content corresponding to the third SR content layer 710c on the table 702. In this example, referring to state 735, the SR content (e.g., associated with a particular campaign) overlaid on the table 702 includes the third SR content layer 710c (e.g., an aerial layer) having SR content elements 716 (e.g., aircraft, spacecraft, missiles, etc.). Figure 7C , the third SR content layer 710c corresponds to the center of gravity of the squadron of aircraft approaching to attack the defensive position in the battlefield. For another example, in state 735, the user 10 may be presented with the viewpoint of a particular pilot in the squadron of aircraft approaching to attack the defensive position on the battlefield (not shown).

[0108] Fig. 8A 800 according to some implementations. Although some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, related (e.g., encapsulated / nested) object enabler (OE) engines 810a, 810b, 810c, and 810d (sometimes collectively referred to herein as OE engines 810) (e.g., similar to Figure 2 The OE package of the role engine 208a-e) in the target generator 807 performs actions based on the target derived from the target generator 807.

[0109] For example, an OE package includes a highest level OE engine 810a associated with a transport ship, a second level OE engine 810b associated with an amphibious landing craft transported by the transport ship, a third level OE engine 810c associated with an infantry platoon or Marine platoon assigned to the amphibious landing craft, and a lowest level OE engine 810d associated with a specific infantryman or Marine. Thus, the various OE engines within an OE package are related in some way because they correspond to connection nodes within a conditionally related SR content thread associated with an event (e.g., as referenced in FIG. 1 ). Fig. 6A According to some implementations, the emerging content architecture 800 is configured to generate concurrent actions for various OE engines within an OE package to achieve consistent content. In various implementations, one of ordinary skill in the art will appreciate that the emerging content architecture 800 may include any number of OE packages. In some implementations, due to the related nature of OEs and OE packages, the same OE engine may be shared between two or more OE packages.

[0110] According to some implementations, a goal generator 807 (e.g., a neural network or other AI construct) generates a target based on a set of predetermined goals, previous goals, source assets, and / or other information (e.g., similar to Figure 2 The emerging content engine 250 in the event management system generates a target for each OE for each time period. According to some specific implementations, the package manager 825 determines the OE package (e.g., the nesting or layering of related OEs). For example, the package manager 825 determines the OE package based on the connection between the story nodes within the conditionally related SR content thread associated with the event. In some specific implementations, the package manager 825 modifies the OE package over time based on the target, source assets and / or other information (e.g., adding or removing layers of the OE).

[0111] In some implementations, the demultiplexer 845 routes the targets to its corresponding OE engine 810 on a per-OE basis. In some implementations, the OE engine 810 performs actions based on its targets. According to some implementations, the actions over a period of time are captured and provided to the display pipeline 855 for rendering and display so as to present SR content associated with the event to the user (e.g., an SR reconstruction or simulation of a campaign or other event based on source assets and then emerging content). According to some implementations, the actions over a period of time are captured and provided to the OE engine 810 as training feedback 865. According to some implementations, the actions are provided to the target generator 807, which in turn generates updated targets for each OE for each time period (e.g., as described above with reference to Figure 2 ).

[0112] Figure 8B 810d (sometimes collectively referred to herein as OE engines 810) (e.g., similar to Figure 2 The OE package of the role engine 208a-e in performs actions based on the initial goals derived from the source asset 805 at time T0 and based on the updated goals derived from the emerging content engine 250 at time T1 and thereafter.

[0113] According to some implementations, the package manager 825 determines OE packages (e.g., nesting or layering of related OEs). For example, the package manager 825 determines OE packages based on connections between story nodes within conditionally related SR content threads associated with an event. In some implementations, the package manager 825 modifies OE packages (e.g., adds or removes layers of OEs) over time based on targets, source assets, and / or other information.

[0114] For example, an OE package includes a highest level OE engine 810a associated with a transport ship, a second level OE engine 810b associated with an amphibious landing craft transported by the transport ship, a third level OE engine 810c associated with an infantry platoon or Marine platoon assigned to the amphibious landing craft, and a lowest level OE engine 810d associated with a specific infantryman or Marine. Thus, the various OE engines within an OE package are related in some way because they correspond to connection nodes within a conditionally related SR content thread associated with an event (e.g., as referenced in FIG. 1 ). Fig. 6A In some implementations, the emerging content architecture 850 is configured to generate concurrent actions for various OE engines within an OE package to achieve consistent emerging content. In various implementations, one of ordinary skill in the art will appreciate that the emerging content architecture 850 may include any number of OE packages. In some implementations, due to the related nature of OEs and OE packages, the same OE engine may be shared between two or more OE packages.

[0115] According to some implementations, initial goals are generated through the operation of the eraser engine 820 and the goal generator 830 based on the source assets 805 associated with the event (e.g., the plan 810a of the event, the ground truth 810b, and the personal records 810c, etc.). In some implementations, the eraser engine 820 performs parsing and understanding operations on the source assets 805 to generate extracted actions for each OE (e.g., the role identified in the source assets 805) for each time period. For example, in some implementations, the eraser engine 820 extracts actions from the source assets 805. Thereafter, in some implementations, the goal generator 830 (e.g., a neural network or other AI construct) generates initial goals for each OE for each time period.

[0116] According to some implementations, the multiplexer 835 enables one of the initial target or the updated target as input to the OE package. In some implementations, the demultiplexer 845 routes the target to its corresponding OE engine 810 on a per-OE basis. In some implementations, the OE engine 810 performs actions based on its target. According to some implementations, the actions over a period of time are captured and provided to the display pipeline 855 for rendering and display so that the SR content associated with the event (e.g., an SR reconstruction or simulation of a campaign or other event based on the source assets and then emerging content) is presented to the user. According to some implementations, the actions over a period of time are captured and provided to the emerging content engine 250 and the OE engine 810 as training feedback 865. According to some implementations, the actions are provided to the emerging content engine 250, which in turn generates updated targets for each OE for each time period (e.g., as described above with reference to Figure 2 ).

[0117] Figure 8C860d (sometimes collectively referred to herein as OE engines 860) (e.g., similar to Figure 2 The role engine 208a-e) in FIG. 1 performs actions based on the initial goals derived from the source asset 805 at time T0 and based on the updated goals derived from the emerging content engine 250 at time T1 and thereafter.

[0118] According to some implementations, initial goals are generated based on source assets 805 (e.g., movies, TV series, audiobooks, novels, magazine articles, etc.) through the operation of the eraser engine 820 and the goal generator 830. In some implementations, the eraser engine 820 performs parsing and understanding operations on the source assets 805 to generate extracted actions for each OE (e.g., characters identified in the source assets 805) for each time period. For example, in some implementations, the eraser engine 820 extracts actions from the source assets 805. Thereafter, in some implementations, the goal generator 830 (e.g., a neural network or other AI construct) generates initial goals for each OE for each time period.

[0119] According to some implementations, the multiplexer 835 enables one of the initial goal or the updated goal as input to the OE engine 860. In some implementations, the demultiplexer 845 routes the goals to its corresponding OE engine 860 on a per-OE basis. In some implementations, the OE engine 860 performs actions based on its goals. According to some implementations, the actions over a period of time are captured and provided to the display pipeline 855 for rendering and display so that SR content associated with the event (e.g., an SR reconstruction or simulation of a campaign or other event based on source assets and then emerging content) is presented to the user. According to some implementations, the actions over a period of time are captured and provided to the emerging content engine 250 and the OE engine 860 as training feedback 865. According to some implementations, the actions are provided to the emerging content engine 250, which in turn generates updated goals for each OE for each time period (e.g., as described above with reference to Figure 2 ).

[0120] 9A to 9FMultiple SR scenes according to different viewing vectors are shown according to some specific implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary implementations disclosed herein. According to some specific implementations, 9A to 9F The multiple SR scenes shown correspond to predetermined content (e.g., scripts, stage plays, movies, television shows, historical events, fictional stories, live events, etc.). According to some implementations, 9A to 9F The plurality of SR scenes shown correspond to emerging content based at least in part on source assets (e.g., original or seed content), such as movies, television series, video games, historical events, etc., as shown in reference Fig. 8A and Figure 8B According to some specific implementations, 9A to 9F The multiple SR scenes shown correspond to Fig. 6A The conditionally related SR content thread 600 is described.

[0121] Fig.9A An exemplary SR scene 900 is shown presented according to a view vector 902 at a time 904. Fig.9A As shown, SR scene 900 corresponds to a first SR scene (e.g., a driving scene) presented by electronic device 103 or HMD 104 (including vehicle 928). According to some implementations, viewing vector 902 is characterized by perspective 906, field of view (FOV) 908, pose 910, and coordinate set 912. Fig.9A 9, viewpoint 906 corresponds to vehicle 928 within the first SR set, FOV 908 corresponds to the size, shape, focal length, etc. of the field of view associated with vehicle 928, pose 910 corresponds to rotational coordinates associated with FOV 908, and coordinate set 912 corresponds to translational coordinates characterizing a position associated with FOV 908 within the first SR set.

[0122] exist Fig.9A, SR scene 900 includes a navigation timeline 920 having a current time indicator 922 and a plurality of view vector affordances 924A, 924B, 924C, 924D, 924E, and 924F (sometimes collectively referred to herein as view vector affordances 924). According to some implementations, each of view vector affordances 924 is associated with a different view vector for a corresponding time on navigation timeline 920. In some implementations, each of view vector affordances is associated with an OE corresponding to a story node within a conditionally related SR content thread 600. In some implementations, each of view vector affordances is associated with an OE package corresponding to a story node within a conditionally related SR content thread 600.

[0123] For example, the electronic device 103 or HMD 104 detects a first input selecting the look vector affordance 924B of the time 904, such as a voice command, gesture input, touch input, gaze direction input, etc., by a user of the electronic device 103 or HMD 104 via one or more input devices (e.g., a microphone, a gaze tracker, a body pose estimator, a touch-sensitive surface, a button, etc.). Continuing with the example, in response to detecting the first input selecting the look vector affordance 924B, Fig. 9B An exemplary SR scene 930A is shown presented according to a view vector 932A associated with a view vector affordance representation 924B at time 934.

[0124] like Fig. 9B As shown, SR set 930A corresponds to a second SR set (e.g., a construction set) presented by electronic device 103 or HMD 104, including a plurality of workers 944A, 944B, and 944C (sometimes collectively referred to as workers 944) and a user's hand holding an SR item 948 within the second SR set, represented as worker 946. In some implementations, the user's hand is mapped to the SR hand of an SR avatar holding the SR item 948 associated with worker 946. In some implementations, the user's hand corresponds to a video pass-through or optical see-through of his / her hand holding the SR item 948 associated with worker 946.

[0125] According to some implementations, viewing vector 932A is characterized by viewing angle 936A, FOV 938A, pose 940A, and coordinate set 942A. Fig. 9B, view angle 936A corresponds to a worker within the second SR set, FOV 938A corresponds to the size, shape, focal length, right-side-up or inverted, etc., of the field of view associated with worker 946, pose 910 corresponds to the rotation coordinates (e.g., heading, angle, and pitch) associated with FOV 938A, and coordinate set 942A corresponds to the position associated with FOV 938A.

[0126] For example, electronic device 103 or HMD 104 detects a second input associated with modifying pose 940A (e.g., rotation coordinates) associated with FOV 938A at time 934, such as a voice command, gesture input, touch input, gaze direction input, etc. from a user of electronic device 103 or HMD 104. For example, the second input corresponds to a gesture modification affordance ( Fig. 9B (not shown).

[0127] In some implementations, the SR scene 930A includes a FOV modification affordance (or a set of controls) provided to modify the FOV associated with the viewing vector 932A. In some implementations, the SR scene 930A includes a pose modification affordance (or a set of controls) provided to change the pose (e.g., rotation coordinates) associated with the viewing vector 932A. In some implementations, the SR scene 930A includes a coordinate modification affordance (or a set of controls) provided to change the coordinates (e.g., translation coordinates) associated with the viewing vector 932A. In some implementations, the SR scene 930A includes a perspective modification affordance (or a set of controls) provided to change the perspective (e.g., from the perspective of worker 946 to the perspective of worker 944A or the perspective of the foreman associated with the center of gravity of worker 944 and worker 946). In some implementations, the SR scene 930A includes a perspective modification affordance (or a set of controls) provided to restore to a previous perspective (e.g., at a current time (e.g., time 934)). Fig.9A In some implementations, the SR scene 930A includes a set of playback controls (e.g., play, pause, rewind, fast forward, etc.).

[0128] Continuing with the above example, in response to detecting a second input associated with changing pose 940A, Fig. 9C An exemplary SR scene 930B is shown rendered according to a view vector 932A' at time 934+1. Fig. 9C The time 934+1 and Fig. 9B The times 934 in FIG. 1 are separated by Δ time periods 935A (e.g., X seconds). Fig. 9CAs shown, SR scene 930B corresponds to a second SR scene presented by the electronic device 103 or HMD 104, including a construction item 949, a plurality of workers 944, and a user's hand represented as worker 946 holding an SR item 948 within the second SR scene.

[0129] According to some implementations, the viewing vector 932A′ is determined by the angle of view 936A, the FOV 938A, the pose 940A′ (e.g., relative to Fig. 9B The changed pose) and coordinate set 942A are represented. Fig. 9C , viewing vector 932A' is still associated with viewpoint 936A (eg, worker 946), FOV 938A, and coordinate set 942A, but viewing vector 932A' is associated with changed pose 940A'. For example, worker 946's head or body has been rotated counterclockwise.

[0130] For example, the electronic device 103 or HMD 104 detects a third input associated with changing the viewing angle 936A at time 934+1, such as a voice command, gesture input, touch input, gaze direction input, etc. from a user of the electronic device 103 or HMD 104, thereby, for example, modifying the affordance ( Fig. 9C Continuing with this example, in response to detecting a third input associated with changing the viewing angle 936A, Fig.9D An exemplary SR scene 930C is shown rendered according to a view vector 932B at time 934+2. Fig.9D The time in 934+2 and Fig. 9B The time 934 in FIG. 1 is separated by a Δ time period 935B (e.g., Y seconds), where the Δ time period 935B is greater than the Δ time period 935A. Fig.9D As shown, the SR scene 930C corresponds to a second SR scene presented by the electronic device 103 or the HMD 104, including a construction item 949, a plurality of workers 944, and a worker 946 holding an SR item 948 within the second SR scene.

[0131] According to some implementations, the viewing vector 932B is characterized by a viewing angle 936B, a FOV 938B, a pose 940B, and a coordinate set 942B. Fig.9D , viewing vector 932B is associated with viewing angle 936B (eg, an omniscient view of the second SR scene), which is Fig. 9B and Fig. 9C 936A (eg, the view of worker 946 within the second SR set).

[0132] For example, electronic device 103 or HMD 104 detects a fourth input selecting the look vector affordance representation 924E at time 934+2, such as a voice command, gesture input, touch input, gaze direction input, etc. from a user of electronic device 103 or HMD 104. Continuing with this example, in response to detecting the fourth input selecting the look vector affordance representation 924E, Fig.9E An exemplary SR scene 930D ​​presented according to view vector 932B at time 934+3 is shown, including a view vector menu 950 having a plurality of selectable view vector affordance options 952A and 952B for times associated with view vector affordance 924E. Fig.9E The time in 934+3 and Fig. 9B 934 in are separated by a Δ time period 935C (e.g., Z seconds), where Δ time period 935C is greater than Δ time periods 935A and 935B. In some implementations, each of the selectable view vector affordance options 952A and 952B within view vector menu 950 is associated with a different OE corresponding to a story node within the conditionally related SR content thread 600. In some implementations, each of the selectable view vector affordance options 952A and 952B within view vector menu 950 is associated with a different OE package corresponding to a story node within the conditionally related SR content thread 600.

[0133] For example, the electronic device 103 or HMD 104 detects a fifth input of the look vector affordance representing option 952A at selection time 934+3, such as a voice command, gesture input, touch input, gaze direction input, etc. from a user of the electronic device 103 or HMD 104. Continuing with this example, in response to detecting the fifth input of selecting the look vector affordance representing option 952A, Fig.9F An exemplary SR scene 970 rendered according to a view vector 972 at time 974 is shown.

[0134] like Fig.9F As shown, SR scene 970 corresponds to a third SR scene presented by electronic device 103 or HMD 104, including a plurality of buildings 984A, 984B, 984C, and 984D (sometimes collectively referred to as buildings 984) and a plurality of vehicles 986 within the third SR scene. According to some implementations, viewing vector 972 is characterized by viewing angle 976, FOV 978, pose 980, and coordinate set 982. Fig.9F, perspective 976 corresponds to a view associated with a commander of multiple vehicles 986, FOV 978 corresponds to the size, shape, focal length, right-side-up or inverted, etc., of the field of view associated with the commander's field of view, pose 980 corresponds to rotation coordinates (e.g., heading, angle, and pitch) associated with FOV 978, and coordinate set 982 corresponds to a position associated with FOV 978.

[0135] Fig.10 1 is a flowchart representation of a method for changing a view vector within an SR scene according to some implementations. In various implementations, the method 1000 is performed by a processor (e.g., a processor) having a non-volatile memory and one or more processors coupled to the non-volatile memory. Figure 1A The controller 102, the electronic device 103 and / or Figure 1B In some implementations, the method 1000 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1000 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0136] As indicated at block 1010, method 1000 includes presenting a synthetic reality (SR) scene associated with predetermined content according to a first viewing vector at a first time, wherein the first viewing vector represents a display of the predetermined content at the first time. Fig.9A , the electronic device 103 or the HMD 104 presents a SR scene 900 according to a view vector 902 at a time 904, wherein the SR scene 900 corresponds to a first SR scene (eg, a driving scene).

[0137] For example, the predetermined content corresponds to a script, a stage play, a movie, a television show, a historical event, a fictional story, a live event, etc. In some implementations, the predetermined content is replaced with emerging content that is based at least in part on a source asset (e.g., an initial or seed content), such as a movie, a television show, a video game, a historical event, etc., as referenced. Fig. 8A and Figure 8B In some embodiments, the predetermined content is replaced by Fig. 6A Conditionally related SR content as described in thread 600.

[0138] In some implementations, the plurality of view vector affordances correspond to predetermined view vectors. For example, the predetermined view vectors correspond to times and perspectives associated with the most important events in the predetermined content based on crowdsourced information (e.g., frequency or volume of favorites, likes, social media impressions, etc.). In another example, the predetermined view vectors correspond to times and perspectives associated with the most important events in the predetermined content manually selected by a content creator, historian, director, etc.

[0139] In some implementations, the plurality of view vector affordances correspond to view vectors determined based on a set of filters. For example, the set of filters is determined based on a user profile associated with a user, a set of questions answered by the user, etc. For example, a plurality of view vectors associated with machinery within predetermined content are selected based on activation of a machinery filter. For another example, a plurality of view vectors associated with naval training within predetermined content are selected based on activation of a naval trainee filter. For another example, a plurality of view vectors associated with weather phenomena within predetermined content are selected based on activation of a measurement filter.

[0140] In some implementations, as shown in block 1010a, the first viewing vector is characterized at least in part by a first field of view (FOV), a first pose, and a first set of SR coordinates within the SR scene. Fig.9A , the electronic device 103 or HMD 104 presents a SR scene 900 according to a view vector 902 characterized by a perspective 906, a FOV 908, a pose 910, and a coordinate set 912.

[0141] In some implementations, the FOV corresponds to parameters that define the shape, size, focal length, etc. of the field of view. In some implementations, the pose corresponds to a set of rotational coordinates associated with the FOV (e.g., pitch, roll, and yaw). In some implementations, the pose indicates the angle of the FOV and whether the FOV is right-side up or inverted. In some implementations, the SR coordinate set corresponds to a set of translational coordinates within the SR scene (e.g., global coordinates relative to the SR scene or environment). In some implementations, the perspective corresponds to a unit, point of view (POV), or objective achiever (OE) associated with predetermined content, such as an omniscient POV within a historical battle, a general's POV, an infantry unit's POV, an individual soldier's POV, etc.

[0142] In some implementations, the first viewing vector is associated with a corresponding viewing vector affordance. In some implementations, the first viewing vector is not associated with a viewing vector affordance in a plurality of viewing vector affordances. For example, the user changes viewing angles multiple times before the first time. For example, Fig.9A, view vector 902 is not associated with a view vector affordance in multiple view vector affordances 924 .

[0143] As shown in block 1012, method 1000 includes presenting a navigation timeline of predetermined content, the navigation timeline including a current time indicator at a first time and a plurality of view vector affordances, each view vector affordance being associated with a different view vector at a corresponding time on the navigation timeline, wherein each view vector characterizes how the SR content is displayed at the corresponding time. In one example, the navigation timeline is overlaid on the SR scenery. In another example, the navigation timeline is presented as part of a separate control interface. For example, in Fig.9A , the electronic device 103 or HMD 104 presents a SR scene 900, including a navigation timeline 920 having a current time indicator 922 and a plurality of view vector affordances 924. In some implementations, each of the view vector affordances is associated with an OE corresponding to a story node within a conditionally related SR content thread 600. In some implementations, each of the view vector affordances is associated with an OE package corresponding to a story node within a conditionally related SR content thread 600.

[0144] As shown in block 1014, method 1000 includes detecting an input selecting a corresponding look vector affordance on a navigation timeline associated with a second time within the predetermined content. In some implementations, the input corresponds to a voice command, a gesture input, a touch input, a gaze direction, etc. For example, referring to Fig.9A , the electronic device 103 or HMD 104 detects the first input of the viewing vector indication 924B at the selection time 904, such as a voice command, gesture input, touch input, gaze direction input, etc., performed by the user of the electronic device 103 or HMD 104 via one or more input devices (e.g., a microphone, a gaze tracker, a body pose estimator, a touch-sensitive surface, a button, etc.).

[0145] In some implementations, the input corresponds to a first input type, and when the user input corresponds to a second input type, a menu of view vector options for that time is presented. In some implementations, the menu is ranked according to the most popular view vectors based on crowdsourced information. In some implementations, the menu is sorted according to a predetermined order. For example, relative to Fig.9A and Fig. 9B , the first input of the selection view vector indication 924B corresponds to the first input type (e.g., a short press, a single click, a first voice command, etc.). Fig.9A and Fig. 9BIn the exemplary sequence of FIG. 1 , detecting a first input associated with a first input type at time 904 causes the electronic device 103 or HMD 104 to change from presenting Fig.9A The SR scene 900 according to the view vector 902 is transformed into a rendering Fig. 9B 930a according to the SR background 932A in FIG. Fig.9D and Fig.9E , the fourth input of the viewing vector affordance 924E corresponds to the second input type (e.g., long press, double click, second voice command, etc.). Fig.9D and Fig.9E In the exemplary sequence in FIG. 9 , detecting a fourth input associated with the second input type at time 934+3 causes the electronic device 103 or HMD 104 to present a view vector menu 950 having multiple selectable view vector affordance options 952A and 952B for the time associated with the view vector affordance 924E.

[0146] As shown at block 1016, method 1000 includes, in response to detecting an input selecting a corresponding view vector affordance on the navigation timeline, updating the SR scenery to present the SR scenery according to a second view vector different from the first view vector, wherein the second view vector represents a display of the predetermined content at the second time. Fig.9A The view vector at time 904 in can represent the first input of 924B, Fig. 9B An exemplary SR scene 930A is shown where the electronic device 103 or HMD 104 presents a view vector 932A associated with a view vector affordance representation 924B as a function of time 934 .

[0147] In some implementations, as shown in block 1016a, the second viewing vector is characterized at least in part by a second FOV, a second pose, a second set of SR coordinates within the SR scene, and a second viewing angle. Fig. 9B , the electronic device 103 or HMD 104 presents a SR scene 930A according to a view vector 932A characterized by a viewing angle 936A, a FOV 938A, a pose 940A, and SR coordinates 942A.

[0148] In some implementations, in response to detecting an input selecting a corresponding view vector affordance on the navigation timeline, the device presents a current time indicator at the second time. Fig.9A The first input of the viewing vector indication 924B of the time 904 in FIG. 1 , the electronic device 103 or the HMD 104 moves the current time indicator 922 to move to Fig. 9B The view vector in can indicate the location near 924B.

[0149] In some implementations, in response to detecting an input selecting a corresponding view vector affordance on the timeline, the device stops displaying the SR scene according to the first view vector. Fig.9A The view vector at time 904 in can represent the first input of 924B, in Fig. 9B , the electronic device 103 or the HMD 104 stops displaying the SR scenery 900 according to the viewing vector 902 and instead displays the SR scenery 930A according to the viewing vector 932A.

[0150] In some implementations, the device further presents a FOV modification affordance provided to modify a current field of view (FOV) associated with the second view vector based on one or more subsequent user inputs. In some implementations, changing the FOV corresponds to increasing or decreasing the field of view (e.g., increasing the field of view from 30 degrees to 180 degrees). In some implementations, changing the FOV corresponds to changing the shape of the field of view (e.g., from an elliptical shape to a rectangular shape). In some implementations, changing the FOV corresponds to changing a focal length of the field of view. In some implementations, changing the FOV corresponds to changing a zoom level of the field of view.

[0151] In some implementations, the device also presents a pose modification affordance that is provided to change a current pose associated with a second viewing vector to a subsequent pose based on one or more subsequent inputs. In some implementations, changing the pose corresponds to changing the roll, pitch, or yaw of a current FOV of a current perspective, such as changing the head tilt or pitch of a unit or OE. For example, changing the pose corresponds to changing the FOV from a soldier's eyes to a FOV from their back. For another example, changing the pose corresponds to changing the FOV from the rear of a vehicle to a FOV from the front of the vehicle. For example, referring to Fig. 9B and Fig. 9C In the sequence, in response to detecting and changing Fig. 9B The electronic device 103 or the HMD 104 presents the second input associated with the posture 940A according to Fig. 9C Continuing with the exemplary sequence, the view vector 932A' is composed of the view angle 936A, the FOV 938A, the pose 940A' (e.g., relative to Fig. 9B The changed pose) and coordinate set 942A are represented. Fig. 9C , viewing vector 932A' is still associated with viewpoint 936A (eg, worker 946), FOV 938A, and coordinate set 942A, but viewing vector 932A' is associated with changed pose 940A'. For example, worker 946's head or body has been rotated counterclockwise.

[0152] In some implementations, the device further presents an SR coordinate modification affordance provided to change a current SR coordinate set associated with the second view vector to a subsequent SR coordinate set within the SR scenery based on one or more subsequent inputs. In some implementations, changing the SR coordinate set corresponds to walking, driving, or otherwise moving a current perspective (e.g., current unit or OE). In some implementations, changing the SR coordinate set corresponds to changing from a first-person view of the current perspective (e.g., current unit or OE) to a third-person view.

[0153] In some implementations, the device further presents a perspective modification affordance provided to change from a current second perspective associated with the second viewing vector to a subsequent perspective based on the options menu. In some implementations, changing the current perspective (e.g., a unit or OE) corresponds to switching from an omniscient point of view (POV) to a POV of a particular unit / OE. For example, referring to Fig. 9C and Fig.9D In the sequence, in response to detecting and changing Fig. 9C The electronic device 103 or the HMD 104 presents the third input associated with the viewing angle 936A in accordance with Fig.9D SR scene 930C is presented by view vector 932B at time 934+2 in FIG. Continuing with the exemplary sequence, view vector 932B is characterized by view angle 936B, FOV 938B, pose 940B, and coordinate set 942B. Fig.9D , viewing vector 932B is associated with viewing angle 936B (eg, an omniscient view of the second SR scene), which is Fig. 9B and Fig. 9C 936A (eg, the view of worker 946 within the second SR set).

[0154] In some implementations, when presenting the predetermined content according to the second viewing vector, the device also presents a previous perspective affordance for viewing the SR scenery from a previous perspective associated with the first viewing vector at the second time. Fig. 9B The previous perspective indication in the associated input transitions the electronic device 103 or HMD 104 from presenting the SR scenery according to the current viewing vector associated with the perspective 936A at time 934 to presenting the SR scenery according to the subsequent vector associated with the perspective 906 at the next time period.

[0155] In some embodiments, the device detects one or more subsequent inputs associated with a time scale. In response to detecting the one or more subsequent inputs, the device changes the time scale associated with the navigation timeline used for the narration, and updates multiple view vector affordances based on the changed time scale. For example, one or more subsequent inputs correspond to a pinch-in or pinch-out gesture. For another example, one or more subsequent inputs correspond to voice commands associated with zooming in or out from a current time scale associated with the navigation timeline.

[0156] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0157] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0158] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0159] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method for presenting a synthetic reality (SR) scene, comprising: At a device including a non-transitory memory and one or more processors coupled to the non-transitory memory: presenting the SR scenery associated with a first portion of predetermined content according to a first viewing vector at a first time, wherein the first viewing vector represents a display of the first portion of the predetermined content at the first time from a first perspective corresponding to a first entity within the SR scenery; Simultaneously presenting a navigation timeline for the predetermined content and a first perspective modification affordance for changing the first perspective, wherein the navigation timeline includes a current time indicator located at the first time and a plurality of view vector affordances, each view vector affordance being associated with a different view vector at a corresponding time on the navigation timeline, wherein each view vector represents how to display the SR content at the corresponding time, and wherein the first perspective modification affordance includes a first selectable perspective corresponding to a different entity associated with the first portion of the predetermined content; detecting a first input selecting a corresponding look vector affordance on the navigation timeline associated with a second portion of the predetermined content at a second time; in response to detecting the first input selecting the corresponding view vector affordance on the navigation timeline, presenting a menu of view vector options associated with the second time; detecting a second input selecting a second view vector included in the menu of view vector options; as well as in response to detecting the second input selecting the second view vector, updating the SR scenery to present the SR scenery according to a second view vector different from the first view vector, wherein the second view vector represents a display of the second portion of the predetermined content at the second time from a second perspective corresponding to a second entity within the updated SR scenery, The method further includes presenting a second perspective modification enable representation, wherein the second perspective modification enable representation is provided to change from the second perspective to a second optional perspective based on a menu of viewing vector options, wherein the second optional perspective corresponds to a different entity associated with the second portion of the predetermined content and is different from the first optional perspective.

2. The method of claim 1 , wherein the first view vector is at least partially characterized by a first field of view (FOV), a first pose, a first set of SR coordinates within the SR scenery, and the first viewing angle, and wherein the second view vector is at least partially characterized by a second FOV, a second pose, a second set of SR coordinates within the SR scenery, and the second viewing angle.

3. The method according to claim 1, further comprising: In response to detecting the first input selecting the corresponding view vector affordance on the navigation timeline, presenting the current time indicator located at the second time.

4. The method according to claim 1, further comprising: In response to detecting the first input selecting the corresponding view vector affordance on the navigation timeline, ceasing to display the SR scenery according to the first view vector.

5. A method according to any one of claims 1 to 4, wherein the multiple view vector affordances correspond to predetermined view vectors.

6. A method according to any one of claims 1 to 4, wherein the multiple view vector affordances correspond to view vectors determined based on a set of filters.

7. The method according to any one of claims 1 to 4, further comprising: Presentation is provided for modifying a current field of view FOV associated with the second view vector based on one or more subsequent inputs.

8. The method according to any one of claims 1 to 4, further comprising: Presenting a pose modification affordance provided for changing a current pose associated with the second view vector to a subsequent pose based on one or more subsequent inputs.

9. The method according to any one of claims 1 to 4, further comprising: Presenting an SR coordinate modification affordance provided for changing a current set of SR coordinates associated with the second view vector to a subsequent set of SR coordinates within the SR scenery based on one or more subsequent inputs.

10. The method according to any one of claims 1 to 4, further comprising: When presenting the SR scenery according to the second viewing vector, a previous perspective affordance is presented for viewing the SR scenery from a previous perspective associated with the first viewing vector.

11. The method according to any one of claims 1 to 4, wherein the predetermined content corresponds to one of the following: a pre-existing story, a movie, a TV series, a book, a stage play, or a historical event.

12. The method of any one of claims 1 to 4, wherein the predetermined content corresponds to emerging content based at least in part on one or more pre-existing source assets.

13. The method of claim 12, wherein the one or more pre-existing source assets correspond to at least one of: a pre-existing story, a movie, a television series, a book, a stage play, or a historical event.

14. The method according to any one of claims 1 to 4, wherein the predetermined content corresponds to a conditionally related SR content thread.

15. The method of any one of claims 1 to 4, wherein the first input and the second input correspond to one of: a voice command, a gesture input, a touch input, or a gaze directional input.

16. The method according to any one of claims 1 to 4, further comprising: detecting one or more subsequent inputs associated with the time scale; as well as In response to detecting the one or more subsequent inputs, a time scale associated with the navigation timeline for the narrative is changed, and the plurality of look vector affordances are updated based on the changed time scale.

17. The method of any one of claims 1 to 4, wherein the device corresponds to a mobile device, and wherein the SR scenery is composited with at least a portion of a physical scenery, and the composite content is presented on a display of the mobile device.

18. The method according to any one of claims 1 to 4, wherein the device corresponds to a head mounted device (HMD).

19. A device comprising: one or more processors; non-transitory memory; as well as One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform or cause the performance of a method according to any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing one or more programs which, when executed by one or more processors of a device, cause the device to perform or cause the performance of the method according to any one of claims 1 to 18.

21. A device comprising: one or more processors; non-transitory memory; as well as A module for causing the device to execute or cause the execution of the method according to any one of claims 1 to 18.

22. A computer program product comprising one or more programs which, when executed by one or more processors of a device, cause the device to perform or cause the performance of a method according to any one of claims 1 to 18.