Content Event Mapping

Abstract

In one implementation, an immersive synthetic reality (SR) map is utilized to supplement the consumption of media content based on the media content (such as video, audio, or text). In various implementations described herein, the SR map includes a plurality of SR environment representations that, when selected by a user, cause a corresponding SR environment to be displayed.

Description

Technical Field

[0001] The present disclosure relates generally to synthetic reality (SR) environments, and in particular, to systems, methods, and apparatus for generating SR environments including content event maps. Background Art

[0002] A physical set refers to a world that an individual can perceive and / or interact with without the aid of an electronic system. A physical set (e.g., a physical forest) includes physical elements (e.g., physical trees, physical structures, and physical animals). An individual can directly interact with and / or perceive a physical set, such as by touch, sight, smell, hearing, and taste.

[0003] In contrast, a synthetic reality (SR) set refers to a set that is fully or partially computer-created that an individual can perceive and / or that the individual can interact with via an electronic system. In SR, a subset of the individual's movements is monitored, and in response thereto, one or more properties of one or more virtual objects in the SR set are changed in a manner that complies with one or more laws of physics. For example, the SR system may detect that the individual has taken a few steps forward, and in response thereto, adjust the graphics and audio presented to the individual in a manner similar to how such scenes and sounds would change in a physical set. Modifications to one or more properties of one or more virtual objects in the SR set may also be made in response to a representation of movement (e.g., an audio command).

[0004] The individual may interact with and / or perceive SR objects using any of his senses, including touch, smell, sight, taste, and sound. For example, the individual may interact with and / or perceive auditory objects that create a multi-dimensional (e.g., three-dimensional) or spatial auditory scenery and / or implement auditory transparency. The multi-dimensional or spatial auditory scenery provides the individual with the perception of discrete auditory sources in a multi-dimensional space. Auditory transparency selectively combines sounds from the physical scenery with or without computer-created audio. In some SR scenery, the individual may interact with and / or perceive only auditory objects.

[0005] An example of SR is virtual reality (VR). A VR set refers to a simulated set designed to include only computer-created sensory input for at least one sense. The VR set includes a plurality of virtual objects that an individual can interact with and / or perceive. The individual can interact with and / or perceive virtual objects in the VR set by simulating a subset of the individual's actions within the computer-created set and / or by simulating the individual or his / her presence within the computer-created set.

[0006] Another example of SR is mixed reality (MR). An MR set refers to a simulated set designed to integrate computer-created sensory input (e.g., virtual objects) with sensory input from a physical set, or representation thereof. On the reality spectrum, a mixed reality set is between VR sets at one end and fully physical sets at the other end, and does not include these sets.

[0007] In some MR sets, computer-created sensory input can adapt to changes in sensory input from the physical set. In addition, some electronic systems used to render MR sets can monitor orientation and / or position relative to the physical set to enable virtual objects to interact with real objects (i.e., physical elements from the physical set or representations thereof). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0008] An example of mixed reality is augmented reality (AR). AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor, the imaging sensor being used to capture images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects may be projected, for example, on a physical surface or as a hologram, such that an individual using the system observes the virtual objects superimposed on the physical scenery.

[0009] An augmented reality set may also refer to a simulated set in which a representation of a physical set is altered by computer-created sensory information. For example, a portion of the representation of a physical set may be graphically altered (e.g., enlarged) such that the altered portion may still represent one or more originally captured images but is not a faithfully reproduced version. For another example, when providing video pass-through, the system may alter at least one of the sensor images to impose a particular viewpoint that is different from the viewpoint captured by one or more image sensors. For another example, the representation of a physical set may be altered by graphically blurring or eliminating portions thereof.

[0010] Another example of mixed reality is augmented virtuality (AV). An AV set refers to a simulated set in which a computer-created set or a virtual set incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may present the color of a physical element captured by one or more imaging sensors. As another example, a virtual object may present features consistent with actual weather conditions in a physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images taken of physical animals.

[0011] Many electronic systems enable individuals to interact with and / or perceive various SR scenery. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smart phone). The head-mounted system may have one or more imaging sensors and / or microphones for taking images / video of the physical scenery and / or capturing audio of the physical scenery, respectively. The head-mounted system may also have a transparent or translucent display. The transparent or translucent display may be combined with a substrate through which light representing the image is directed to the individual's eyes. The display may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one embodiment, the transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. As another example, the electronic system may be a projection-based system. A projection-based system may project an image onto the retina of an individual using retinal projection. Alternatively, the projection system may also project the virtual object into a physical setting (e.g., onto a physical surface or as a hologram). Other examples of SR systems include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, headphones or earbuds, a speaker arrangement, an input mechanism (e.g., a controller with or without tactile feedback), a tablet, a smartphone, and a desktop or laptop computer.

[0012] In various implementations, SR content is utilized to supplement traditional media content to provide a more immersive user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0014] Figure 1A is a block diagram of an exemplary operating architecture according to some specific implementations.

[0015] Figure 1B is a block diagram of an exemplary operating architecture according to some specific implementations.

[0016] Figure 2 is a block diagram of an example controller according to some implementations.

[0017] Figure 3 is a block diagram of an exemplary head mounted device (HMD) according to some implementations.

[0018] Figure 4 A scene is illustrated along with an electronic device that surveys the scene.

[0019] FIG. 5A to FIG. 5F Illustrated Figure 4 A portion of a display of an electronic device that displays an image of a representation of a scene including a first SR map.

[0020] Figure 6 Illustrated Figure 4 A scene together with electronic equipment for surveying the scene.

[0021] 7A to 7E Illustrated Figure 4 A portion of a display of an electronic device that displays an image of a representation of a scene including a second SR map.

[0022] Fig. 8A Media content objects according to some specific implementations are illustrated.

[0023] Figure 8B An event table according to some specific implementations is illustrated.

[0024] Figure 8C An SR mapping object according to some specific implementations is illustrated.

[0025] Fig. 9 is a flowchart representation of a method of generating an SR map according to some specific implementations.

[0026] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0027] Various specific implementations disclosed herein include devices, systems, and methods for generating SR maps. In various specific implementations, the method is performed at a device having one or more processors and non-volatile memory. The method includes obtaining media content data represented by a timeline, wherein the media content data includes event markers representing multiple events of a story frame. The method includes determining multiple event definitions corresponding to multiple events and multiple corresponding event times in the timeline from the media content data based on the event markers. The method includes obtaining multiple SR environment representations based on multiple event definitions. The method includes determining a path and multiple corresponding positions of the multiple SR environment representations, wherein the path is defined by a set of ordered positions including the multiple corresponding positions, and the multiple corresponding positions are in an order based on multiple corresponding event times. The method includes displaying an SR map, the SR map including multiple SR environment representations displayed at the multiple corresponding positions, wherein each SR environment representation in the multiple SR environment representations is associated with an enable representation, which enables the display of the corresponding SR environment when selected.

[0028] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0029] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0030] In various implementations, the consumption of media content (such as video, audio, or text) is not truly immersive and / or is not customized to the user environment. Therefore, in various implementations described herein, immersive SR mapping is utilized to supplement traditional media content based on the media content. In various implementations described herein, the SR mapping includes multiple SR environment representations that, when selected by a user, cause the corresponding SR environment to be displayed.

[0031] Figure 1A 1 is a block diagram of an exemplary operational architecture 100A according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operational architecture 100A includes an electronic device 120A.

[0032] In some implementations, the electronic device 120A is configured to present SR content to the user. In some implementations, the electronic device 120A includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120A presents the SR content to the user via a display 122 when the user is physically present in a physical environment 103, which includes a table 107 located within a field of view 111 of the electronic device 120A. Therefore, in some implementations, the user holds the electronic device 120A in his / her hand. In some implementations, when providing augmented reality (AR) content, the electronic device 120A is configured to display an AR object (e.g., an AR cube 109) and implement video pass-through of the physical environment 103 (e.g., including a representation 117 of the table 107) on the display 122.

[0033] Figure 1B 1 is a block diagram of an exemplary operating environment 100B according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100B includes a controller 110 and a head mounted device (HMD) 120B.

[0034] In some implementations, the controller 110 is configured to manage and coordinate SR presentation to users. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 2The controller 110 is described in more detail. In some implementations, the controller 110 is a computing device that is located locally or remotely relative to the scene 105. For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some implementations, the controller 110 is communicatively coupled to the HMD 120B via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included in the housing of the HMD 120B.

[0035] In some implementations, the HMD 120B is configured to present SR content to the user. In some implementations, the HMD 120B includes a suitable combination of software, firmware, and / or hardware. Figure 3 HMD 120B is described in greater detail. In some implementations, the functionality of controller 110 is provided by and / or combined with HMD 120B.

[0036] According to some implementations, HMD 120B presents SR content to the user while the user is virtually and / or physically present within scene 105 .

[0037] In some implementations, the user wears the HMD 120B on his / her head. Thus, the HMD 120B includes one or more SR displays provided for displaying SR content. For example, in various implementations, the HMD 120B surrounds the user's field of view. In some implementations, such as in Figure 1A In some implementations, a handheld device (such as a smartphone or tablet) configured to present SR content is used in place of HMD 120B, and the user no longer wears HMD 120B but holds the device with the display facing the user's field of view and the camera facing the scene 105. In some implementations, the handheld device can be placed in a housing that can be worn on the user's head. In some implementations, HMD 120B is replaced with an SR pod, housing, or chamber configured to present SR content, wherein the user no longer wears or holds HMD 120B.

[0038] Figure 2is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0039] In some implementations, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0040] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and an SR experience module 240.

[0041] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the SR experience module 240 is configured to manage and coordinate single or multiple SR experiences of one or more users (e.g., a single SR experience of one or more users, or multiple SR experiences of corresponding groups of one or more users). To this end, in various implementations, the SR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0042] In some specific implementations, the data acquisition unit 242 is configured to at least Figure 1B The HMD 120B obtains data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various specific implementations, the data acquisition unit 242 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0043] In some implementations, the tracking unit 244 is configured to map the scene 105 and track at least the HMD 120B relative to the scene 105. Figure 1B The location / position of the scene 105. To this end, in various specific implementations, the tracking unit 244 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0044] In some implementations, the coordination unit 246 is configured to manage and coordinate the SR experience presented to the user by the HMD 120B. To this end, in various implementations, the coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0045] In some implementations, the data transfer unit 248 is configured to transfer data (e.g., presentation data, position data, etc.) to at least the HMD 120B. To this end, in various implementations, the data transfer unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0046] Although the data acquisition unit 242, tracking unit 244, coordination unit 246, and data transfer unit 248 are illustrated as residing on a single device (e.g., controller 110), it should be understood that in other implementations, any combination of the data acquisition unit 242, tracking unit 244, coordination unit 246, and data transfer unit 248 may be located in separate computing devices.

[0047] also, Figure 2More as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic diagram of the structure of the specific implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0048] Figure 3 1 is a block diagram of an example of an HMD 120B according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some specific implementations, the HMD 120B includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE802.3x, IEEE802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more SR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0049] In some implementations, one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0050] In some implementations, one or more SR displays 312 are configured to present SR content to the user. In some implementations, one or more SR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more SR displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, HMD 120B includes a single SR display. In another example, HMD 120B includes an SR display for each eye of the user. In some implementations, one or more SR displays 312 can present MR and / or VR content.

[0051] In some implementations, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and thus may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to the scene that the user would see when the HMD 120B is not present (and thus may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0052] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or a subset thereof, including an optional operating system 330 and an SR presentation module 340.

[0053] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the SR presentation module 340 is configured to present SR content to a user via one or more SR displays 312. To this end, in various implementations, the SR presentation module 340 includes a data acquisition unit 342, an SR presentation unit 344, an SR map generation unit 346, and a data transmission unit 348.

[0054] In some implementations, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of FIG1 . To this end, in various implementations, the data acquisition unit 342 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0055] In some implementations, the SR presentation unit 344 is configured to present the SR content via the one or more SR displays 312. To this end, in various implementations, the SR presentation unit 344 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0056] In some implementations, the SR map generation unit 346 is configured to generate a SR map based on the media content data. To this end, in various implementations, the SR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0057] In some implementations, the data transfer unit 348 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the controller 110. To this end, in various implementations, the data transfer unit 348 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0058] Although the data obtaining unit 342, the SR rendering unit 344, the SR map generation unit 346, and the data transmission unit 348 are shown as residing in a single device (eg, Figure 1B However, it should be understood that in other specific implementations, any combination of the data acquisition unit 342, the SR presentation unit 344, the SR map generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0059] also, Figure 3 More as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic diagram of the structure of the specific implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0060] Figure 4 A scene 405 is illustrated along with an electronic device 410 surveying the scene 405. The scene 405 includes a table 408 and a wall 407.

[0061] Electronic device 410 displays a representation of scene 415 on a display, including a representation of table 418 and a representation of wall 417. In various implementations, the representation of scene 415 is generated based on an image of the scene captured with a scene camera of electronic device 410, the scene camera having a field of view toward scene 405. The representation of scene 415 also includes SR map 409 displayed on the representation of table 418.

[0062] As the electronic device 410 moves around the scene 415, the representation of the scene 415 changes according to the change in viewing angle of the electronic device 410. In addition, the SR map 409 changes accordingly according to the change in viewing angle of the electronic device 410. Thus, as the electronic device 410 moves, the SR map 409 appears in a fixed relationship relative to the representation of the table 418.

[0063] In various specific implementations, the SR map 409 corresponds to the media content data represented by the timeline, so that at each time, the corresponding portion of the media content data will be presented. In some embodiments, the media content data includes video content data represented by the timeline, so that at each time, a corresponding image will be displayed, and at the same time, optionally, a sound corresponding to the corresponding image will be played simultaneously. In some embodiments, the media content data includes audio content data, wherein the corresponding sound will be played at the corresponding time. In various specific implementations, the audio content data includes music and / or spoken narrative. In some embodiments, the media content data includes text content data, which is represented by the timeline by virtue of the character position of each portion of the text in the text content data or the estimated text-to-speech time of each portion of the text in the text content data (if read by the text-to-speech module).

[0064] The media content data includes event markers representing multiple events of the story frame. In various specific implementations, the event markers are portions of the media content data that meet event marker criteria. In various specific implementations, event markers are detected by image analysis of the video content data. For example, in some embodiments, the event marker criteria include criteria that are met when an object is present in an image of the video content data (e.g., a building, a skylight, or a planet). In some embodiments, the event marker criteria include criteria that are met when the color palette of the image is significantly different from a previous image (e.g., indicating a scene change).

[0065] In various specific implementations, event markers are detected by audio analysis of the audio content data (or the audio component of the video content data). For example, in some embodiments, the event marker criteria include criteria that are met when a music cue is present. In some embodiments, the event marker criteria include criteria that are met when the volume of the audio violates a threshold or the rate of change of the volume violates a threshold.

[0066] In various implementations, event markers are detected by text analysis of text content data (or text recognized from audio content data or an audio component of video content data). For example, in some embodiments, event marker criteria include criteria that are met when certain words (e.g., one or more words that meet the event marker word criteria) are present.

[0067] Figure 5A A portion of a display of an electronic device 410 is illustrated, the display displaying at a first time a first image 500A of a representation of a scene 415 including an SR map 409. In various implementations, the first image 500A is displayed concurrently with playback of the media content data (or an audio component thereof), and the first image 500A is displayed at a first time in a timeline of the media content data that corresponds to a first event marker representing a first event of a story frame.

[0068] exist Figure 5A , SR map 409 includes SR map representation 510 (representation of a mountain) and path representation 511 (representation of a walking path winding around the mountain). In various implementations, SR map representation 510 is a default SR map representation or a SR map representation selected by a user. In various implementations, SR map representation 510 is obtained based on media content data. For example, in some embodiments, SR map representation 510 is selected from a plurality of stored SR map representations based on the media content data. In some embodiments, SR map representation 510 is generated based on the media content data (e.g., from an image in the media content data, such as a map detected in an image in the media content data).

[0069] The SR map 409 includes a first SR environment representation 501A (e.g., a representation of a house) displayed along the path representation 511. In various specific implementations, the first SR environment representation 501A is obtained based on the media content data, in particular, a portion of the media content data associated with the first event (e.g., a location associated with the first event). For example, in some embodiments, the first SR environment representation 501A is selected from a plurality of stored SR environment representations based on the portion of the media content data associated with the first event. For another example, in some embodiments, the first SR environment representation 501A is generated based on the portion of the media content data associated with the first event (e.g., based on one or more images of locations in the portion of the media content data associated with the first event).

[0070] Figure 5B A portion of a display of an electronic device 410 is illustrated, the display displaying a second image 500B of a representation of a scene 415 including an SR map 409 at a second time. In various implementations, the second image 500B is displayed concurrently with playback of the media content data (or its audio component), and the second image SR is displayed at a second time in a timeline of the media content data, the second time corresponding to a second event marker representing a second event of the story frame.

[0071] and Figure 5A In contrast, the SR map 409 also includes a second SR environment representation 501B (eg, a representation of a school) displayed further along the path representation 511. In various implementations, the second SR environment representation 501B is obtained in a similar manner as the first SR environment representation 501A.

[0072] Figure 5C A portion of a display of an electronic device 410 is illustrated, the display displaying a third image 500C of a representation of a scene 415 including an SR map 409 at a third time. In various implementations, the third image 500C is displayed concurrently with playback of the media content data (or an audio component thereof), and the third image 500C is displayed at a third time in a timeline of the media content data that corresponds to a third event marker representing a third event of the story frame.

[0073] and Figure 5B In contrast, the SR map 409 also includes a third SR context representation 501C (eg, a representation of the Parthenon) displayed further along the path representation 511. In various implementations, the third SR context representation 501C is obtained in a similar manner as the first SR context representation 501A.

[0074] The third SR environment representation 501C is associated with a third enable representation, which, when selected, causes the third SR environment to be displayed. Similarly, the first SR environment representation 501A is associated with a first enable representation, which, when selected, causes the first SR environment to be displayed, and the second SR environment representation 501B is associated with a second enable representation, which, when selected, causes the second SR environment to be displayed.

[0075] Figure 5D A portion of a display of electronic device 410 is illustrated that, at a fourth time and in response to detecting selection of a third affordance, displays a fourth image 500D of a representation of scene 415 including a third SR environment 520. In various implementations, third SR environment 520 is obtained in a manner similar to third SR environment representation 501C.

[0076] In various implementations, the third SR environment 520 includes a representation of the location and also includes virtual objects corresponding to the media content data. For example, in some embodiments, the third SR environment 520 is populated with virtual objects corresponding to the portion of the media content data associated with the third event, such as one or more images based on people or objects in the portion of the media content data associated with the third event.

[0077] In various embodiments, in response to detecting a selection of the third enable representation, the display of the SR mapping 409 is stopped. In various embodiments, in response to detecting a selection of the third enable representation, the simultaneous playback of the media content data is paused. However, in various embodiments, in response to detecting a selection of the third enable representation, the simultaneous playback of the media content data is continued.

[0078] In response to a user selection to return to the SR map 409 via a gesture or selection of a return affordance in the third SR environment 520, the third SR environment 520 ceases to be displayed (and, if hidden, the SR map 409 is redisplayed).

[0079] Figure 5E A portion of a display of an electronic device 410 is illustrated, the display displaying a fifth image 500E at a fifth time that includes a representation of a scene 415 of the SR map 409. In various implementations, the fifth image 500E is displayed concurrently with playback of the media content data (or an audio component thereof), and the fifth image 500E is displayed at a fourth time in a timeline of the media content data that corresponds to a fourth event marker representing a fourth event of the story frame.

[0080] and Figure 5CIn contrast, the SR map 409 also includes a fourth SR environment representation 501D (eg, a representation of a skyscraper) displayed further along the path representation 511. In various implementations, the fourth SR environment representation 501D is obtained in a similar manner as the first SR environment representation 501A.

[0081] Fig. 5F A portion of a display of an electronic device 410 is illustrated, the display displaying a sixth image 500F at a sixth time that includes a representation of a scene 415 of the SR map 409. In various implementations, the sixth image 500F is displayed concurrently with playback of the media content data (or an audio component thereof), and the sixth image 500F is displayed at a fifth time in a timeline of the media content data that corresponds to a fifth event marker representing a fifth event of the story frame.

[0082] and Figure 5E In contrast, the SR map 409 also includes a fifth SR context representation 501E (eg, a representation of the Capitol Building) displayed further along the path representation 511. In various implementations, the fifth SR context representation 501E is obtained in a similar manner as the first SR context representation 501A.

[0083] As an illustrative example, FIG. 5A to FIG. 5FCorresponding to an SR map associated with media content data, the media content data includes video content data of an interview of a senator by a reporter that takes place on a hiking trail. Based on analysis of the video data, the device detects the scene as a hiking trail and selects an SR map representation 510 of a high mountain. At a first time in the interview, the senator discusses his humble childhood growing up in a small one-bedroom home with two young siblings. The device detects that the words "one-bedroom home" are spoken and selects a first SR context representation 501A of the small house to be displayed along the path representation 511 at the first time. At a second time in the interview, the senator discusses his high school football career at a local school, and the video includes an image of the school. The device detects that the words "high school" are spoken and generates a second SR context representation 501B of the local school to be displayed along the path representation 511 at the second time. At a third time in the interview, the senator discusses studying abroad in Greece and is impressed by the magnificence of the Parthenon. The device detects that the words "Parthenon" are spoken and selects a third SR context representation 501C of the Parthenon to be displayed along the path representation 511 at the third time. At a fourth time in the interview, the Senator discusses working at a prominent New York law firm. The device detects that the words "work," "law firm," and "New York" are spoken, and selects a fourth SR context representation 501D of a skyscraper to be displayed along the path representation 511 at the fourth time. At a fifth time in the interview, the Senator discusses his election to the U.S. Senate. The device detects that the words "election" and "Senate" are spoken, and selects a fifth SR context representation 501E of the Capitol Building to be displayed along the path representation 511 at the fifth time.

[0084] While the media content data is being played simultaneously, the device displays the SR map 409 including the SR map representation 510. At corresponding times, the device displays the SR map 409 including corresponding SR representations 501A to 501E.

[0085] Figure 6 Illustrated Figure 4 The scene 405 is shown together with an electronic device 410 that surveys the scene 405. As described above, the scene 405 includes a table 408 and a wall 407.

[0086] exist Figure 6 4, electronic device 410 displays a representation of scene 415 on a display, including a representation of table 418 and a representation of wall 417. In various implementations, the representation of scene 415 is generated based on an image of the scene captured by a scene camera of electronic device 410, the scene camera having a field of view toward scene 405. The representation of scene 415 also includes SR map 609 displayed on the representation of wall 417.

[0087] As the electronic device 410 moves around the scene 405, the representation of the scene 415 changes according to the change in viewing angle of the electronic device 410. In addition, the SR map 609 changes accordingly according to the change in viewing angle of the electronic device 410. Thus, as the electronic device 410 moves, the SR map 609 appears in a fixed relationship relative to the representation of the wall 417.

[0088] Similar to Figure 4 In various implementations, the SR map 609 corresponds to media content data represented by a timeline. The media content data includes event markers representing multiple events of a story frame.

[0089] Fig. 7A A portion of a display of an electronic device 410 is illustrated, the display displaying a first image 700A of a representation of a scene 415 including an SR map 609 at a first time.

[0090] exist Fig. 7A , SR map 609 includes SR map representation 710 (representation of a paper map). In various implementations, SR map representation 710 is a default SR map representation or a SR map representation selected by a user. In various implementations, SR map representation 710 is obtained based on media content data. For example, in some embodiments, SR map representation 710 is selected from a plurality of stored SR map representations based on the media content data. In some embodiments, SR map representation 710 is generated based on media content data (e.g., from an image in the media content data, such as a map detected in an image in the media content data).

[0091] However, in Figure 5A In the illustrated first time, the SR map 409 includes only the first SR environment representation 501A. Fig. 7A At the first time shown, the SR map 609 includes multiple SR environment representations 701A to 701E. The multiple SR environment representations 701A to 701E include a first SR environment representation 701A, a second SR environment representation 701B, a third SR environment representation 701C, a fourth SR environment representation 701D, and a fifth SR environment representation 701E. At least one SR environment representation of the multiple SR environment representations 701A to 701E is associated with an affordance that, when selected, causes the corresponding SR environment to be displayed. In some embodiments, each SR environment representation of the multiple SR environment representations is associated with an affordance that, when selected, causes the corresponding SR environment to be displayed.

[0092] In various implementations, each of the plurality of SR environment representations 701A to 701E is represented by Figure 5A The first SR environment representation 501A is obtained in a similar manner.

[0093] SR map 609 includes a first object representation 720A displayed at the location of first SR environment representation 701A and a second object representation 720B displayed at the location of fifth SR environment representation 701E. In various implementations, first object representation 720A represents a first character, vehicle, or prop of media content data. In various implementations, first object representation 720A is obtained based on the media content data. Similarly, second object representation 720B represents a second character, vehicle, or prop of media content data. In various implementations, second object representation 720A is obtained based on the media content data.

[0094] Figure 7B A portion of a display of the electronic device 410 is illustrated, the display displaying a second image 700B of a representation of the scene 415 including the SR map 609 at a second time.

[0095] exist Figure 7B , the first object representation 720A is displayed at the location of the second SR environment representation 701B, and the second object representation 720B is displayed at the location of the fourth SR environment representation 701D. In addition, the SR map 609 includes a first path representation 730A located between the first SR environment representation 701A and the second SR environment representation 701B, the first path representation indicating that the first object representation 720A has moved from the first SR environment representation 701A to the second SR environment representation 701B. The SR map 609 includes a second path representation 730B located between the fifth SR environment representation 701E and the fourth SR environment representation 701D, the second path representation indicating that the second object representation 720B has moved from the fifth SR environment representation 701E to the fourth SR environment representation 701D.

[0096] Figure 7C A portion of a display of the electronic device 410 is illustrated, the display showing a third image 700C of a representation of the scene 415 including the SR map 609 at a third time.

[0097] exist Figure 7C, the first object representation 720A is displayed at the location of the fourth SR environment representation 701D, and the second object representation 720B is displayed at the location of the fifth SR environment representation 701E. In addition, the first path representation 730A is extended to include a portion located between the second SR environment representation 701B and the fourth SR environment representation 701D, which indicates that the first object representation 720A has moved from the second SR environment representation 701B to the fourth SR environment representation 701D. The second path representation 730B is extended to include a portion located between the fourth SR environment representation 701D and the fifth SR environment representation 701E, which indicates that the second object representation 720B has moved from the fourth SR environment representation 701D to the fifth SR environment representation 701E.

[0098] Fig.7D A portion of a display of the electronic device 410 is illustrated, the display showing a fourth image 700D of a representation of the scene 415 including the SR map 609 at a fourth time.

[0099] exist Fig.7D , the first object representation 720A is displayed at the location of the second SR environment representation 701B, and the second object representation 720B is also displayed at the location of the second SR environment representation 701B. In addition, the first path representation 730A is extended to include a portion located between the fourth SR environment representation 701D and the second SR environment representation 701B, which indicates that the first object representation 720A has moved from the fourth SR environment representation 701D to the second SR environment representation 701B. The second path representation 730B is extended to include a portion located between the fifth SR environment representation 701E and the second SR environment representation 701B, which indicates that the second object representation 720B has moved from the fifth SR environment representation 701E to the second SR environment representation 701B.

[0100] Fig. 7E A portion of a display of the electronic device 410 is illustrated, the display showing a fifth image 700E of a representation of the scene 415 including the SR map 609 at a fifth time.

[0101] exist Fig. 7E, the first object representation 720A is displayed at the location of the third SR environment representation 701C, and the second object representation 720B is also displayed at the location of the third SR environment representation 701C. In addition, the first path representation 730A is extended to include a portion located between the second SR environment representation 701B and the third SR environment representation 701C, which indicates that the first object representation 720A has moved from the second SR environment representation 701B to the third SR environment representation 701C. The second path representation 730B is extended to include a portion located between the second SR environment representation 701B and the third SR environment representation 701C, which indicates that the second object representation 720B has also moved from the second SR environment representation 701B to the third SR environment representation 701C.

[0102] As an illustrative example, 7A to 7E Corresponding to an SR map associated with media content data, the media content data comprising an electronic book about a child looking for his mother. The device selects an SR map representation 710 of a paper map as a default SR map representation. At a first time in the book, the mother works in an office and the child is at home. The device detects the word "office" associated with the mother, selects a first SR environment representation 701A of an office building, and associates a first object representation 720A with the first SR environment representation 701A and the first time. Similarly, the device detects the word "home" associated with the child, selects a fifth SR environment representation 701E of the home, and associates a second object representation 720B with the fifth SR environment representation 701E and the first time.

[0103] At a second time in the book, the mother goes to lunch, and the child looks for her mother at the police station. The device detects the word "lunch", selects the second SR context representation 701B of the restaurant, and associates the first object representation 720A with the second SR context representation 701B and the second time. Similarly, the device detects the word "police station" associated with the child, selects the fourth SR context representation 701D of the police station, and associates the second object representation 720B with the fourth SR context representation 701D and the second time.

[0104] At the third time in the book, the mother (having received a missed call from the police) goes to the police. However, at the third time in the book, the police have already escorted the child home. The device detects the word "Police" associated with the mother and associates the first object representation 720A with the fourth SR environment representation 701D and the third time. Similarly, the device detects the word "Home" associated with the child and associates the second object representation 720B with the fifth SR representation 701E and the third time.

[0105] At the fourth time in the book, the mother (who has been informed by the police that the child was escorted home) returns for lunch. However, at the fourth time in the book, the child is in the restaurant looking for his mother. The device detects the word "lunch" associated with the mother and associates the first object representation 720A with the second SR environment representation 701B and the fourth time. Similarly, the device detects the word "restaurant" associated with the child and associates the second object representation 720B with the second SR representation 701B and the fourth time.

[0106] At the fifth time in the book, the mother and child have reunited and go to buy ice cream together. The device detects the word "ice cream" associated with the mother and child, selects the third SR environment representation 701C of the ice cream shop, and associates both the first object representation 720A and the second object representation 720B with the third SR environment representation 701C and the fifth time.

[0107] Fig. 8A A media content object 810 according to some implementations is illustrated. The media content object 810 includes media content data 812 associated with a timeline, such as video, audio, and / or text. The method content object 810 includes metadata 811 describing the media content data, such as the title, author, genre, or release year of the media content data.

[0108] In various implementations, the event indicator is embedded in metadata 811 indicating the time in the timeline of the event. In various implementations, the media content data 812 includes event markers representing multiple events of the story frame. In various implementations, the event marker is a portion of the media content data 812 that meets the event marker criteria.

[0109] In various implementations, event markers are detected by image analysis of the video content data. For example, in some embodiments, event marker criteria include criteria that are met when an object is detected in an image of the video content data (e.g., a building, a skylight, or a planet). In some embodiments, event marker criteria include criteria that are met when the color palette of the image is significantly different from a previous image (e.g., indicating a scene change).

[0110] In various specific implementations, event markers are detected by audio analysis of audio content data (or audio components of video content data). For example, in some embodiments, event marker criteria include criteria that are met when a music cue is detected. In some embodiments, event marker criteria include criteria that are met when the volume of the audio violates a threshold or the rate of change of the volume violates a threshold.

[0111] In various implementations, event markers are detected by text analysis of text content data (or text recognized from audio content data or an audio component of video content data). For example, in some embodiments, event marker criteria include criteria that are met when certain words (e.g., one or more words that meet event marker word criteria) are detected.

[0112] In various specific implementations, the device determines a plurality of event definitions from the media content data based on event markers embedded in the media content data. In some embodiments, the plurality of event definitions are stored as an event table.

[0113] Figure 8B An event table 820 according to some specific implementations is illustrated. The event table 820 includes a plurality of entries respectively associated with a plurality of events. Each entry includes an event identifier 821 of the event, an event time 822 in the timeline of the event, and an event definition 823 of the event.

[0114] In various implementations, event identifier 821 is a unique name or number of an event. In various implementations, event definition 823 indicates which event marking criteria (or criteria) are met. For example, in various implementations, event definition 823 indicates that a building is detected in an image of video content data at event time 822. For another example, in various implementations, event definition 823 indicates that a dramatic music cue is detected in audio content data at event time 822. For another example, in various implementations, event definition 823 indicates that the word "police station" is detected in text content data at event time 822.

[0115] In various implementations, the device generates the SR map object based in part on the event table 820 .

[0116] Figure 8C An SR map object 830 according to some implementations is illustrated. The SR map object 830 includes an SR map metadata field 837 that includes metadata for the SR map object 830. In various implementations, the metadata indicates the media content data to which the SR map corresponds. In various implementations, the metadata indicates the date and / or time when the SR map object 830 was created and / or modified.

[0117] The SR map object 830 includes an SR map representation field 831, which includes data indicating the SR map representation. In various implementations, the SR map representation field 831 includes an SR map representation, such as Figure 5A The SR mapping representation 510 or Fig. 7A810. In various implementations, the SR map representation field 831 includes a reference SR map representation that is stored separately from the SR map object 830, either locally with the SR map object 830 or remotely on another device such as a network server.

[0118] The SR map object 830 includes a path representation field 832 that includes data indicating a path. The path includes a set of ordered positions (e.g., with reference to an SR map representation or SR coordinate space). In various implementations, the number of ordered positions is greater than the number of entries in the event table 820 (e.g., ten times or one hundred times).

[0119] The SR map object 830 includes an SR context representation table 833 that includes a plurality of entries, each entry corresponding to one of the entries in the event table 820. In various implementations, the number of entries in the SR context representation table 833 is less than the number of entries in the event table 820. For example, in various implementations, no entry in the SR context representation table corresponds to a particular entry in the event table 820.

[0120] Each entry of the SR environment representation table 833 includes an event identifier 841 of the corresponding entry of the event table 820 and an event time 842 of the corresponding entry of the event table 820. Each entry of the SR environment representation table 833 includes an SR environment representation field 843, which includes data indicating the SR environment representation. In various specific implementations, the SR environment representation field 843 includes an SR environment representation, such as Fig. 5F The SR environment indicates 501A to 501E or Fig. 7A In various implementations, the SR environment representation field 843 includes a reference SR environment representation that is stored separately from the SR map object 830, either locally with the SR map object 830 or remotely on another device such as a network server.

[0121] Each entry of the SR environment representation table 833 includes an SR environment representation location field 844 that includes data indicating the location of the SR environment representation (e.g., with reference to an SR mapping representation or an SR coordinate space). Because the SR environment representation is located along a path, each position in the SR environment representation location field 844 is a position in a set of ordered positions of the path indicated by the path representation field 832.

[0122] Each entry of the SR context representation table 833 includes an SR context field 845 that includes data indicating the SR context corresponding to the SR context representation. In various implementations, the SR context field 845 includes an SR context, such as Figure 5D The third SR context 520. In various implementations, the SR context field 845 includes a reference SR context that is stored separately from the SR mapping object 830, either locally with the SR mapping object 830 or remotely on another device such as a network server.

[0123] Fig. 9 900 is a flowchart representation of a method 900 for generating an SR map according to some specific implementations. In various specific implementations, the method 900 is performed by an electronic device (e.g., Figure 3 HMD 120B or Figure 4 In some implementations, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 900 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0124] In block 910 , method 900 begins with the device obtaining media content data represented by a timeline, wherein the media content data includes event markers representing a plurality of events of a story frame.

[0125] In various specific implementations, the media content data is characterized by a timeline so that the corresponding parts of the media content data are presented at different times. In some embodiments, the media content includes video content data represented by a timeline so that at each time, a corresponding image will be displayed, and at the same time, optionally, the sound corresponding to the corresponding image is played simultaneously. In some embodiments, the media content data includes audio content data, wherein the corresponding sound will be played at the corresponding time. In various specific implementations, the audio content data includes music and / or spoken narrative. In some embodiments, the media content data includes text content data, which is represented by a timeline by the estimated text-to-speech time (if read by a text-to-speech module) of the character positions of the various parts of the text in the text content data or the various parts of the text in the text content data.

[0126] In various implementations, an event marker is a portion of the media content data that satisfies event marker criteria. For example, in some embodiments, the event marker criteria include criteria that are satisfied when an object is present in an image of the video content data (e.g., a building, a skylight, or a planet). In some embodiments, the event marker criteria include criteria that are satisfied when the color palette of the image is significantly different from a previous image (e.g., indicating a scene change).

[0127] For example, in some embodiments, the event marking criteria include criteria that are met when there is a music cue. In some embodiments, the event marking criteria include criteria that are met when the volume of the audio violates a threshold or the rate of change of the volume violates a threshold.

[0128] For example, in some embodiments, event marking criteria include criteria that are met when certain words and / or phrases are present (e.g., one or more words that meet the event marking word criteria).

[0129] At block 920, the method 900 continues where the device determines, from the media content data, a plurality of event definitions corresponding to the plurality of events and corresponding event times in the timeline based on the event markers.

[0130] In various specific implementations, the device determines event definitions and corresponding event times by detecting event markers in the media content. In various specific implementations, event markers are detected by image analysis of video content data. In various specific implementations, event markers are detected by audio analysis of audio content data (or audio components of video content data). In various specific implementations, event markers are detected by text analysis of text content data (or text identified from audio content data or audio components of video content data). In various specific implementations, the device detects event markers by determining that a portion of the media content data associated with a specific time in a timeline meets a specific event marker criterion.

[0131] In various specific implementations, the device determines the event definition and the corresponding event time by determining an event definition indicating a specific event marking criterion and determining the corresponding event time as the specific time. For example, in various specific implementations, the event definition indicates that a building is detected in an image of the video content data at the corresponding event time. For another example, in various specific implementations, the event definition indicates that a dramatic music cue is detected in the audio content data at the corresponding event time 822. For another example, in various specific implementations, the event definition indicates that the word "police station" is detected in the text content data at the corresponding event time.

[0132] At box 930, the method 900 continues, where the device obtains multiple SR environment representations based on multiple event definitions. In various specific implementations, the first SR environment representation is obtained based on the first event definition. For example, in some embodiments, the first SR environment representation is selected from multiple SR environment representations stored (locally or remote from the device) based on the first event definition. For example, when the first event definition indicates that the word "Washington Monument" is detected, the device obtains the SR environment representation of the Washington Monument. For another example, when the first event definition indicates that a building is detected in an image of video content data of media content data, a first SR environment representation is generated based on the image of the video content data of the media content data. Therefore, in various specific implementations, the first SR environment representation is generated based on the media content data.

[0133] In some cases, the number of SR environment representations is less than the number of event definitions. For example, in some embodiments, an event definition indicates that a dramatic music cue was detected at a corresponding event time, but a suitable SR environment could not be generated from the corresponding video. As another example, in some embodiments, multiple event definitions correspond to the same SR environment representation. For example, in some embodiments, a first event definition indicates that the word "home" was detected at a first event time, and a second event definition indicates that the word "home" was detected at a second event time. In response, the device may obtain a single SR environment representation for both event definitions.

[0134] At block 940 , the method 900 continues with the device determining a path and a plurality of corresponding locations represented by the plurality of SR environments, wherein the path is defined by an ordered set of locations including the plurality of corresponding locations, the plurality of corresponding locations being in an order based on a plurality of corresponding event times.

[0135] In various implementations, for example, the path includes, in order, a starting location, a first location, a second location, a third location, and an ending location. In some embodiments, the path includes a plurality of locations between the starting location and the first location, a plurality of locations between the first location and the second location, a plurality of locations between the second location and the third location, and / or a plurality of locations between the third location and the ending location. Thus, the second location is further along the path than the first location, and the third location is further along the path than the second location.

[0136] The multiple corresponding event times include: a first event time associated with a first event definition, which is associated with a first SR environment representation; a second event time (later than the first event time) associated with a second event definition, which is associated with a second SR environment representation; and a third event time (later than the second event time) associated with a third event definition, which is associated with a third SR environment representation.

[0137] In various implementations, determining the path and the plurality of corresponding locations includes determining the path and determining the plurality of corresponding locations after determining the path. For example, in various implementations, the device obtains an SR mapping representation (e.g., Figure 5A The SR mapping representation 510 includes a predefined path (eg, corresponding to the path representation 511). After obtaining the predefined path, the device determines a plurality of corresponding locations as locations of the path.

[0138] In addition to the above examples, the device selects a location along the path (e.g., the first location) as a first corresponding location for a first SR context representation, selects a location farther along the path (e.g., the second location) as a second corresponding location for a second SR context representation because the second SR context representation is associated with a time later than the first SR representation, and selects a location even farther along the path (e.g., the third location) as a third corresponding location for a third SR context representation because the third SR context representation is associated with a time later than the second SR representation. In various implementations, the distance along the path is proportional to the amount of time between the start of the timeline and the corresponding event time.

[0139] In various specific implementations, determining the path and the plurality of corresponding positions includes determining the plurality of corresponding positions, and determining the path after determining the plurality of corresponding positions. For example, in various specific implementations, the device obtains the SR mapping representation (e.g., Fig. 7A SR mapping representation 710) and determine multiple corresponding positions. In some embodiments, the SR mapping representation includes predefined candidate positions, and the device selects multiple corresponding positions from the predefined candidate positions. In some embodiments, the device randomly determines multiple corresponding positions. In some embodiments, the device determines multiple corresponding positions according to a space filling algorithm. After obtaining the multiple corresponding positions, the device determines a path that passes through the multiple corresponding positions in sequence based on the multiple corresponding event times.

[0140] In addition to the above examples, the device determines the first position of the path as the position of the first SR environment representation, determines the second position of the path as the position of the second SR environment representation because the second SR environment representation is associated with a time later than the first SR representation, and determines the third position of the path as the position of the third SR environment representation because the third SR environment representation is associated with a time later than the second SR representation.

[0141] In various implementations, the path returns to the same location at different points along the path. Fig. 7E, the first path representation 730A passes through the location of the second SR environment representation 701B twice. Thus, in various implementations, the path is defined by a set of ordered locations that include the same location twice or more. For example, in various implementations, the path includes, in order, a first location of the first SR environment representation associated with a first event time, a second location of the second SR environment representation associated with a second event time that is later than the first event time, and a third location that is the same as the first location because the first SR environment representation is also associated with a third event time that is later than the second event time. For example, a first event definition is associated with the first event time and the first SR environment representation, and a third event definition is associated with the third event time and also with the first SR environment representation.

[0142] In various implementations, determining the path and the plurality of corresponding locations includes determining the path and the plurality of corresponding locations simultaneously (eg, iteratively selecting the path and the plurality of corresponding locations).

[0143] At box 950, method 900 continues, where the device displays an SR map, which includes a plurality of SR environment representations displayed at the plurality of corresponding locations, wherein each of the plurality of SR environment representations is associated with an enable representation that, when selected, causes display of the corresponding SR environment.

[0144] For example, in Fig. 5F In the embodiment, the electronic device 410 displays the SR map 409, which includes a plurality of SR environment representations 501A to 501E displayed at a plurality of corresponding positions. In addition, when selected, when the corresponding SR environment (e.g., Figure 5D 520), each of the plurality of SR environment representations is associated with the affordance representation. Fig. 7E , the electronic device 410 displays a SR map 609 that includes a plurality of SR environment representations 701A to 701E displayed at a plurality of corresponding locations.

[0145] In various implementations, displaying the SR map includes displaying a path representation of the path. In some embodiments, the path representation is embedded in the SR map representation.

[0146] In various implementations, displaying the SR map includes generating the SR map. For example, in various implementations, the device generates an SR map object, such as Figure 8C SR map object 830. In various implementations, the device generates a SR map rendering (eg, an image or overlay) that can be displayed on a display.

[0147] In various specific implementations, displaying the SR map includes simultaneously presenting the media content data according to the timeline. In some embodiments, displaying the SR map includes displaying corresponding SR representations of the plurality of SR representations at corresponding times of the plurality of corresponding times in the timeline.

[0148] In various implementations, method 900 includes receiving user input indicating a selection of an associated affordance representation of a particular SR environment representation among a plurality of SR environment representations, and in response to receiving the user input, displaying the corresponding SR environment.

[0149] In various implementations, displaying the corresponding SR environment includes stopping displaying the SR mapping. In various implementations, displaying the corresponding SR environment includes pausing playback of the media content data. In various implementations, displaying the corresponding SR environment includes displaying at least one virtual object based on the media content data.

[0150] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0151] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0152] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0153] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method, comprising: At an electronic device including a processor and non-transitory memory and located in a physical scene: Obtaining media content data associated with a timeline, wherein the media content data includes event markers representing a plurality of events; determining, from the media content data, a plurality of event definitions corresponding to the plurality of events and a plurality of corresponding event times in the timeline based on the event markers; Obtaining a plurality of synthetic reality SR environment representations based on the plurality of event definitions; The method is characterized in that: determining a path and a plurality of corresponding locations along the path in the physical scene represented by the plurality of SR environments, wherein the path is defined by an ordered set of locations located in the physical scene including the plurality of corresponding locations, the plurality of corresponding locations being in an order based on the plurality of corresponding event times and at corresponding distances along the path that are proportional to an amount of time between a start of the timeline and the corresponding event times; and displaying, in association with the physical scene, a representation of an SR map at a location in the physical scene, a representation of a path at the set of ordered locations in the physical scene, and a plurality of SR environment representations at the plurality of corresponding locations along the path in the physical scene, wherein each of the plurality of SR environment representations is associated with an affordance that, when selected, causes display of a corresponding SR environment; Wherein, the event marker is a portion of the media content data that meets an event marker standard; Wherein, determining an event definition in the plurality of event definitions and a corresponding event time in the plurality of corresponding event times comprises: determining that a portion of the media content data associated with a particular time in the timeline satisfies particular event marking criteria; and The event definition indicating the specific event marking criteria is determined and the corresponding event time is determined to be the specific time. 2 . The method of claim 1 , wherein the media content data comprises at least one of video content data, audio content data, or text content data.

3. The method of claim 1 , wherein obtaining a first one of the multiple SR environment representations based on a first event definition among the multiple event definitions comprises selecting the first one of the multiple SR environment representations from a plurality of stored SR environment representations based on the first event definition. 4 . The method of claim 1 , wherein obtaining a first SR environment representation of the plurality of SR environment representations based on a first event definition of the plurality of event definitions comprises generating the first SR environment representation based on the media content data. The method of claim 1 , wherein the number of SR environment representations is less than the number of event definitions. 6 . The method of claim 1 , wherein determining the path and the plurality of corresponding locations comprises determining the path before determining the plurality of corresponding locations and determining the plurality of corresponding locations after determining the path. 7 . The method of claim 1 , wherein determining the path and the plurality of corresponding locations comprises determining the plurality of corresponding locations before determining the path and determining the path after determining the plurality of corresponding locations.

8. The method of claim 1, wherein the path is defined by a set of ordered positions that includes the same position two or more times. 9 . The method of claim 1 , wherein displaying the SR map representation comprises generating a SR map rendering displayable on a display.

10. The method of claim 1, wherein displaying the SR map representation comprises simultaneously presenting the media content data according to the timeline. 11 . The method of claim 10 , wherein displaying the SR map representation comprises displaying a respective one of the plurality of SR environment representations at a corresponding one of a plurality of respective times in the timeline.

12. The method according to claim 1, further comprising: receiving user input indicating a selection of the affordance; as well as In response to receiving the user input, content is displayed. The method of claim 12 , wherein displaying the content comprises ceasing to display a map. The method of claim 12 , wherein displaying the content comprises pausing playback of the media content data.

15. The method of claim 12, wherein displaying the content comprises displaying a computer-created set. The method of claim 12 , wherein displaying the content comprises displaying at least one virtual object based on the media content data.

17. A device for positioning in a physical scene, the device comprising: non-transitory memory; and One or more processors configured to: Obtaining media content data associated with a timeline, wherein the media content data includes event markers representing a plurality of events; determining, from the media content data, a plurality of event definitions corresponding to the plurality of events and a plurality of corresponding event times in the timeline based on the event markers; obtaining a plurality of SR environment representations based on the plurality of event definitions; Characterized in that the one or more processors are further configured to: determining a path and a plurality of corresponding locations along the path in the physical scene represented by the plurality of SR environments, wherein the path is defined by an ordered set of locations located in the physical scene including the plurality of corresponding locations, the plurality of corresponding locations being in an order based on the plurality of corresponding event times and at corresponding distances along the path that are proportional to an amount of time between a start of the timeline and the corresponding event times; and displaying, in association with the physical scene, a representation of an SR map at a location in the physical scene, a representation of a path at the set of ordered locations in the physical scene, and a plurality of SR environment representations at the plurality of corresponding locations along the path in the physical scene, wherein each of the plurality of SR environment representations is associated with an affordance that, when selected, causes display of a corresponding SR environment; Wherein, the event marker is a portion of the media content data that meets an event marker standard; Wherein, determining an event definition in the plurality of event definitions and a corresponding event time in the plurality of corresponding event times comprises: determining that a portion of the media content data associated with a particular time in the timeline satisfies particular event marking criteria; and The event definition indicating the specific event marking criteria is determined and the corresponding event time is determined to be the specific time.

18. A non-transitory computer-readable medium having encoded thereon instructions that, when executed by one or more processors of a device for positioning in a physical scene, cause the device to: Obtaining media content data associated with a timeline, wherein the media content data includes event markers representing a plurality of events; determining, from the media content data, a plurality of event definitions corresponding to the plurality of events and a plurality of corresponding event times in the timeline based on the event markers; obtaining a plurality of SR environment representations based on the plurality of event definitions; Characterized in that the one or more processors are further configured to: determining a path and a plurality of corresponding locations along the path in the physical scene represented by the plurality of SR environments, wherein the path is defined by an ordered set of locations located in the physical scene including the plurality of corresponding locations, the plurality of corresponding locations being in an order based on the plurality of corresponding event times and at corresponding distances along the path that are proportional to an amount of time between a start of the timeline and the corresponding event times; and displaying, in association with the physical scene, a representation of an SR map at a location in the physical scene, a representation of a path at the set of ordered locations in the physical scene, and a plurality of SR environment representations at the plurality of corresponding locations along the path in the physical scene, wherein each of the plurality of SR environment representations is associated with an affordance that, when selected, causes display of a corresponding SR environment; Wherein, the event marker is a portion of the media content data that meets an event marker standard; Wherein, determining an event definition in the plurality of event definitions and a corresponding event time in the plurality of corresponding event times comprises: determining that a portion of the media content data associated with a particular time in the timeline satisfies particular event marking criteria; and The event definition indicating the specific event marking criteria is determined and the corresponding event time is determined to be the specific time.

Images