Shared event recording and rendering

Abstract

The invention relates to shared event recording and rendering. Various implementations record shared events for providing / rendering a view of the recorded events by generating a time-based three-dimensional (3D) representation. For example, a method may include determining that a second device is currently in an event at a physical environment. The method may also include providing a notification to the second device of an option to authorize use of sensor data based on determining that the second device is currently in the event. The method may also include receiving authorization to use the sensor data. The method may also include obtaining the sensor data from the second device in accordance with the authorization. The method may also include generating a time-based 3D representation of the event based on the sensor data obtained from the second device and additional sensor data obtained from one or more other devices in the event.

Description

Technical Field

[0001] The present disclosure generally relates to electronic devices that provide views of a multi-user environment, including views with time-based 3D representations of shared experiences or events. Background Art

[0002] When a user is using a device such as a head mounted device (HMD), it may be desirable to generate and display a time-based three-dimensional (3D) representation, such as a 3D model, of a shared experience or event (e.g., multiple users at a concert). However, existing systems may not utilize data that may be obtained from other sources to generate such representations. Summary of the invention

[0003] Various specific implementations disclosed herein include devices, systems, and methods for providing a depiction of a shared experience or event within a multi-user 3D environment such as an extended reality (XR) environment. An exemplary process records a shared (e.g., multi-user) event (e.g., a concert, a birthday party, etc.) by generating a time-based 3D representation for providing / rendering views of the recorded event from multiple perspectives. For example, for sixty-two recorded events, the time-based 3D representation can be used to provide different perspectives, wherein a first view is provided for a first time point during the event, a second view is provided for a next time point during the event, and the like. Users at the event can optionally participate in the shared event recording process by contributing sensor data (e.g., RGB images / videos, depth data, location information, etc.) of the user device to the recording of the shared event. Based on consent to participate (e.g., authorization to use data captured by the user device) and / or facilitating the shared event recording, the user / device can be granted access to the generated time-based 3D representation of the event so that the user can relive the event from multiple different perspectives.

[0004] Additionally, various implementations disclosed herein include devices, systems, and methods for rendering views corresponding to multiple time instances during a time period of a recorded event (e.g., providing 60 seconds of different views corresponding to a 60-second recorded event, wherein a first view is provided for a first time point during the event, a second view is provided for a next time point during the event, etc.). In some implementations, a unique viewing perspective of a recorded event is generated based on a previously recorded viewpoint of the event, and the unique viewing perspective need not correspond to a location of a previously recorded event. In some implementations, the viewpoint may be specified by the renderer (e.g., changes based on the current renderer HMD pose).

[0005] In some implementations, a view may be generated using a volume rendering model, such as a neural rendering model (e.g., a neural radiance field (NeRF) model), that is trained using sensor data (e.g., image data) captured from multiple locations / devices during an event. For example, for each rendering time point in a plurality of rendering time points (e.g., rendering frames), a rendering device may "query" the model by specifying inputs to the model (e.g., a time point and a viewpoint) and receive outputs from the model. For example, the output of the model may include information about the visual appearance of the queried point of the event view at that time and from that viewpoint. In some instances, based on the event view at the queried time and time point from the specified input, the query point may be located on a ray corresponding to a pixel.

[0006] In one exemplary implementation, the model additionally returns an "expiration" value (ε) as part of the output (e.g., the expiration time of each ray). The renderer uses this information from the expiration value (ε) to limit its queries to the volume rendering model. For example, if the viewpoint has not changed from one time to the next (within a threshold), the renderer does not need to query the model for certain new values ​​(e.g., rays that have not yet expired). This can reduce the number of queries when rendering dynamic scenes with many static parts from a fixed viewpoint. The renderer only queries the volume rendering model at locations that have changed in the scene, and it avoids repeatedly querying the volume rendering model for parts of the scene that have not changed, which significantly reduces the number of queries required, reduces computational complexity, and can enable rendering at higher frame rates, at higher resolutions, or with less power.

[0007] In some implementations, the expiration value determined by the volume rendering model can be based on a dissimilarity threshold on color / density-transparency differences, such as a standard deviation of values, or a difference between a minimum and a maximum value. Additionally or alternatively, in some implementations, the expiration value determined by the volume rendering model can be based on ray differences (e.g., angle between two rays, ray origin differences, etc.). Additionally or alternatively, in some implementations, the expiration value determined by the volume rendering model can be based on a per-frame expiration budget (similar number of rays expire simultaneously) and / or based on prioritization of visible scene changes.

[0008] In some specific implementations, the shared event recording and rendering process described herein can be used for live broadcast, sampling rate control, and some other possible implementation details. For example, for live broadcast, the shared event recording and rendering process needs to understand the future (e.g., how long this part of the scene remains unchanged), so that the image data is very suitable for offline batch processing of the recorded content. If the final rendering allows a small delay, the shared event recording and rendering process can also be used for live broadcast of events using NeRF plus expiration values ​​(e.g., one NeRF per frame, or one NeRF per second (which encodes 90 frames)). For example, if the live content is delayed by one second, the shared event recording and rendering algorithm can look forward one second into the future, then the expiration value can be any value less than or equal to one second, and for static parts of the scene, the expiration value will be one second. Compared with other NeRF protocols, these rays must be sampled only once per second based on the query, instead of 90 times per second, which significantly improves efficiency.

[0009] Additionally, for example, for sampling rate control, the shared event record and rendering process can utilize ray expiration dates, and determine such expiration values ​​so that the number of all rays expiring at any given timestamp is limited to a certain value (e.g., a per-frame expiration budget). Any rays that should also expire at the same timestamp but will exceed the per-frame expiration budget can be assigned a slightly later expiration date (e.g., one timestamp later). Sorting dynamic scene changes by their difference can be used to prioritize correctly setting expiration dates for strong scene changes, while expiration dates for less severe scene changes can be slightly delayed. In some specific implementations, the determination of expiration dates attempts to achieve a similar number of expiring rays at each given frame, thereby resulting in a more constant computational load over time. This can also be achieved by prioritizing the temporal accuracy of expiration dates for highly visible scene changes, while reducing the priority of less obvious scene changes, which can be assigned slightly later expiration dates, causing the renderer to update them slightly later than when they actually occurred.

[0010] Generally speaking, an innovative aspect of the subject matter described in this specification may be embodied in methods performed at a first device having one or more processors, the methods comprising the following actions: determining, based on one or more event criteria, that a second device is currently in an event at a physical environment; providing a notification to the second device based on the determination that the second device is currently in the event, the notification providing an option to authorize the use of sensor data obtained by the second device during the event to generate a time-based three-dimensional (3D) representation of the event; receiving authorization to generate the time-based 3D representation of the event using the sensor data, wherein the authorization is based on user input received at the second device in response to the notification; obtaining the sensor data from the second device in accordance with the authorization; and generating the time-based 3D representation of the event based on the sensor data obtained from the second device and additional sensor data obtained from one or more other devices in the event.

[0011] These and other embodiments can optionally include one or more of the following features.

[0012] In some aspects, access rights to use the time-based 3D representation of the event are based on the authorization. In some aspects, based on the authorization, a user who facilitates the time-based 3D representation is provided access rights to use the time-based 3D representation of the event.

[0013] In some aspects, the additional sensor data is obtained from the one or more other devices at the event based on receiving additional authorization from the one or more other devices at the event to use the additional sensor data to generate the time-based 3D representation of the event.

[0014] In some aspects, the time-based 3D representation is a neural rendering model. In some aspects, the neural rendering model is a neural radiance field (NeRF) model. In some aspects, the NeRF model is configured to receive input corresponding to a 3D viewpoint and a time point during the event. In some aspects, the NeRF model is configured to output a plurality of ray values ​​corresponding to a view of the event from the 3D viewpoint during the event and the time point. In some aspects, the NeRF model is configured to output an expiration corresponding to one or more of the ray values.

[0015] In some aspects, the expiration is determined based on a dissimilarity threshold corresponding to ray color or ray density. In some aspects, the expiration is determined based on a dissimilarity threshold corresponding to ray position or orientation difference. In some aspects, the expiration is determined based on a per-time point expiration budget. In some aspects, the expiration is determined based on prioritization, the prioritization being determined based on determining visible scene changes.

[0016] In some aspects, the method further comprises the acts of updating a portion of the time-based 3D representation of the event by obtaining user-based image content from a user database; and augmenting the portion of the time-based 3D representation of the event based on the user-based image content.

[0017] Generally speaking, an innovative aspect of the subject matter described in this specification can be embodied in methods performed at an electronic device having a processor, the methods comprising the following actions: for each rendering time point in a plurality of rendering time points, determining a three-dimensional (3D) event viewpoint corresponding to a 3D location in a physical environment of a recorded event; determining an event time point corresponding to a time during a time period of the recorded event; inputting the 3D event viewpoint and the event time point into a neural rendering model, the neural rendering model being trained using sensor data captured by a plurality of devices during the recorded event; receiving an output from the neural rendering model, the output comprising (i) a plurality of ray values ​​corresponding to a view of the recorded event from the 3D viewpoint and the time point during the recorded event; and (ii) an expiration corresponding to one or more of the ray values; and providing the view of the recorded event based on the received output.

[0018] These and other embodiments can optionally include one or more of the following features.

[0019] In some aspects, at least one view for the rendering time point reuses rays from a previous view based on expiration of the rays from the previous view. In some aspects, the method further includes an act of inputting ray data into the neural rendering model, wherein the ray data identifies a first subset of less than all rays in the view to be included in the output or a second subset of less than all rays in the view to be excluded from the output. In some aspects, the first subset or the second subset is determined based on expiration data corresponding to expiration of one or more rays output by the neural rendering model for the previous rendering time point.

[0020] In some aspects, the expiration is determined based on a dissimilarity threshold corresponding to ray color. In some aspects, the expiration is determined based on a dissimilarity threshold corresponding to ray density or transparency. In some aspects, the expiration is determined based on a dissimilarity threshold corresponding to ray position or orientation difference. In some aspects, the expiration is determined based on a per-time point expiration budget. In some aspects, the expiration is determined based on prioritization, the prioritization being determined based on determining visible scene changes.

[0021] In some aspects, a portion of the time-based 3D representation of the event is updated by obtaining user-based image content from a user database; and the portion of the time-based 3D representation of the event is augmented based on the user-based image content.

[0022] In some aspects, generating the time-based 3D representation of the event based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices in the event includes: determining synchronization data based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices based on one or more synchronization algorithms; and generating the time-based 3D representation of the event based on the determined synchronization data. In some aspects, the one or more synchronization algorithms are based on: common clock synchronization associated with the second device and the one or more other devices; image content obtained from the second device corresponding to additional image content obtained from the one or more other devices; audio content obtained from the second device corresponding to additional audio content obtained from the one or more other devices; or a combination thereof.

[0023] In some aspects, generating the time-based 3D representation of the event based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices in the event includes: determining refined image data based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices based on one or more transformation algorithms; and generating the time-based 3D representation of the event based on the determined refined image data.

[0024] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0026] Figure 1 are examples of multiple devices used within a physical environment according to some implementations.

[0027] Figure 2AExample electronic devices operating in the same physical environment while recording content of a shared event are illustrated according to some implementations, wherein a view of a first device is within an extended reality (XR) environment.

[0028] Figure 2B Example electronic devices operating in the same physical environment are illustrated according to some implementations, wherein a view of a second device includes a request to opt-in to share content of an event.

[0029] Figure 3 Example electronic devices are illustrated that operate in the same physical environment and record content of shared events and an associated location map based on the location of the electronic device according to some implementations.

[0030] Figure 4 Examples are provided according to some specific implementations of FIG. 2A to FIG. 2B Example electronic devices operating in different physical environments.

[0031] Figure 5A According to some specific implementations, Figure 4 An exemplary 3D environment generated by a physical environment and a portal displaying portal content within the 3D environment.

[0032] Figure 5B Examples are provided according to some specific implementations and in Figure 5A Example interactions of portal content displayed within a portal.

[0033] Figure 6 An environment for implementing a process for generating a time-based 3D representation of a shared event based on sensor data from two or more devices according to an embodiment of the present invention is illustrated.

[0034] Figure 7 is a process flow diagram illustrating an exemplary process for generating a time-based 3D representation of a shared event from a volume rendering model based on an expiration value according to some implementations.

[0035] Figure 8 is a flow chart illustrating a method for generating a time-based 3D representation of a shared event based on sensor data from two or more devices according to some implementations.

[0036] Fig. 9 is a flow chart illustrating a method for providing a view of recorded events based on output from a neural rendering model according to some implementations.

[0037] Fig.10 is a block diagram of an electronic device according to some specific implementations.

[0038] Fig.11is a block diagram of a head mounted device (HMD) according to some specific implementations.

[0039] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0040] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0041] Figure 1 An example environment 100 is illustrated of exemplary electronic devices 105, 165, 175, and 185 operating in a physical environment 102. Additionally, the example environment 100 includes an information system 104 that communicates with one or more of the electronic devices 105, 165, 175, and 185. In some implementations, the electronic devices 105, 165, 175, and 185 may be able to share information with each other or with an intermediate device (such as the information system 104). Additionally, the physical environment 102 includes a user 110 wearing the device 105, a user 160 holding the device 165, a user 170 holding the device 175, and a user 180 holding the device 185. In some implementations, the devices are configured to present views of an extended reality (XR) environment that may be based on the physical environment 102, and / or include added content (such as virtual elements that provide text narration).

[0042] exist Figure 1In the example of , the physical environment 102 is a room including physical objects such as wall hangings 120, plants 125, and tables 130. Each electronic device 105, 165, 175, and 185 may include one or more cameras, microphones, depth sensors, motion sensors, or other sensors that can be used to capture information about the physical environment 102 and objects therein, as well as information about each user 110, 160, 170, and 180 of the electronic devices 105, 165, 175, and 185 and evaluate the information. Information about the physical environment 102 and / or each user 110, 160, 170, and 180 can be used to provide visual and audio content during the recording of the shared event or experience. For example, the shared experience / event session can provide a view of a 3D environment that is generated based on camera images and / or depth camera images from one or more electronic devices of the physical environment 102, based on the captured camera images and / or depth camera images of the environment. One or more electronic devices may provide a view of a 3D environment that includes representations of users 110 , 160 , 170 , and 180 .

[0043] exist Figure 1 In an example of , the first device 105 includes one or more sensors 116 that capture light intensity images, depth sensor images, audio data, or other information about the user 110 and the physical environment 100. For example, the one or more sensors 116 may capture images of the user's forehead, eyebrows, eyes, eyelids, cheeks, nose, lips, chin, face, head, hands, wrists, arms, shoulders, torso, legs, or other body parts. As an example, sensor data about the user's eyes 111 may indicate various user characteristics, such as the user's gaze direction 119 over time, the user's scanning behavior over time, the user's eye dilation behavior over time, etc. The one or more sensors 116 may capture audio information, including the user's voice and sounds made by other users and sounds within the physical environment 100.

[0044] One or more sensors, such as one or more sensors 115 on device 105, may identify user information based on proximity or contact with a portion of user 110. For example, one or more sensors 115 may capture sensor data that may provide bio-information related to the user's cardiovascular state (e.g., pulse), body temperature, respiratory rate, etc.

[0045] One or more sensors 116 or one or more sensors 115 may capture data from which a user orientation 121 within the physical environment may be determined. In this example, user orientation 121 corresponds to the direction in which the torso of user 110 is facing.

[0046] Some specific implementations disclosed herein determine user understanding based on sensor data obtained by a device worn by a user (such as the first device 105). Such user understanding may indicate a user state associated with providing user assistance. In some examples, the user's appearance or behavior or understanding of the environment may be used to identify the need or desire for assistance so that the user can obtain such assistance. For example, based on determining such a user state, an augmentation may be provided to assist the user by enhancing or supplementing the user's capabilities (e.g., providing guidance or other information about the environment to a disabled / impaired person).

[0047] The content may be visible (e.g., displayed on a display of device 105) or audible (e.g., produced as audio 118 by a speaker of device 105). In the case of audio content, the audio 118 may be produced in a manner such that only user 110 is likely to hear the audio 118 (e.g., via a speaker close to the user's ear 112 or at a volume below a threshold), making it less likely that nearby people (e.g., users 160, 170, etc.) will hear it. In some implementations, the audio mode (e.g., volume) is determined based on determining whether the other person is within a threshold distance or based on how close the other person is relative to user 110.

[0048] In some implementations, the content provided by the device 105 and the sensor features of the device 105 may be provided using components, sensors, or software modules that are small enough and efficient in power consumption and use to fit and otherwise be used in lightweight, battery-powered wearable products such as wireless earbuds or other ear-hook devices or head-mounted devices (HMDs) such as smart / augmented reality (AR) glasses. A combination of multiple devices may be used to facilitate features. For example, a smartphone (wirelessly connected and interoperable with a wearable device) may provide computing resources, connections to cloud or Internet services, location services, etc.

[0049] Figure 2A Example electronic devices operating in the same physical environment while recording content of a shared event are illustrated according to some implementations, wherein the view of a first device is within an extended reality (XR) environment. Specifically, Figure 2A Exemplary environment 200A illustrates an exemplary view 205A of physical environment 102 provided by electronic device 105 during an event (eg, a party) where user 160 uses device 165. View 205A is a view of 3D environment 250 based on physical environment 102. Figure 2A and Figure 2BThe example illustrates recording content during an event and determining whether to agree to share the content of the recorded event with other devices. For example, user 110 views representations of other users 160 and a representation of user 110's physical environment 102 (e.g., user 110's office). View 205A includes representation 225 of plant 125, representation 220 of wall hanging 120, and representation 230 of table 130. Additionally, Figure 2A represents the view of user 110 after user 110 initiates recording of event content. Figure 2A As illustrated in FIG. 2 , a notification recording symbol 280 is provided to the user 110 as a notification to the user that the device 105 is participating in a live recording of the event / experience.

[0050] Figure 2B An exemplary electronic device operating in the same physical environment during an event is illustrated, wherein a view of the device includes a request to opt-in to share content of the shared event, according to some implementations. Figure 2B An exemplary operating environment 200B is illustrated of an exemplary view 205B of an electronic device 165 recording an event from his or her perspective while a user 110 using the device 105 is recording the same event (e.g., a party) from his or her perspective. The view 205B is a view of a 3D environment 255 as a representation of the physical environment 102 of the device 165. The view 205B includes a representation 220 of the wall 120 and a representation 230 of the table 130. Specifically, Figure 2B A text transcript (e.g., transcript bubble 290) initiating an opt-in request is illustrated to request permission from user 160 to participate in sharing recorded content of a particular shared event. The request may be initiated by another device (e.g., device 105) recording information about the same event, or may be initiated by another source such as information system 104 (e.g., an external system that determines that the device is recording the same event). Thus, by selecting to share content of the event, user 160 initiates the transmission of recorded content from the event to be shared. Additionally or alternatively, by providing access to the recorded event, the system may provide user 160 with access to view a time-based volume rendering of the shared event at a later time, as further discussed herein.

[0051] Figure 3 An exemplary electronic device is illustrated that operates in the same physical environment and records the content of a shared event and an associated location map based on the location of the electronic device according to some specific implementations. Specifically, Figure 3 Illustrated Figure 1300 of a physical environment 102 of the information system 104, a location network map 310 (e.g., a mesh network map) generated from a geolocation instruction set 304 based on the determined locations of devices in the physical environment 102, and an image content database 330 for collecting recorded content 335 from one or more devices. For example, a geolocation network (e.g., a mesh network) can be utilized based on location / location data of multiple devices (e.g., devices 105, 165, 175, 185, etc.) in a room, while the identity of each device remains anonymous (e.g., via anonymization, tokenization, etc.) as the information system 104 records and collects image content from each device.

[0052] In an exemplary implementation, the content instruction set 306 of the information system 104 collects recorded content 335 from one or more devices and stores the content in one or more image content databases 330. For example, the information system 104 can generate a complete 3D representation of the event based on 2D and / or 3D images captured by the device by rendering views corresponding to multiple time instances during the time period of the recorded event (e.g., different views corresponding to 60 seconds of recorded events can be provided, wherein a first view is provided for a first time point during the event, a second view is provided for a next time point during the event, etc.). In some implementations, a unique viewing perspective of the recorded event can be generated by the content instruction set 306 based on a previously recorded viewpoint of the event, and the unique viewing perspective does not need to correspond to the location of the previously recorded event. In some implementations, the viewpoint can be specified by the renderer (e.g., changes based on the current renderer HMD pose). In some implementations, the 3D representation can include an expiration value of another device to more efficiently render the content (as will be discussed in more detail herein).

[0053] The location map 310 illustrates a two-dimensional (2D) top view of the location of device representations or other object representations within a 3D environment. In this example, during an example of recording event content, a recording or viewing session instruction set executed on an electronic device (e.g., device 105) or networked through an external server can generate the location map 310 based on detected devices (e.g., devices 105, 165, 175, etc.). For example, as illustrated in the location map 310, the location of device 105 indicated by location indicator 305 is {ABCD}, the location of device 165 indicated by location indicator 365 is {EFGH}, the location of device 175 indicated by location indicator 375 is {MNOP}, and the location of device 185 indicated by location indicator 385 is {IJKL}. In some specific implementations, the location of each device can be determined and / or approximated based on the location of another device at a particular time (e.g., based on short-range sensor data, GPS coordinates, WiFi location, SLAM positioning technology, combinations thereof, etc.). In some implementations, the location of each device may be determined and / or approximated based on identifying one or more objects within the view of the acquired image. For example, if the event is a party, the person having a birthday may be used to detect the locations of other devices that have accepted to share data from the same event. Additionally or alternatively, a static object may be used as an anchor, such as table 130. Thus, as the user / device moves throughout the environment, as new content is acquired, the static object (table) may be used as an anchor when analyzing and combining different subsets of the RGB image data during the recording of the party (e.g., a 60 second clip of watching a birthday party blowing out candles may be recreated from content from multiple devices).

[0054] In an exemplary implementation, the physical environment includes a third user (or more users) associated with a third device (e.g., device 175 of user 170, device 185 of user 180 within physical environment 102) to be depicted in the view of the 3D environment by the first device. In some implementations, determining location data indicating a location of another device relative to the first device is based on identifying a mesh network associated with location information of the first device, the second device, and the third device. For example, Figure 3 As illustrated in , the mesh network can be used to continuously update the location of each device. Specifically, for physical environment 102, location indicator 305 depicts the location of device 105 at Loc:{ABCD}, location indicator 365 depicts the location of device 165 at Loc:{EFGH}, location indicator 385 depicts the location of device 185 at Loc:{IJKL}, and location indicator 375 depicts the location of device 175 at Loc:{MNOP}.

[0055] Figure 4An exemplary electronic device 405 is illustrated operating in a physical environment 400. Specifically, Figure 4 Example of Figure 1 102, the physical environment of the device 105, 165, 175, and / or 185. In other words, this different environment is where a user (e.g., user 402) may want to view and replay a shared event recorded by device 105, 165, 175, and / or 185. Additionally, by wearing the HMD, user 402 may be able to fully immerse in the view of the party / event (e.g., may walk around based on different poses of device 405, and then may update the applicable scene reconstruction). Alternatively, as Figure 5A , Figure 5B As illustrated in , only a partial immersive view may be used, such as a view through a portal.

[0056] exist Figure 4 , the physical environment 400 is a room including a sofa 420, a wall hanging 450, and a television screen 470. The electronic device 405 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture and evaluate information about the physical environment 400 and objects therein, as well as information about the user 402 of the electronic device 105. The information about the physical environment 400 and / or the user 402 can be used to provide visual and audio content and / or to identify the current location of the physical environment 400 and / or the location of the user within the physical environment 400.

[0057] In some implementations, a view of an XR environment may be provided to one or more participants (e.g., user 402 and / or other participants not shown, such as user 110) via an electronic device 405 (e.g., a wearable device such as an HMD and / or a handheld device such as a mobile device, a tablet computing device, a laptop computer, etc. (e.g., device 165)). Such an XR environment may include a view of a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 400 and a representation of the user 402 based on the camera image and / or the depth camera image of the user 402. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system (e.g., 3D space) associated with the XR environment, which 3D coordinate system may correspond to the 3D coordinate system of the physical environment 400.

[0058] In some embodiments, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 405) and used to render an XR environment. In other embodiments, optical perspective may be used to render an XR environment by overlaying virtual content on a view of the physical environment seen through a translucent or transparent display. In some embodiments, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment may be generated based on, in particular, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., so that the 3D representation is configured to align with corresponding features of the physical environment. In some embodiments, a viewpoint within a 3D coordinate system may be determined based on the position of the electronic device within the physical environment. The viewpoint may be determined based on, in particular, image data, depth sensor data, motion sensor data, etc., which may be retrieved via a virtual inertial odometer system (VIO), a simultaneous localization and mapping (SLAM) system, etc.

[0059] Figure 5A According to some specific implementations, Figure 4 4. An exemplary 3D environment 500 generated from a physical environment 400 and a portal 580 that displays content 585 within the 3D environment. Portal 580 may also be referred to herein as a projection of a 3D image. 3D environment 500 includes representations 525, 550, and 570 of sofa 420, wall hanging 450, and TV screen 470 of physical environment 400, respectively. 3D environment 500 also includes content 585 that is displayed to form the shape of portal 580. In some specific implementations, the shape of portal 580 can be any geometric shape that can project content (e.g., a 3D virtual shape such as a hemisphere, also known as a "snowball" view). The content 585 that portal 580 is displaying constitutes a portal (e.g., a projection of an image), as discussed herein.

[0060] The electronic device 405 provides views of the physical environment 400, which include a depiction of the 3D environment 500 from a viewpoint 520 (e.g., also referred to herein as a viewer location), which in this example is determined based on the location of the electronic device 405 in the physical environment 400 (e.g., the viewpoint of the user 502, also referred to herein as the "viewer location" or "viewer viewpoint"). Therefore, as the user 402 moves with the electronic device 405 (e.g., HMD) relative to the physical environment 400, the viewpoint 520 corresponding to the location of the electronic device 405 moves relative to the 3D environment 500. The view of the 3D environment provided by the electronic device changes based on the change of the viewpoint 520 relative to the 3D environment 500. In some specific implementations, the 3D environment 500 does not include a representation of the physical environment 500, for example, only includes virtual content corresponding to a virtual reality environment.

[0061] Figure 5B Examples are provided according to some specific implementations and in Figure 5A 5 illustrates an exemplary "indirect" interaction involving selecting or interacting with content of a user interface (e.g., content 585) within portal 580 while gazing along gaze direction 590. In this example, user 402 is using device 405 to view and interact with an XR environment, which may include a user interface (e.g., content 585 within portal 580) within a view of XR environment 510. The direct interaction identification process may use sensor data and / or UI information to determine, for example, which UI element a user's hand is virtually touching and / or where on the UI element an interaction is occurring. Direct interaction may additionally (or alternatively) involve evaluating user activity to determine the user's intent, for example, whether the user intends to perform a direct tap gesture through a UI element or a sliding / scrolling motion along the UI element. Such identification may utilize information about the UI element, for example, regarding the position, size, element type, types of interactions that can be performed on the element, types of interactions enabled on the element, which of a set of potential target elements for user activity accepts which types of interactions, and the like.

[0062] Figure 5B Further illustrated Figure 5A A view of an XR environment 510 provided via device 405 of virtual elements within a 3D physical environment of a user 402, wherein a user 402 may perform interactions. In this example, the user 402 makes gestures relative to content presented in the view of the XR environment 510 provided by a device (e.g., device 405). The view of the XR environment 510 includes an exemplary user interface of an application (e.g., content 585 within portal 580) and a drawing 570 of a television screen 470 (e.g., a representation of a physical object, which may be viewed as a pass-through video or may be a direct view of the physical object through a transparent or semi-transparent display). Additionally, the view of the XR environment 510 may include a representation of the hand / arm of the user 402. Providing such a view may involve determining 3D properties of the physical environment 400 and positioning virtual content (e.g., a user interface within portal 580) in a 3D coordinate system corresponding to the physical environment 400.

[0063] exist Figure 5BIn the example of , the user interface within portal 580 includes various content items, including a background portion, an application portion, control elements, and scroll bars (not shown). The application portion (e.g., content 585) can be displayed in a view provided by device 405 together with a 3D effect. For illustrative purposes, the user interface within portal 580 (e.g., portal content 585 displayed within 580) is simplified, and the user interface can actually include any level of complexity, any number of content items, and / or a combination of 2D and / or 3D content. The user interface of portal 580 can be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0064] In some implementations, the position and / or orientation of such one or more user interfaces may be determined to facilitate visibility and / or use. Within the 3D environment, one or more user interfaces may be in a fixed position and orientation. In such a case, user movement will not affect the position or orientation of the user interface within the 3D environment. Additionally, the view of the XR environment 510 may change based on the immersion level. For example, Figure 5B , user 402 may clearly view his or her background environment outside portal 580 and therefore would be considered to be at a lower level of immersion. However, the rendered 3D content 585 may be scaled to be more immersive so that user 402 may be able to walk around in his or her physical environment but be able to view and interact with the recording of the shared event based on head pose and position to view different viewpoints of the rendered event that may not be captured from the original source viewpoint but rather synthesized (e.g., creating a 3D view from a series of 2D images). View synthesis may be accomplished using a series of photographs that show an object from multiple angles, creating a hemispherical plane of the object, and placing each image in an appropriate place around the object. The view synthesis function attempts to predict depth given a series of images depicting different perspectives of an object.

[0065] The position of the user interface within the 3D environment may be based on determining the distance of the user interface from the user (e.g., from an initial or current user position). The position and / or distance from the user may be determined based on a variety of criteria, including but not limited to criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, environment lighting, the presence of other people in the environment, use of applications or content by multiple users, user preferences, user input, and numerous other factors. Figure 1In the example of FIG. 5 , the electronic devices 165, 175, and 185 are illustrated as handheld devices, and the electronic devices 105 and 405 are illustrated as head mounted devices (HMD). However, each of the electronic devices 105, 165, 175, 185, and 405 may be a mobile phone, a tablet computer, a laptop computer, and the like. In some implementations, the electronic devices 105, 165, 175, 185 may be worn by a user, such as the devices 105 and 405. For example, the electronic devices 105, 165, 175, 185, and 405 may be a watch, an HMD, a head mounted device (glasses), a headset, an ear mounted device, and the like. In some implementations, the functions of the electronic devices 105, 165, 175, 185, and 405 are implemented via two or more devices, such as a mobile device and a base station or a head mounted device and an ear mounted device. Various functions can be distributed among multiple devices, including but not limited to power functions, CPU functions, GPU functions, storage functions, memory functions, visual content display functions, audio content production functions, etc. Multiple devices that can be used to implement the functions of electronic devices 105, 165, 175, 185, and 405 can communicate with each other via wired or wireless communications. In some specific implementations, each device communicates with a separate controller or server to manage and coordinate the user's experience (e.g., information system / server). Figure 6 As further illustrated in , such a controller or server may be located in the physical environment 102 or may be remote from the physical environment.

[0066] Figure 6 An example environment 600 for implementing a process for generating a time-based 3D representation of a shared event based on sensor data from two or more devices according to some implementations is illustrated. The example environment 600 includes one or more client devices 610 (e.g., electronic devices 105, 165, 175, 185, 405, etc.) and an information system server 620, which communicate via a data communication network 602 (e.g., a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof).

[0067] The electronic device 610 (e.g., an electronic device used by a user, such as the user device 105 used by the user 110) can be a mobile phone, a tablet computer, a laptop, an HMD, etc. In some specific implementations, each electronic device 610 can be worn by a user. For example, the electronic device 610 can be a watch, a head-mounted device (HMD), a head-mounted device (glasses), a headset, an ear-hook device, etc. In some specific implementations, the functions of the device 610 are implemented by two or more devices, such as a mobile device and a base station or a head-mounted device and an ear-hook device. Various functions can be distributed among multiple devices, including but not limited to power functions, CPU functions, GPU functions, storage functions, memory functions, visual content display functions, audio content production functions, etc. Multiple devices that can be used to implement the functions of the electronic devices 105, 165, 175, 185, and 405 can communicate with each other via wired or wireless communications through the network 602. In some specific implementations, each device communicates with a separate controller or server to manage and coordinate the user experience (e.g., an information system server 620 using a communication session instruction set 616). Such a controller or server may be located in the physical environment of device 105 (eg, physical environment 102) or may be remote from the physical environment.

[0068] The example system flow of the example environment 600 includes a client device 610 acquiring light intensity image data (e.g., a live camera feed such as RGB from a light intensity camera), depth image data (e.g., RGB-D from a depth camera), and images from a physical environment (e.g., Figure 1 The client device 610 may also include a rendering instruction set 616 that generates 3D representation data (e.g., a representation of an event) based on a volume rendering model (e.g., a NeRF model) from an information system server 620.

[0069] For positioning information, the client device 610 may also include a geolocation instruction set 612. The geolocation instruction set 612 and the device 610 may include a VIO system to determine equivalent odometer information using sequence camera images (e.g., light intensity image data) and motion data (e.g., obtained from an IMU / motion sensor) to estimate the driving distance. In some specific implementations, other sources of physical environment information (e.g., camera positioning information, such as location and orientation data from a position sensor) may be obtained in contrast to using a VIO system. Alternatively, some specific implementations of the present disclosure may include a simultaneous positioning and simultaneous construction (SLAM) system (e.g., a positioning sensor). The SLAM system may include a multi-dimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous positioning and mapping. The SLAM system may generate and manage data of a very accurate point cloud generated by reflections of laser scans from objects in the environment. Over time, the movement of any point in the point cloud is accurately tracked so that the SLAM system can use the points in the point cloud as reference points for the location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0070] In an example implementation, the client device 610 includes a geolocation instruction set 612 configured with instructions executable by a processor to obtain sensor data (e.g., RGB data, depth data, etc.) and track the position of a mobile device (e.g., device 610, device 105) in a 3D coordinate system using one or more techniques. For example, the geolocation instruction set 612 analyzes an RGB image from a light intensity camera using a sparse depth map from a depth camera (e.g., a time-of-flight sensor), plane extraction data (e.g., plane estimation parameters), and other sources of physical environment information (e.g., camera positioning information, such as VIO data or a camera's SLAM system, etc.) to generate position data for 3D reconstruction of a scene (e.g., representation) by tracking the device position information. Figure 1 A 3D model of one or more objects in the physical environment of the device).

[0071] In an example implementation, the device 610 includes a shared content instruction set 614 configured with instructions executable by a processor to obtain sensor data (e.g., RGB data, depth data, etc.) and location data from a geolocation instruction set 612 using one or more techniques and generate 3D representation data. For example, the shared content instruction set 614 utilizes a sparse depth map from a depth camera (e.g., a time-of-flight sensor, a passive or active stereo sensor such as a structured light depth camera, etc.) and other sources of physical environment information (e.g., camera positioning information, such as VIO data or a camera's SLAM system, etc.) to analyze an RGB image from a light intensity camera to generate 3D representation data. For example, as Figure 2AAs illustrated in the example view of environment 200A in , the 3D representation data may include a representation 230 of table 130 and a representation 220 of wall hanging 120 .

[0072] In an example implementation, the device 610 includes a rendering instruction set 616 configured with instructions executable by the processor to obtain a volume rendering model from the information system server. For example, in some implementations, the device 610 obtains the volume rendering from the information system server 620 via the rendering instruction set 616 via a volume rendering model instruction set 622 (which may store such a model in a representation model database 640) and a time-based 3D representation instruction set of the query volume rendering model instruction set 622, as described herein with reference to Figure 7 Further described.

[0073] In an example implementation, the device 610 also includes an augmentation instruction set 618 configured with instructions executable by a processor to facilitate obtaining client-specific images (e.g., a personal photo library) from a locally stored database (e.g., a user content database 619) or from an online media library (e.g., a cloud database) to help synthesize data that may be missing from a shared recorded event. In other words, the augmentation instruction set 618 can facilitate access to a client-specific image library storing personal pictures to potentially fill in images of recorded events that may be missing data. For example, personal images can be used to augment events that are recreated only for specific users with personal photos. For example, if the perceived view of the event is from a location that was not originally captured in the initial recording (e.g., the front of a friend's head is captured), then if the perspective of the replay is from a location where the back of the friend's head (not originally captured) is seen, the client device 610 can utilize the stored personal image of the back of the friend's head (captured at a completely different time and place) to fill in the content during the replay.

[0074] The information system server 620 (e.g., a server within the information system 104) is an external server configured to facilitate the generation of a time-based 3D representation system based on image data from two or more client devices 610. In some implementations, the information system server 620 determines that a user has selected to receive image data for a particular shared event. For example, the information system server 620 can access all shared content associated with the event from the image content database 630.

[0075] In an example implementation, the information system server 620 includes a volume rendering model instruction set 622 configured with instructions executable by a processor to facilitate the exchange of input data (e.g., 5D input data including position (x, y, z) and 2D viewing direction {θ, Φ}) and output data (e.g., 5D output data including color {RGB}, density {σ}, and expiration value {ε}) based on a query from a time-based 3D representation instruction set 624, which will be referred to herein as Figure 7 Further description. The volume rendering model instruction set 622 can be a NeRF neural network. For example, NeRF is a fully connected neural network that can generate novel views of a complex 3D scene based on a partial 2D image set. NeRF is trained to reproduce an input view of a scene using a rendering loss, and works by taking input images representing the scene and interpolating between them to render a complete scene. NeRF is an efficient way to generate images for synthetic data. However, one advantage of the present disclosure over a typical NeRF model is that the volume rendering model instruction set 622 can also output an expiration value {ε}, which will be referred to herein. Figure 7 Further discussion.

[0076] In an example implementation, the information system server 620 also includes a synchronization instruction set 626 configured with instructions executable by a processor to synchronize data from multiple devices (e.g., looking up timestamps of frames from different devices based on a common clock). For example, content data (e.g., images, videos, etc.) obtained from multiple client devices 610 is non-synchronized data and needs to be synchronized before a volume rendering model is generated by the volume rendering model instruction set 622. The synchronization of data received from multiple non-synchronized devices can be based on a common clock synchronization such as an Internet Time Service (ITS), captured image / video content, captured audio, or a combination thereof.

[0077] Additionally or alternatively, in some implementations, synchronization of data may be based on audio recorded via one or more techniques. For example, synchronization between two recordings may be based on audio classification by associating audio classes that require temporal overlap to determine synchronization points. Additionally, synchronization between two recordings may be based on audio fingerprints to represent the recorded audio. Additionally or alternatively, in some implementations, synchronization of data may be based on tracking moving objects and identifying inflection points, and using consensus-based matching heuristics to refine estimates of synchronization to find moving features from still image features that are most consistent with pre-computed camera geometry. Additionally or alternatively, in some implementations, synchronization of data may be based on detecting spatio-temporal interest points in the data and using a uniform search with a uniform sampling algorithm in each video using a scale-adaptive technique to select the strongest interest points to form a distribution of spatio-temporal interest points. This distribution becomes a descriptor of the temporal features of the video, and a correlation algorithm may associate these distributions and estimate temporal differences between videos.

[0078] In some implementations, even after synchronization (e.g., finding timestamps for frames from different devices based on a common clock), the actual camera integration times of different cameras are likely to be misaligned, so frames from different devices may be close in time, but may not capture the same point in time. In addition, images from different devices may have passed through different imaging pipelines, and therefore, even if each client device 610 observes common portions of the scene, these common portions may look very different in the images from the two cameras based on different camera properties. For example, even two photos taken at the same time and location may exhibit considerable variation, i.e., exposure, color correction, and tone mapping may all vary from camera to camera and post-processing. Therefore, in an exemplary implementation, the synchronization instruction set 626 and / or the volume rendering model instruction set 622 may also be configured to address these issues related to the temporal aspects of dynamic reconstruction.

[0079] In some implementations, to account for potential misalignment of camera integration times of different cameras (e.g., frames from different devices may be close in time but may not capture the same point in time), the synchronization instruction set 626 and / or the volume rendering model instruction set 622 may be configured to use temporal interpolation (with a suitable motion model) to synthesize camera frames at a desired timestamp based on at least one earlier frame and one later frame.

[0080] In some implementations, in order to account for variations in camera properties from two or more devices (e.g., different imaging pipelines, filtering techniques, lighting issues, etc.), the synchronization instruction set 626 and / or the volume rendering model instruction set 622 may be configured to remove any filtering properties from the RGB photo (e.g., based on metadata embedded with each image). In other words, the synchronization and volume rendering model systems described herein may modify the obtained RGB images including various imaging properties or filters, and refine these images to account for variations in camera properties from two or more devices due to different imaging pipelines, filtering techniques, lighting issues, etc. For example, since almost all of these types of image transformations are lossy and irreversible, the system may attempt to use inverse tone mapping and approximation of image filters to make them as comparable as possible between image frames and to make them closer to the original image. In some embodiments, if the system can control the recording application software and hardware (e.g., a camera), additional metadata is recorded to make the approximation of the inversion as accurate as possible, and / or fewer processing effects are applied to the image frames or video, so that the image frames or video may not look as good when observed without further processing (e.g., without an image filter), but the image frames or video data can retain more of the original information and thus do not need to be reconstructed afterwards with a lossy process.

[0081] In an example implementation, the information system server 620 also includes a geolocation instruction set 628 configured with instructions executable by a processor to facilitate receiving location and / or position information from a plurality of client devices 610 (e.g., user devices 105, 165, 175, 185, 405, etc.). In an example implementation, the information system server 620 also includes a mesh / tokenization instruction set configured with instructions executable by a processor to facilitate receiving location and / or position information received from a plurality of anonymized devices from the geolocation instruction set 628, facilitating a tokenization process for maintaining anonymization between devices, and / or facilitating a mesh network between a plurality of devices. For example, a mesh network may be utilized based on location / location data of a plurality of devices (e.g., devices 105, 165, 175, 185, 405, etc.) in a room, while the identity of each device remains anonymous (e.g., via anonymization, tokenization, etc.). In an exemplary implementation, the physical environment includes a third user (or more users) associated with a third device (e.g., device 175 of user 170, device 185 of user 180 within physical environment 102) to be depicted in the view of the 3D environment by the first device. In some implementations, determining location data indicating a location of the second device relative to the first device is based on identifying a mesh network associated with location information of the first device, the second device, and the third device. For example, Figure 3As illustrated in , the mesh network can be used to continuously update the position of each device and thus obtain the current viewpoint or pose of each device for each captured image. For example, device A (e.g., device 105) may send a message to a server (e.g., information system 104) saying "I saw a device with anonymous token 1234 at location X" (e.g., device 105 identifies device 165 at location {EFGH}). Device B (e.g., device 165) may later say "I saw a device with anonymous token 5678 at location X'". For example, because there is a rotating token system, anonymous token 5678 refers to the same device as token 1234 (e.g., device 185 identifies device 165 at location {EFGH}). Thus, when device C (e.g., device 175) sees a device (e.g., device 165) at location X”, it may send location X” to the server, which will retrieve and return user settings associated with the device (e.g., device 165) at that location, which were previously determined to be around location X / X’ at location {EFGH} (which should correspond to location X”), as anonymously determined by the tokens sent by devices A and B.

[0082] Figure 7 An example environment 700 is illustrated for implementing a process for generating a time-based 3D representation of a shared event from a volume rendering model based on an expiration value according to some specific implementations. The example environment 700 may include a viewing device 405, a time-based 3D representation instruction set 624, and a volume rendering model instruction set 622 that communicates via a data communication network (e.g., network 602) (e.g., a LAN, WAN, the Internet, a mobile network, or a combination thereof). The time-based 3D representation instruction set 624 and the volume rendering model instruction set 622 may each be hosted externally (e.g., hosted at an information system server 620), or may be executed on a viewing / rendering device (e.g., device 405).

[0083] In the example system flow of the example environment 700, the synchronization instruction set 626 obtains content data 745 from multiple content 740 from two or more different devices (e.g., client devices 610) that obtain sensor data at a specific event. As discussed herein, the synchronization instruction set 626 can solve one or more problems related to associating and synchronizing content data from two or more devices (e.g., common clock synchronization, integration of precise time points, and / or different imaging pipelines). For example, the content data (e.g., images, videos, etc.) obtained from multiple client devices 610 is non-synchronized data and needs to be synchronized before the volume rendering model is generated by the volume rendering model instruction set 622. The synchronization of the data received from multiple non-synchronized devices can be based on common clock synchronization such as Internet Time Service (ITS), captured image / video content, captured audio, or a combination thereof.

[0084] After synchronizing the image data from the two or more devices, the volume rendering model instruction set 622 obtains the synchronized content data 747 from the synchronization instruction set 626 and synthesizes the synchronized content data 747 to generate a sampled 3D point set (e.g., point 715) by marching camera rays through the scene. After the event has been synthesized (e.g., the shared experience is ready for viewing), the rendering device (e.g., device 405) initiates the process of viewing and interacting with the shared recorded event (e.g., viewing from a different viewpoint) via event request 752. For example, user 402 may have attended the event (e.g., as Figure 1 ) and wish to replay the experience from a different viewpoint than the one that user 402 may have recorded during the event. Alternatively, if user 402 did not attend the event, another user may provide user 402 with access to the shared event.

[0085] In an exemplary implementation, a volume rendering model instruction set 622 (such as a neural rendering model (e.g., a NeRF model) trained using sensor data (e.g., image data) captured from multiple locations / devices (e.g., devices 105, 165, 175, 185, etc.) during an event) may be used to generate a view at a rendering device (e.g., device 405), such as Figure 1 . For example, for each rendering time point in a plurality of rendering time points (e.g., rendering frames), the rendering device 405 may "query" the model (e.g., query 760 at t) by specifying query input data 750 (e.g., a time point and a viewpoint) to the model and receive output from the model (e.g., a ray corresponding to a pixel of the event view at that time and from the viewpoint). For example, an event request 752 may initiate rendering of an event at a particular time point, and the query input data 750 may include 5D inputs such as a spatial position 754 (e.g., position coordinates (x, y, z)) and a viewpoint / pose 756 (e.g., a 2D viewing direction (θ, Φ)). In Figure 7 In the example of , a representation 702 of an object (e.g., a plant on top of a stool) is used to illustrate the volume rendering process using a NeRF model, which is used to generate 2D images 704, 706 for multiple points along the direction / viewpoint of the camera image and illustrated with 3D coordinate points 715.

[0086] In an exemplary implementation, the volume rendering of the volume rendering model instruction set 622 enables the creation of a 2D projection of a 3D discrete sampled data set, as shown in the 5D input illustration 710. As output, for a given camera location (e.g., spatial position 754 and viewpoint / pose 756) associated with an event request (e.g., time period), the volume rendering algorithm of the volume rendering model instruction set 622 obtains the RGBσ (red, green, blue and density channels) of each voxel in the space through which the ray from the camera (e.g., the cast ray 714, 716) is cast. The RGBσ color is converted to RGB color and recorded in the corresponding pixel of the 2D image (e.g., 2D images 704, 706). This process is repeated for each pixel until the entire 2D image is rendered. The output of the volume rendering model instruction set 622 will include a 5D output illustration 720 (e.g., RGBσ color) and an illustration 730 (e.g., expiration value (ε)).

[0087] In an exemplary implementation, the volume rendering model instruction set 622 additionally determines and returns an "expiration" value (ε) as part of the output (e.g., the expiration time of each ray). The renderer (device 405) uses the expiration value (ε) to limit its queries to the volume rendering model instruction set 622. For example, if the viewpoint has not changed from one time (t1) to the next time (t2) (e.g., within a threshold), the renderer may not need to query the model for certain new values ​​(e.g., rays that have not yet expired). For these instances where the renderer does not need to query the volume rendering model instruction set 622, the number of queries may be reduced when rendering a dynamic scene with many static parts from a fixed viewpoint. The renderer (e.g., the device 405 via the time-based 3D representation instruction set 624) only queries the volume rendering model instruction set 622 at locations in the scene that have changed, and avoids repeatedly querying the volume rendering model instruction set 622 for parts of the scene that have not changed, which significantly reduces the number of queries required, reduces computational complexity, and can enable rendering at higher frame rates, at higher resolutions, or with less power.

[0088] In some embodiments, the renderer may cache and reuse the density and color returned for the query (or even the color for the pixel accumulated from multiple query results) until the expiration date, and only begin querying the same ray again after the expiration date. In this way, the renderer (device 405) only queries the volume rendering model instruction set 622 (e.g., a NeRF model with an expiration value) at locations in the scene that have changed, and it avoids repeatedly querying the volume rendering model instruction set 622 for parts of the scene that have not changed. Therefore, after a query is input at t1 (e.g., query 760 input at t), the volume rendering model instruction set 622 returns output data 765 (e.g., emission color (RGB), volume density (σ), and expiration value (ε)). The time-based 3D representation instruction set 62 may then use the output data 765 to generate a time-based 3D representation 770 (e.g., as Figure 1 ), user 402 can view different viewpoints synthesized from different content from different viewpoints in a dynamic scene.

[0089] In some implementations, the expiration value determined by the volume rendering model instruction set 622 may be based on a dissimilarity threshold on color / density-transparency differences, such as a standard deviation of values, or a difference between minimum and maximum values. To determine whether a given ray produces sufficiently similar densities and colors over a given series of timestamps, a threshold on statistics of these densities and colors may be used, such as a standard deviation of values, or a difference between minimum and maximum values. If such statistics exceed a defined threshold, this may be considered a scene change, and the expiration date of the previous ray is set to this scene change time.

[0090] In some implementations, a threshold on a ray dissimilarity metric may be used to determine whether a first ray and a second ray are sufficiently similar so that a cached result for the first ray may be used as a substitute for the second ray, i.e., so that the second ray does not need to be queried. The ray dissimilarity metric may include an angle between two rays, a Euclidean distance between the origins of the two rays, a Euclidean distance from the origin of the first ray to a line formed by the second ray. For example, if the origins of two rays differ by less than 1 cm and their directions differ by less than 1 degree, they may be considered sufficiently similar.

[0091] Additionally or alternatively, in some implementations, the expiration value determined by the volume rendering model can be based on ray differences (e.g., angle between two rays, ray origin differences, etc.). Additionally or alternatively, in some implementations, the expiration value determined by the volume rendering model can be based on a per-frame expiration budget (similar number of rays are due simultaneously) and / or based on prioritization of visible scene changes.

[0092] In some implementations, for image content that is not captured by multiple devices (e.g., the back of a user in a video), the information system server may be able to utilize content that may have been designated as "shared content" if the user sharing the content gives permission (e.g., saving the image data locally on their own device). Thus, if the renderer changes to a viewpoint that may not have been synthesized (e.g., the back of a person's head) due to lack of information, the system may be able to query from another source (e.g., saved locally, or a personal cloud server). In another example, a device may augment image content locally without the device sharing personal content to the cloud.

[0093] In some specific implementations, the shared event recording and rendering process described herein can be used for live broadcast, sampling rate control, and some other possible implementation details. For example, for live broadcast, the shared event recording and rendering process needs to understand the future (e.g., how long this part of the scene remains unchanged), so that the image data is very suitable for offline batch processing of the recorded content. If the final rendering allows a small delay, the shared event recording and rendering process can also be used for live broadcast of events using NeRF plus expiration values ​​(e.g., one NeRF per frame, or one NeRF per second (which encodes 90 frames)). For example, if the live content is delayed by one second, the shared event recording and rendering algorithm can look forward one second into the future, then the expiration value can be any value less than or equal to one second, and for static parts of the scene, the expiration value will be one second. Compared with other NeRF protocols, these rays must be sampled only once per second based on the query, instead of 90 times per second, which significantly improves efficiency.

[0094] In some implementations, for sampling rate control, the shared event record and rendering process may utilize ray expiration dates, and determine such expiration values ​​so that the number of all rays expiring at any given timestamp is limited to a certain value (e.g., a per-frame expiration budget). Any rays that should also expire at the same timestamp but will exceed the per-frame expiration budget may be assigned a slightly later expiration date (e.g., one timestamp later). Sorting dynamic scene changes by their difference can be used to prioritize correctly setting expiration dates for strong scene changes, while expiration dates for less severe scene changes can be slightly delayed. In some implementations, expiration date determination attempts to achieve a similar number of expiring rays at each given frame, thus resulting in a more constant computational load over time. This can also be achieved by prioritizing the temporal accuracy of expiration dates for highly visible scene changes, while deprioritizing less obvious scene changes, which can be assigned slightly later expiration dates, causing the renderer to update them slightly later than when they actually occurred.

[0095] Figure 8800 is a flowchart illustrating a method 800 for generating a time-based 3D representation of a shared event based on sensor data from two or more devices according to some specific implementations. In some specific implementations, a device (such as electronic device 105, or electronic device 165, 175, 185, 405, etc., or a combination of any one / each) performs method 800. In some specific implementations, method 800 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, an ear-hook device, or a server device (e.g., information system server 620), or a combination thereof. Method 800 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 800 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0096] At block 810, method 800 determines at a first device including one or more processors that a second device is currently in (e.g., near) an event at a physical environment based on one or more event criteria. The first device may be a server or may be one of the participant devices. The one or more event criteria may be based on physical location, timeline constraints, geo-fencing, metadata (e.g., tagging an event or sharing an event).

[0097] At block 820, method 800 provides a notification to the second device based on determining that the second device is currently in an event, the notification providing an option to authorize the use of sensor data obtained by the second device during the event to generate a time-based 3D representation of the event. Figure 2B As illustrated in FIG. 2 , a transcription bubble 290 is provided for requesting consent from the user 160 to authorize permission to participate in sharing recorded content of a particular event (eg, a party, a concert, etc.).

[0098] At block 830, method 800 receives authorization to use the sensor data to generate a time-based 3D representation of the event, wherein the authorization is based on user input received at the second device in response to the notification. Figure 2B As illustrated in , user 160 may select “yes” in transcription bubble 290, where selecting “yes” may be based on gaze and hand movement, direct interaction with transcription bubble 290 (e.g., the user reaches out and attempts to interact / touch the user interface in 3D space), or may be based on a verbal command uttered by user 160 “Yes, I agree” or something similar.

[0099] At block 840, method 800 obtains sensor data (e.g., images, videos, depth images, etc.) from the second device in accordance with the authorization. At block 850, method 800 generates a time-based 3D representation of the event based on the sensor data obtained from the second device and additional sensor data obtained from one or more other devices in the event. For example, the system may require at least a third source in order to generate a time-based 3D representation of the event (e.g., more viewpoints for a multi-user event).

[0100] In some implementations, additional sensor data is obtained from one or more other devices at the event based on receiving additional authorization from the one or more other devices at the event to use the additional sensor data to generate a time-based 3D representation of the event.

[0101] In some implementations, method 800 also includes updating a portion of the time-based 3D representation of the event by obtaining user-based image content from a user database, and augmenting the portion of the time-based 3D representation of the event based on the user-based image content. For example, if the perceived view of the event is from a location that was not originally captured in the initial recording (e.g., the front of a friend's head was captured), then if the perspective of the replay is from a position where the back of the friend's head is seen (not initially captured), then the device (e.g., client device 610 utilizing augmentation instruction set 618 and user content database 619) can utilize a stored personal image of the back of the friend's head (captured at a completely different time and place) to fill in the content during the replay.

[0102] In some implementations, the time-based 3D representation is a neural rendering model. In some implementations, the neural rendering model is a neural radiance field (NeRF) model. In some implementations, the NeRF model is configured to receive input corresponding to a 3D viewpoint and a time point during an event. In some implementations, the NeRF model is configured to output a plurality of ray values ​​corresponding to a view of the event from a 3D viewpoint at a time point during the event.

[0103] In some implementations, a NeRF model (e.g., a time-based 3D representation) is configured to output an expiration corresponding to one or more of the ray values. In some implementations, the expiration is determined based on a dissimilarity threshold corresponding to ray color or ray density. In some implementations, the expiration is determined based on a dissimilarity threshold corresponding to a ray position or orientation difference. In some implementations, the expiration is determined based on an expiration budget per time point. In some implementations, the expiration is determined based on a prioritization, the prioritization being determined based on determining a visible scene change.

[0104] In some implementations, access rights to use the time-based 3D representation of the event are based on authorization. In some implementations, based on authorization, only users who facilitate the time-based 3D representation (e.g., by providing sensor data) are granted access rights to use the time-based 3D representation of the event.

[0105] In some implementations, the information system / server determines that the second user has provided user consent to receive user preference settings associated with the second user at the first device based on receiving an affirmative response to the consent request from the second user (or from a device associated with the second user) at the first device via the information system. Figure 2B As illustrated in , a notification bubble 290 is provided to user 160 in which user 160 needs to affirmatively select whether to allow consent to participate in the sharing event (e.g., via an input device such as a mouse, via an interactive element on the display of selection device 165, via an audio prompt such as saying "yes", etc.).

[0106] In some implementations, generating a time-based 3D representation of an event based on sensor data obtained from a second device and additional sensor data obtained from one or more other devices in the event includes: determining synchronization data based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices based on one or more synchronization algorithms; and generating a time-based 3D representation of the event based on the determined synchronization data. In some implementations, the one or more synchronization algorithms are based on: common clock synchronization associated with the second device and the one or more other devices; image content obtained from the second device corresponding to additional image content obtained from the one or more other devices; audio content obtained from the second device corresponding to additional audio content obtained from the one or more other devices; or a combination thereof. For example, Figure 7 As illustrated in FIG, a synchronization instruction set 626 obtains content data 745 from multiple asynchronous client devices 610 and synchronizes the data (synchronized content data 757), which may be based on an ITS associated with each device, image / video content obtained from each device (e.g., associated with identifying an object within the captured image / video), captured audio (e.g., associated with identifying a specific audio segment within the captured video), or a combination thereof.

[0107] In some embodiments, generating a time-based 3D representation of an event based on sensor data obtained from a second device and additional sensor data obtained from one or more other devices in the event includes: determining refined image data (e.g., raw RGB data) based on the sensor data obtained from the second device and the additional sensor data obtained from one or more other devices based on one or more transformation algorithms; and generating a time-based 3D representation of the event based on the determined refined image data. For example, in order to address variations in camera properties from two or more devices due to different imaging pipelines, filtering techniques, lighting issues, etc., the techniques described herein may adjust one or more frames of images or video content from multiple devices so that each frame includes the same or very similar image content (such as raw RGB data). For example, the image data obtained may include filter and / or tone mapping attributes. The refined image data image may be attributed to the intensity transformed to a common nominal space (or domain). The purpose of the refined image data is to update the same pixel values ​​in two images so that they correspond to the same amount of incident radiation. The transformation may use a linear space, use gamma correction / encoding, etc., as long as it is consistent between multiple images used for synchronization. As described herein with reference Figure 6 and Figure 7 As discussed, the synchronization instruction set 626 and / or the volume rendering model instruction set 622 may determine refined image data (eg, raw RGB data) before the volume rendering model instruction set 622 generates a time-based 3D representation.

[0108] In some implementations, the information system is a server external to the first device (e.g., information system server 620). Alternatively, in some implementations, the information system is located at the first device. For example, a device (e.g., device 105) may be configured with a volume rendering model instruction set 622 and a time-based 3D representation instruction set for a rendering instruction set 616, which has access to a representation model database 640.

[0109] Fig. 9 is a flowchart illustrating a method 900 for providing a view of recorded events based on output from a neural rendering model according to some specific implementations. In some specific implementations, a device (such as electronic device 105 or electronic device 165, 175, 185, 405, etc., or a combination of each / any one) performs method 900. In some specific implementations, method 900 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, an ear-hook device, or a server device (e.g., an information system server 620), or a combination thereof. Method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 800 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0110] Method 900 is executed at an electronic device having a processor, and each of the boxes (e.g., boxes 910-950) described herein for method 900 is processed for each of a plurality of rendering time points (e.g., rendering frames). In an exemplary implementation, method 900 renders views corresponding to a plurality of times during a time period of a recorded event (e.g., providing a view corresponding to a 60-second recorded event for 60 seconds, wherein a first view is provided for a first time point during the event, a second view is provided for a next time point during the event, and the like). Each of the views of the recorded event can be generated based on a viewpoint within the scene of the recorded event, and the viewpoint need not correspond to the location of the recorded event in the scene. Instead, the viewpoint can be specified by the renderer (e.g., changed based on the current renderer HMD pose). A neural rendering model (e.g., a NeRF model) can be used to generate the view, which is trained using sensor data (e.g., image data) captured from multiple locations / devices during the event. For example, at each time, a rendering device may "query" the model by specifying inputs to the model (ie, a point in time and a viewpoint) and receiving outputs from the model (eg, rays corresponding to pixels of a view of the event at that time and from that viewpoint).

[0111] In an exemplary implementation, method 900 focuses on the volume rendering model returning "expiration" as additional output data (e.g., the expiration time of each ray). The renderer can use this information to limit its queries to the volume rendering model. For example, if the viewpoint has not changed from one time to the next (within a threshold), the renderer does not need to query the model for certain new values ​​(e.g., rays that have not yet expired). This can reduce the number of queries when rendering dynamic scenes with many static parts from a fixed viewpoint. The renderer only queries the volume rendering model at locations that have changed in the scene, and then avoids repeatedly querying the volume rendering model for parts of the scene that have not changed, which will significantly reduce the number of queries required, reduce computational complexity, and can enable rendering at higher frame rates, at higher resolutions, or with less power. The expiration value can be based on a dissimilarity threshold, namely: (a) color / density-transparency difference (e.g., standard deviation of values, or difference between minimum and maximum values); and / or (b) ray difference (e.g., angle between two rays, ray origin difference, etc.). Expiration values ​​may be based on (c) a per-frame expiration budget (similar number of rays expiring at the same time) and / or (d) prioritization based on visible scene changes.

[0112] At block 910, method 900 determines a 3D event viewpoint corresponding to a 3D location in the physical environment of the recorded event. Figure 7As illustrated in , the neural rendering model may include a continuous scene construction that takes as input a 5D vector-valued function having the following properties for that input: a 3D position x=(x; y; z) and a 2D viewing direction (θ; Φ). In some implementations, the event viewpoint may be specified by the renderer; the viewpoint may be based on the rendering device pose, and thus may change for different rendering times based on the current rendering device location and / or orientation (e.g., current HMD pose).

[0113] At block 920, method 900 determines an event time point corresponding to a time during the time period of the recorded event. For example, a first rendering time point may use a first event time point, a second rendering time point may use a second event time point, and so on.

[0114] At block 930 , the method 900 inputs the 3D event viewpoint and the event timepoint into a neural rendering model trained using sensor data captured by multiple devices (e.g., multiple mobile devices / phones, multiple HMDs, etc.) during the recorded event.

[0115] At box 940, method 900 receives output from the neural rendering model, the output including (i) a plurality of ray values ​​corresponding to a view of the recorded event from a 3D viewpoint at a point in time during the recorded event, and (ii) an expiration corresponding to one or more of the ray values.

[0116] In some implementations, expiration is determined based on a dissimilarity threshold corresponding to ray color. Additionally or alternatively, in some implementations, expiration is determined based on a dissimilarity threshold corresponding to ray density or transparency. Additionally or alternatively, in some implementations, expiration is determined based on a dissimilarity threshold corresponding to ray position or orientation differences. Additionally or alternatively, in some implementations, expiration is determined based on an expiration budget per time point. Additionally or alternatively, in some implementations, expiration is determined based on prioritization, which is determined based on determining visible scene changes. Therefore, the expiration value may be based on a dissimilarity threshold regarding color / density-transparency differences, such as a standard deviation of values, or a difference between a minimum and a maximum value. Additionally or alternatively, in some implementations, the expiration value may be based on ray differences, such as an angle between two rays, a difference in ray origins, etc. Additionally or alternatively, in some implementations, expiration values ​​can be based on a per-frame expiration budget (e.g., similar numbers of rays expire simultaneously), and rays can be prioritized for update based on visible scene changes (e.g., a viewer turns his or her head to get a different viewing perspective).

[0117] At block 950, method 900 provides a view of the recorded event based on the received output. For example, the rendering system may provide a view for each time point. In some implementations, at least one view for rendering a time point reuses a ray from a previous view based on the expiration of the ray from the previous view.

[0118] In some implementations, method 900 further includes an act of inputting ray data into the neural rendering model, wherein the ray data identifies a first subset of less than all rays in the view to be included in output, or a second subset of less than all rays in the view to be excluded from output. In some implementations, the first subset or the second subset is determined based on expiration data corresponding to an expiration of one or more rays output by the neural rendering model for a previous rendering time point.

[0119] In some implementations, privacy and / or audio notification settings are automatically set based on the context of the determined physical environment. For example, in a quiet environment such as a library, the font may be minimized or less distracting. Or, in a noisy environment, such as at a concert, the font of the audio transcription may be larger and it may be easier to notice that someone is speaking to the audience. The privacy and / or audio notification settings may be adjusted automatically at the speaker device and / or the listener device or by each user (e.g., the listener's speakers).

[0120] Fig.10 1 is a block diagram of an electronic device 1000. Device 1000 illustrates an exemplary device configuration of an electronic device, such as devices 105, 165, 175, 185, 405, etc. Although some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 1000 includes one or more processing units 1002 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1006, one or more communication interfaces 1008 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1010, one or more displays 1012 or other output devices, one or more internal-facing and / or external-facing image sensor systems 1014, a memory 1020, and one or more communication buses 1004 for interconnecting these and various other components.

[0121] In some implementations, the one or more communication buses 1004 include circuitry that interconnects system components and controls communications between system components. In some implementations, the one or more I / O devices and sensors 1006 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0122] In some implementations, the one or more output devices 1012 include one or more displays configured to present a view of a 3D environment to a user. In some implementations, the one or more devices 1012 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, the one or more displays correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 1000 includes a single display. As another example, the device 1000 includes a display for each eye of the user.

[0123] In some specific implementations, one or more output devices 1012 include one or more audio generating devices. In some specific implementations, the one or more output devices 1012 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sounds such as 3D audio effects. Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sounds can involve converting sound waves (e.g., using head-related transfer functions (HRTFs), reverberation, or elimination techniques) to simulate natural sound waves (including reflections from walls and floors), which are emitted from one or more points in a 3D environment. Spatialized sounds can induce the listener's brain to interpret the sound as if the sound occurs at one or more points in a 3D environment (e.g., from one or more specific sound sources), even if the actual sound may be generated by speakers in other locations. One or more output devices 1012 can be configured to generate tactile sensations in addition or alternatively.

[0124] In some implementations, the one or more image sensor systems 1014 are configured to obtain image data corresponding to at least a portion of a physical environment. For example, the one or more image sensor systems 1014 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 1014 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 1014 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0125] Memory 1020 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, memory 1020 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 1020 optionally includes one or more storage devices remotely located from one or more processing units 1002. Memory 1020 includes non-transitory computer-readable storage media.

[0126] In some implementations, the memory 1020 or a non-transitory computer-readable storage medium of the memory 1020 stores an optional operating system 1030 and one or more instruction sets 1040. The operating system 1030 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 1040 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 1040 is software that can be executed by one or more processing units 1002 to implement one or more of the techniques described herein.

[0127] The instruction set 1040 includes a volume rendering model instruction set 1042, a time-based 3D representation instruction set 1044, a geographic location instruction set 1046, and a content / rendering instruction set 1048. The volume rendering model instruction set 1042 may be configured to determine, when executed, a volume rendering model as described herein. The time-based 3D representation instruction set 1044 may be configured to determine, when executed, a time-based 3D representation as described herein. The geographic location instruction set 1046 may be configured to determine, when executed, a geographic location of a device as described herein. The content / rendering instruction set 1048 may be configured to determine, when executed, content and / or rendering instructions for a device as described herein. The instruction set 1040 may be embodied as a single software executable file or multiple software executable files.

[0128] Although instruction set 1040 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. In addition, the drawings serve more as functional descriptions of the various features present in a particular implementation, as opposed to the structural schematics of the implementations described herein. As will be appreciated by those of ordinary skill in the art, the items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how the features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0129] Fig.11 A block diagram of an exemplary head mounted device 1100 according to some implementations is illustrated. The head mounted device 1100 includes a housing 1101 (or case) that houses various components of the head mounted device 1100. The housing 1101 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 110) end of the housing 1101. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 1100 in place on the face of the user 110 (e.g., around the eyeballs of the user 110).

[0130] Housing 1101 houses display 1110, which displays an image and emits light toward or onto the eyes of user 110. In various implementations, display 1110 emits light through an eyepiece having one or more optical elements 1105, which refract the light emitted by display 1110 so that the display appears to user 110 as a virtual distance farther than the actual distance from the eye to display 1110. For example, optical element 1105 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. In order to enable user 110 to focus on display 1110, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). In addition, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0131] The housing 1101 also houses a tracking system that includes one or more light sources 1122, a camera 1124, a camera 1132, a camera 1134, a camera 1136, and a controller 1180. The one or more light sources 1122 emit light onto the eyes of the user 110, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by the camera 1124. Based on the light pattern, the controller 1180 can determine the eye tracking characteristics of the user 110. For example, the controller 1180 can determine the gaze direction and / or blink state (eyes open or closed) of the user 110. As another example, the controller 1180 can determine the pupil center, pupil size, or focus point. Thus, in various implementations, light is emitted by the one or more light sources 1122, reflected from the eyes of the user 110, and detected by the camera 1124. In various implementations, the light from the eyes of the user 110 reflects from the hot mirror or passes through the eyepiece before reaching the camera 1124.

[0132] The display 1110 emits light within a first wavelength range, and the one or more light sources 1122 emit light within a second wavelength range. Similarly, the camera 1124 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400 nm-700 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700 nm-1400 nm within the near infrared spectrum).

[0133] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 110 selecting an option on display 1110 by looking at it), provide foveated rendering (e.g., rendering a higher resolution in the area of ​​display 1110 that user 110 is looking at and a lower resolution elsewhere on display 1110), or correct distortion (e.g., for images to be provided on display 1110).

[0134] In various implementations, one or more light sources 1122 emit light toward the eyes of the user 110, which is reflected in the form of multiple flashes.

[0135] In various implementations, the camera 1124 is a frame / shutter based camera that generates images of the user's 110 eyes at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, which pixels correspond to locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0136] In various implementations, camera 1124 is an event camera comprising multiple light sensors (e.g., a light sensor matrix) at multiple corresponding positions, which generates an event message indicating a specific position of a particular light sensor in response to the particular light sensor detecting a change in light intensity.

[0137] In various implementations, camera 1132, camera 1134, and camera 1136 are frame / shutter-based cameras that can generate images of the face of user 110 or capture the external physical environment at a specific time point or multiple time points at a frame rate. For example, camera 1132 captures images of the user's face below the eyes, camera 1134 captures images of the user's face above the eyes, and camera 1136 captures the user's external environment (e.g., Figure 1 The images captured by camera 1132, camera 1134, and camera 1136 may include light intensity images (eg, RGB) and / or depth image data (eg, time of flight, infrared, etc.).

[0138] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0139] As described above, one aspect of the present technology is to collect and use sensor data that may include user data to improve the user experience of electronic devices. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include motion data, physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0140] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0141] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any necessary steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0142] Regardless of the foregoing, the present disclosure also contemplates implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0143] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0144] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identifying and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access their stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0145] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0146] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0147] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0148] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0149] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0150] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0151] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0152] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0153] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every respect and not restrictive, and the scope of the present invention disclosed herein is to be determined not only by the detailed description of the illustrative specific implementations but by the full breadth allowed by the patent law.

[0154] It will be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method, comprising: At a first device comprising one or more processors: determining, based on one or more event criteria, that the second device is currently in an event at the physical environment; providing a notification to the second device based on determining that the second device is currently at the event, the notification providing an option to authorize use of sensor data obtained by the second device during the event to generate a time-based three-dimensional (3D) representation of the event; receiving authorization to generate the time-based 3D representation of the event using the sensor data, wherein the authorization is based on user input received at the second device in response to the notification; obtaining the sensor data from the second device according to the authorization; as well as The time-based 3D representation of the event is generated based on the sensor data obtained from the second device and additional sensor data obtained from one or more other devices at the event. 2 . The method of claim 1 , wherein access rights to use the time-based 3D representation of the event are based on the authorization. 3 . The method of claim 1 , wherein based on the authorization, a user facilitating the time-based 3D representation is provided access rights to use the time-based 3D representation of the event.

4. The method of claim 1 , wherein the additional sensor data is obtained from the one or more other devices at the event based on receiving additional authorization from the one or more other devices at the event to use the additional sensor data to generate the time-based 3D representation of the event. The method of claim 1 , wherein the time-based 3D representation is a neural rendering model. The method of claim 5 , wherein the neural rendering model is a Neural Radiance Field (NeRF) model.

7. The method of claim 5, wherein the NeRF model is configured to receive input corresponding to a 3D viewpoint and a time point during the event.

8. The method of claim 6, wherein the NeRF model is configured to output a plurality of ray values ​​corresponding to a view of the event from the 3D viewpoint at the point in time during the event. 9 . The method of claim 8 , wherein the NeRF model is configured to output an expiration corresponding to one or more of the ray values. 10 . The method of claim 9 , wherein the expiration is determined based on a dissimilarity threshold corresponding to ray color or ray density. The method of claim 9 , wherein the expiration is determined based on a dissimilarity threshold corresponding to a ray position or orientation difference. The method of claim 9 , wherein the expiration is determined based on a budget expiration per time point.

13. The method of claim 9, wherein the expiration is determined based on prioritization, the prioritization being determined based on determining a visible scene change.

14. The method of claim 1, further comprising updating a portion of the time-based 3D representation of the event by: Obtaining user-based image content from a user database; and The portion of the time-based 3D representation of the event is augmented based on the user-based image content.

15. The method of claim 1, generating the time-based 3D representation of the event based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices at the event comprises: determining synchronization data based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices based on one or more synchronization algorithms; as well as The time-based 3D representation of the event is generated based on the determined synchronization data.

16. The method of claim 15, wherein the one or more synchronization algorithms are based on: synchronizing with a common clock associated with the second device and the one or more other devices; image content obtained from the second device corresponding to additional image content obtained from the one or more other devices; audio content obtained from the second device corresponding to additional audio content obtained from the one or more other devices; or A combination of them.

17. The method of claim 1, generating the time-based 3D representation of the event based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices at the event comprises: determining, based on one or more transformation algorithms, refined image data based on the sensor data obtained from the second device and the additional sensor data obtained from the one or more other devices; as well as The time-based 3D representation of the event is generated based on the determined refined image data.

18. A first device, comprising: one or more sensors; a non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations including: determining, based on one or more event criteria, that the second device is currently in an event at the physical environment; providing a notification to the second device based on determining that the second device is currently at the event, the notification providing an option to authorize use of sensor data obtained by the second device during the event to generate a time-based three-dimensional (3D) representation of the event; receiving authorization to generate the time-based 3D representation of the event using the sensor data, wherein the authorization is based on user input received at the second device in response to the notification; obtaining the sensor data from the second device according to the authorization; as well as The time-based 3D representation of the event is generated based on the sensor data obtained from the second device and additional sensor data obtained from one or more other devices at the event.

19. The apparatus of claim 18, wherein access rights to use the time-based 3D representation of the event are based on the authorization.

20. The apparatus of claim 18, wherein based on the authorization, a user facilitating the time-based 3D representation is provided access rights to use the time-based 3D representation of the event.

Images