3D photo

Abstract

Various implementations disclosed herein include devices, systems, and methods that determine how to present a three-dimensional (3D) photograph in an extended reality (XR) environment (e.g., present in 3D mode, present in 2D mode, present in a blurred mode, or not present at all) based on a viewing position of a user active in the XR environment relative to placement of the 3D photograph in the XR environment. In some implementations, at an electronic device with a processor, a 3D photograph is obtained as an incomplete 3D representation created based on one or more images captured by an image capture device. In some implementations, a viewing position of the electronic device relative to a placement position of the 3D photograph is determined, and a presentation mode of the 3D photograph is determined based on the viewing position. In some implementations, the 3D photograph is provided at the placement position based on the presentation mode in the XR environment.

Description

Technical Field

[0001] This disclosure relates in its entirety to systems, methods, and apparatus for providing or displaying photographic and three-dimensional (3D) content. Background Technology

[0002] A photograph is typically a two-dimensional (2D) representation of the physical environment captured from a single viewpoint. Such photographs are often printed in 2D or viewed on electronic devices, and therefore lack depth, making the viewpoint of the person holding the photograph or device seem irrelevant. Summary of the Invention

[0003] The various embodiments disclosed herein include devices, systems, and methods capable of creating and viewing 3D photographs that provide an incomplete 3D representation of a physical environment. The amount of viewable content represented by the 3D photograph can depend on the viewer's viewpoint, for example, providing a complete view from a frontal viewpoint, a slightly incomplete view when viewed from a side angle, and a very incomplete view when viewed from a rear viewpoint. In some embodiments, the 3D photograph is presented in an XR environment based on the viewer's viewpoint relative to the 3D photograph within an extended reality (XR) environment. For example, a 3D photograph of a person captured with a camera facing the person may only have frontal data and no back data because the camera cannot see the back of the person. In some embodiments, the 3D photograph will appear realistic and 3D when viewed in a viewing area similar to the posture (e.g., position and orientation) or movement of the image capturing device during the creation of the 3D photograph. For example, the average position of all image capture positions of the image capture device during the creation of a 3D photograph, and the well-defined volume around the average capture position of the 3D photograph relative to the 3D photograph, can define the 3D viewing area (e.g., the effective 3D viewing area) of the 3D photograph based on the placement of the 3D photograph in the XR environment.

[0004] In some implementations, mitigation techniques modify the 3D depiction of a 3D photograph in an XR environment based on the user's viewing position relative to the placement of the viewing electronic device within the XR environment. In some implementations, the display of the 3D photograph is based on the viewing angle of the viewing electronic device relative to the placement (e.g., pose) of the 3D photograph in the XR environment. In some implementations, the display of the 3D photograph is based on the spatial distance from the viewing position of the viewing electronic device to the displayed 3D photograph within the XR environment. In some implementations, the 3D photograph is rendered as 3D when the user's viewpoint is within a first viewing area (e.g., the effective viewing area) and rendered as a 2D image or blurred (e.g., to prevent the user from viewing missing parts of objects in the 3D photograph) when the user is outside the effective viewing area of ​​the XR environment. In some implementations, the 3D photograph is depicted on or within a virtual object in the XR environment. In some implementations, the 3D photograph is depicted on or within a representation of a physical object in the XR environment.

[0005] In some implementations, only certain portions of a 3D photograph (e.g., 3D subvolumes or 3D cutouts) are extracted from the 3D photograph and placed into an XR environment. In other implementations, only context-relevant portions of a 3D photograph (e.g., individuals extracted from a fixed background) are "cut out" and displayed in the XR environment. For example, when the 3D photograph is of people in an office with a table and chairs, the furniture remains unchanged during the capture of the images and data used to create the 3D photograph; instead, a 3D cutout of the 3D photograph, including a 3D image of the person, is extracted and placed in the XR environment.

[0006] The various embodiments disclosed herein include devices, systems, and methods for determining how to present a 3D photograph in an XR environment (e.g., in 3D mode, in 2D mode, in a blurred mode, or not at all) based on the viewing position of a user active in an XR environment relative to the placement of the 3D photograph within the XR environment. In some embodiments, a 3D photograph is obtained at an electronic device having a processor as an incomplete 3D representation created based on one or more images captured by an image capturing device. In some embodiments, a viewing position of the electronic device relative to the placement of the 3D photograph is determined, and a presentation mode of the 3D photograph is determined based on the viewing position. In some embodiments, the 3D photograph is provided at the placement position based on the presentation mode in the XR environment.

[0007] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and means for performing or causing to perform any of the methods described herein. Attached Figure Description

[0008] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0009] Figure 1 This is a block diagram illustrating an exemplary system according to some specific implementations.

[0010] Figure 2 This is a block diagram illustrating an exemplary controller according to some specific implementations.

[0011] Figure 3 This is a block diagram illustrating an exemplary electronic device according to some specific implementations.

[0012] Figure 4 This is a block diagram illustrating an exemplary environment.

[0013] Figure 5 This shows that, according to some specific implementations, Figure 4 A block diagram of a mobile device capturing frames in a frame sequence within an environment.

[0014] Figures 6A to 8 This is a schematic diagram illustrating an exemplary 3D photograph presented in an XR environment according to some specific implementations.

[0015] Figure 9 This is a flowchart illustrating an exemplary method, according to some specific implementations, for detecting input to an XR environment by recognizing a handheld object and detecting hand movement relative to the object.

[0016] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0017] Numerous details have been described to provide a thorough understanding of the exemplary embodiments shown in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will recognize that other effective aspects or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein. Although Figures 1 to 3 Exemplary specific implementations relating to electronic devices are described, including but not limited to watches and other wearable electronic devices, mobile devices, laptop computers, desktop computers, HMDs, gaming devices, home automation devices, accessory devices, and other devices that include or use image capture devices.

[0018] Figure 1 This is a block diagram of an exemplary operating environment 100 according to some specific implementations. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for brevity and to avoid obscuring further relevant aspects of the exemplary implementations disclosed herein. Therefore, as a non-limiting example, operating environment 100 includes a controller 110 and an electronic device (e.g., a laptop computer) 120, one or both of which may be located in a physical environment 105. A physical environment refers to the physical world that people can sense and / or interact with without the assistance of electronic systems. Physical environments such as physical parks include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with physical environments through senses such as sight, touch, hearing, taste, and smell.

[0019] In some implementations, controller 110 is configured to manage and coordinate the user's extended reality (XR) environment. In some implementations, controller 110 includes a suitable combination of software, firmware, or hardware. See below for reference. Figure 2 The controller 110 is described in more detail. In some implementations, the controller 110 is a computing device located locally or remotely relative to the physical environment 105.

[0020] In one example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to a corresponding electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0021] In some implementations, controller 110 and corresponding electronic devices (e.g., 120) are configured to present an XR environment to the user together.

[0022] In some implementations, electronic device 120 is configured to present an XR environment to a user. In some implementations, electronic device 120 includes a suitable combination of software, firmware, or hardware. See below for reference. Figure 3 The electronic device 120 is described in more detail. In some specific embodiments, the functions corresponding to the controller 110 are provided by or combined with the electronic device 120, for example, in the case of the electronic device being used as a stand-alone unit.

[0023] According to some specific implementations, when a user is present within physical environment 105, electronic device 120 presents an XR environment to the user. An extended reality (XR) environment refers to a fully or partially simulated environment that a person can interact with and / or sense using an electronic device. For example, an XR environment may include virtual reality (VR) content, augmented reality (AR) content, mixed reality (MR) content, etc. Using an XR system, a portion of a person's body movement or its representation can be tracked. In response, one or more characteristics of virtual objects simulated in the XR environment can be adjusted such that they conform to one or more physical laws. For example, an XR system can detect the user's head movement and, in response, adjust the graphical and auditory content presented to the user in a manner similar to how views and sounds change in the physical environment. In another example, an XR system can detect the movement of an electronic device (e.g., a laptop, mobile phone, tablet, etc.) presenting the XR environment and, in response, adjust the graphical and auditory content presented to the user in a manner similar to how views and sounds change in the physical environment. In some cases, an XR system may adjust one or more characteristics of the graphical content in the XR environment in response to a representation of physical motion (e.g., a voice command).

[0024] Various electronic systems enable a person to interact with and / or sense an XR environment. Examples include projection-based systems, head-mounted systems, head-up displays (HUDs), windows with integrated displays, vehicle windshields with integrated displays, displays designed to be placed on a user's eyes (e.g., similar to contact lenses), speaker arrays, headsets / earpieces, input systems (e.g., wearable or handheld controllers with or without haptic feedback), tablets, smartphones, and desktop / laptop computers. A head-mounted system may include an integrated opaque display and one or more speakers. In other examples, the head-mounted system may accept an external device with an opaque display (e.g., a smartphone). The head-mounted system may include one or more image sensors and / or one or more microphones to capture images or video and / or audio of the physical environment. In other examples, the head-mounted system may include a transparent or translucent display. The medium through which light representing the image is guided may be included within the transparent or translucent display. The display may utilize OLED, LED, uLED, digital light projection, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The medium can be a holographic medium, an optical combiner, an optical waveguide, an optical reflector, or a combination thereof. In some examples, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can use retinal projection techniques to project graphic images onto a user's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as onto a physical surface or as a hologram.

[0025] Figure 2This is a block diagram of an example controller 110 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0026] In some embodiments, the one or more communication buses 204 include circuitry for communication between interconnecting system components and control system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.

[0027] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures or subsets thereof, including optional operating system 230, extended reality (XR) module 240, and 3D photo rendering unit 250.

[0028] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks.

[0029] In some implementations, XR module 240 is configured to create, edit, present, or experience an XR environment. XR module 240 is configured to present virtual content to one or more users that will be used as part of the XR environment. For example, users can view and otherwise experience an XR-based user interface that allows users to select, place, move, and otherwise present the XR environment based on the location of the virtual content, such as through gestures, voice commands, or input devices.

[0030] In some embodiments, the 3D photo rendering unit 250 is configured to render 3D photos in an XR environment. In some embodiments, the 3D photo rendering unit 250 is configured to determine the rendering mode of the 3D photo based on the viewing position relative to the placement position of the 3D photo in the XR environment. In some embodiments, the 3D photo rendering unit 250 can be used to acquire or create 3D photos. Although these modules and units are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of these modules and units may reside in a separate computing device.

[0031] also, Figure 2 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware chosen for that particular implementation.

[0032] Figure 3This is a block diagram of an example of an electronic device 120 according to some specific embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, SPI, I2C, or similar interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external sensor systems 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0033] In some implementations, one or more communication buses 304 include circuitry for interconnecting and communicating between system components. In some implementations, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0034] In some embodiments, one or more displays 312 are configured to present an XR environment to a user. In some embodiments, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), or similar display types. In some embodiments, one or more displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, an electronic device may include a single display. Alternatively, an electronic device may include a display for each of the user's eyes.

[0035] In some embodiments, one or more internally or externally oriented sensor systems 314 include an image capture device or array that captures image data or an audio capture device or array (e.g., a microphone) that captures audio data. In some embodiments, the one or more image sensor systems 314 are configured to acquire image data corresponding to at least a portion of a user's face, including the user's eyes. For example, the one or more image sensor systems 314 include one or more RGB cameras (e.g., those with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, event-based cameras, etc. In various embodiments, the one or more image sensor systems 314 also include an illumination source, such as a flash or flash source, that emits light (e.g., to a portion of the subject or user's face).

[0036] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330, extended reality (XR) module 340, and 3D photo rendering unit 350.

[0037] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks.

[0038] In some implementations, XR module 340 is configured to create, edit, present, or experience an XR environment. XR module 340 is configured to present virtual content that will be used as part of an XR environment to one or more users. For example, users can view and otherwise experience an XR-based user interface that allows users to select, place, move, and otherwise present the XR environment based on the location of the virtual content, such as through gestures, voice commands, or input devices.

[0039] In some embodiments, the 3D photo rendering unit 350 is configured to render 3D photos in an XR environment. In some embodiments, the 3D photo rendering unit 350 is configured to determine the rendering mode of the 3D photo based on the viewing position relative to the placement position of the 3D photo in the XR environment. In some embodiments, the 3D photo rendering unit 350 can be used to acquire or create 3D photos. Although these modules and units are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of these modules and units may reside in a separate computing device.

[0040] also, Figure 3 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 3 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware chosen for that particular implementation.

[0041] The various specific implementations disclosed herein include devices, systems, and methods for rendering 3D photographs in XR environments. In some specific implementations, examples of 3D photographs include parallax 3D photographs, 1-to-1 scale 3D photographs, and multi-view 3D photographs. Figure 33D reconstructed photographs include (i) 3D reconstruction using multiple different image sensors with overlapping images (e.g., a wedding scene) or (ii) multiple different overlapping images from a single image sensor, 3D panoramic photographs, etc. In some implementations, 3D photographs are typically captured from a specific viewpoint and will lack data from other views. For example, a 3D photograph of a person captured with a camera facing the person may only have frontal data and no back data because the camera cannot see the back of the person. In some implementations, the viewing area of ​​the 3D photograph visible in 3D is determined by the movement of the image capture device during the creation of the 3D photograph. For example, the average position of all image capture positions of the image capture device during the creation of the 3D photograph can define the effective viewing area of ​​the 3D photograph (e.g., a well-defined 3D volume) based on the placement of the 3D photograph in the XR environment. In some implementations, the rendering mode for the 3D photograph in the XR environment (e.g., 3D mode, 2D mode, blurred mode, or no rendering at all) is based on the viewing position of the user's electronic device relative to the placement of the 3D photograph in the XR environment. In some implementations, the rendering mode is determined by detecting the amount or percentage of effective pixels viewed from the viewing location of the 3D photo and comparing it to a standard or threshold.

[0042] In some embodiments, a 3D photograph is an incomplete 3D representation because, from at least some viewpoints, a large number of views will be incomplete (e.g., missing data). In some embodiments, a 3D photograph is an incomplete 3D representation with multiple viewing modes. In some embodiments, a 3D photograph is an incomplete 3D representation based on the placement of the 3D photograph in an XR environment relative to a viewing electronic device, having a first viewing mode (e.g., shown in 3D), a second viewing mode (e.g., shown in 2D or blurred), and a third viewing mode (e.g., not shown at all) in the XR environment. In some embodiments, the image capture device includes a frame-based camera, an event camera (e.g., a dynamic vision sensor), an RGB camera, an RGB-D camera, a depth sensor, a motion sensor, an accelerometer, an orientation sensor, or a combination thereof.

[0043] In some implementations, a parallax 3D photograph includes at least one frame of image data with corresponding depth information. Due to the corresponding depth information, the 3D parallax photograph will display three dimensions from a limited range of viewpoints.

[0044] In some embodiments, a 1-1 scale 3D photograph includes at least image data, corresponding depth data, and additional information providing the actual scale (e.g., actual size) of at least some content in the 3D photograph. In one embodiment, one or more additional sensors on the image capture device are used to determine the actual size of some content in the 3D photograph. In some embodiments, the additional sensors may detect the actual distance of content in the image used to create the 3D photograph. Thus, the actual size can be determined based on the relative size of the content in the image and the actual distance of the content from the image capture device. Alternatively, some sensors or the image capture device may provide metadata to identify the actual size of the content in the captured image used to create the 1-1 scale 3D photograph. For example, the image capture device may include an inherent scale that allows size information to be added to the captured image.

[0045] In some implementations, 3D panoramic photographs are created by capturing multiple images and depth data as the image capture device moves (e.g., scanning in a single direction). In some implementations, the image capture device for 3D panoramic photographs has an additional sensor that captures its movement during a panoramic data capture session used to create the 3D panoramic photograph. In some implementations, multiple images, depth data, and translation or rotation of the image capture device during capture make the 3D panoramic photograph visible in 3D for a limited range of viewpoints.

[0046] In some specific implementations, 3D photos are multi-view... Figure 3 D-reconstructed photographs. In some specific implementations, multi-view... Figure 3 A D-photo includes multiple image capture devices that simultaneously or approximately simultaneously capture photographs (e.g., image and depth data) of the same object (e.g., overlapping content or features) from different angles. In one specific implementation, multi-view... Figure 3 Multi-view reconstruction of wedding scenes realizes a wedding scene where numerous different individuals (e.g., using image capture devices) simultaneously take photos of themselves from different distances and angles (e.g., cutting a wedding cake). Similarly, different individuals may also take photos of themselves from different distances and angles (e.g., poses) at approximately the same time (e.g., a first kiss, a first dance, a best man's toast, a maid of honor's toast, etc.). By identifying common features (e.g., overlapping content) in multiple photos in such multi-view reconstruction of wedding scenes, the pose (e.g., position and orientation) of each image capture device can be determined and combined with corresponding sensor data (e.g., image, depth, or motion data), and corresponding sensor data from all (or more) individual image capture devices are used to create multi-view images. Figure 3 D-reconstructed photograph.

[0047] Or, in some specific implementations, multiple perspectives Figure 3The reconstructed photograph was generated by a single image capture device. In such a single image capture device, multiple views... Figure 3 In 3D reconstruction, multiple images of a single scene or individual are acquired from slightly different poses. For example, a person might take 10 slightly different photos of a friend standing next to a tree before being satisfied with just one of 10 photos. In this case, common features in the overlapping 10 images can be used to create a single 3D photo because the poses of the multiple 10 photos are different. In some single-image capture devices, multi-view... Figure 3 In D reconstruction, the movement of a single image capture device is also used for multi-view reconstruction. Figure 3 D-reconstructed photograph.

[0048] In some implementations, for any type of 3D photograph, the image capture device or additional sensors can capture lighting information simultaneously or approximately simultaneously during the creation of the 3D photograph. For example, during the image capture process of a 3D photograph, specular highlights can be detected in at least some images and used to determine the lighting of the physical environment. In some implementations, shadows from multiple images used to create the 3D scene can provide lighting information about the physical environment. Alternatively, metadata about the lighting conditions can be recorded in addition to the data captured by the image capture device. For example, time and location information during the day (e.g., outdoors at 3 o'clock on a specific date in a city in a state) can be used to determine the lighting information of the physical environment. In some implementations, the image capture device includes additional sensors (such as ambient light sensors, white balance sensors) to capture lighting conditions simultaneously or approximately simultaneously.

[0049] In some implementations, for any type of 3D photograph, the image capture device or additional sensors are used to determine the spatial relationship between the 3D photograph and a positional reference, or the spatial relationship between an individual / cropped portion of the 3D photograph and a positional reference. In some implementations, the spatial relationship between the 3D photograph and a positional reference (e.g., when created) is used to orient or pose the 3D photograph in an XR environment (e.g., using a corresponding positional reference) for later viewing. For example, when a 3D photograph is created, the actual size or spatial relationship between the 3D photograph and the ground plane of the physical environment (e.g., the surface on which the individual in the 3D photograph appears to stand, the floor of a room) can be determined, and then the corresponding size or spatial relationship can be mimicked in the XR environment relative to the ground plane (e.g., the floor of an office, a football field, or a similar surface on which the individual in the 3D photograph appears to stand). Thus, in some implementations, the ground plane alignment between the 3D photograph subjects in the physical environment during capture can be replicated during viewing for rendering the 3D photograph in the XR environment. In some implementations, the ground plane is at a 90-degree angle to a drawing plane or flat 2D surface on which 2D / 3D images are drawn or projected in perspective. In another example, when captured as an individual in a 3D photograph, a person leans against a wall in a room, and this leaning spatial relationship is then mimicked relative to a tree in the XR environment (e.g., an office wall, or a similar surface the person in the 3D photograph might lean against). Thus, in some implementations, during viewing, the spatial relationship between the rendered 3D photograph and its positional reference in the physical environment during capture can be replicated for the 3D photograph rendered in the XR environment.

[0050] In some implementations, for any type of 3D photograph, only certain portions of the 3D photograph are extracted and placed or used as the 3D photograph in an XR environment. For example, when an individual is a person sitting behind a desk in an office, the surrounding furniture in the office does not change much during the image capture session of the 3D photograph. Therefore, in some implementations, a “cutout” of the person behind the desk can be extracted and used as the cutout portion of the 3D photograph in the XR environment. In some implementations, the 3D cutout portion can have any three-dimensional shape (e.g., sphere, oval, box, etc.) or size. In some implementations, the three-dimensional shape of the 3D cutout portion is based on context or semantic segmentation. In some implementations, the three-dimensional shape of the 3D cutout portion is based on the movement of the image capture device during the image capture of the 3D photograph. In some implementations, lighting conditions in the XR environment (e.g., light sources or shadows) are used to render the 3D photograph cutout portion.

[0051] In some implementations, 3D photographs are created from images and depth data captured in a physical environment in a conventional manner. In some implementations, the created 3D photographs have a limited range of 3D viewpoints (e.g., 3D viewing areas). For example, when a 3D photograph is created from an image captured facing the front of a soda can, the information about the back of the soda can in the 3D photograph will be limited. Therefore, as the viewpoint of the 3D photograph of the soda can in an XR environment moves from the viewpoint of the front of the 3D photograph directly facing the soda can, the amount or percentage of missing pixels in the 3D photograph of the soda can will increase.

[0052] In some implementations, various techniques are used to present or render 3D photographs in an XR environment. In some implementations, a presentation mode is used to present or render 3D photographs in an XR environment based on the viewing position of the viewing electronic device relative to the placement of the 3D photograph in the XR environment. In some implementations, the presentation mode is based on viewpoint (POV) constraints or a combination of POV constraints and distance. In some implementations, the presentation mode includes 3D mode, 2D mode, blurred mode, or no presentation at all. In some implementations, the presentation mode is determined by detecting the amount or percentage of effective pixels viewed from the viewing position of the 3D photograph and comparing it to a standard or threshold.

[0053] In some implementations, 3D photographs are depicted on or inside virtual objects within an XR environment. In some implementations, 3D photographs are depicted on or inside a representation of a physical object within an XR environment. In some implementations, the rendering mode of the 3D photograph can be altered when depicted on or inside a virtual or physical object within an XR environment.

[0054] In some implementations, a 3D photograph is considered an entry point into the scene of the 3D photograph. In other words, the entry point displaying a 3D photograph may include a 2D plane within the XR environment, which, when viewed by the user, displays the scene of the 3D photograph in 3D. In some examples, the entry point displaying a 3D photograph will show the scene of the 3D photograph in 3D form, while the viewpoint entering the entry point is within the effective viewing area. However, when the viewpoint is outside the effective viewing area, the entry point will switch to display a 2D representation of the 3D scene of the 3D photograph to prevent the user from seeing missing data about objects in the 3D photograph.

[0055] refer to Figure 4The physical environment 405 includes a first electronic device 420 used by a first user and a second electronic device 425 used by a second user. In this example, environment 405 is a room including a drawing 430 and a table 435. The two devices 420 and 425 can operate independently or interact with additional devices (not shown) to capture images of the environment, detect or track objects in those images, or present XR content based on the images and the detected / tracked objects. Each of the two devices 420 and 425 can communicate wirelessly or via a wired connection with a separate controller (not shown) to perform one or more functions. Similarly, each of the two devices 420 and 425 can store information for the XR environment or communicate with a separate device storing that information, such as a server or other computing device.

[0056] In some embodiments, the device, such as device 425, is a head-mounted device (HMD) such as XR glasses worn by a second user. In some embodiments, the device (such as first device 420) is a handheld electronic device (e.g., a smartphone or tablet) configured to present an XR environment to a first user. First device 420 and second device 425 are each configured to provide an XR environment using images or other real-world information detected by a camera or other sensors.

[0057] In some implementations, the first device 420 or the second device 425 enables its user to change the viewpoint or otherwise modify or interact with the XR environment. In some implementations, the first device 420 or the second device 425 is configured to receive user input for interaction with the displayed XR content. For example, virtual objects, such as 3D representations of real-world people or objects, or information displays each with interactive commands, may be presented in the XR content. The user can reposition the virtual object or information display relative to the depicted real object, or interact with the interactive commands, by providing user input on the respective device or otherwise using the respective device.

[0058] Figure 5 It is based on some specific implementations of the display 410 Figure 4 The physical environment 405 shows a block diagram of the first device 420 or the second device 425 of the XR environment 500. For example, when an application is executed in the physical environment 405, the XR environment 500 may be generated from frames in a frame sequence captured by the first device 420 or the second device 425. Figure 5 As shown, the first device 420 or the second device 425 displays an XR environment 500 including a drawing 530 of a picture 430 and a drawing 535 of a table 435. In some embodiments, the XR environment 500 is shared by the first device 420 and the second device 425.

[0059] Figures 6A to 8 This is a schematic diagram illustrating exemplary 3D photographs rendered in an XR environment according to some specific implementations. For example... Figures 6A to 8 As shown, the exemplary 3D photo can be any type of 3D photo or a cutout thereof.

[0060] like Figures 6A to 6B As shown, the 3D photograph 650 is presented on the depiction 530 of the drawing 430 in the XR environment 500. In some embodiments, an indication 655 is provided indicating that the 3D photograph 650 is located in the XR environment 500. Figure 6A As shown, when the viewing electronic device is within the viewing area 660, the 3D photograph 650 is presented to the viewing electronic device in a 3D presentation mode. For example, as the viewing electronic device moves within the viewing area 660, the environment depicted by the 3D photograph 650 will be presented by the viewing electronic device, making it respond as if viewed from different perspectives (e.g., with parallax). Figure 6B As shown, when the viewing electronic device is outside the viewing area 660 in the XR environment 500, the 3D photograph 650 is presented to the viewing electronic device in a "2D" mode. For example, when the viewing electronic device moves outside the viewing area 660, the environment depicted by the 3D photograph 650 is presented by the viewing electronic device in such a way that it does not respond as a typical 3D environment viewed from different perspectives (e.g., without parallax), but is presented as a flat 2D image.

[0061] In some examples, the position and size of the viewing area 660 can be selected such that the viewing electronics can meet viewing standards, or represent the amount or percentage of effective pixels of the 3D photograph 650 when it is within the viewing area 660. In some examples, the position and size of the viewing area 660 may depend on the capture position of the image capture device during the capture session used to create the 3D photograph and the position and orientation of the 3D photograph within the XR environment. For example, the capture position of the image capture device during a capture session used to create a 3D photograph relative to one or more captured objects can be used to determine the relative offset between the 3D photograph and the viewing area 660, as the image capture device is able to obtain the effective pixel data of the 3D photograph from that angle.

[0062] In some examples, the user is shown an indication to view area 660. In other examples, the user is not shown an indication to view area 660.

[0063] In some examples, light or shadow within the 3D photo 650 can affect the rendering of the XR environment 500, or both. For example, if the 3D photo 650 contains bright red light, some of that red light can be projected onto portions of the XR environment 500 as if the red light source were positioned at the location of the 3D photo 650. This makes the 3D photo 650 appear as if it were actually within the XR environment 500. In some examples, lighting or occlusion effects can be applied only if the size of the 3D photo 650 within the XR environment 500 is greater than a threshold, or if the intensity or characteristics of the light or shadow of the 3D photo 650 are greater than a threshold. Similarly, in some examples, lighting or occlusion effects can be applied to the 3D photo 650 only if the size of the light source or shadow in the XR environment 500 is greater than a threshold, or if the intensity or other characteristics of the light or shadow in the XR environment 500 are greater than a threshold.

[0064] like Figure 7 As shown, the 3D photograph 750 is rendered as a virtual image frame 530' in the XR environment 500. (As...) Figure 7 As shown, when the viewing electronic device is within the first viewing area 760, the 3D photograph 750 is presented to the viewing electronic device in a 3D presentation mode. In some embodiments, when the viewing electronic device is outside the first viewing area 760 but inside the second viewing area 770 in the XR environment 500, the 3D photograph 750 is presented to the viewing electronic device in a blurred mode. In some examples, a blurred presentation mode may include blurring all of the 3D photograph 750. In other examples, a blurred presentation mode may include presenting portions of the 3D photograph 750 with valid pixels without a blur effect, but blurring portions of the 3D photograph 750 without valid pixels to hide missing information. In some examples, a gradual transition between blurred and unblurred portions of the 3D photograph may be applied. In some embodiments, when the viewing electronic device is outside the second viewing area 770 in the XR environment 500, the 3D photograph 750 is presented to the viewing electronic device in a "not shown" mode, and the viewing electronic device will only present the depiction 530 on the wall.

[0065] In some examples, the viewing area 760 may be determined in the same or similar manner as described above regarding viewing area 660. In some examples, an indication of viewing area 760 is presented to the user. In other examples, no indication of viewing area 760 is presented to the user.

[0066] In some examples, the position and size of the viewing area 770 can be selected such that the viewing electronics can meet viewing criteria, or represent the amount or percentage of effective pixels of the 3D photograph 750 when it is within the viewing area 770. The viewing criteria or percentage of effective pixels may be lower than the viewing criteria or percentage used to determine the viewing area 760. In some examples, the position and size of the viewing area 770 may depend on the capture position of the image capture device during the capture session used to create the 3D photograph and the position and orientation of the 3D photograph within the XR environment. For example, the capture position of the image capture device during a capture session used to create a 3D photograph relative to one or more captured objects can be used to determine the relative offset between the 3D photograph and the viewing area 770, as the image capture device is able to obtain the effective pixel data of the 3D photograph from that angle.

[0067] In some examples, the user is shown an indication to view area 770. In other examples, the user is not shown an indication to view area 770.

[0068] In some examples, light or shadow within the 3D photo 750 can affect the rendering of the XR environment 500, or affect both in a similar or identical manner to that described above with respect to the 3D photo 650.

[0069] like Figure 8 As shown, a 3D photograph 850 is presented within a virtual object 830 in the XR environment 500. In some embodiments, an indication 855 of the 3D photograph 850 within the XR environment 500 is provided. In some embodiments, the virtual object 830 is opaque. Figure 8 In this context, the 3D photo 850 is presented in a 3D presentation mode. In some implementations, the 3D photo 850 is placed on the bottom surface of the recess 830. Therefore, based on the dimensions of the recess 835 in the virtual object 830 (e.g., the shape, depth, and angle of the sidewalls), the viewing area of ​​the 3D photo 850 (e.g., by viewing an electronic device) is limited within the XR environment 500. In other words, when the viewing electronic device is outside the effective viewing area of ​​the 3D photo 850, the 3D photo 850 is occluded by the virtual object 830. Therefore, in some examples, no viewing area (similar to viewing areas 660, 760, or 770) is identified or presented to the user.

[0070] In some examples, light or shadow within the 3D photo 850 can affect the rendering of the XR environment 500, or affect both in a similar or identical manner to that described above with respect to the 3D photo 650.

[0071] Figure 9This is a flowchart illustrating exemplary methods for presenting 3D photographs in an XR environment according to some specific embodiments. In some embodiments, the 3D photograph is an incomplete 3D representation because a large number of views will be incomplete from at least some viewpoints. In some embodiments, the 3D photograph is created based on one or more images captured by an image capture device. In some embodiments, it is presented by an electronic device (e.g., Figures 1 to 3 Method 900 is executed by a controller 110 or electronic device 120. Method 900 may be executed at a mobile device, HMD, desktop computer, laptop computer, server device, or by multiple devices communicating with each other. In some embodiments, method 900 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 900 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0072] At box 910, method 900 obtains a 3D photograph comprising an incomplete three-dimensional (3D) representation created based on one or more images captured by an image capture device. In some specific implementations, the 3D photograph includes parallax 3D photographs (e.g., 2D images with depth), 1-to-1 scale 3D photographs, multi-view 3D photographs, etc. Figure 3 3D reconstructed photographs (e.g., wedding scenes or multiple slightly different images of the same scene taken by a single device) or 3D panoramic photographs. In some implementations, 3D photographs are incomplete because a large number of views will be incomplete from at least some viewpoints. For example, when viewed from the front and 5 feet, a 3D photograph may include enough data to provide values ​​for 99% of the pixels of the view, but when viewed from a 45° side angle and 10 feet, a 3D photograph may only include enough data to provide values ​​for 65% of the pixels of the view. In some implementations, 3D photographs can be created based on at least one image and additional sensor data (e.g., depth or motion data), based on image and depth data and pixel positions determined by triangulation (e.g., approximately using multiple photographs from the same or multiple users simultaneously), or from multiple photographs captured from panoramic type photographs.

[0073] At box 920, method 900 determines the viewing position of the device relative to the placement location of the 3D photograph. In some embodiments, the spatial position (e.g., pose and orientation) of the 3D photograph in the XR environment is determined. For example, at box 920, method 900 determines the location where the 3D photograph will be placed in the XR environment. Furthermore, for the XR environment, the spatial position (e.g., pose and orientation) of the viewing electronic device is known. In some embodiments, the viewing position (e.g., relative positioning) determines the distance or direction from the viewing electronic device to the 3D photograph.

[0074] At box 930, method 900 determines the rendering mode of the 3D photograph based on the viewing position. In some embodiments, the 3D photograph may be associated with one or more viewing areas (e.g., ideal viewing area, acceptable viewing area, invalid viewing area, etc.), and the rendering mode (e.g., 3D, blurred, 2D, not shown) may be determined accordingly. In some embodiments, the viewing area may be based on capture conditions during the creation of the 3D photograph (e.g., where one or more images used to create the 3D photograph were captured from, how much movement was present, etc.). For example, the effective viewing range may be based on the capture position of each image during the capture session used to create the 3D photograph or the average capture position of the image capture device. In some embodiments, the viewing position is within the effective viewing area of ​​the 3D photograph. In some embodiments, the viewing position may be directly evaluated by determining the amount or percentage of effective pixels viewing the 3D photograph from the viewing position and comparing it to a standard or threshold. For example, when more than 95% of the voxels have data, the 3D photo is rendered in 3D; when 70%-95% of the voxels have data, the 3D photo is rendered in 2D or with a blurred effect; and when less than 70% of the voxels have data, the 3D photo is removed or not rendered at all. In some implementations, the user viewing the electronic device may be provided with a way to switch between the rendering modes of the 3D photo.

[0075] In some implementations, the viewing position of the device relative to the placement of the 3D photograph is determined based on the movement of the image capture device during the capture of one or more images used to create the 3D photograph, either within or outside a first-type viewing area. In some implementations, the viewing position within the first-type viewing area uses the 3D rendering mode of the 3D photograph. In some implementations, the viewing area is a 3D viewing area based on all device capture positions during the creation of the 3D photograph and a well-defined volume (such as a sphere, oval, or box) surrounding those capture positions.

[0076] At box 940, method 900 provides a 3D photograph at the placement location based on the presentation mode in the XR environment. For example, the XR environment could be VR or MR. In some specific implementations, the electronic device is a head-mounted device worn by a person.

[0077] In some implementations, the 3D photograph is further modified, or the presentation mode is further modified based on a second viewing position relative to the placement of the 3D photograph. In some implementations, the presentation mode includes a 3D presentation mode, a blurred presentation mode, a 2D presentation mode, or removal from the XR environment.

[0078] In some implementations, the lighting or occlusion of the 3D photograph affects the XR environment because the size of the 3D photograph in the XR environment exceeds a specified threshold. In some implementations, the lighting or occlusion of the XR environment affects the 3D photograph. In some implementations, the 3D photograph is a 3D clipped portion of an incomplete 3D representation created from an image captured by an image capture device. In some implementations, the 3D clipped portion is based on depth data, 3D boundary regions, or 3D boundaries. In some implementations, the 3D photograph is a 3D clipped portion of an incomplete 3D representation created from contextual or semantic data determined from an image captured by an image capture device. In some implementations, the lighting or occlusion of the XR environment affects the 3D clipped portion.

[0079] In some implementations, during viewing, the spatial relationship between the rendered 3D photograph and its positional reference in the physical environment during capture can be replicated for the 3D photograph rendered in the XR environment. In some implementations, the capture relationship between the 3D photograph and the capture ground plane is determined using one or more images captured by an image capture device during the creation of the 3D photograph. In some implementations, during playback, the ground plane capture relationship of the 3D photograph is aligned with the ground plane of the XR environment. For example, when a 3D photograph is created, the actual size or spatial relationship between the 3D photograph and the ground plane of the physical environment (e.g., the surface on which the individual in the 3D photograph appears to stand, the floor of a room) can be determined, and then the corresponding size or spatial relationship can be mimicked in the XR environment relative to the ground plane (e.g., the floor of an office, a football field, or a similar surface on which the individual in the 3D photograph appears to stand).

[0080] In some implementations, missing pixels / voxels in a view of a 3D object from the viewing location are filled based on smoothing techniques (e.g., image, depth, additional, or related sensor data). In some implementations, missing pixels / voxels in a view of a 3D object from the viewing location are filled based on nearby pixels / voxels with image or depth values ​​(e.g., nearby color values, average color, nearby depth values, or depth values ​​with a confidence threshold). In some implementations, filling missing pixels in a view of a 3D object provides an artistic treatment of gaps or missing pixels in a 3D photograph. In some implementations, blurring missing pixels in a view of a 3D object provides an artistic treatment of gaps or missing pixels in a 3D photograph. In some implementations, filling missing pixels in a view of a 3D object is performed only when the number of missing pixels is below a preset threshold.

[0081] In some implementations, the method operates on an electronic device where the XR environment processing takes place. In some implementations, the electronic device is the same electronic device including an image sensor (e.g., an HMD). In some implementations, the electronic device is a different electronic device receiving data from an electronic device with an image sensor (e.g., a server separate from the HMD). In some implementations, a single electronic device including a processor implements the XR environment (e.g., an HMD). In some implementations, the XR environment is implemented by more than one electronic device.

[0082] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

[0083] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “computing,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0084] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein can be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0085] Specific implementations of the methods disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be varied; for example, the boxes can be reordered, combined, or divided into sub-boxes. Certain boxes or processes can be executed in parallel. In some implementations, the system includes a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform the exemplary methods disclosed herein. In some implementations, the non-transitory computer-readable storage medium stores program instructions executable on a computer to perform operations including the exemplary methods disclosed herein, these program instructions being implemented on an electronic device including a processor.

[0086] The use of “applies to” or “configured to” in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Similarly, the use of “based on” implies openness and inclusivity, as processes, steps, calculations, or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included in this document are for illustrative purposes only and are not intended to be restrictive.

[0087] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0088] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the term “comprising” as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0089] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

[0090] The foregoing description and summary of the present invention should be understood as illustrative and exemplary in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative specific embodiments, but also by the full extent permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method comprising: In electronic devices with processors: Obtain a 3D photograph comprising an incomplete 3D representation created from one or more images captured by an image capture device, wherein the incomplete 3D representation lacks data for providing a complete view from one or more viewpoints; Determine the viewing position of the electronic device relative to the placement of the 3D photograph within the extended reality (XR) environment; The rendering mode of the 3D photograph is determined based on an estimate of the effectiveness of viewing the 3D object from the viewing location, wherein estimating the effectiveness of viewing the 3D object includes: Detect the amount or percentage of effective pixels viewed from the viewing location; and Compare the amount or percentage of valid pixels viewed to a threshold; as well as The 3D photograph is presented at the placement location based on the presentation mode in the XR environment.

2. The method according to claim 1, further comprising: In response to a change in the viewing position of the electronic device relative to the placement position of the 3D photo to a second viewing position, the 3D photo or the presentation mode is modified.

3. The method according to claim 1, wherein the presentation mode of the 3D photograph is changed to a blurred presentation mode.

4. The method according to claim 1, wherein the presentation mode of the 3D photograph is changed to a 2D presentation mode.

5. The method of claim 1, wherein the presentation mode of the 3D photograph is changed to an invisible presentation mode.

6. The method of claim 2, wherein when at least a first threshold amount of pixels in the view of the 3D photograph includes an image value and a depth value, the second viewing position is determined to be located in a first type of viewing region; when at least a second threshold amount of pixels in the view of the 3D photograph that is less than the first threshold amount includes the image value and the depth value, the second viewing position is determined to be located in a second type of viewing region; otherwise, the second viewing position is determined to be located in a third type of viewing region.

7. The method of claim 2, wherein the second viewing position is determined to be located within or outside the first type of viewing area based on the movement of the image capturing device during the capture of the one or more images for creating the 3D photograph.

8. The method of claim 1, wherein the lighting or occlusion of the 3D photograph affects the XR environment, or the lighting or occlusion of the XR environment affects the 3D photograph.

9. The method according to claim 1, further comprising: The capture relationship between the 3D photograph and the ground plane is determined using one or more images captured by an image capture device; as well as The ground plane capture relationship of the 3D image is matched with the ground plane of the XR environment.

10. The method of claim 1, wherein the 3D photograph is a 3D cut-out portion of the incomplete 3D representation created based on the one or more images captured by the image capturing device.

11. The method of claim 10, wherein the 3D cutout portion is based on depth data, a 3D boundary region, or a 3D boundary, and wherein the lighting or occlusion of the XR environment affects the 3D cutout portion.

12. The method of claim 1, wherein pixels missing from the viewing location in the view of the 3D photograph are filled based on nearby pixels having image values ​​or depth values.

13. The method of claim 1, wherein at least one image and depth information are used to create the 3D photograph.

14. The method of claim 1, wherein the 3D photograph includes scaling information for scaling the 3D photograph in the XR environment.

15. The method of claim 1, wherein the 3D photograph is created using multiple images from a single electronic device, wherein the multiple images originate from a single capture event of a panoramic type.

16. The method of claim 1, wherein the 3D photograph is created using a plurality of images from a plurality of different electronic devices, wherein each of the plurality of images comprises at least a portion of a physical object in a physical environment.

17. The method of claim 1, wherein the 3D photograph is created using a plurality of images from a single electronic device, wherein the plurality of images are from a plurality of different capture events, each comprising at least a portion of a physical object in a physical environment.

18. The method of claim 1, wherein the 3D photograph is displayed within or on a virtual object in the XR environment.

19. A system comprising: Non-transitory computer-readable storage medium; as well as One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: Obtain a 3D photograph comprising an incomplete 3D representation created from one or more images captured by an image capture device, wherein the incomplete 3D representation lacks data for providing a complete view from one or more viewpoints; Determine the viewing position of the electronic device relative to the placement of the 3D photograph within the extended reality (XR) environment; The rendering mode of the 3D photograph is determined based on an estimate of the effectiveness of viewing the 3D object from the viewing location, wherein estimating the effectiveness of viewing the 3D object includes: Detect the amount or percentage of effective pixels viewed from the viewing location; and Compare the amount or percentage of valid pixels viewed to a threshold; as well as The 3D photograph is presented at the placement location based on the presentation mode in the XR environment.

20. A non-transitory computer-readable storage medium storing processor-executable instructions for performing operations, the operations including: Obtain a 3D photograph comprising an incomplete 3D representation created from one or more images captured by an image capture device, wherein the incomplete 3D representation lacks data for providing a complete view from one or more viewpoints; Determine the viewing position of the electronic device relative to the placement of the 3D photograph within the extended reality (XR) environment; The rendering mode of the 3D photograph is determined based on an estimate of the effectiveness of viewing the 3D object from the viewing location, wherein estimating the effectiveness of viewing the 3D object includes: Detect the amount or percentage of effective pixels viewed from the viewing location; and Compare the amount or percentage of valid pixels viewed to a threshold; as well as The 3D photograph is presented at the placement location based on the presentation mode in the XR environment.

Images