A technique for capturing and editing dynamic depth images

Abstract

To provide a user interface which enables selection of an object image.SOLUTION: A method includes capturing image data using one or more cameras, wherein the image data includes a primary image and associated depth values. The method further includes encoding the image data in an image format. The encoded image data includes the primary image encoded in the image format and image metadata that includes a device element that includes a profile element indicative of an image type and a first camera element, wherein the first camera element includes an image element and a depth map based on the depth values. The method further includes, after the encoding, storing the image data in a file container based on the image format. The method further includes causing the primary image to be displayed.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] Related Applications This application claims the benefit of priority to U.S. Provisional Application No. 62 / 827,739, filed April 1, 2019, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] background Users capture images using a variety of devices, e.g., mobile phones, wearable devices, smart appliances, smart speakers, tablets, computers, standalone cameras, etc. Many devices allow for capturing images with enhanced information. For example, a user may capture a still image and depth information associated with the scene depicted in the still image. In another example, a user may capture a still image and one or more alternative representations of the still image, e.g., a burst mode image including multiple image frames. In yet another example, a user may capture an image with a small amount of motion, e.g., motion captured over a short period of time, e.g., 0-3 seconds while the camera is running. In yet another example, a user may obtain an augmented reality (AR) image by augmenting a scene with objects, e.g., by inserting a virtual object into the scene before capturing the image.

[0003] A user may view and edit images using different types of devices, e.g., mobile phones, wearable devices, smart appliances, smart speakers, tablets, computers, standalone cameras, etc. Different devices may have different capabilities for displaying images, e.g., two-dimensional or three-dimensional display. Different software applications used to edit images may include different features, e.g., portrait effects (one or more objects in an image identified as the subject of the image are in focus and other objects are blurred), bokeh effects (selected objects are in focus and other parts of the image are blurred), etc. Some software applications may also include features that perform image segmentation to identify one or more objects, for example. Such applications allow a user to modify the identified objects, for example, by applying effects to the identified objects (e.g., changing to black and white or sepia, softening edges, etc.) while leaving the rest of the image intact, extract the identified objects, for example, by a cut or paste operation, or insert the objects into another image.

[0004] The background discussion provided herein is intended to generally present the contents of the present disclosure, and the currently known work of the inventors to the extent described in the background section, and aspects of the discussion (unless deemed prior art at the time of filing) are not expressly or impliedly indicated or admitted as prior art to the present disclosure. Summary of the Invention

[0005] overview Some embodiments relate to a computer-implemented method that includes capturing image data using one or more cameras. The image data includes a primary image and associated depth values. The method further includes encoding the image data into an image format. The encoded image data includes the primary image encoded into the image format and image metadata, the image metadata including a device element including a profile element indicating the image format, and a first camera element. The first camera element includes an image element, and a depth value. The method further includes storing the image data in a file container based on the image format. The method further includes displaying the main image.

[0006] In some implementations, the depth map includes a plurality of pixel values, each pixel value indicating a distance from a particular camera of the one or more cameras. In some implementations, the depth map further includes a lens focus model that defines values ​​of a radius of a circle of confusion corresponding to a plurality of distances from the particular camera. In some implementations, the plurality of distances includes a near point distance and a far point distance, the near point distance being less than the far point distance. In some implementations, the plurality of distances further includes a focal length greater than the near point distance and less than the far point distance.

[0007] In some implementations, the method further includes obtaining a plurality of pixel values ​​by converting the depth values ​​to an integer format and compressing the converted depth values ​​based on an image format.

[0008] In some embodiments, the image data may further include one or more sub-images. In some embodiments, each sub-image may be taken by a different one of the one or more cameras than the particular camera. In some embodiments, the image metadata may further include one or more additional camera elements, each of which includes an image element corresponding to at least one of the one or more sub-images and including a pointer to the at least one image. In some embodiments, the one or more additional camera elements are sequentially organized in the image metadata after the first camera element. In some embodiments, storing the image data in the file container includes concatenating the main image and the one or more sub-images encoded in the image format. In some embodiments, the order of the main image and the one or more sub-images in the file container is the same as the order of the one or more additional camera elements in the image metadata.

[0009] In some implementations, the image data may further include one or more sub-images, each captured by a camera included in the one or more cameras. In some implementations, the one or more sub-images may include a depth image captured by a depth camera or video.

[0010] Some embodiments relate to a computer-implemented method that includes obtaining image data from a file container. The image data includes a plurality of pixel values ​​corresponding to a primary image and image metadata including a lens focus model. The method further includes causing the primary image to be displayed. The method further includes receiving a user input indicating a target focal length. The method further includes modifying one or more pixel values ​​of the primary image based on the lens focus model in response to the user input such that an object at the target focal length of the modified image is in focus to obtain a modified image. The method further includes causing the modified image to be displayed.

[0011] In some implementations, the lens focus model may define values ​​for the radius of the circle of confusion corresponding to multiple distances from the camera that captured the primary image. In these implementations, modifying the one or more pixel values ​​blurs one or more objects in the primary image, where the one or more objects are associated with depth data indicative of one or more objects at a distance from the camera that captured the primary image that is different from the target focal distance.

[0012] In some implementations, the lens focus model may be stored in a file container of a depth map element in the image metadata. In some implementations, the depth map element may correspond to the camera that captured the main image.

[0013] Some embodiments relate to a computer-implemented method, the method including obtaining image data from a file container. The image data includes a plurality of pixel values ​​corresponding to a main image and image metadata including a depth map. The method further includes displaying the main image. The method further includes receiving user input indicating at least one of a crop operation on the main image or a resize operation on the main image. The method further includes modifying the main image based on the user input to obtain a modified image. The method further includes calculating an updated depth map, the updated depth map including depth data corresponding to the modified image. The method further includes updating the file container to replace the main image with the modified image and replacing the depth map with the updated depth map.

[0014] Some embodiments relate to a computer-implemented method, the method including obtaining image data from a file container. The image data includes a plurality of pixel values, each pixel value corresponding to a particular pixel of a plurality of pixels of a primary image, and image metadata including a plurality of depth maps. The method further includes generating a three-dimensional image based on the primary image and the plurality of depth maps. The method further includes displaying the three-dimensional image.

[0015] In some implementations, the method further includes receiving a user input indicating a tilt or pan operation relative to the three-dimensional image. The method further includes, in response to receiving the user input, generating an updated three-dimensional image based on the user input, the main image, and the plurality of depth maps. The method further includes causing the updated three-dimensional image to be displayed.

[0016] In some implementations, multiple depth maps may be obtained by using depth from a single camera motion technique. The depth map of the numbers may be obtained via one or more depth cameras.

[0017] Some embodiments relate to a computer-implemented method, the method including obtaining image data from a file container. The image data includes a plurality of pixel values, each pixel value corresponding to a pixel of a plurality of pixels of a main image, and image metadata including a depth map. The method further includes displaying the main image. The method further includes receiving user input indicating a selection of an object appearing in the main image. The method further includes generating a segmentation mask based on the depth map. The segmentation mask may identify a subset of pixels of the main image that correspond to the object. The method further includes obtaining an object image of the object based on the segmentation mask. The method further includes providing a user interface to enable selection of the object image.

[0018] In some implementations, the depth map includes a plurality of depth values, each corresponding to a particular pixel of the main image, In some implementations, generating the segmentation mask includes selecting pixels of the main image having depth values ​​within a threshold range of depth values.

[0019] Some embodiments relate to a computing apparatus, computer system, or computing device for performing any of the methods described above. Some embodiments relate to a computer readable storage medium having stored thereon instructions or a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform the methods described above. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram of an exemplary network environment that may be used in one or more implementations described herein. [Figure 2A] FIG. 1 illustrates an example of image capture using a camera, according to some embodiments. [Figure 2B] FIG. 2 illustrates an exemplary image captured using a camera and a corresponding depth image. [Diagram 3] FIG. 2 illustrates an exemplary image file including a metadata structure for storing dynamic depth information. [Figure 4] FIG. 1 is a flow diagram illustrating an exemplary method according to some embodiments. [Diagram 5] FIG. 1 is a flow diagram illustrating another exemplary method according to some implementations. [Figure 6] FIG. 1 is a block diagram of an example device that may be used in one or more implementations described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description Depth imaging and augmented reality imaging are increasingly prevalent camera applications across mobile devices and operating systems. However, there is a lack of standardization in taking, storing, sharing, and editing such images. Depth images created using a camera application may not be editable in other applications.

[0022] The techniques described herein provide a uniform and consistent framework within the camera framework used by various client devices to store image metadata and / or additional images in one concatenated file container. Any application can access the images stored using the techniques described herein to view and / or edit the images. Thus, the techniques described herein can improve interoperability between applications and / or devices when taking, storing, sharing, or editing depth and / or augmented reality images.

[0023] These techniques are easy to implement, can be quickly analyzed, and can extend existing image formats across a variety of operating systems used in cameras, including mobile device cameras, and in software, including mobile and personal computing device software. These techniques allow device-related metadata, such as depth information, imaging model of the device camera, etc., to be stored in the image metadata of a captured image along with any number of sub-images. The stored metadata and sub-images can be utilized by any image viewing or editing application.

[0024] Like reference numbers are used in FIG. 1 to refer to like elements. A letter following a reference number, such as "156a," indicates that the text is specifically referring to the element having that particular reference number. A reference number in text that is not followed by a letter, such as "156," refers to any or all elements having that reference number in the drawings (e.g., "156" in the text refers to reference numbers "156a" and / or "156b" in the figures).

[0025] 1 is a block diagram of an exemplary network environment 100 that may be used in some implementations described herein. In some implementations, the network environment 100 may include one or more server systems, such as the example server system 102 of FIG. The server system 102 may be in communication with the network 130, for example. The server system 102 may include a server device 104 and a storage device (not shown). The server system 102 may store one or more image files 106b. In some implementations, the server device 104 may provide an image management application 156b. The image management application 156b may access each image file 106b with the permission of the user who provided the image file 106b.

[0026] The network environment 100 may also include one or more client devices, e.g., client devices 120, 122, 124, and 126. The client devices 120, 122, 124, and 126 may communicate with each other and / or with the server system 102 via a network 130. The network 130 may be any type of communication network including one or more of the Internet, a local area network (LAN), a wireless network, a switched or hub connection, and the like. In some implementations, the network 130 may include peer-to-peer communication between devices, for example, using peer-to-peer wireless protocols (e.g., Bluetooth, Wi-Fi Direct, and the like). An example of peer-to-peer communication between two client devices 120 and 122 is illustrated by arrow 132.

[0027] For ease of explanation, FIG. 1 shows one block of server system 102 and server device 104, and four blocks of client devices 120, 122, 124, and 126. Server blocks 102 and 104 may represent multiple systems, multiple server devices, and multiple network databases, and these blocks may be provided in configurations different than those shown. For example, server system 102 may represent multiple server systems that can communicate with other server systems over network 130. In some implementations, server system 102 may include, for example, a cloud hosting server. In some examples, one or more storage devices may be provided in a server system block(s) separate from server device 104 and that can communicate with server device 104 and other server systems over network 130.

[0028] There may be any number of client devices. Each client device may be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smart phone, a tablet computer, a television, a TV set-top box or entertainment device, a wearable device (e.g., viewing glasses or goggles, a watch, a headset, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming console, etc. Some client devices may also include local image files 106a, for example, taken with a camera of the client device or obtained in other ways, such as by downloading an image file from a server. In some implementations, the network environment 100 may not have all of the illustrated components and / or may have other elements, including other types of elements instead of or in addition to the elements described herein.

[0029] In various implementations, end users U1, U2, U3, and U4 may utilize their respective client devices 120, 122, 124, and 126 to communicate with the server system 102 and / or with each other. In some examples, users U1, U2, U3, and U4 may interact with each other through applications running on their respective client devices and / or the server system 102, via a network service implemented on the server system 102, such as a social networking service, an image hosting service, or other type of network service. For example, each client device 120, 122, 124, and 126 may communicate data to or from one or more server systems, such as system 102.

[0030] In some implementations, the server system 102 may provide appropriate data to the client devices to enable each client device to receive the communicated content or the shared content uploaded to the server system 102 and / or the network service. In some examples, the users U1-U4 may interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications. The network services implemented by the server system 102 may include systems that allow users to communicate, link, associate, upload and post shared content, such as images, text, video, audio, and other types of content, and / or perform other functions. For example, the client devices may display received data, such as content posts that are sent or streamed to the client device and that originate from (or directly from) different client devices via the server and / or network service or that originate from the server system and / or network service. In some implementations, the client devices may communicate directly with each other, for example, using the peer-to-peer communication between the client devices described above. In some implementations, a "user" may include one or more programmatic virtual entities and any person connected to a system or network.

[0031] In some implementations, any of client devices 120, 122, 124, and / or 126 may provide one or more applications. For example, as shown in FIG. 1, client device 120 may provide a camera application 152 and an image management application 156a. Client devices 122-126 may also provide similar applications. For example, camera application 152 may provide users of each client device (e.g., users U1-U4) with the ability to take images using one or more cameras of the client device. For example, camera application 152 may be a software application running on client device 120.

[0032] In some implementations, camera application 152 may provide a user interface. For example, the user interface may enable a user of client device 120 to select an image capture mode, such as a still image (or photo) mode, a burst mode (e.g., taking multiple images in rapid succession), a motion picture mode, a video mode, a high dynamic range (HDR) mode, etc. In some implementations, camera application 152 may implement (e.g., in part or in whole) the methods described herein with reference to Figures 4 and 5. In some implementations, image management application 156a and / or image management application 156b may implement (e.g., in part or in whole) the methods described herein with reference to Figures 4 and 5.

[0033] The camera application 152 and the image management application 156a may be implemented using hardware and / or software on the client device 120. In different implementations, the image management application 156a may be a standalone application running on, for example, any of the client devices 120-124, or may operate in conjunction with an image management application 156b provided on the server system 102. The application 156 may allow a user to view and / or edit an image, such as an image file 106.

[0034] With user permission, the image management application 156 may perform one or more automatic functions, such as an automatic function to store (e.g., back up) an image on a server, an automatic function to edit an image, an automatic function to automatically enhance an image, an automatic function to stabilize an image, an automatic function to recognize one or more features in an image, such as a face, a body, a type of object, a type of motion, etc.

[0035] Image management application 156 may also provide image management functions such as image management functions for displaying images in a user interface (e.g., in a one-up view with a single image, a grid view with multiple images, etc.), editing images (e.g., adjusting image settings, applying filters, changing the focus of an image, applying portrait effects, applying bokeh effects, applying other effects, viewing three-dimensional images using pan / tilt operations, extracting objects from images using image segmentation techniques, etc.), sharing images with other users (e.g., of client devices 120-126), archiving images (e.g., storing images so that they do not appear in the main user interface), and generating image-based works (e.g., motion-based artifacts such as collages, photo books, animations, stories, video loops, etc.). In some implementations, to generate image-based works, image management application 156 may utilize one or more labels associated with the images or image metadata stored in the images.

[0036] User interfaces on client devices 120, 122, 124, and / or 126 may enable the display of user and other content, including images, videos, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using software on the client device, software on the server device, and / or a combination of client and server software running on server device 104, e.g., application software or client software in communication with server system 102. The user interfaces may be displayed on a display device of the client device or server device, e.g., a touch screen or other display screen, a projector, etc. In some implementations, application programs running on the server system may communicate with the client devices to receive user input at the client devices and output data, such as image data, audio data, etc., at the client devices.

[0037] In some implementations, the server system 102 and / or any of one or more of the client devices 120-126 may provide a communications application program. This communications program may enable a system (e.g., a client device or a server system) to provide options for communicating with other devices. The communications program may provide one or more associated user interfaces for display on a display device associated with the server system or client device. The user interface may provide the user with various options for selecting a communication mode, a user or device to communicate with, etc. In some examples, the communications program may provide an option for sending or broadcasting a content post, for example to a broadcast area, and / or provide a notification that a content post has been received by the device, e.g., a notification that the device is in a defined broadcast area for the post. The communications program may display or output the sent and received content posts, for example, in various formats. Content posts may include, for example, pictures that have been shared with other users.

[0038] Other implementations of the features described herein can utilize any type of system and / or service. For example, other network-connected (e.g., Internet-connected) services can be used instead of or in addition to social networking services. Any type of electronic device can utilize the features described herein. Some implementations can provide one or more features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device that includes or has a display device connected thereto can display data (e.g., content) that is stored on a storage device local to the client device and that was previously received, e.g., over a communication network.

[0039] FIG. 2A is a diagram illustrating an example of capturing an image using a camera. As shown in FIG. 2A, an image may be captured using a camera having a lens 202. Although FIG. 2A illustrates a lens 202, the camera may include other elements not shown in FIG. 2A, such as an image sensor, a focal length adjustment device, and the like. When capturing an image, the focus of the camera is on a focal plane 208. The captured image may include multiple objects at different distances from the lens 202 (different distances from the camera). For example, the captured image may include a near view, a near field ... The scene may include one or more objects in the background plane 206, the focal plane 208, a far plane 210, and any intermediate planes.

[0040] A device used for image capture (e.g., any of client devices 120-126) may include one or more cameras or one or more image sensors. For example, a device may include one or more cameras that can be used to capture color images (e.g., RGB cameras), infrared cameras, etc. For example, the one or more cameras may include cameras with different configurations, such as telephoto cameras, wide-angle cameras, etc. Other configurations of cameras may be used. Each camera may generate a corresponding image. In some implementations, image data acquired from different cameras of a device may be combined to obtain a main image. For example, the main image may be displayed on a screen of the device. Furthermore, the main image may be stored in an image format, such as JPEG, PNG, TIFF, HEIF, etc., using techniques described herein. Furthermore, one or more images acquired from each camera may be stored in a container, as described below with reference to FIG. 3, using techniques described herein.

[0041] 2B illustrates an example image 220 captured using a camera and a corresponding depth image 230. For example, the depth image may be captured using an infrared camera or other depth sensor. In another example, the depth image may be generated based on image data captured using the camera that captured image 220.

[0042] As can be seen in Figure 2B, image 220 depicts a scene including a flower closer to the camera than leaves that are seen in the background. Correspondingly, in depth image 230, the petals closer to the camera appear darker than the other petals further away from the camera and the leaves further away from the camera. In depth image 230, darker shades correspond to objects in the scene that are closer to the camera and lighter shades correspond to objects that are further away from the camera.

[0043] Depth image 230 may be an image of scalar values ​​(e.g., integer, real, or floating point values, etc.) that represent the distance from a camera viewpoint to objects seen in a corresponding image (e.g., image 220). The definition of a depth value may vary based on the type of depth sensor. By way of example, two common definitions are the depth along the optical axis (usually the Z axis) from the camera that captured the image, and the depth along the ray that passes through each pixel during capture of the image.

[0044] FIG. 3 illustrates an exemplary image file 300 that includes a metadata structure for storing dynamic depth information. Structured metadata can be utilized for images that include depth information. In some implementations, the image file 300 may be stored as a linked file container. A linked file container consists of a composite or linked file in which a primary image has no secondary media files attached to it, or in which a primary image has one or more secondary media files attached to it. In the example shown in FIG. 3, a primary image 340 and associated metadata 310, as well as eXtensible Metadata Platform (XMP) metadata 320 are shown. In some implementations, the secondary media files may include another representation of the primary image or associated media, such as depth data.

[0045] The image file 300 may be any type of image, such as, for example, JPEG, PNG, TIFF, HEIF, etc. The image file 300 includes a main image 340. The image file 300 may further include metadata 310, such as Exchangeable Image File Format (EXIF) data. The EXIF ​​data may include one or more settings, such as ISO speed, shutter speed, aperture, white balance, lens type, focal length, date and time, etc.

[0046] In some implementations, the image file 300 may be stored as an ISO Box Media File Format Container in accordance with ISO / IEC 14496-12. In these implementations, the XMP metadata in the container includes a ContainerXMP metadata directory element where each item in the directory references a box in the ISO / IEC 14496-12 container using a Uniform Resource Identifier (URI).

[0047] In some implementations, a container element encoded in the XMP metadata of an image file may define a directory of media items in the container. In some implementations, the concatenated media items are located in the container file in the same order as the media item elements in the directory. In some implementations, the concatenated media items are tightly packed, e.g., the file container has data corresponding to each different media item arranged contiguously with no gaps. For example, an image file may include bytes corresponding to the concatenated media items concatenated after the main image (e.g., main image 340).

[0048] The main image 340 may be a display-ready image, such as a JPEG image or an image in another format. The image file 300 may also include secondary images, such as depth data, intermediate images, or other representations of the main image, or may include other media items (e.g., one or more videos, text, etc.). As described above, secondary images and / or media items may be stored in a linked file container. In some implementations where the image file 300 is an ISO / IEC 14496-12 ISO Box media file format container, secondary images may be stored in the container as other boxes.

[0049] The image file 300 may further include eXtensible Metadata Platform (XMP) metadata 320. The XMP metadata 320 may include one or more The metadata may include data specifying the device, e.g., Device element 324. For example, the metadata may be numbered sequentially by conforming to ISO16684-1:2011(E)XMP Specification Part 1, e.g., Adobe XMP The metadata may be embedded in the image file 300 as described in Specification Part 3 Storage in Files. In some embodiments, the image file 300 may include the following items formatted as Resource Description Framework / eXtensible Markup Language (RDF / XML):

[0050] In some implementations, the XMP metadata 320 (also referred to as the container XMP metadata directory) may specify the order and properties of subsequent media files in the concatenated file container. Each element may be an element of the structure Container:Item and may specify the layout and contents of the container. The container type may be, for example, a "Device" element, a "Camera" element, etc. Each file in the concatenated container (e.g., other images, video, text, etc.) may have a corresponding media item in the directory. The media item may describe the location in the file container and basic properties of each concatenated file contained in the concatenated file container. In some implementations, the media items in the container directory may be referenced by an ItemURI (Uniform Resource Identifier) ​​attribute of an Image element or a Depth Map element in the metadata. The URI may be based on a namespace, for example, with Container as the default namespace prefix. The directory contains only one main image, which is the first item contained in the directory. The Item element contains a description of each item available to the application.

[0051] In some implementations, the root metadata object in the XMP data 320 may include a Device element (324). The Device element (324) is also referred to as a device 324. In some implementations, the Device element 324 is required, e.g., at least one Device 324 must be included in the XMP data 320. In some implementations, more than one Device element can be included.

[0052] The Device element 324 may be specified according to one or more profiles or uses. Each profile may have a corresponding set of required elements and data. The profile may be usable by an application, such as a camera application that generated the image file 300, or by another application that accesses the image file 300, such as an image backup application, an image viewing application, or an image editing application. For example, the profile may be usable to identify uses that a particular image file can support. The image file 300 may include one or more profiles. In some implementations, each profile may correspond to a particular use and may have a corresponding set of required elements and data. For example, an application that accesses the image file 300 to allow a user to view or edit an image may be able to utilize the profile(s) to identify uses that a given image file can support.

[0053] In the example shown in FIG. 3, the XMP metadata 320 for the image file 300 includes one device element 324 that includes a Profile element 326 (also referred to as Profile326) of profile type "DepthPhoto." Furthermore, Profile326 indicates the number of cameras ("cameraindices=i", where i is an index value) that captured the image data for the image file 300, e.g., when the image was captured (or created). For example, in some implementations where the camera index is 0, the primary image is captured by the camera. The image referenced in the Camera element may be the same as the image referenced in the Camera element. In some implementations, a profile of profile type "DepthPhoto" may be required to include at least one Profile element 326 and at least one camera element (e.g., "Cameras 328"). In the example shown in FIG. 3, Cameras 328 (also referred to as Camera element 328) is shown as including camera element 330 (Camera i). In some implementations, Cameras 328 is a sequence of one or more camera elements.

[0054] In some implementations, each camera element (e.g., Camera element 330) includes an image element 332 for an image provided by the camera and a depth map element 334 (also referred to as Depth Map 334). The image element 332 may include an attribute called "Primary" to indicate whether the image element corresponds to a primary image 340 in the linked file container. In some implementations, the image element 332 may include an attribute called "Original" to indicate whether the media item to which the image element 332 points is an original version of an image, e.g., an unfiltered color image that can be used to render depth effects. In some implementations, the image element 332 may include an attribute called "Raw" to indicate that the media item, e.g., the image to which the image element 332 points is a raw version of an image that is not display-ready and can be processed to obtain a display-ready image. In some implementations, the image element 332 may include an attribute called "Depth" to indicate that the media item, e.g., the image to which the image element 332 points contains a depth image.

[0055] The Depth Map 334 may include depth-related information and / or a depth map of an image provided by a corresponding camera. In some implementations, the Image element 332 and / or the Depth Map element 334 may include a Uniform Resource Identifier (URI) that points to the location of the corresponding image data and / or depth data stored in the image file 300, e.g., in a linked file container. The depth data of the first camera should not have holes. The application generating the image file encodes predicted values ​​in areas of the image where depth values ​​are not available or can not be calculated.

[0056] In some implementations, the depth image includes multiple values, each value corresponding to a particular pixel of the depth image. For example, the depth information in floating point or 32-bit integer format is converted to integer format and compressed using an image codec supported by the file container type (e.g., JPEG). The conversion to integer format can be done in different ways, including linear range conversion and inverse range conversion. For example, an inverse conversion may be used when the depth map loses precision when encoded, e.g., converting floating point depth values ​​to 16-bit integers.

[0057] Linear transformation (RangeLinear): d is the depth distance value of a pixel, with minimum and maximum possible depth values ​​for the near and far points. First, we normalize the depth values ​​to the [0,1] range using the following formula:

[0058]

number

[0059] The depth values ​​are then quantized to 16 bits using the following formula:

[0060]

number

[0061] The quantized depth value d is calculated using the following formula: n Recover depth from

[0062]

number

[0063] Inverse Transformation (RangeInverse): RangeInverse assigns more bits to the near-point depth value and fewer bits to the far-point value. First, the depth values ​​are normalized to the [0,1] range using the following formula:

[0064]

number

[0065] The depth values ​​are then quantized to 16 bits using the following formula:

[0066]

number

[0067] The quantized depth value d is calculated using the following formula: n Recover depth from

[0068]

number

[0069] In some implementations, the Depth Map 334 may include a lens focal model of the depth data. In some implementations, the lens model may include metadata stored, for example, in a FocalTable attribute. The metadata may be used by an application to render, for example, depth-of-field effects. The lens model defines the radius of the circle of confusion at different distances from the viewer, for example, the camera that captured the image. The FocalTable may include distance and radius pairs that can be used to build a lookup table that defines the circle of confusion as a function of depth distance values.

[0070] In some implementations, the FocalTable attribute may be a string value consisting of Base64 encoded little-endian floating-point pairs. These floating-point pairs are the actual distance values. <distance> 、 <radius>The pair defines a look-up table that can be used, for example, by an image display or editing application to calculate the radius of the circle of confusion at distance values ​​between the near and far point values ​​of the depth map.

[0071] In some implementations, the distance and radius pairs are stored in ascending order sorted by distance value. In some implementations, the distance coordinates may be specified in units of the distance values ​​of the depth map. In some implementations, the radius values ​​may be specified in pixel coordinates. Each radius value is a value equal to or greater than zero. A radius value of zero may represent a depth distance that is in focus on the focal plane of the image. The lookup table includes at least two pairs, e.g., a first pair corresponding to a near distance of the depth map and a second pair corresponding to a far distance of the depth map. In some implementations, the radius values ​​between the points defined in the FocalTable may be interpolated (e.g., linearly interpolated) to obtain the radius of the circle of confusion at any focal distance from the camera that captured the image.

[0072] In some implementations, the FocalTable may include three radius values, a near point value for the near view, a focal plane value for the focal plane, and a far point value for the far view. An image display application may access the FocalTable to draw objects that are in focus at the focal plane depth value. In some implementations, the FocalTable may include two or more distance values ​​that correspond to, for example, focal planes that are in focus.

[0073] In some implementations, in the Camera element 328, the main image 340 corresponds to a first camera. In some implementations, the first camera element described in Cameras 328 may be a main camera, for example, a main camera of a device, such as a smartphone or tablet terminal. In some implementations, additional camera elements may correspond to other cameras of the device, for example, an infrared camera of a smartphone or tablet device. The order of the other cameras may be arbitrary. In some implementations, each additional Camera element may include a corresponding Pose element. The Pose element indicates the pose of the camera with respect to the device. In some implementations, all images and depth maps included in one Camera element may be rectified by that same camera. For example, the elements have the same pose, the same aspect ratio, and the same field of view.

[0074] For example, if the image file 300 is modified by an image editing application, such as cropping, resizing, etc., one or more Camera elements may be updated to reflect the modifications to the image, e.g., updating the depth map and image elements of the Camera element.

[0075] In some implementations, the image file 300 may include information about the image capture, such as pose, which includes the position and orientation of the image sensor(s) or camera(s) that captured the image relative to the scene. Including this information allows an application to use images from multiple cameras together (e.g., images from a color camera and depth images from a depth sensor). For example, the pose information allows an application to map depth data (which may have different resolutions) onto the image.

[0076] In some implementations, Device 324 may include one or more other elements, such as a Container element that is an ordered directory of concatenated files in a file container, a VendorInfo element that contains vendor-related information for the device, an AppInfo element that contains application-specific or rendering information for the device, etc.

[0077] In some implementations, the camera element 330 may include, for example, a VendorInfo element that contains vendor-related information for the camera, an AppInfo element that contains application-specific or rendering information for the camera, an Image element that contains the image provided by the camera (e.g., a depth The ImagingModel element may include one or more additional elements, such as a focal length along the X and Y axes, expressed as a real number, normalized by the image sensor dimensions; a principal point (X and Y positions) where the optical axis of the camera intersects the center of the image plane of the camera along the X and Y axes, respectively, normalized by the height of the sensor; an image width and height, expressed in pixels; a camera tilt, expressed in degrees, e.g., the negative angle that the Y axis of the image is tilted from the vertical; a pixel aspect ratio of pixel width to pixel height; and one or more distortion parameters. If the image is edited, e.g., resized or cropped, the ImagingModel element may be updated accordingly.

[0078] The Item structure may specify the layout and contents of the container in which the items are contained, for example, in the form Container:Item. For example, a container element called "Profile" may contain an item called "type" and an item called "camera indices" as shown in Figure 3. In another example, an element called "Camera" may contain an item called "image" that contains a URI to a corresponding image in a file container, an item called "depth map" that contains a depth map of the image, etc. Each item element contains information that enables an image backup application, an image viewing application, or an image editing application to, for example, allow a user to view or edit the image.

[0079] In some implementations, the first media item in the directory is a main image 340. The main image element has an associated field to indicate the MIME type. This field may be a simple string indicating the MIME type of the media items in the container. For example, the MIME type value for a main image may be:

[0080] [Table 1]

[0081] Additionally, the value of the length field in the main image element (if included) may be set to 0. In embodiments where the main image is the first file in a file container, an application may determine the length of the main image by parsing it from the beginning of the file container based on the indicated MIME type.

[0082] The first media item of XMP metadata may optionally contain a Padding attribute that specifies additional padding between the end of the encoded primary image and the beginning of the first secondary image.

[0083] In some implementations, if the media items are tightly packed, subsequent sub-images Media items do not include a Padding attribute. Each media item (e.g., an image other than the primary image 340) has a corresponding MIME type attribute and a Length attribute. The length attribute may be an integer value and may be required for secondary media items. In some implementations, the length attribute may be a simple string with a positive integer length indicating the number of bytes of the item contained in the corresponding file in the concatenated file container. In general, an Item element includes a Mime attribute that is one of the image MIME types listed in Table 1. In some implementations, consecutive media items may share resource data in a file container. The first media item may indicate the location of the resource in the file container and subsequent shared media items may have the length attribute set to 0. If the resource data is itself a container, the DataURI may be used to locate the media item data in the resource. An application may locate a media item resource in the concatenated file container by adding the length of the primary image plus the padding value of the primary image (if specified) to the length of the preceding secondary item resource.

[0084] The following table describes examples of the various attributes that can be included in an Item element.

[0085] [Table 2]

[0086] In some implementations, the XMP metadata may specify parameters of an augmented reality (AR) image. The profile type of the AR image may be "ARPhoto". Such parameters may include, for example, the pose of the client device that captured the AR image, lighting estimation information, horizontal and / or vertical surface planes (e.g., floors, walls, etc.) included in the scene, camera internal parameters, etc. Additionally, the AR parameters may include application-specific metadata, for example provided by the image capture application or the augmented reality application. Such application-specific metadata may include three-dimensional (3D) asset identifier(s), for example, one or more virtual objects utilized to augment the real-world scene captured in the AR image, and the corresponding pose. The object may include

[0087] Furthermore, in some implementations, one or more images can be embedded within an image file used for an augmented reality image. An exemplary application of such embedding is when the embedded image is an embedded image without 3D assets and the container, e.g., the main image, includes the 3D assets. This allows a user to preview thumbnails with 3D assets, e.g., in an image gallery, while allowing a display or editing application to provide a user interface that allows the user to interact with and move virtual objects in the augmented reality image. In this example, the main image may be updated when the user saves the augmented reality image after editing it. In this example, the XMP metadata stores the embedded image in the camera with an index greater than 0. If there is no embedded image, camera 0 includes the AR metadata.

[0088] Also, other types of profiles other than, for example, "depthphoto" and "ARphoto", may be specified in the XMP metadata to support other features and uses.

[0089] In some implementations, for example, an application such as a camera application on a mobile device may generate image file 300 using camera hardware on a client device (e.g., any of client devices 120-126). In these implementations, the camera application may generate image file 300 in the format described above. To generate image file 300, image data may be obtained from an image sensor, for example a hardware sensor (e.g., CCD, CMOS, infrared, etc.).

[0090] Image data acquired from an image sensor may be modified, e.g., by a camera application, so that the pose and aspect ratio of the primary image and one or more other images (e.g., each corresponding to camera i) included in image file 300 are the same. For example, the primary image and one or more other images may be cropped to have the same aspect ratio. In some implementations, image file 300 may include a depth map, e.g., if depth information is available from one or more cameras. In some implementations, the resolution of the depth map may be the same as the resolution of the primary image and one or more other images. In some implementations, the resolution of the depth map may be different from the resolution of the primary image and one or more other images. In some implementations, the one or more other images may be stored in image file 300 and referenced in additional camera elements.

[0091] An image file, being a concatenated file container with XMP metadata as described herein, allows an image backup application, an image viewing application, or an image editing application to present media, e.g., depth photos, augmented reality photos, etc., in a manner consistent with the generation of the image, e.g., capture with one or more image sensors of the client device. Additionally, including the main image in the file container allows applications or services that cannot parse the XMP metadata to obtain a display-ready version of the image.

[0092] 4 is a flow diagram illustrating an example method 400 according to some embodiments. In some embodiments, the method 400 may be implemented, for example, on the server system 102 shown in FIG. 1. In some embodiments, the method 400 may be implemented on one or more of the client devices 120, 122, 124, or 126, one or more server devices, and / or a server system shown in FIG. Some or all of method 400 may be implemented on both the server device(s) and the client device(s). In the described example, the implementation system comprises one or more digital processors or processing circuits ("processors") and one or more storage devices. In some implementations, different components of one or more servers and / or one or more clients may perform different blocks or other portions of method 400. In some examples, a first device is described as performing blocks of method 400. Some implementations may have one or more blocks of method 400 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0093] In various implementations, client device 120 may be a standalone camera, another device with a camera (e.g., a smartphone, a tablet, a computer, a wearable device such as a smart watch, a headset, etc.), or other client device capable of receiving images or video captured by another device. In some implementations, client device 120 may be a capture-only device, e.g., a camera without a screen. In some implementations, client device 120 may be a display-only device, e.g., a device with a screen capable of displaying images or video, but without camera functionality or other functionality for capturing images or video. In some implementations, client device 120 may have both capture and display capabilities.

[0094] In some implementations, client device 120 may include one camera for capturing images or videos. In some implementations, client device 120 may include multiple cameras (or lenses). For example, a smartphone or other device may include one or more front-facing cameras (on the same side as the device's screen) and / or one or more rear-facing cameras. In some implementations, the one or more front-facing or rear-facing cameras may work in tandem during capture, e.g., a first camera may capture depth information and a second camera may capture image pixels for an image or video. In some implementations, different cameras (e.g., telephoto lenses, wide-angle lenses, etc.) may be used for different types of image capture or video capture, e.g., with different zoom levels. In some implementations, client device 120 may be configured to capture 360-degree images or videos. In some implementations, a camera or lens may capture images using one image sensor (e.g., a CCD or CMOS sensor) or multiple sensors. In some implementations, other sensors, e.g., depth sensors, may be used in conjunction with one or more cameras during image capture.

[0095] In some implementations, client device 120 may combine raw image data captured by the image sensors of one or more of the cameras (or lenses) with other data obtained from other sensors (e.g., accelerometers, gyroscopes, position sensors, depth sensors, etc.) to form an image and store the image in a file container.

[0096] Client device 120 allows a user to capture images in different modes, such as, for example, a still image (or photo) mode for capturing a single frame, a burst or video mode for capturing multiple frames, a video mode for capturing a video containing multiple frames, etc. In some implementations, method 400 may be performed at image capture time, which is when the camera captures the image.

[0097] The client device 120 allows the user to, for example, view images or videos taken by the client device 120 or related to the user across different user interfaces. The images may be viewed on a single interface. For example, a one-up or slideshow mode may be provided that allows a user to view and / or edit one image or one video at a time. In another example, a gallery mode may be provided that allows a user to view and / or edit multiple images simultaneously, for example, as an image grid. In some implementations, client device 120 may perform method 400. In another example, a client device or a server device may perform method 400. In some implementations, method 400 may be implemented by a server device.

[0098] Block 402 verifies whether user consent (e.g., user permission) has been obtained to utilize user data in implementing method 400. For example, user data may include images or videos taken by the user using a client device, images or videos stored or accessed by the user using, for example, a client device, image metadata / video metadata, user data related to the use of messaging applications, user preferences, user biometric information, user characteristics (e.g., identity, name, age, gender, occupation, etc.), information about the user's social network and contacts, social or other types of behavior and activities, user-created or written content, ratings, and opinions, user's current location, past user data, images generated, received, and / or accessed by the user, images viewed or shared by the user, etc. One or more blocks of the methods described herein may use such user data in some implementations.

[0099] If user consent has been obtained from an associated user whose user data may be used in method 400, then at block 404 it is determined that the blocks of the method herein may be performed with the user data available as described above, and the method continues to block 410. If user consent has not been obtained, then at block 406 it is determined that the blocks will be performed without utilizing user data, and the method continues to block 410. In some embodiments, if user consent has not been obtained, then the blocks are performed without utilizing user data, and utilizing synthetic data and / or common or commonly accepted and commonly available data. In some embodiments, if user consent has not been obtained, then method 400 is not performed.

[0100] In block 410, image data is captured, for example, using one or more cameras of a client device. In some implementations, the captured image data may include a main image and associated depth values. In some implementations, the captured image data includes one or more sub-images. In some implementations, the main image may be a color image. In some implementations, the one or more sub-images may include a color image (e.g., captured from a different perspective or at a different time than the main image), a grayscale image, a monochrome image, a depth image (e.g., an image that includes depth information but does not include a human-visible depiction of the scene), etc. In some implementations, the one or more sub-images may be captured by a different camera than the camera that captured the main image. In some implementations, each of the one or more sub-images may be captured by a corresponding one of the one or more cameras. Block 410 may be followed by block 412.

[0101] At block 412, the image data may be encoded into an image format, for example, as a JPEG image, a TIFF image, a HEIF image, a PNG image, etc. In some implementations, the encoded image data may include a main image encoded into an image format. The encoded image data may further include image metadata. In some implementations, the image metadata may be encoded as XMP metadata. In some implementations, the image metadata may be encoded into the image format, for example, as a Depth Photo image, ... PNG image, etc. , a device element including a profile element indicating an augmented reality (AR) photo, etc. In some implementations, the image metadata may further include a camera element including an image element and a depth map obtained based on the depth values.

[0102] In some implementations, the depth map may include a number of pixel values ​​each indicating a distance from a particular camera of one or more cameras used to capture the image data. In some implementations, the depth may further include a lens focus model. In some implementations, the lens focus model may define respective radius values ​​of a circle of confusion corresponding to a number of distances from the particular camera. In some implementations, the number of distances may include a near point distance (corresponding to a near view) and a far point distance (corresponding to a far view), where the near point distance is smaller than the far point distance (the near view is closer to the camera than the far view). In some implementations, the number of distances may further include focal lengths larger than the near point distance and smaller than the far point distance. In some implementations, the number of pixel values ​​may be obtained by converting the depth values ​​into an integer format (e.g., 16-bit integer) and compressing the converted depth values ​​based on an image format (e.g., JPEG).

[0103] In embodiments including one or more sub-images, the image metadata may further include one or more additional camera elements. Each additional camera element may include an image element corresponding to at least one of the one or more sub-images and including a pointer (e.g., a Uniform Resource Identifier) ​​to at least one image in the file container. In some embodiments, the one or more additional camera elements may be sequentially organized in the image metadata after the initial camera element. Block 412 may be followed by block 414.

[0104] At block 414, the encoded image data may be stored in a file container based on the image format. In some implementations, storing the image data in the file container may include concatenating the primary image and one or more secondary images encoded in the image format. The order of the primary image and the one or more secondary images in the file container may be the same as the order of the one or more additional camera elements in the image metadata. Block 414 may be followed by block 416.

[0105] In block 416, the main image may be displayed. FIG. 5 is a flow diagram illustrating another exemplary method 500 according to some embodiments. In some embodiments, the method 500 can be implemented, for example, on the server system 102 shown in FIG. 1. In some embodiments, some or all of the method 500 can be implemented on one or more client devices 120, 122, 124, or 126 shown in FIG. 1, one or more server devices, and / or both the server device(s) and the client device(s). In the described example, the implementation system includes one or more digital processors or processing circuits ("processors") and one or more storage devices. In some embodiments, different components of one or more servers and / or one or more clients can perform different blocks or other portions of the method 500. In some examples, a first device is described as performing blocks of the method 500. Some embodiments may have one or more blocks of the method 500 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0106] In various implementations, client device 120 may be a standalone camera, another device equipped with a camera (e.g., a smartphone, a tablet, a computer, a wearable device such as a smart watch, a headset, etc.), or any other client device capable of receiving images or video captured by another device. In some implementations, client device 120 may be a capture-only device, e.g., a camera with no screen. In some implementations, client device 120 may be a display-only device, e.g., a device with a screen capable of displaying images or video, but without camera functionality or other functionality for capturing images or video. In some implementations, client device 120 may have both capture and display capabilities.

[0107] In some implementations, client device 120 may include one camera for capturing images or videos. In some implementations, client device 120 may include multiple cameras (or lenses). For example, a smartphone or other device may include one or more front-facing cameras (on the same side as the device's screen) and / or one or more rear-facing cameras. In some implementations, the one or more front-facing or rear-facing cameras may work in tandem during capture, e.g., a first camera may capture depth information and a second camera may capture image pixels for an image or video. In some implementations, different cameras (e.g., telephoto lenses, wide-angle lenses, etc.) may be used for different types of image capture or video capture, e.g., with different zoom levels. In some implementations, client device 120 may be configured to capture 360-degree images or videos. In some implementations, a camera or lens may capture images using one image sensor (e.g., a CCD or CMOS sensor) or multiple sensors. In some implementations, other sensors, e.g., depth sensors, may be used in conjunction with one or more cameras during image capture.

[0108] In some implementations, client device 120 may combine raw image data captured by the image sensors of one or more of the cameras (or lenses) with other data obtained from other sensors (e.g., accelerometers, gyroscopes, position sensors, depth sensors, etc.) to form an image and store the image in a file container.

[0109] Client device 120 allows a user to capture images in different modes, such as, for example, a still image (or photo) mode for capturing a single frame, a burst or video mode for capturing multiple frames, a video mode for capturing a video including multiple frames, etc. In some implementations, method 500 may be performed at the time of image capture, when the camera captures the image, after capture is completed, or at a later time (e.g., when client device 120 is used by a user to view and / or edit the image).

[0110] Client device 120 may allow a user to view, for example, images or videos taken by client device 120 or related to the user in different user interfaces. For example, a one-up or slideshow mode may be provided that allows a user to view / edit one image or video at a time. In another example, a gallery mode may be provided that allows a user to view / edit multiple images simultaneously, for example as an image grid.

[0111] In some implementations, the client device 120 may perform the method 500. In another example, the client device or the server device may perform the method 500. In some implementations, the method 500 may be implemented by the server device.

[0112] Block 502 verifies whether user consent (e.g., user permission) has been obtained to utilize user data in implementing method 500. For example, user data may include images or videos taken by a user using a client device, e.g. This may include images or videos stored or accessed by the user using the client device, image / video metadata, user data related to usage of messaging applications, user preferences, user biometric information, user characteristics (e.g., identity, name, age, gender, occupation, etc.), information about the user's social network and contacts, social or other types of behavior and activity, user-created or written content, ratings, and opinions, the user's current location, past user data, images generated, received, and / or accessed by the user, images viewed or shared by the user, etc. One or more blocks of the methods described herein may use such user data in some embodiments.

[0113] If user consent has been obtained from an associated user whose user data may be used in method 500, then at block 504 it is determined that the blocks of the method herein may be performed with the user data available as described above, and the method continues to block 510. If user consent has not been obtained, then at block 506 it is determined that the blocks are to be performed without the user data, and the method continues to block 510. In some implementations, if user consent has not been obtained, then the blocks are performed without the user data, and with synthetic data and / or common or commonly accepted and commonly available data. In some implementations, if user consent has not been obtained, then method 500 is not performed. For example, if a user denies permission to access one or more images, then method 500 is not performed, or is stopped after performing block 506.

[0114] In block 510, image data is obtained from a file container, for example a file container that stores images in a particular image format, as described with reference to Figure 3. In different implementations, block 510 may be followed by block 512, block 522, block 542, or block 562.

[0115] In some implementations, block 510 may be followed by block 512, which may include, for example, displaying a primary image from the image data on a display screen of a client device. In some implementations, the primary image may be displayed in a user interface that allows a user to provide user input. Block 512 may be followed by block 514.

[0116] In block 514, a user input may be received indicating a target focal length. For example, the user interface may include a user interface element (e.g., a slider) that allows a user to select a target focal length by moving the slider between distance values ​​that correspond to near and far distance values ​​in the captured image. In some implementations, a user input may be received directly, such as, for example, a touch input, a gesture input, or a voice input, that identifies a particular portion of the primary image, and the focal length may be determined based on the particular portion. Block 514 may be followed by block 516.

[0117] In block 516, the primary image may be modified based on, for example, a lens focus model stored in the file container as part of the image metadata. For example, the primary image may be modified to apply, for example, a portrait effect that blurs objects in the image that are at a distance greater than a target focal length; a bokeh effect that blurs objects in the image that are at a distance different from the target focal length from the camera that captured the primary image; other selective blur effects; etc. Other types of effects based on the target distance may also be applied, for example, color effects, filters, etc.

[0118] In some implementations, the effects are stored in a depth map in the image metadata. The radius of the circle of confusion may be obtained by a simple search in the FocalTable. If the target focal length corresponds to a distance included in the FocalTable, the radius of the circle of confusion may be obtained by a simple search in the FocalTable. If the target focal length is not included in the FocalTable, the radius value corresponding to an available distance, e.g., the distance closest to the target focal length, may be obtained. For example, if the FocalTable includes distance and radius pairs for near and far views, these values ​​are obtained. The radius value for the target focal length is then determined based on an interpolation between the obtained values. For example, a linear interpolation may be performed to obtain the radius value for the target focal length. The radius value thus determined is used to apply an effect, e.g., to blur the main image. Block 516 may be followed by block 518.

[0119] The modified image is displayed at block 518. In some implementations, block 518 may be followed by block 514 to have the user provide further input indicating, for example, a different target focal length.

[0120] In some implementations, block 510 may be followed by block 522, which may include, for example, displaying a primary image from the image data on a display screen of a client device. In some implementations, the primary image may be displayed in a user interface that allows a user to provide user input. Block 522 may be followed by block 524.

[0121] At block 524, user input indicating a crop or resize operation may be received. For example, the user interface may include one or more user interface elements that allow a user to indicate a crop or resize operation. For example, a user may indicate a crop (removing one or more portions of the primary image) operation by drawing a shape (e.g., a crop rectangle) to select a subset of pixels of the primary image. In another example, a user may indicate a resize (increasing the size of the primary image) operation by dragging one or more corners or edges of the primary image. Other types of user interfaces that allow a user to input a crop or resize operation may be provided. Block 524 may be followed by block 526.

[0122] At block 526, the main image is modified (cropped and / or resized) based on the user input received at block 524. Block 526 may be followed by block 528.

[0123] At block 528, the depth map is updated based on the modified main image. For example, portions of the depth map that correspond to the cropped (removed) portions of the main image may be removed. In the case of a resize operation, the depth map may be updated based on the resize applied to the modified main image. Block 528 may be followed by block 530.

[0124] At block 530, the file container may be updated to store the modified main image and the updated depth map.

[0125] In some implementations, block 510 may be followed by block 542, which generates a three-dimensional image using, for example, computer vision techniques. The three-dimensional image is generated using image data obtained from a file container, which may include a main image, one or more sub-images, and depth information. For example, the depth information may include multiple depth maps. Each depth map may be generated from the motion of one camera when capturing an image and embedded in the image metadata. In another example, one or more depth cameras may be used to generate a three-dimensional image. A digital camera may be used to generate multiple depth maps. The multiple depth maps may be utilized to reconstruct the 3D scene. Block 542 may be followed by block 544.

[0126] In block 544, the three-dimensional image is displayed. Block 544 may be followed by block 546.

[0127] In block 546, user input indicative of a tilt or pan operation may be received. For example, the user input may be received via one or more sensors of the client device, such as a gyroscope, an accelerometer, a gesture sensor, a touch screen, or other sensors. Block 546 may be followed by block 548.

[0128] Block 548 may generate an updated three-dimensional image based on user input. The three-dimensional nature of the image allows objects to have a sense of depth and allows images to be displayed such that as objects move around they occlude other objects in the image. In some implementations, the effect is similar to looking into a 3D diorama. Block 548 may be followed by block 550.

[0129] The updated three-dimensional image may be displayed in block 550. In some implementations, block 550 may be followed by block 546 to receive further user input.

[0130] In some implementations, block 510 may be followed by block 562, which may include, for example, displaying a primary image from the image data on a display screen of a client device. In some implementations, the primary image may be displayed in a user interface that allows a user to provide user input. Block 522 may be followed by block 524.

[0131] At block 564, user input may be received indicating a selection of one or more objects included in the image. For example, a user may select the objects using a pointing device, a touch screen, a gesture interface, voice, etc. Block 564 may be followed by block 566.

[0132] At block 566, a segmentation mask is generated that corresponds to the object(s) selected by the user. The segmentation mask may identify a subset of pixels of the main image that correspond to the object. The segmentation mask may be generated using a depth map that includes a plurality of depth values. In some implementations, each depth value may correspond to a particular pixel of the main image. In these implementations, generating the segmentation mask may include selecting pixels of the main image that each have a depth value within a threshold range of depth values. For example, the threshold range of depth values ​​may be a range between a minimum depth value and a maximum depth value that corresponds to an object identified using an image segmentation technique that utilizes, for example, the depth information and information regarding a depth of field effect applied by a camera application used during capture of the image. An object image is obtained based on the segmentation mask. For example, the object may include only a subset of pixels of the main image identified by the segmentation mask. For example, with reference to FIG. 2B, a dark flower in the depth image 230 may be utilized to segment a flower in the image 220. Block 566 may be followed by block 568 .

[0133] At block 568, the user may select the object image to perform one or more operations, such as a cut operation to remove the object from the displayed image, a copy operation to copy the object, or a combination of operations. The present invention provides a user interface that allows a user to perform a copy operation to insert an object into a different image or a paste operation to insert an object into a different position in a displayed image.

[0134] FIG. 6 is a block diagram of an exemplary device 600. The device 600 may be used to implement one or more features described herein. In one example, the device 600 may be used to implement a client device, such as any of the client devices (120, 122, 124, 126) shown in FIG. 1. Alternatively, the device 600 may implement a server device, such as the server 104. In some implementations, the device 600 may be used to implement a client device, a server device, or both a client and a server device. The device 600 may be any suitable computer system, server, or other electronic device or hardware device described above.

[0135] One or more methods described herein may be a standalone program that can run on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a mobile phone, a smart phone, a tablet computer, a wearable device (e.g., a watch, an arm band, jewelry, a hat, virtual reality goggles or glasses, augmented reality goggles or glasses, a head mounted display, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within the mobile app (and / or other apps) on the mobile computing device. In another example, computations may be shared between the mobile computing device and one or more server devices.

[0136] In some implementations, device 600 includes a processor 602, a memory 604, an input / output (I / O) interface 606, and a camera 616. In some implementations, camera 616 may include multiple cameras or multiple image sensors. For example, if device 600 is a mobile phone, tablet, or computing device, camera 616 may include a front camera and a rear camera. In another example, camera 616 may include, for example, a camera that captures RGB images, a depth camera that captures depth information using, for example, infrared or other techniques, a grayscale camera, a monochrome camera, etc.

[0137] The processor 602 may be one or more processors and / or one or more processing circuits for executing program code and controlling basic operations of the device 600. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing a function, a specialized processor for performing neural network model-based processing, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some implementations, the processor 602 may include one or more co-processors for implementing neural network processing. In some implementations, the processor 602 may be a processor that processes data to generate a probability-based output, e.g., the output generated by the processor 602 may be inaccurate or accurate within a particular range from an expected output. The processing need not be limited to a particular geographic location or be time-bound. For example, the processor may perform its functions in "real-time", "offline", "batch mode", etc. Portions of the processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0138] Memory 604 is typically provided within device 600 for access by processor 602. Memory 604 may be any suitable processor-readable storage medium suitable for storing instructions executed by a processor, such as Random Access Memory (RAM), Read-Only Memory (ROM), Electrical Erasable Read-only Memory (EEPROM), flash memory, etc., and may be located separately from and / or integrated with processor 602. Memory 604 may store software operated on server device 600 by processor 602, including an operating system 608, machine learning applications 630, other applications 612, and application data 614.

[0139] The other applications 612 may include applications such as a camera application, an image gallery, an image management or image library application, a media sharing application, a data display engine, a web hosting engine, an image display engine, a notification engine, a social networking engine, etc. In some implementations, the machine learning application 630 and the other applications 612 may each include instructions that enable the processor 602 to perform some or all of the functions described herein, e.g., the methods of Figures 4 and 5. One or more methods disclosed herein can operate in a number of environments and platforms, e.g., as a standalone computer program that can operate on any type of computing device, as a web application having a web page, as a mobile application ("app") that operates on a mobile computing device, etc.

[0140] In various implementations, the machine learning application 630 may utilize a Bayesian classifier, a support vector machine, a neural network, or other learning techniques. In some implementations, the machine learning application 630 may include a trained model 634, an inference engine 636, and data 632.

[0141] In some implementations, data 632 may include training data, e.g., data used to generate trained model 634. For example, training data may include any type of data, such as text, images, audio, video, etc. Training data may be obtained from any source, e.g., a data repository specifically marked for training, data with permission to be used as training data for machine learning, etc. In implementations where one or more users have authorized the use of their user data to train a machine learning model, e.g., trained model 634, training data may include user data. In implementations where users have authorized the use of their user data, data 632 may include images / videos or image / video metadata (e.g., videos, data regarding sharing of videos with other users, labels associated with the videos, whether a video-based work was generated from the video, e.g., video collages, stories, etc.), communications (e.g., emails; chat data, e.g., text messages, audio, video, etc.), documents, and the like. It may also include authorization data (e.g., a spreadsheet, a text document, a presentation, etc.).

[0142] In some implementations, the training data may include synthetic data generated for training purposes, such as data that is not based on user input or activity in the context being trained, e.g., data generated from simulated or computer-generated video. In some implementations, the machine learning application 630 excludes the data 632. For example, in these implementations, the trained model 634 may be generated, for example, on a different device and provided as part of the machine learning application 630. In various implementations, the trained model 634 may be provided as a data file that includes a model structure or model form (e.g., specifying the number and type of neural network nodes, the connectivity between the nodes, and the organization of the nodes into multiple layers) and associated weights. The inference engine 636 may read the data file for the trained model 634 and implement a neural network with node connectivity, layers, and weights based on the model structure or model form specified in the trained model 634.

[0143] The machine learning application 630 may also include a trained model 634. In some implementations, the trained model may include one or more model forms or one or more model structures. For example, the model forms or structures may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence or frames in a video, and produces a sequence of results as output), etc.

[0144] The model topology or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, nodes in a first layer (e.g., an input layer) may receive data as input data 632 or application data 614. Such data may include, for example, one or more pixels per node, for example, when the trained model is used to analyze a video that includes multiple frames. Subsequent intermediate layers may receive as input the output of the nodes of the previous layer according to the connectivity specified in the model topology or structure. These layers are also referred to as hidden layers. The final layer (e.g., an output layer) generates the output of the machine learning application. In some implementations, the model topology or structure also specifies the number and / or type of nodes in each layer.

[0145] In different implementations, the trained model 634 may include one or more models. One or more of the models may include multiple nodes arranged in multiple layers according to a model structure or model morphology. In some implementations, the node may be, for example, a memoryless computational node configured to process a unit of input and generate a unit of output. The computation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output. In some implementations, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some implementations, the step / activation function may be a nonlinear function. In various implementations, such computation may include operations such as matrix multiplication. In some implementations, the computation may include, for example, a multi-core processor. It can use multiple processor cores in the system, or individual processors in the GPU or specialized neural networks. In some implementations, the nodes may include memory, e.g., to store one or more previous inputs for use in processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. The LSTM node may use memory to maintain a "state" that allows the node to operate like a Finite State Machine (FSM). A model with such nodes may be useful for processing sequential data, e.g., multiple words in a sentence or paragraph, multiple frames in a video, conversation or other audio.

[0146] In some implementations, the trained model 634 may include embeddings or weights for individual nodes. For example, the model may be initialized as a number of nodes organized into layers as specified by the model topology or model structure. At initialization, weights may be applied to pairs of nodes connected according to the model topology, e.g., connections between pairs of nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The model may then be trained, e.g., with data 632, to generate results.

[0147] For example, the training step may include applying a supervised learning technique. In supervised learning, the training data may include multiple inputs (e.g., a set of videos) and corresponding expected outputs for each input (e.g., one or more labels for each video). For example, weight values ​​are automatically adjusted based on comparing the model's outputs to expected outputs to increase the probability that the model will generate the expected outputs given similar inputs.

[0148] In some implementations, the training step may include applying an unsupervised learning technique, in which only the input data is given and a model may be trained to discriminate between the data, e.g., cluster the input data into groups, each group containing input data that is similar in some way.

[0149] In various implementations, the trained model includes a set of weights or a set of embeddings that correspond to the model structure. In implementations in which data 632 is excluded, the machine learning application 630 may include a trained model 634 that is based on prior training, e.g., by a developer of the machine learning application 630, a third party, etc. In some implementations, the trained model 634 may include a fixed set of weights, e.g., downloaded from a server that provides the weights.

[0150] The machine learning application 630 also includes an inference engine 636. The inference engine 636 is configured to apply the trained model 634 to data, such as the application data 614 (e.g., video), to provide inferences. In some implementations, the inference engine 636 may include software code executed by the processor 602. In some implementations, the inference engine 636 may specify circuitry (e.g., a programmable processor, a field programmable gate array (FPGA), etc.) that enables the processor 602 to apply the trained model. In some implementations, the inference engine 636 may include software instructions, hardware instructions, or a combination. In some implementations, the inference engine 636 may provide an application programming interface (API) that the operating system 608 and / or other applications 612 can use to call the inference engine 636, for example, to apply the trained model 634 to the application data 614 to generate inferences.

[0151] In some implementations, the machine learning application 630 may be implemented offline. In these implementations, the trained model 634 may be generated in a first stage and provided as part of the machine learning application 630. In some implementations, the machine learning application 630 may be implemented online. For example, in such implementations, an application that invokes the machine learning application 630 (e.g., the operating system 608, one or more of the other applications 612, etc.) may use the inferences generated by the machine learning application 630, e.g., provide the inferences to a user, and generate a system log (e.g., actions taken by the user based on the inferences, if permitted by the user; or results of further processing, if used as input for such further processing). The system log may be generated periodically, e.g., hourly, monthly, quarterly, etc., and may be used to update the trained model 634, e.g., update embeddings for the trained model 634, if permitted by the user.

[0152] In some implementations, the machine learning application 630 may be implemented to adapt to the particular configuration of the device 600 on which it executes. For example, the machine learning application 630 may determine a computation graph that utilizes available computational resources, e.g., the processor 602. For example, if the machine learning application 630 is implemented as a distributed application on multiple devices, the machine learning application 630 may determine that the computations are performed in a manner that optimizes the computations on each individual device. In another example, the machine learning application 630 may determine that the processor 602 includes a GPU with a particular number of GPU cores (e.g., 1000) and implement the inference engine accordingly (e.g., as 1000 individual processes or threads).

[0153] In some implementations, the machine learning application 630 may implement a set of trained models. For example, the trained models 634 may include multiple trained models, each applicable to the same input data. In these implementations, the machine learning application 630 may choose a particular trained model based on, for example, available computational resources, success rate with previous inferences, etc. In some implementations, the machine learning application 630 may execute an inference engine 636 such that multiple trained models are applied. In these implementations, the machine learning application 630 may combine the outputs from applying the individual models, for example, using a majority vote to score the outputs from applying each trained model, or by choosing one or more particular outputs. Furthermore, in these implementations, the machine learning application may apply a time threshold (e.g., 0.5 ms) to apply the individual trained models and only utilize the individual outputs that are available within the time threshold. Outputs not received within the time threshold may not be used, e.g., may be discarded. For example, such an approach may be appropriate when there is a time limit specified during invocation of the machine learning application, for example, by the operating system 608 or one or more applications 612. In some implementations, the machine learning application 630 may generate output based on a format specified by an invoking application, e.g., the operating system 608 or one or more applications 612. In some implementations, the invoking application may be another machine learning application. For example, such a configuration may be used in a generative adversarial network where the invoking machine learning application is trained with output from the machine learning application 630 or where the machine learning application 630 is trained with output from the invoking machine learning application.

[0154] The software in memory 604 may alternatively be stored on other suitable storage locations or computer-readable media. Additionally, memory 604 (and / or other connected storage device(s)) may store one or more messages, one or more taxonomies, an electronic encyclopedia, a dictionary, a thesaurus, a knowledge base, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. Memory 604 and other types of storage (such as magnetic disks, optical disks, magnetic tapes, or other tangible media) may be considered "storage" or "storage devices."

[0155] The I / O interface 606 can provide functionality that allows the server device 600 to interface with other systems and devices. The interfaced devices can be included as part of the device 600. Or, the interfaced devices can be separate and in communication with the device 600. For example, network communication devices, storage devices (e.g., memory), and input / output devices can communicate through the I / O interface 606. In some implementations, the I / O interface can connect to interface devices, such as input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0156] Some examples of interfaced devices that may be connected to the I / O interface 606 include one or more display devices 620 that may be used to display content, such as images, videos, and / or user interfaces of output applications described herein. The display devices 620 may be connected to the device 600 via a local connection (e.g., a display bus) and / or via a networked connection and may be any suitable display device. The display devices 620 may include any suitable display device, such as an LCD, LED, or plasma display screen, a CRT, television, monitor, touch screen, 3D display screen, or other display device. For example, the display devices 620 may be a flat display screen provided on a mobile device, multiple display screens included in a goggle or headset device, or a monitor screen of a computing device.

[0157] The I / O interface 606 can interface to other input and output devices. Some examples include one or more cameras capable of capturing images. Some implementations may also include a camera or other devices capable of capturing images (e.g., a portion of the captured image, voice commands, etc.). A microphone for recording sound, an audio speaker device for outputting sound, or other input and output devices may be provided.

[0158] The camera 616 may be any type of camera capable of capturing video including multiple frames. As used herein, a camera may include any imaging device. In some implementations, the camera 616 may include multiple lenses or multiple image sensors with different capabilities, such as, for example, front vs. rear lenses, different zoom levels, image resolutions of captured images, etc. In some implementations, the device 600 may include one or more sensors, such as a depth sensor, an accelerometer, a position sensor (e.g., GPS), a gyroscope, etc. In some implementations, one or more of these sensors may be operated in conjunction with the camera to obtain sensor measurements corresponding to different frames of a video captured with the camera.

[0159] For ease of illustration, FIG. 6 illustrates a processor 602, a memory 604, an I / O interface 606, and a 6 illustrates a block for each of the face 606, the camera 616, and the software blocks 608, 612, and 630. These blocks may represent one or more processors or processing circuits, one or more operating systems, one or more memories, one or more I / O interfaces, one or more applications, and / or one or more software modules. In other implementations, the device 600 may not have all of the components illustrated, and / or may have other elements, including other types of elements instead of or in addition to the elements illustrated herein. Although some components are described as performing the blocks and operations described in some implementations herein, any suitable component or combination of components of the network environment 100, the device 600, a similar system, or any suitable processor or processors associated with such a system may perform the described blocks and operations.

[0160] The methods described herein can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored on a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer disk, Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, and the like. Also, the program instructions can be included or provided as electrical signals, for example, in a Software as a Service (SaaS) format delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more of the methods can be implemented in hardware (e.g., logic gates) or a combination of hardware and software. Exemplary hardware may include programmable processors (e.g., Field-Programmable Gate Arrays (FPGAs), Complex Programmable Logic Devices (CPLDs)), general purpose processors, graphics processors, Application Specific Integrated Circuits (ASICs), etc. One or more of the methods may be implemented as a component or part of an application running on the system, or may be implemented as an application or software operating in conjunction with other applications and the operating system.

[0161] Although a description of specific implementations has been provided, these specific implementations are exemplary only and not limiting, and the illustrated concepts may be applied to other examples and implementations.

[0162] In situations where certain implementations described herein may collect or use personal information about a user (e.g., user data, information about the user's social network, the user's location and the time when the user is at that location, the user's biometric information, the user's activity and demographic information), the user is given one or more opportunities to regulate whether information may be collected, whether the personal information may be stored, whether the personal information may be used, and how information about the user is collected, stored, and used. That is, the systems and methods described herein specifically do so upon receiving explicit permission from the relevant user to collect, store, and / or use user personal information.

[0163] For example, a program or feature may use user information about a user or A particular user may regulate whether user information is collected about the user's system or other users related to that characteristic. Each user about whom personal information is collected may be presented with one or more options, such as options that allow the collection of information related to that user to be regulated, options that allow the information to be collected, and options that give permission or approval as to whether the information may be collected and what parts of the information are collected. For example, a user may be given one or more such regulation options over a communications network. In addition, certain data may be treated in one or more ways before being stored or used, such that personally identifiable information is removed. As one example, the identity of the user may be treated such that personally identifiable information cannot be determined. As another example, the geographic location of the user device may be generalized to a larger region such that the user's specific location cannot be determined.

[0164] However, as will be appreciated by those skilled in the art, the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks. Any suitable programming language and programming techniques may be used to implement the routines of a particular embodiment. Different programming techniques, such as procedural or object-oriented, may be used. The routines may be executed on one processing device or multiple processors. Although steps, operations, or computations are presented in a particular order, this order may be changed in different particular embodiments. In some embodiments, multiple steps or operations shown in a sequence in this specification may be performed simultaneously.< / radius> < / distance>

Claims

1. 1. A computer-implemented method comprising: obtaining image data from a file container, the image data including a plurality of pixel values ​​corresponding to a main image, the method further comprising: displaying the main image; receiving user input indicating at least one of a target focal length and a selection of an object to be captured in the primary image; If the user input indicates the target focal length, the method further comprises: obtaining image metadata from said file container, said image metadata including: i) a depth map corresponding to said primary image; and ii) a lens focus model defining circle of confusion radius values ​​corresponding to a number of distances from a camera that captured said primary image; and modifying one or more pixel values ​​of the main image based on the lens focus model to obtain the modified image such that an object at the target focal length is in focus in the modified image, wherein modifying the one or more pixel values ​​of the main image includes: i) obtaining a distance from the camera of at least one other object in the main image based on the depth map; and ii) blurring the at least one other object in the modified image based on a value of the radius of the circle of confusion at the distance, the method further comprising: displaying the modified image; If the user input indicates the selection of an object appearing in the primary image, the method further comprises: obtaining image metadata from the file container, the image metadata including the depth map; and generating a segmentation mask based on the depth map, the segmentation mask identifying a subset of pixels of the main image that correspond to the object, the method further comprising: obtaining an object image of the object based on the segmentation mask; and providing a user interface to enable selection of the object image.

2. 2. The computer-implemented method of claim 1, further comprising managing one or more file containers by using an image management application, the image management application providing management functionality for sharing the one or more file containers among multiple users.

3. 2. The computer-implemented method of claim 1, wherein the lens focus model is stored in a depth map element in image metadata of the file container, the depth map element corresponding to a camera that captured the primary image.

4. 2. The computer-implemented method of claim 1, wherein the depth map includes a plurality of depth values ​​each corresponding to a particular pixel of the main image, and generating the segmentation mask includes selecting pixels of the main image having depth values ​​within a threshold range of depth values.

5. A computing device comprising means for carrying out the method according to any one of claims 1 to 4.

6. A computer system comprising means for carrying out the method according to any one of claims 1 to 4.

7. A computer program comprising instructions which, when executed by one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 4.

8. one or more processors; A computing device comprising: a memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 4.

Images