Generating non-destructive synthetic lens blur with edge rendering in focus

Abstract

Embodiments of the present disclosure relate to generating non-destructive synthetic lens blur with edge rendering in focus. Methods, systems, and non-transitory computer-readable storage media are disclosed for generating lens blur effects in a digital image with in-focus edge rendering. The disclosed system generates a focus mask indicating a focus range of depth values of a digital image based on a focus region of the digital image and a depth map. The disclosed system generates a hierarchical depth map from a depth map and a focus mask, the hierarchical depth map including foreground depth values and background depth values for pixels on a digital image. The disclosed system also renders a digital image to include lens blur effects by determining a combination of foreground depth values and background depth values using the focus mask and the hierarchical depth map in conjunction with a splash operation.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 585,369, filed on September 26, 2023, which is incorporated herein by reference in its entirety. Background Art

[0003] Improvements in computer processing techniques have led to significant advances in the field of image processing. In particular, many industries have implemented various image editing tools and processes for generating and manipulating digital images for various purposes. In addition, computer-aided image editing tools have increased the availability of digital image editing applications for more users with different levels of expertise. For example, many mobile devices (e.g., smart phones with digital cameras) provide users with tools for editing digital images to imitate the effects usually achieved with optical zoom lenses, such as zoom or focus effects. Due to the diversity of objects and scenes captured in digital images, the difficulty of identifying the boundaries between objects in digital images, and the diversity of the depth of object placement in digital images, modifying digital images to accurately represent lens blur effects is a challenging task. Conventional systems have many disadvantages in accurately representing effects based on synthetic depth or based on blur in digital images. Summary of the invention

[0004] One or more embodiments described herein provide benefits and / or solve one or more problems in the art by implementing artificial intelligence models to facilitate flexible and efficient scene-based image editing systems, methods, and non-transitory computer-readable media. To illustrate, in one or more embodiments, the system utilizes one or more machine learning models to learn / identify the characteristics of digital images, predict potential edits to digital images, and / or generate supplementary components that can be used for various edits. Therefore, the system obtains an understanding of the two-dimensional image as if it were a real scene, with different semantic regions reflecting the real world (e.g., three-dimensional) conditions. In addition, the system enables the two-dimensional image to be edited so that these changes automatically and consistently reflect the corresponding real-world conditions without relying on additional user input. The system also provides realistic editing of two-dimensional objects in the two-dimensional image based on the three-dimensional characteristics of the two-dimensional scene, such as by generating one or more three-dimensional grids based on the two-dimensional image. Therefore, the system promotes flexible and intuitive editing of digital images while effectively reducing the user interaction usually required for such editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0006] Figure 1An example environment is shown in which a lens blur system operates in accordance with one or more implementations.

[0007] Figure 2 An overview diagram of a lens blur system for applying a lens blur effect to a digital image is shown in accordance with one or more implementations.

[0008] Figure 3 A diagram illustrating a process for generating a focal matte for a digital image in accordance with one or more implementations.

[0009] Figure 4 A diagram illustrating a process for generating a layered depth map for a digital image in accordance with one or more implementations.

[0010] Figure 5 A diagram illustrating a process for rendering a lens blur effect for a digital image in accordance with one or more implementations.

[0011] Figure 6 A diagram illustrating a process for correcting lens blur effects of a digital image in response to user interaction in accordance with one or more implementations.

[0012] Figure 7 A diagram illustrating a process for modifying depth values ​​of a digital image in response to user interaction in accordance with one or more implementations.

[0013] Figure 8 A diagram illustrating a process for modifying depth values ​​of a digital image via a focus mode in accordance with one or more implementations.

[0014] Fig. 9 A diagram illustrating a process for modifying depth values ​​of a digital image via a blur pattern in accordance with one or more implementations.

[0015] Fig.10 A diagram illustrating a process for interpolating depth values ​​used to render a digital image based on a selected blur amount in accordance with one or more implementations.

[0016] Fig.11 A diagram illustrating a process for boosting the brightness of light sources in a digital image in accordance with one or more implementations.

[0017] Fig.12 A diagram illustrating a process for generating a highlight guide image for a digital image according to one or more implementations.

[0018] Fig.13 A diagram illustrating a process for learning parameters of a machine learning model for generating a highlight guidance image in accordance with one or more implementations.

[0019] Fig.14A diagram illustrating a process for modifying a lens blur effect in a digital image by boosting brightness values ​​of light sources in accordance with one or more implementations.

[0020] Fig.15 A graphical user interface for modifying a digital image via one or more lens blur settings is shown in accordance with one or more implementations.

[0021] Fig.16 A graphical user interface for displaying a depth map for a digital image is shown in accordance with one or more implementations.

[0022] Fig.17 A graphical user interface for applying a lens blur effect to a digital image is shown in accordance with one or more implementations.

[0023] Fig.18 A graphical user interface for selecting a focus area of ​​a digital image is shown in accordance with one or more implementations.

[0024] Fig.19 A graphical user interface for modifying a brightness boost value of a light source in a digital image is shown in accordance with one or more implementations.

[0025] Fig. 20 A graphical user interface for correcting depth values ​​in a depth map of a digital image is shown in accordance with one or more implementations.

[0026] Fig.21 A graphical user interface for displaying a modified depth map for a digital image is shown in accordance with one or more implementations.

[0027] Fig. 22 A graphical user interface for displaying a modified depth map for a digital image is shown in accordance with one or more implementations.

[0028] Fig.23 A graphical user interface for displaying a modified depth map and a modified focus area of ​​a digital image is shown in accordance with one or more implementations.

[0029] Fig.24 An example schematic diagram of a lens blur system is shown in accordance with one or more implementations.

[0030] Fig.25 A flow chart illustrating a series of acts for generating a lens blur effect via in-focus edge rendering of a digital image in accordance with one or more implementations.

[0031] Fig.26 A flow chart illustrating a series of acts for interactively correcting depth values ​​in a depth map in connection with generating a lens blur effect for a digital image in accordance with one or more implementations.

[0032] Fig. 27 A flow chart illustrating a series of acts for generating and utilizing a highlight guide image to boost brightness values ​​of light sources in connection with generating a lens blur effect for a digital image in accordance with one or more implementations.

[0033] Fig.28 A block diagram of an example computing device is shown in accordance with one or more implementations. DETAILED DESCRIPTION

[0034] One or more embodiments of the present disclosure include a lens blur system that utilizes a non-destructive lens blur pipeline to generate a lens blur effect based on synthetic depth. In particular, the lens blur system utilizes a non-destructive lens blur pipeline to perform high-quality in-focus edge rendering, interactive depth map refinement, and / or interactive adjustment of the brightness of a light source associated with a depth-based blur effect. For example, the lens blur system utilizes one or more estimated depth maps generated from a single digital image to apply different amounts of blur to different areas of the digital image based on the estimated depth value. In addition, the lens blur system provides tools within a graphical user interface for interactively indicating that portions of a digital image are to be blurred or deblurred (e.g., in a non-destructive blur pipeline) and / or increasing the brightness of a light source. Specifically, the non-destructive processing pipeline relates to a lens blur system that applies an editable lens blur effect to a digital image (e.g., by inserting a blur effect into a digital image or modifying a lens blur effect previously applied by a lens blur system).

[0035] As mentioned, in one or more embodiments, a lens blur system generates a lens blur effect for a digital image via a non-destructive lens blur pipeline. Specifically, the lens blur system utilizes a depth map of the digital image to generate a focus mask that represents a focus range of depth values ​​corresponding to a selected focus area of ​​the digital image. For example, the lens blur system provides one or more tools for selecting a specific depth value range that corresponds to a focus point relative to the depth map. The lens blur system also generates a focus mask by identifying edges corresponding to discontinuities in the depth map to refine the focus mask based on the selected focus area.

[0036] In addition, in one or more embodiments, the lens blur system generates a layered depth map for the digital image. In particular, the lens blur system generates a set of foreground depth values ​​and a set of background depth values ​​for pixels on the digital image. For example, the lens blur system phenomenates foreground depth values ​​for portions of the digital image outside of an area masked by the focus matte. The lens blur system also phenomenates background depth values ​​within the area masked by the focus matte. The lens blur system combines the foreground depth values ​​and the background depth values ​​into a single depth map (e.g., a layered depth map).

[0037] In some embodiments, the lens blur system uses the layered depth map and the focus matte to render the digital image with the lens blur effect. For example, the lens blur system uses the focus matte to determine a blended depth value by interpolating the foreground depth value and the background depth value in the layered depth map. In addition, the lens blur system uses the blended depth value to determine parameters for a splatter operation to render the digital image with the lens blur effect.

[0038] In at least some embodiments, the lens blur system provides tools for customizing or correcting various depth values ​​in the lens blur effect of a digital image. Specifically, the lens blur system uses a fused depth map generated from a layered depth map and a focus matte of the digital image to modify the depth values ​​of one or more selected portions. For example, the lens blur system determines the one or more selected portions in response to a user interaction with the selected portion(s) (e.g., with a stroke or brush stroke or other selection operation). The lens blur system also determines a selected correction mode for modifying the depth values ​​of the selected portion(s).

[0039] To illustrate, in conjunction with detecting a focus mode to deblur one or more selected portions of a digital image in a non-destructive processing pipeline, a lens blur system determines a focus region of the digital image. The lens blur system modifies a depth value of the digital image (e.g., in a layered depth map and / or a fused depth map) by moving the selected portion toward the foreground or toward the background to deblur the selected portion based on the focus region. Furthermore, in conjunction with detecting a blur mode to blur one or more selected portions of the digital image, the lens blur system uses the focus region of the digital image to determine whether to adjust the depth value of the portion toward the foreground or toward the background to blur the selected portion based on the focus region. Thus, the lens blur system provides a tool for correcting depth maps of a digital image via a general selection of a portion and based on information of surrounding portions of the digital image.

[0040] In an additional embodiment, the lens blur system provides tools for adjusting the light sources of a digital image in connection with a lens blur effect. In particular, the lens blur system generates a highlight guide image that indicates the light sources in the digital image. For example, the lens blur system generates the highlight guide image based on detecting the light sources in a brightness map of the digital image (e.g., via edges) via generating various masks. Alternatively, the lens blur system generates the highlight guide image using a trained machine learning model based on a ground-truth highlight guide image of the digital image. The lens blur system stores the highlight guide image with the digital image for current or later access in connection with applying the lens blur effect in one or more digital image applications.

[0041] In one or more embodiments, during rendering of a digital image, a lens blur system utilizes a highlight guide image to boost the brightness of pixels corresponding to detected light sources. Specifically, the lens blur system extracts brightness values ​​of pixels in the digital image and determines a brightness boost value based on the highlight guide image, the blur value of the pixel, and / or the highlight value of the pixel from the highlight guide image. In addition, the lens blur system provides tools to modify the brightness boost value based on a selected boost amount. Thus, the lens blur system provides tools to apply realistic blur effects to light sources in digital images based on information extracted from the digital image and user interaction with various tools.

[0042] Some conventional systems that provide synthetic lens blur effects for digital images lack the accuracy and flexibility of lens blur systems. In particular, some conventional systems utilize an estimated depth map of a digital image to generate a lens blur effect by blurring pixels according to their corresponding depths (e.g., via a Gaussian blur operation). Although such conventional systems provide an approximate blur effect in a digital image by utilizing an estimated depth map, the resulting blur effect often does not accurately simulate realistic lens blur. For example, conventional systems often produce inaccurate blur, blooming, or image artifacts at edges and depth transitions where there are discontinuities between foreground and background elements. The inaccurate results are often due to inaccuracies in the estimated depth map and the difficulty that many image processing operations have in accurately detecting depth at transition regions.

[0043] In addition, conventional systems lack flexibility in customizing the lens blur effect in the digital image. Specifically, by relying on per-pixel blur based on an estimated depth map of the digital image, the conventional system produces a generally consistent blur of out-of-focus elements of the digital image. However, the conventional system lacks certain capabilities for refining or customizing the lens blur effect. Thus, the conventional system does not have the ability to apply certain additional modifications or otherwise customize / correct the blur effect in the digital image based on contextual foreground depth information and background depth information of the entire digital image.

[0044] The lens blur system provides a number of advantages over conventional systems associated with generating lens blur effects in digital images. For example, the lens blur system improves the accuracy of a computational system that generates lens blur effects in digital images. Compared to conventional systems that apply per-pixel blur operations to digital images using only an estimated depth map, the lens blur system utilizes a multi-layered depth map with a selected focus depth range to refine the depth-based blur effect. In particular, the lens blur system provides focus edge rendering in a non-destructive lens blur pipeline by generating a focus matte based on a selectable range of focus depth values ​​with fused foreground / background depth values ​​to improve splatter operations during rendering.

[0045] In some embodiments, the lens blur system also improves the accuracy of the lens blur effect by improving the accuracy of light source blur in depth-based blur operations of digital images. Specifically, the lens blur system utilizes brightness edge detection to identify light sources (e.g., point light sources) in the digital image, and generates a highlight guide image to be stored with the digital image. The lens blur system utilizes the highlight guide image and user input via a graphical user interface tool to provide accurate, consistent rendering of non-destructive blur edits applied at various resolutions and / or across multiple digital images. More specifically, the lens blur system utilizes layered depth information with customizable parameters via a non-destructive processing pipeline to provide realistic blur and visibility of light sources whose light is scattered via the lens blur effect to enhance pixels corresponding to the light sources in the digital image.

[0046] In addition, in one or more embodiments, the lens blur system provides improved flexibility in applying synthetic lens blur effects to digital images. In particular, compared to conventional systems that only utilize an estimated depth map of a digital image to apply a per-pixel Gaussian blur effect, the lens blur system utilizes multiple depth maps with user input via various blur customization tools to provide a variety of different lens blur effects. For example, the lens blur system provides customization of in-depth range values ​​for indicating a specific focus range of a digital image. In addition, the lens blur system provides user customization of focus areas to correct errors in lens blur effects caused by inaccurate depth map estimates. The lens blur system also provides customization of light source blur via various light source enhancement tools to accurately represent lens blur effects on light sources detected within a digital image.

[0047] Turning now to the accompanying drawings, Figure 1 An embodiment of a system environment 100 is included in which a lens blur system 102 is implemented. In particular, the system environment 100 includes a server device 104 and a client device 106 that communicate via a network 108. Furthermore, as shown, the server device(s) 104 include a digital imaging system 110 that includes the lens blur system 102. Furthermore, the client device 106 includes a digital imaging application 112 that optionally includes the digital imaging system 110 and the lens blur system 102.

[0048] like Figure 1As shown, the client device 106 or (multiple) server devices 104 include or host a digital image system 110. The digital image system 110 includes one or more systems that implement digital image editing operations, or is part of one or more systems that implement digital image editing operations. For example, the digital image system 110 provides tools for performing various operations on digital images. For illustration, the digital image system 110 communicates with the client device 106 via the network 108 to provide tools for displaying and interacting via the digital image application 112 at the client device 106. In addition, in some embodiments, the digital image system 110 receives a request to access a stored digital image (e.g., at (multiple) server devices 104 or at another device such as a digital content database) and / or a request to store a digital image. In some embodiments, the digital image system 110 receives interactive data for viewing, generating, or editing a digital image or data associated with the digital image, processes the interactive data (e.g., to view, generate, or edit a digital image), and provides the result of the interactive data for display via the digital image application 112 or providing to a third-party system.

[0049] In one or more embodiments, the digital image comprises a digital raster image having image content representing one or more objects. For example, the digital image comprises one or more foreground objects and / or background objects within a scene. Furthermore, in some embodiments, the digital image comprises a captured image of a real-world scene. Alternatively, the digital image comprises a fully or partially synthetic (e.g., computer) generated image of a fully or partially synthetic scene. In some embodiments, the digital image comprises a modified version of another digital image (e.g., in response to one or more digital image editing operations).

[0050] According to one or more embodiments, the digital image system 110 utilizes the lens blur system 102 to generate a synthetic lens blur effect for a digital image in a non-destructive processing pipeline (e.g., by applying the blur effect during rendering). In particular, the digital image system 110 utilizes the lens blur system 102 to determine and utilize multi-layered depth values ​​of pixels of the digital image and indicative portions of the digital image to produce a realistic lens blur effect in an out-of-focus area. For example, as shown in more detail below, the digital image system 110 utilizes the lens blur system 102 to provide accurate lens blur for depth discontinuities in the digital image. In addition, the digital image system 110 utilizes the lens blur system 102 to correct or refine depth values ​​associated with applying the lens blur effect to the digital image. In addition, the digital image system 110 utilizes the lens blur system 102 to provide light source blur (e.g., via a brightness boost) in the digital image via the generation and use of a highlight guide image. Therefore, the digital image system 110 utilizes the lens blur system 102 to synthetically generate a realistic lens blur effect in a digital image with the aid of a tool that allows for increased control over the focus range and other customizable parameters of the depth value.

[0051] like Figure 1 As shown, the lens blur system 102 can be implemented on the client device 106 or on the server device(s) 104. In particular, in some implementations, the lens blur system 102 on the server device(s) 104 supports the lens blur system 102 on the client device 106. For example, the server device 104 generates or obtains the lens blur system 102 for the client device 106 (e.g., as part of a software application or suite). The server device(s) 104 provide the lens blur system 102 to the client device 106 for performing digital image editing at the client device 106. In other words, the client device 106 obtains (e.g., downloads) the lens blur system 102 from the server device(s) 104. At this point, the client device 106 is able to utilize the lens blur system 102 to edit digital images independently of the server device(s) 104.

[0052] In additional embodiments, although Figure 1 The server device(s) 104 and the client device(s) 106 are shown communicating via the network 108, but the various components of the system environment 100 communicate and / or interact via other methods (e.g., the server device(s) 104 communicate directly with the client device(s) 106). Figure 1The lens blur system 102 is shown as being implemented by specific components and / or devices within the system environment 100, but the lens blur system 102 is implemented in whole or in part by other computing devices and / or components in the system environment 100. For example, in some embodiments, server device(s) 104 include or host the digital image system 110 and / or the lens blur system 102.

[0053] For illustration, the lens blur system 102 includes a web hosting application (e.g., in a software as a service implementation) that allows a client device 106 to interact with content and services hosted on a server device 104. For illustration, in one or more implementations, the client device 106 accesses a web page supported by the server device(s) 104. The client device 106 provides input to the server device(s) 104 to perform digital image processing operations (e.g., differential captioning), and in response, the lens blur system 102 or the digital image system 110 on the server device(s) 104 performs operations to generate a lens blur effect for a digital image. The server device(s) 104 provide output or results of the operations to the client device 106.

[0054] In one or more embodiments, the server device(s) 104 include various computing devices, including the following references: Fig.28 Those computing devices described. For example, (multiple) server device 104 includes one or more servers for storing and processing data associated with editing digital images. In some embodiments, server device 104 also includes multiple computing devices that communicate with each other, such as in a distributed storage environment. In some embodiments, (multiple) server device 104 includes a content server. (Multiple) server device 104 also optionally includes an application server, a communication server, a network hosting server, a social network server, a digital content activity server, or a digital communication management server.

[0055] In addition, if Figure 1 As shown, the system environment 100 includes a client device 106. In one or more embodiments, the client device 106 includes, but is not limited to, a mobile device (e.g., a smart phone or a tablet computer), a laptop computer, a desktop computer, including the following reference Fig.28 Furthermore, although Figure 1Although not shown, client device 106 may be operated by a user (e.g., a user included in or associated with system environment 100) to perform various functions. In particular, client device 106 performs functions such as, but not limited to, accessing, viewing, and interacting with digital images. In some embodiments, client device 106 also performs functions for generating, capturing, or accessing data to provide to digital image system 110 and lens blur system 102 associated with digital image editing. For example, client device 106 communicates with server device(s) 104 via network 108 to provide information associated with digital content (e.g., user interactions). Although Figure 1 The system environment 100 is shown with a single client device, but in some embodiments, the system environment 100 includes a different number of client devices.

[0056] In addition, if Figure 1 As shown, the system environment 100 includes a network 108. The network 108 enables communication between components of the system environment 100. In one or more embodiments, the network 108 may include the Internet or the World Wide Web. In addition, the network 108 optionally includes various types of networks using various communication technologies and protocols, such as a company intranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a cellular network, a wide area network (WAN), a metropolitan area network (MAN), or a combination of two or more such networks. In practice, (multiple) server devices 104 and client devices 106 communicate via the network using one or more communication platforms and technologies suitable for transmitting data and / or communication signals, including any known communication technologies, devices, media, and protocols that support data communication, examples of which are referenced herein. Fig.28 Give a description.

[0057] As mentioned, the lens blur system 102 provides tools for generating realistic lens blur effects in digital images. Specifically, the lens blur system 102 utilizes estimated depth values ​​of pixels in the digital image and one or more selected regions to apply various lens blurs to the digital image. Figure 2 An overview of a lens blur system 102 is shown for applying a lens blur effect to a digital image using a depth map and a focus matte of the digital image.

[0058] In one or more embodiments, Figure 2 As shown, the lens blur system 102 determines a digital image 200 to be modified via one or more depth-based blur operations. In particular, as mentioned, the digital image 200 includes a photograph or other image including various objects arranged in a scene. For example, the digital image 200 includes one or more foreground objects and one or more background objects.

[0059] In addition, if Figure 2 As shown, the lens blur system 102 determines the depth map 202 of the digital image 200 by estimating the depth map 202 including depth values ​​corresponding to the pixels of the digital image. For example, the lens blur system 102 utilizes a depth estimation machine learning model to generate the depth map 202 of the digital image 200. Alternatively, the lens blur system 102 utilizes data from a camera that captured the digital image 200 to generate (or otherwise obtain) the depth map 202 of the digital image 200. The depth map 202 includes a depth value for each pixel in the digital image 200, so that the lens blur system 102 determines the approximate depth of each object (and each part of an object) relative to the viewpoint (e.g., camera view) of the digital image 200.

[0060] In one or more embodiments, the lens blur system 102 also utilizes the depth map 202 to determine a focus mask 204. Specifically, the lens blur system 102 determines the focus mask 204 to indicate specific portions of the digital image 200 that are in focus. Figure 3 Described in more detail, the lens blur system 102 generates a focus matte 204 using a depth map 202 having one or more selected portions (e.g., a selected range of in-focus depth values) of the digital image 200. To illustrate, the lens blur system 102 generates the focus matte 204 in a matte operation (e.g., via a matte model) that combines a customized focus range of depth values ​​for content in the digital image 200 to refine edges and capture details of objects with soft or irregular boundaries (e.g., hair).

[0061] In at least some embodiments, the lens blur system 102 utilizes the depth map 202 and / or the focus mask 204 to generate a lens blur effect 206 for the digital image 200. In particular, the lens blur system 102 utilizes one or more lens blur operations to generate a depth-based blur effect on the digital image 200. For example, as described below with respect to Figure 5 , Fig.10 , Fig.14 ,and Figures 17 to 20 As described in more detail, the lens blur system 102 applies the lens blur effect 206 by blurring or de-blurring one or more portions of the digital image 200 and / or boosting highlights caused by one or more light switches in the digital image 200. As also described in more detail below, the lens blur system 102 utilizes one or more additional masks, depth maps, or guide images to apply the lens blur effect 206 to the digital image 200. Thus, the lens blur system 102 generates a modified digital image 208 by rendering the digital image 200 with the lens blur effect 206.

[0062] As mentioned, the lens blur system 102 modifies the digital image to include a lens blur effect via a non-destructive processing pipeline. In particular, the lens blur system 102 utilizes depth values ​​of pixels in the digital image of the in-focus range having depth values ​​to determine parameters of a splatter operation for rendering the digital image with the lens blur effect. As further mentioned, Figure 3 A process of generating a focus matte of a digital image to represent a focus range of depth values ​​of the digital image is shown.

[0063] In one or more embodiments, Figure 3 As shown, the lens blur system 102 determines a digital image 300 to be rendered with a lens blur effect. For example, the lens blur system 102 receives a request to apply a lens blur effect to the digital image 300, such as via a digital image application for viewing and editing digital images. In response to the request, the lens blur system 102 obtains a depth map 302 for the digital image 300. As mentioned, the lens blur system 102 determines the depth map 302 including depth values ​​of pixels in the digital image 300 relative to a viewpoint (e.g., a camera viewpoint) of the digital image 300.

[0064] For example, in some embodiments, the lens blur system 102 utilizes a monocular depth estimation model to generate the depth map 302. For illustration, the depth estimation model includes a single image depth estimation model (SIDE) having a convolutional neural network architecture. In additional embodiments, the lens blur system 102 utilizes a transformer model and / or utilizes a self-attention layer to generate the depth map 302. For illustration, the lens blur system 102 utilizes a depth estimation model as described in U.S. patent application Ser. No. 17 / 186,436, filed on Feb. 26, 2021, “Generating Depth Images Utilizing A Machine-Learning Model BuiltFrom Mixed Digital Image Sources And Multiple Loss Function Sets,” which is incorporated herein by reference in its entirety. Thus, the lens blur system 102 utilizes a depth estimation machine learning model to process the digital image 300 and generate the depth map 302.

[0065] In one or more embodiments, the lens blur system 102 also determines a focus region 304 that indicates that one or more portions of the digital image 300 are "in focus." Specifically, the focus region includes a specific depth (or depth range) corresponding to one or more objects in the digital image that are to be focused with a synthetic lens blur effect. For example, the lens blur system 102 determines the focus region 304 in response to a user interaction with one or more portions of the digital image 300. Alternatively, the lens blur system 102 determines the focus region 304 in conjunction with an object detection operation and / or a default focus region (e.g., corresponding to a nearest depth value or nearest detected foreground object).

[0066] In response to determining the depth map 302 and the focus area 304, the lens blur system 102 determines a focus mask 306 for the digital image 300. For example, the lens blur system 102 generates a binary mask that indicates one or more foreground areas and one or more background areas. To illustrate, the lens blur system 102 generates the focus mask 306 by thresholding the depth values ​​in the depth map 302 according to the focus area 304. In particular, the lens blur system 102 thresholds the in-focus depth to a first value (e.g., 1) and the out-of-focus depth to a second value (e.g., 0). Therefore, the focus mask 306 includes initial estimates of the foreground and background portions of the digital image 300 based on the estimated depth values ​​in the depth map 302.

[0067] In one or more embodiments, the lens blur system 102 utilizes one or more additional processes to modify the initial estimates of foreground and background portions in the focus mask 306 by refining details of edges in the depth map 302. For example, the lens blur system 102 utilizes a machine learning model that is trained to refine the edges of the focus mask 306 to better match content in the digital image 300 (e.g., by capturing details of hair or other irregular boundaries). To illustrate, the lens blur system 102 generates a ternary map segmentation 308 from the depth map 302 (and, in some cases, from the focus mask 306 as an additional input to the depth map 302) using an edge detection operation. Specifically, the ternary map segmentation 308 includes information indicating discontinuities in the depth map 302 via unknown regions that correspond to pixels that may include depth values ​​in the foreground or background.

[0068] In one or more embodiments, the lens blur system 102 generates the ternary map segmentation 308 by flooding portions of the focus mask 306 (e.g., extending the focus values) until an edge or extended focus range is encountered in the depth map 302. In some embodiments, the lens blur system 102 further generates the ternary map segmentation 308 by eroding and dilating the resulting flooded portions of the focus mask 306 to create unknown regions. The lens blur system 102 combines the unknown regions with the flooded portions of the focus mask 306 to create the ternary map segmentation 308, which thus includes a foreground region, a background region, and an unknown region. In some embodiments, the lens blur system utilizes a ternary map segmentation model to generate a ternary map segmentation, as described in U.S. Patent No. 11,393,100, filed on August 7, 2020, “AUTOMATICALLY GENERATING A TRIMAP SEGMENTATION FOR A DIGITALIMAGE BY UTILIZING A TRIMAP GENERATION NEURAL NETWORK,” which is incorporated herein in its entirety.

[0069] In response to generating the ternary map segmentation 308, the lens blur system 102 generates a focus matte 310 for the digital image 300. In particular, the lens blur system 102 blends (or otherwise combines) the ternary map segmentation 308 with the focus mask 306 to generate the focus matte 310. Thus, the focus matte 310 includes masked regions that indicate in-focus regions that correspond to the focus regions 304 where the alpha values ​​correspond to the blended boundary regions in the depth map 302. Furthermore, as described above, by flooding the focus regions 304 during the generation of the ternary map, the lens blur system 102 utilizes a matte model that is applied to in-focus pixels as well as slightly out-of-focus pixels.

[0070] In one or more embodiments, the lens blur system 102 utilizes the depth information and a focus matte corresponding to a selected focus area of ​​the digital image to generate a layered depth map that includes background and foreground depth information across the entire digital image. Figure 4 As shown, the lens blur system 102 utilizes machine learning to utilize depth values ​​of the digital image and the selected focused portion of the digital image to generate foreground and background information for various portions of the digital image. As described in more detail below, the lens blur system 102 utilizes a mask-guided machine learning model to determine the foreground and background information.

[0071] like Figure 4As shown, the lens blur system 102 provides a digital image 400, a depth map 402 of the digital image 400, and a focus mask 404 of the digital image 400 to a refinement machine learning model 406. In one or more embodiments, the lens blur system 102 utilizes a machine learning model, as described in U.S. Patent Application No. 17 / 658,873, filed on April 12, 2022, "UTILIZING MACHINE-LEARNING MODELS TO GENERATE REFINED DEPTH MAPS WITH SEGMENTATION MASK GUIDANCE," which is incorporated herein by reference in its entirety. Specifically, as mentioned, the lens blur system 102 uses foreground information for out-of-focus portions and background information for in-focus portions to pseudo-image on the digital image.

[0072] To illustrate, the lens blur system 102 utilizes the refinement machine learning model 406 to determine a background depth map 408 and a foreground depth map 410 based on the digital image 400, the depth map 402, and the focus matte 404. In particular, the lens blur system 102 utilizes the focus matte 404 as a guiding mask for the refinement machine learning model 406. For example, the lens blur system 102 utilizes the refinement machine learning model 406 to phenomenologically visualize background depth values ​​within areas masked by the focus matte 404, corresponding to in-focus areas. The lens blur system 102 also utilizes the refinement machine learning model 406 to phenomenologically visualize foreground depth values ​​outside of areas masked by the focus matte 404, corresponding to out-of-focus areas.

[0073] Thus, the lens blur system 102 generates a background depth map 408 to include background depth values ​​for pixels on the digital image 400, including areas that are not visible in the digital image 400 (e.g., behind one or more foreground objects). In addition, the lens blur system 102 generates a foreground depth map 410 to include foreground depth values ​​for pixels on the digital image 400, including areas that visually correspond to background scenery. The lens blur system 102 generates a layered depth map 412 that includes the background depth map 408 and the foreground depth map 410. For illustration, the layered depth map 412 includes a data structure (e.g., a vector or a matrix) that includes background depth values ​​and foreground depth values ​​for pixels in the digital image 400. Alternatively, the layered depth map 412 includes an index mapping value from the background depth map 408 and the foreground depth map 410 to each pixel.

[0074] In one or more embodiments, lens blur system 102 utilizes depth information from a layered depth map and a range of in-focus depth values ​​to render a digital image with a lens blur effect. Figure 51 shows a process in which the lens blur system 102 renders a digital image with a lens blur effect via splashing in a non-destructive processing pipeline. Specifically, Figure 5 As shown, the lens blur system 102 utilizes a focus matte and a layered depth map to determine splatter parameters for rendering a lens blur effect in a digital image.

[0075] like Figure 5 As shown, the lens blur system 102 applies a lens blur effect to a digital image using a focus matte 500 and a layered depth map 502 of the digital image. Figure 3 As described above, the lens blur system 102 determines a focus matte 500 for the digital image. Figure 4 As depicted, lens blur system 102 determines a layered depth map 502 for a digital image.

[0076] In one or more embodiments, the lens blur system 102 generates a fused depth map 504 based on the focus matte 500 and the layered depth map 502. In particular, the fused depth map 504 includes a combination of foreground depth values ​​and background depth values ​​from the layered depth map 502. For example, the lens blur system 102 generates the fused depth map 504 by combining background depth values ​​and foreground depth values ​​of a pixel according to the position of the pixel in the focus matte 500. To illustrate, the lens blur system 102 linearly interpolates between the background depth value and the foreground depth value of the pixel using the focus matte value of the pixel from the focus matte 500 (e.g., by weighting the combination toward the focus matte value). Therefore, when generating the fused depth value for the pixel, the lens blur system 102 can take into account whether the particular pixel is in the focus area or out of the focus area.

[0077] Additionally, in some embodiments, the lens blur system 102 utilizes a rendering engine 506 to render a lens blur effect in a digital image. Specifically, the lens blur system 102 provides the fused depth map 504 to the rendering engine 506. In some implementations, the rendering engine 506 comprises an engine within a digital image application that utilizes filters and / or other editing information of a digital image to determine how to present or render the digital image within a graphical user interface. Thus, the rendering engine 506 utilizes a splatter operation 508 that combines various textures (e.g., a digital image and blur parameters) using a point spread function to produce a rendered digital image 510 having a lens blur effect applied to the digital image.

[0078] In one or more embodiments, the lens blur system 102 utilizes the rendering engine 506 to implement the splash operation 508 by using the fused depth map 504 to determine a blur kernel for the splash operation 508. For example, the splash operation 508 involves determining a splash size or weight based on the fused depth value. To illustrate, in response to determining that the focus matte value of the pixel is less than 1 and greater than a threshold value ∈ that is higher than 0, the lens blur system 102 modulates the splash weight by the focus matte value. In addition, in response to determining that the focus matte value of the pixel is 1 or 0 (or less than ∈), the lens blur system 102 does not modulate the splash weight by the focus matte value.

[0079] In one or more embodiments, the splatter operation 508 utilizes a point spread function that describes the response of a single white pixel to a given radius, as well as other characteristics (e.g., shape, boundary strength, aspect ratio, and orientation). The lens blur system 102 normalizes the intensity of the point spread function to preserve energy (e.g., not generate additional light). In addition, in some embodiments, the lens blur system 102 reduces the amount of splatter with a sparse point spread function by integrating the splatter values ​​corresponding to pixels from the gradient domain according to the splatter size / weight in a single pass to obtain a rendered digital image 510. In addition, in some embodiments, the lens blur system 102 divides the depth range into multiple layers (e.g., multiple different depth ranges) for separately integrating the depth splatter and blending the integrated layers. For example, the lens blur system 102 utilizes one or more alpha blending and / or alpha normalization operations to generate the rendered digital image 510 using the integrated values ​​of the layers.

[0080] Furthermore, in one or more embodiments, the lens blur system 102 utilizes the a posteriori blur kernels to improve processing performance of rendered lens blur effects. For example, the lens blur system 102 performs a posteriori on each blur kernel before determining the point spread function gradient. To illustrate, by retaining fewer unique pixel values ​​in the a posteriori blur kernels, the lens blur system 102 generates sparse gradients, which results in fewer computational operations and lower computational cost for the blur kernels while maintaining accurate blur results in the lens blur effect.

[0081] As previously mentioned, the lens blur system 102 provides tools for interactively correcting or refining lens blur effects via one or more corrections to depth values ​​associated with a digital image. Figure 6An overview of the process by which the lens blur system 102 corrects portions of a lens blur effect applied to a digital image is shown. Specifically, the lens blur system 102 applies the corrections by modifying depth values ​​associated with the lens blur effect in response to various user interactions via the digital image. Additionally, in some embodiments, the lens blur system 102 applies a series of corrections to the lens blur effect by modifying the depth values ​​in response to a series of user interactions via the digital image.

[0082] like Figure 6 As shown, the lens blur system 102 determines the digital image 600 in conjunction with a request to apply a lens blur effect to the digital image 600 (e.g., a request to match the depth value of a portion with the depth value of its surroundings). In one or more embodiments, the lens blur system 102 determines one or more sets of depth values ​​associated with the digital image 600. To illustrate, the lens blur system 102 determines the depth values ​​from an initially estimated depth map, determines the background / foreground depth values ​​from a layered depth map, and / or determines the fused depth values ​​from a fused depth map, as described above.

[0083] In one or more embodiments, the lens blur system 102 determines a selected portion 602 of the digital image 600 indicating one or more regions of the digital image 600 to refine, correct, or otherwise adjust a blur effect in the one or more regions. For example, the lens blur system 102 determines the selected portion 602 in response to a user interaction with one or more tools to correct a lens blur effect. For illustration, as described with respect to Figure 7 As described in more detail, the lens blur system 102 determines one or more portions of one or more depth maps based on a focus area of ​​the digital image 600 to modify a blurred portion or a deblurred portion of the digital image 600 during a non-destructive processing pipeline.

[0084] In conjunction with correcting or refining the depth map(s) used to apply the lens blur effect to the digital image 600, the lens blur system 102 also determines a selected correction mode 604. In particular, the lens blur system 102 determines whether the selected correction mode 604 is a blur mode that blurs the selected portion 602 in a non-destructive blurring process or a focus mode that deblurs the selected portion 602. Figures 8 to 9 In one or more described embodiments, lens blur system 102 determines how to modify the lens blur effect based on selected correction mode 604 (eg, based on user input or a default correction mode).

[0085] Additionally, in one or more embodiments, the lens blur system 102 generates a modified depth value 606 for the selected portion 602 according to the selected correction mode 604. Specifically, the lens blur system 102 generates the modified depth value 606 for the selected portion 602 in conjunction with the focus mode to be closer to the in-focus range of the depth. Alternatively, the lens blur system 102 generates the modified depth value 606 for the selected portion 602 in conjunction with the blur mode to be further away from the in-focus range of the depth. Figures 8 to 9 and corresponding descriptions provide additional details associated with generating modified depth value 606 .

[0086] In some embodiments, the lens blur system 102 utilizes the modified depth values ​​606 to generate a corrected lens blur effect 608. For example, the lens blur system 102 utilizes the modified depth values ​​606 to render the corrected lens blur effect 608 in the digital image, such as by rendering based on the initial / original depth values ​​of the pixels in the selected portion 602 and each corrected modified depth value 606. Further, in some embodiments, the lens blur system 102 utilizes the selected blur amount to determine how to render the corrected lens blur effect 608. Thus, the lens blur system 102 provides the corrected lens blur effect 608 for one or more indicated portions of the digital image 600. Fig.10 and corresponding descriptions provide additional details associated with rendering corrected lens blur effects for digital images.

[0087] As mentioned, Figure 7 1 shows a process by which the lens blur system 102 generates modified depth values ​​for correcting a lens blur effect. In particular, the lens blur system 102 determines a digital image 700 in conjunction with a request to apply a lens blur effect to the digital image 700. Furthermore, as previously mentioned, the lens blur system 102 determines a layered depth map 702 and a focus matte 704 for the digital image 700 based on an initial estimated depth map of the digital image and an indicated focus area. Furthermore, as previously mentioned, the lens blur system 102 generates a fused depth map 706 by combining foreground / background depth values ​​using the focus matte 704.

[0088] In one or more embodiments, the lens blur system 102 determines a correction mask 708 corresponding to one or more indicated portions of the digital image 700. For example, the lens blur system 102 determines one or more portions of depth values ​​to correct the digital image 700 in response to interaction via a focus blur modification tool indicating the one or more portions. To illustrate, the lens blur system 102 detects the selected portion of the digital image 700 and identifies corresponding portions of the layered depth map 702 and the fused depth map 706 in response to interaction via the focus blur modification tool. Additionally, in some embodiments, the lens blur system 102 includes an alpha value in the correction mask 708 to scale the blur / deblur effect of the one or more portions according to the correction amount value.

[0089] In one or more embodiments, as described in more detail below, the lens blur system 102 detects the interaction via a parameterized pen touch or stroke. In alternative embodiments, the lens blur system 102 detects the interaction via a selection (e.g., lasso) tool, pen input, or other method of selecting a portion of a digital image. The lens blur system 102 generates the correction mask 708 by assigning values ​​(e.g., binary values ​​or alpha values) to selected areas or non-selected areas according to parameters (e.g., stroke parameters) of a focus blur modification tool.

[0090] In one or more embodiments, the lens blur system 102 generates a modified depth value 710 for a selected portion of the digital image 700 according to the correction mask 708. For example, the lens blur system 102 modifies the depth values ​​of the layered depth map 702 and / or the fused depth map 706 according to the correction mask 708 and the correction mode. To illustrate, as previously mentioned, the lens blur system 102 modifies the depth values ​​of the layered depth map 702 and the fused depth map 706 according to the correction mode and the selected focus range to blur or deblur a portion of the digital image 700. In some embodiments, the lens blur system 102 also modifies the focus matte 704 in conjunction with modifying the depth values ​​according to the correction mode and / or other parameters for correcting the lens blur effect.

[0091] Figure 8 An example of a process in which the lens blur system 102 modifies the depth values ​​of the correction pattern to focus (e.g., deblur) one or more selected portions of a digital image is shown. For example, as shown, the lens blur system 102 determines that the correction pattern includes a focus pattern 800 to deblur one or more blurred portions in a non-destructive blur process. Specifically, the focus pattern indicates to the lens blur system 102 to focus the one or more selected portions relative to a focus range. As a result, the lens blur system 102 obtains a depth value of the digital image including the fused depth map 802 and the layered depth map 804, the depth value including a background depth value 806 and a foreground depth value 808.

[0092] In addition, in one or more embodiments, the lens blur system 102 calculates new depth values ​​for pixels in the selected portion(s) by determining a focus range of depth values ​​corresponding to the focus range. For example, the lens blur system 102 determines depth values ​​corresponding to the focus area from a minimum depth value to a maximum depth value relative to the camera viewpoint of the digital image. For illustration, the lens blur system 102 determines the focus range of depth values ​​based on the selected range via a tool in a graphical user interface that displays the digital image. Alternatively, the lens blur system 102 determines the focus range of depth values ​​based on a default set of depth values ​​(e.g., a set of depth values ​​of the camera viewpoint closest to the digital image).

[0093] In addition, for a given pixel in the selected portion of the digital image, the lens blur system 102 determines a focus depth value 810 that is closest to the depth value of the pixel. To illustrate, in response to determining that the depth value of the particular pixel is above the focus range of depth values ​​(e.g., the pixel is indicated as being in the background), the lens blur system 102 selects the focus depth value 810 as the deepest depth value from the focus range of depth values. Alternatively, in response to determining that the depth value of the pixel is below the focus range of depth values ​​(e.g., the pixel is indicated as being in the foreground), the lens blur system 102 selects the focus depth value 810 as the shallowest depth value from the focus range of depth values.

[0094] In addition, if Figure 8 As shown, the lens blur system 102 generates a modified depth value 812 based on the in-focus depth value of the pixel in the depth map(s). In particular, the lens blur system 102 modifies the depth values ​​in the fused depth map 802 and the layered depth map 804 for the pixels in the selected portion(s). For example, the lens blur system 102 generates a plurality of modified depth values ​​for a particular pixel in the selected portion by setting the background depth value, the foreground depth value, and the fused depth value of the pixel to the in-focus depth value 810 to move the pixel into the in-focus range of the depth value. Additionally, in one or more embodiments, the lens blur system 102 also generates a modified focus matte value 814 in response to determining that the focus matte value for the pixel focus matte is greater than zero. For illustration, the lens blur system 102 generates the modified focus matte value 814 by setting the focus matte value to 1 at the pixel location (e.g., indicating that the pixel is in a focused region).

[0095] Fig. 9An example of a process in which the lens blur system 102 modifies the depth values ​​of the correction pattern to blur one or more selected portions of a digital image is shown. For example, as shown, the lens blur system 102 determines that the correction pattern includes a blur pattern 900. Specifically, the blur pattern indicates to the lens blur system 102 that the one or more selected portions are out of focus relative to the focus range. Therefore, the lens blur system 102 obtains a depth value for the digital image including the fused depth map 902 and the layered depth map 904, the depth value including a background depth value 906 and a foreground depth value 908.

[0096] In one or more embodiments, in response to determining that blur mode 900 is the selected correction mode, the lens blur system 102 determines whether defocusing the selected portion involves moving it closer to the foreground or closer to the background. In particular, if the focus area for the blur effect is in the background, indicating foreground defocus 910, blurring the selected portion involves moving the depth value of the selected portion closer to the foreground (e.g., by setting the depth value to the foreground depth value). Alternatively, if the focus area for the blur effect is in the foreground, indicating background defocus 912, blurring the selected portion involves moving the depth value of the selected portion closer to the background (e.g., by setting the depth value to the background depth value). Furthermore, if the focus area for the blur effect is somewhere between the foreground and the background, blurring the selected portion may involve moving the depth value closer to the background or the foreground depending on the current depth value of the selected portion.

[0097] To determine where to move a particular pixel toward the foreground or background, the lens blur system 102 compares the fused depth value from the fused depth map 902 for the pixel to the focus range of depth values. In one or more embodiments, the lens blur system 102 compares the fused depth value to the minimum, maximum, median, and / or average of the focus range of depth values. The lens blur system 102 thus determines how to modify the depth value of a given pixel based on the depth of the given pixel relative to the focus area.

[0098] In some embodiments, in response to determining that blur mode 900 indicates foreground defocus 910, lens blur system 102 determines a foreground depth value 914 for modifying the depth value of the depth map. For example, lens blur system 102 selects a depth value that is equal to a depth value in foreground depth values ​​908 of layered depth map 904 and that is at least halfway between the closest value in the in-focus range of depth values ​​and the absolute foreground depth value (e.g., 0). Furthermore, in some embodiments, in response to determining that the foreground depth value of a pixel from layered depth map 904 is not at least halfway between the closest value in the in-focus range of depth values ​​and the absolute foreground depth value, lens blur system 102 instead selects a depth value that is halfway.

[0099] Additionally, in some embodiments, in response to determining that blur mode 900 indicates background defocus 912, lens blur system 102 determines a background depth value 916 for modifying the depth value of the depth map. For example, lens blur system 102 selects a depth value that is equal to a depth value in background depth values ​​906 of layered depth map 904 and that is at least halfway between the closest value in the in-focus range of depth values ​​and an absolute background depth value (e.g., 1). In some embodiments, in response to determining that the background depth value of a pixel from layered depth map 904 is not at least halfway between the closest value in the in-focus range of depth values ​​and the absolute background depth value, lens blur system 102 sets background depth value 916 to a midpoint.

[0100] In one or more embodiments, in response to determining a specific type of defocus operation (e.g., foreground or background) for a pixel in the selected area, the lens blur system 102 generates a modified depth value. For example, the lens blur system 102 generates a modified depth value for the pixel by increasing or decreasing the depth value based on a comparison of the fused depth value to the focus range of the depth value. Specifically, the lens blur system 102 modifies the fused depth value and the depth values ​​(e.g., background depth value and foreground depth value) from the layered depth map 904 corresponding to the pixel to the corresponding selected depth value. For illustration, the lens blur system 102 sets the depth value corresponding to the pixel to the foreground depth value 914 for the foreground defocus 910, or the background depth value 916 for the background defocus 912. Furthermore, in some embodiments, the lens blur system 102 does not change the focus matte value corresponding to the pixel. Therefore, the lens blur system 102 generates a modified depth value 918 for each pixel in the selected portion.

[0101] In response to determining modified depth values ​​for pixels in the one or more selected portions of the digital image, the lens blur system 102 utilizes the modified depth values ​​to correct the lens blur effect. Fig.10 The lens blur system 102 utilizes the modified depth values ​​of one or more portions of a digital image to determine how to render a correction for the lens blur effect. Specifically, the lens blur system 102 utilizes the modified depth values ​​in conjunction with other parameters to render a corrected blur effect in a digital image.

[0102] like Fig.10 As shown, the lens blur system 102 determines initial depth values ​​1000 corresponding to pixels in the selected portion. In particular, the initial depth values ​​1000 include depth values ​​used by the lens blur system 102 to generate an initial lens blur effect for the digital image. For example, the lens blur system 102 determines an initial background depth value and an initial foreground depth value from the layered depth map. In addition, the lens blur system 102 determines an initial fused depth value from the fused depth map.

[0103] In one or more embodiments, the lens blur system 102 also determines the blur amount 1004 in conjunction with the particular correction. Specifically, the lens blur system 102 determines the blur amount 1004 as an amount of correction to be applied to the blur effect in the selected portion. For example, the lens blur system 102 determines the blur amount 1004 in response to an amount indicated via a user interface tool. To illustrate, the lens blur system 102 can detect an interaction with a user interface tool (e.g., a slider or an amount field) indicating the blur amount 1004 in conjunction with determining the selected portion.

[0104] like Fig.10 1 , the lens blur system 102 uses the blur amount 1004 to determine the interpolated depth value 1006 for correcting the lens blur effect. In particular, the lens blur system 102 determines whether the blur amount 1004 is within a first range (e.g., between 0 and 1). If the blur amount 1004 is within the first range, the lens blur system 102 generates the interpolated depth value 1006 using the blur amount 1004 by linearly interpolating between the initial depth value 1000 and the modified depth value 1002 for the corresponding fused depth value, foreground depth value, background depth value, and focus matte value.

[0105] In addition, the lens blur system 102 determines whether the blur amount 1004 is within a second range (e.g., between 1 and 2), which is suitable for correction in the blur mode. In response to the blur amount 1004 being within the second range, the lens blur system 102 generates an interpolated depth value 1006 by linearly interpolating between the modified depth value 1002 and an absolute foreground depth value (e.g., 0) or an absolute background depth value (e.g., 1). More specifically, the lens blur system 102 linearly interpolates the modified depth value in response to determining whether the blur mode results in foreground defocus or background defocus, as previously described.

[0106] like Fig.10 1008 . As shown, the lens blur system 102 generates a rendered blur effect 1008 with applicable corrections using the interpolated depth value 1006. In one or more embodiments, for each correction (e.g., in a series of separate corrections), the lens blur system 102 determines an interpolated blended depth value, a foreground depth value, a background depth value, and a focus matte value. Thus, in response to updating the layered depth map and focus matte for the first correction, as indicated above, the lens blur system 102 updates the layered depth map and focus matte for the second correction in a similar manner. The lens blur system 102 thus builds a corrected lens blur effect for the second correction on top of the corrected lens blur effect for the first correction. In response to processing all corrections, the lens blur system 102 generates a rendered blur effect 1008 using the modified layered depth map.

[0107] As mentioned, in one or more embodiments, lens blur system 102 applies a lens blur effect in a digital image with realistic blurring of light sources detected in the digital image. Fig.11 An overview of a process for boosting the brightness of a light source of a digital image via a highlight guide image is provided. In particular, Fig.11 The lens blur system 102 is shown generating a highlight guide image in conjunction with detected light sources of a digital image for storage with the digital image and for use in conjunction with modifying the digital image.

[0108] like Fig.11 As shown, the lens blur system 102 determines the digital image 1100 in conjunction with a request to apply a lens blur effect to the digital image 1100. For example, as described above, the lens blur system 102 determines that the request relates to applying a lens blur effect to the digital image. In one or more embodiments, the lens blur system 102 detects a light source 1102 in the digital image 1100. Specifically, the lens blur system 102 detects a portion of the digital image corresponding to a light source visible within the digital image, including a point light source, a global light source (e.g., sunlight), or other light sources present within the digital image 1100.

[0109] In one or more embodiments, the lens blur system 102 generates a highlight guide image 1104 based on the light source 1102 of the digital image 1100. In particular, the highlight guide image 1104 includes a visual representation of the light source 1102 detected in the digital image 1100. For example, the highlight guide image 1104 includes a mask that highlights pixels of the digital image 1100 corresponding to the light source 1102 for modifying the blur effect applied to the digital image 1100. By generating and storing the highlight guide image 1104 with the digital image 1100, the lens blur system 102 provides a realistic blur effect to the light source 1102 by applying additional modifications to the portion of the digital image 1100 corresponding to the light source 1102. Figure 12-13 and corresponding descriptions provide additional details associated with generating a highlight guide image for a digital image via various highlight guide image generation processes.

[0110] For example, Fig.11 As shown, the lens blur system 102 determines the brightness boost 1106 of the light source 1102 based on the highlight guide image 1104. In one or more embodiments, the lens blur system 102 uses the highlight guide image 1104 to determine specific pixels in the digital image 1100 for boosting brightness values. Fig.14As described in more detail, lens blur system 102 determines brightness boost 1106 based on highlight guide image 1104 and various blur parameters. Additionally, as shown, lens blur system 102 generates modified digital image 1108 by boosting brightness values ​​of portions of digital image 1100 corresponding to light source 1102 in conjunction with a lens blur effect.

[0111] As mentioned, in one or more embodiments, lens blur system 102 generates a highlight guide image for a digital image using one or more different processes. Fig.12 An example is shown in which the lens blur system 102 generates a highlight guide image of a digital image based on luminance values ​​extracted from the digital image. In particular, the lens blur system 102 generates a plurality of masks based on the extracted luminance values ​​of the digital image 1200, and generates a highlight guide image based on the plurality of masks.

[0112] In one or more embodiments, the lens blur system 102 determines a luminance map 1202 for the digital image 1200. Specifically, the lens blur system generates the luminance map 1202 by extracting luminance values ​​of pixels in the digital image 1200. For example, the lens blur system 102 downsamples the digital image 1200 and determines the luminance values ​​of the pixels from the downsampled digital image. To illustrate, downsampling the digital image 1200 provides a reduced size for generating a highlight guide image for the digital image 1200. Additionally, sampling the digital image 1200 provides a consistent light source boost when rendering the lens blur effect at a different resolution than the digital image 1200. In an alternative embodiment, the lens blur system 102 determines the luminance map 1202 at the original resolution of the digital image 1200.

[0113] In one or more embodiments, the lens blur system utilizes the brightness map 1202 to generate multiple masks. Fig.12As shown, the lens blur system 102 utilizes the brightness map 1202 to generate a gradient mask 1204 of the digital image 1200. For illustration, the lens blur system 102 determines the gradient mask 1204 by applying a gradient filter (or gradient detection filter) to the brightness map 1202. In one or more embodiments, the lens blur system 102 applies a Laplacian filter (or other gradient filters such as a Sobel filter) to the brightness map 1202 to identify high-contrast edges in the brightness map 1202, which indicate boundaries that may belong to light sources (e.g., point-like light sources) in the digital image 1200. The lens blur system 102 also generates the gradient mask 1204 by selecting a subset of gradients that meet a threshold gradient value. In addition, in one or more embodiments, the lens blur system 102 generates the gradient mask 1204 (or a modified gradient mask) by dilating the output of the gradient filter to compensate for certain point-like structures detected in the brightness map 1202 becoming ring-shaped, assuming that the gradient magnitude is sometimes highest along its boundaries.

[0114] In at least some embodiments, the lens blur system 102 also determines a highlight mask 1206 based on the luminance map 1202 of the digital image 1200. In particular, the lens blur system 102 generates the highlight mask 1206 to mask out pixels from the luminance map 1202 that most likely correspond to light sources. For example, the lens blur system 102 determines such pixels by comparing the luminance values ​​of the pixels in the luminance map 1202 to a threshold luminance, thereby retaining only luminance values ​​that satisfy the threshold luminance. By thresholding the luminance values, the lens blur system 102 eliminates values ​​corresponding to certain non-light source image components. Additionally, in some embodiments, the lens blur system 102 applies a tone curve (e.g., a tone curve filter) to the thresholded values ​​to further emphasize the brightest values ​​in the highlight mask 1206.

[0115] In one or more embodiments, the lens blur system 102 generates a combined mask based on the gradient mask 1204 and the highlight mask 1206. Specifically, the lens blur system 102 generates a gradient highlight mask 1208 that combines details in the gradient mask 1204 with details in the highlight mask 1206. For example, the lens blur system 102 generates the gradient highlight mask 1208 by selecting only pixels that are in both the gradient mask 1204 and the highlight mask 1206. By selecting pixels that are in both the gradient mask 1204 and the highlight mask 1206 to generate the gradient highlight mask 1208, the lens blur system 102 removes small high-contrast objects that are also not light sources, such as dust spots that have high contrast to the background (e.g., dust spots created by dust on the sensor) or other similar visual elements.

[0116] In addition, the lens blur system 102 utilizes the gradient highlight mask 1208 to generate a highlight guide image 1210 of the digital image 1200. In particular, the lens blur system 102 generates the highlight guide image 1210 to indicate the light sources in the digital image 1200 for boosting the brightness values ​​of pixels corresponding to the light sources during the blur operation. For example, the lens blur system 102 generates the highlight guide image 1210 based on a combination of the gradient highlight mask 1208 and one or more additional masks. For illustration, the lens blur system 102 generates an inverse gradient highlight mask 1212 by inverting the gradient highlight mask 1208, which provides shadow scaling for the light sources. In addition, the lens blur system 102 generates the highlight guide image 1210 by combining the gradient highlight mask, the inverse gradient highlight mask 1212, and the highlight mask 1206 (e.g., for scaling highlights not in the gradient mask 1204).

[0117] In one or more embodiments, the lens blur system generates the highlight guide image 1210 as a weighted sum of the gradient highlight mask 1208, the inverse gradient highlight mask 1212, and the highlight mask 1206. For example, the lens blur system 102 generates the highlight guide image 1210 based on the weighted sum for boosting certain light sources more than other light sources during rendering. To illustrate, the lens blur system 102 utilizes the weighted sum in the highlight guide image 1210 to boost softer light sources while boosting well-defined point light sources more than soft light sources during rendering.

[0118] In at least some embodiments, the lens blur system 102 generates the highlight guide image 1210 as a separate image. For example, the lens blur system 102 generates the highlight guide image 1210 for storage as an image file separate from the digital image 1200. The lens blur system 102 also generates a mapping between the highlight guide image 1210 and the digital image 1200. Alternatively, the lens blur system 102 generates the highlight guide image 1210 as metadata attached to the digital image 1200. In either case, the lens blur system 102 provides the highlight guide image 1210 for accessing the digital image 1200 and / or one or more versions of the digital image 1200 (e.g., different resolutions or edited versions of the digital image 1200) associated with editing the digital image 1200 via one or more digital image applications.

[0119] To illustrate, in response to a request to open the digital image 1200 within the digital image application, the lens blur system 102 accesses the highlight guide image 1210 mapped to the digital image for editing the digital image 1200. The lens blur system 102 also accesses the highlight guide image 1210 for additional versions of the digital image 1200 in the digital image application or another digital image application. Thus, in some embodiments, the lens blur system 102 renders multiple versions of the digital image 1200 with various lens blur effects based on the highlight guide image 1210.

[0120] In addition to using multiple masks to generate a highlight guide image of a digital image from a brightness map, in additional embodiments, the lens blur system 102 generates a highlight guide image (or one or more of the masks) using a machine learning model. For example, Fig.13 An embodiment is shown in which the lens blur system 102 utilizes a machine learning model to generate a highlight guide image for a digital image. In addition, Fig.13 The lens blur system 102 is shown training a machine learning model (eg, by learning parameters of the machine learning model) based on ground truth highlight guidance data for a digital image.

[0121] In one or more embodiments, the lens blur system 102 determines a digital image dataset 1300 for training a machine learning model 1302. For example, the dataset includes a set of training digital images. In additional embodiments, the dataset includes a test digital image set and / or a validation digital image set for testing and / or validating the machine learning model.

[0122] In at least some embodiments, the machine learning model 1302 includes a computer representation that is fine-tuned (e.g., trained) based on input to approximate the unknown function. For example, the machine learning model includes a neural network having one or more layers or artificial neurons that approximate the unknown function by analyzing known data at different levels of abstraction. In some embodiments, the machine learning model 1302 includes one or more neural network layers, including but not limited to convolutional neural networks, recurrent neural networks, transformer-based neural networks, or feedforward neural networks. In one or more embodiments, the machine learning model 1302 includes but is not limited to multiple separate neural network branches, such as a neural network encoder for encoding image features associated with the highlight guide image and a neural network decoder for decoding the encoded image features associated with the highlight guide image. In one or more embodiments, the machine learning model 1302 includes one or more neural network layers to illusory details in the brightness map (e.g., fill in the details in the brightness map) by generating original scene-referenced brightness values ​​of one or more light sources before being clipped by the limited capacity of the camera sensor that captured the digital image. In addition, in some embodiments, the lens blur system 102 utilizes the machine learning model to generate a mask, matte, or map associated with applying a lens blur effect to a digital image.

[0123] In one or more embodiments, the lens blur system 102 utilizes the machine learning model 1302 to generate a predicted highlight guide image 1304 for the digital image 1300. For example, the lens blur system 102 utilizes the machine learning model 1302 to process each digital image in the digital images 1300 and generates a predicted highlight guide image 1304 for the corresponding digital image. Thus, the lens blur system 102 utilizes the machine learning model 1302 to generate multiple highlight guide images for the digital image 1300 to provide a visual representation of various light sources in the digital image 1300.

[0124] In some embodiments, the lens blur system 102 improves the performance of the machine learning model 1302 by comparing the predicted highlight guidance image 1304 with the ground truth highlight guidance image 1306. For example, the lens blur system 102 improves the performance of the machine learning model 1302 by comparing the predicted highlight guidance image 1304 with the ground truth highlight guidance image 1306. Fig.12 The described process generates or determines the ground truth highlight guide image 1306. Alternatively, the lens blur system 102 determines the ground truth highlight guide image 1306 via a human labeled highlight guide image.

[0125] In one or more embodiments, the lens blur system 102 compares the predicted highlight guidance image 1304 to the ground truth highlight guidance image 1306 to determine a loss 1308. For example, the lens blur system 102 determines the loss 1308 as a cross entropy loss or an L2 loss between the predicted highlight guidance image 1304 and the corresponding ground truth highlight guidance image 1306. The lens blur system 102 utilizes the loss 1308 to modify parameters of the machine learning model 1302 based on the difference between the predicted highlight guidance image 1304 and the ground truth highlight guidance image 1306. To illustrate, the lens blur system 102 trains the machine learning model 1302 via back propagation of the loss 1308 through a neural network encoder and / or a neural network decoder of the machine learning model 1302.

[0126] In one or more embodiments, in response to generating a highlighted guide image of a digital image (e.g., via the above description of Fig.12 or Fig.13 The lens blur system 102 utilizes the highlight guide image to modify the brightness value of one or more light sources in the digital image. In particular, Fig.14 As shown, lens blur system 102 determines a highlight guide image 1402 indicating one or more light sources for digital image 1400. During rendering of the lens blur effect of digital image 1400, lens blur system 102 utilizes highlight guide image 1402 to determine brightness boost values ​​1404 for the light source(s).

[0127] In one or more embodiments, the lens blur system 102 determines the brightness boost value 1404 by determining the contribution of a given pixel based on the brightness value of the pixel in the digital image 1400, the highlight value of the pixel in the highlight guide image 1402, and the blur value of the lens blur effect of the pixel (e.g., a local blur radius). For example, the lens blur system 102 extracts the highlight value of the pixel from the highlight guide image 1402, which indicates whether the pixel belongs to a light source in the digital image 1400. In addition, the lens blur system 102 determines the brightness value of the pixel directly from the digital image 1400 or from a brightness map of the digital image 1400. The lens blur system 102 determines the local blur radius based on the lens blur effect and the corresponding blur parameters (e.g., the depth of the pixel relative to the focus range and the blur strength) used to apply the lens blur effect to the digital image 1400.

[0128] In one or more embodiments, the lens blur system 102 also determines a boost amount 1406 for determining the brightness boost value 1404. For example, the lens blur system 102 determines the boost amount 1406 in response to a selection of the boost amount 1406 via a boost setting within a graphical user interface (e.g., based on user input). For illustration, the boost amount 1406 indicates a brightness threshold value corresponding to a highlight value that is considered for boosting. More specifically, increasing the boost amount 1406 causes the lens blur system 102 to decrease the brightness threshold value while increasing the amount by which the lens blur system 102 scales the brightness boost value 1404. Specifically, the lens blur system 102 selects a highlight value based on the brightness threshold value determined from the boost amount, and determines the scaling amount based on the boost amount. Thus, decreasing the boost amount 1406 causes the lens blur system 102 to increase the brightness threshold value while decreasing the amount by which the lens blur system 102 scales the brightness boost value 1404.

[0129] In one or more embodiments, the lens blur system 102 utilizes a local blur radius to apply a realistic brightness boost in the lens blur effect of the light sources in the modified digital image 1408. Furthermore, the dependence of the brightness boost value 1404 on the local blur radius increases the visibility of light sources whose light is scattered more by a larger blur radius. Thus, an increased or decreased blur radius, which varies according to the parameters of the lens blur effect and the depth of each pixel relative to the focus range of the depth value, also affects whether the lens blur system 102 increases or decreases the brightness boost values ​​of pixels corresponding to light sources at different depths.

[0130] Figures 15 to 23 A number of graphical user interfaces of a client device for performing various depth-based lens blur effects on a digital image are shown. Fig.15 1 shows a graphical user interface displaying a digital image 1500 and a plurality of blur parameters for applying a lens blur effect to the digital image 1500. In particular, as shown in FIG. Fig.15 As shown, the client device displays a digital image 1500 within a graphical user interface for applying various edits to the digital image 1500. For example, the client device displays the digital image 1500 within a digital image application that provides a lens blur tool 1502 for generating a synthetic lens blur effect that simulates a realistic lens blur effect generated when a digital image is captured according to various lens parameters and scene depth of field.

[0131] For illustration, the client device provides the lens blur tool 1502 with a blur amount tool 1504 for indicating the amount of blur to be applied to the digital image 1500 in the lens blur effect. In one or more embodiments, the client device provides the lens blur system 102 with a blur amount based on input via the blur amount tool 1504. The lens blur system 102 utilizes blur amounts related to blurred light sources, out-of-focus depth values, etc. to render the lens blur effect via a non-destructive processing pipeline. In some embodiments, the blur amount tool 1504 also includes other parameters, such as a bokeh parameter and / or a boost parameter.

[0132] also, Fig.15 The lens blur tool 1502 is shown to include a focus range selection tool 1506. In particular, the lens blur system 102 provides the focus range selection tool 1506 to indicate an adjustable depth value range associated with generating a lens blur effect. For example, the focus range selection tool 1506 includes a slider element (e.g., a sliding window 1508) and / or other range selection elements (e.g., an input field set 1510) to indicate a minimum depth value and a maximum depth value of a focus range relative to the values ​​of the depth map of the digital image 1500. For illustration, moving the sliding window 1508 from left to right changes the focus area of ​​the focus range indicating the depth value from closer to the camera viewpoint to farther away from the camera viewpoint (e.g., from the foreground to the background).

[0133] In one or more embodiments, lens blur system 102 provides tools for visualizing depth information of a digital image. For example, Fig.16 A method for displaying and recording digital images (e.g., Fig.15 In one or more embodiments, the client device displays a graphical user interface for a depth map 1600 corresponding to a digital image 1500 (e.g., a digital image 1500). In particular, the client device displays a lens blur tool 1602 including a depth visualization element 1604. In response to selection of the depth visualization element 1604, the client device obtains and displays the depth map 1600 (or a visual representation of the depth map). In one or more embodiments, the client device displays the depth map 1600 based on an initial estimated depth map for the digital image or a fused depth generated from a layered depth map for the digital image.

[0134] For illustration, the depth map 1600 displayed in the client device includes color values ​​representing the depth values ​​of pixels in the digital image. For example, brighter values ​​in the depth map 1600 correspond to pixels in the foreground (e.g., closer to the camera viewpoint), while darker values ​​in the depth map 1600 correspond to pixels in the background (e.g., farther away from the camera viewpoint). By visualizing the depth map 1600 within the graphical user interface, the lens blur system 102 can provide the user with the ability to quickly identify estimates of the depth values ​​of various portions of the digital image, including possible errors in the depth map 1600 for correction.

[0135] Fig.17 1700. In particular, as shown, the lens blur system 102 utilizes blur parameters from the lens blur tool 1702 to modify the digital image 1700 by applying a modified lens blur effect to the digital image (e.g., Fig.15 1500) to generate a modified digital image 1700. For example, the lens blur system 102 determines a focus area (e.g., based on a selected focus range indicated by the position of the sliding window 1704) and an amount of blur (e.g., indicated by a value input via the blur amount tool 1706). As shown, the lens blur system 102 determines that the focus area corresponds to a foreground focus and a background defocus.

[0136] Thus, the lens blur system 102 generates a modified digital image 1700 to apply a lens blur effect that blurs background elements based on the relative depth values ​​of pixels in the background, and the amount of blur. For example, the lens blur system 102 generates a plurality of masks for the digital image using the corresponding depth map. To illustrate, the lens blur system 102 generates a focus mask based on a selected focus range and / or a default focus range (e.g., based on an object mask corresponding to a detected foreground object), generates a focus matte based on the focus mask, generates a layered depth map that pseudo-perspectives foreground and background depth information across the entire digital image, and generates a fused depth map that combines information from the layered depth map and the focus matte. The lens blur system 102 uses various masks (e.g., the focus matte and the fused depth map) to render the modified digital image 1700 with a lens blur effect via a splatter operation.

[0137] In one or more embodiments, modifying the focus area causes the lens blur system 102 to modify the lens blur effect on the digital image. For example, Fig.18 shows a method for displaying Fig.17 1700 of the modified digital image 1700 and a graphical user interface of the modified digital image 1800 of different focus areas. In particular, the lens blur system 102 determines that the focus area corresponds to a range of depth values ​​in the background of the digital image. For illustration, the lens blur system 102 determines that the sliding window 1802 indicates a focus range of values ​​corresponding to higher depth values ​​(e.g., depth values ​​that are farther away in the background). Therefore, by providing tools for modifying the focus range of depth values ​​and other lens blur parameters, the lens blur system 102 provides customizable depth-based lens blur and renders the lens blur effect in real-time and non-destructively.

[0138] In additional embodiments, the lens blur system 102 also provides tools for customizing the brightness boost value of realistic light source blur associated with the lens blur effect. In particular, Fig.19A graphical user interface is shown for displaying a modified digital image 1900 including a lens blur effect on the digital image having a light source boost.For example, the client device displays a boost tool 1902 for customizing the amount of boost to be applied to the brightness boost value.

[0139] To illustrate, the lens blur system 102 generates a highlight guide image for a digital image using the previously described process. The system 102 determines a boost amount according to the boost tool 1902 (e.g., in response to input via a slider element or a manually entered numeric value). The lens blur system 102 uses the boost amount with the highlight guide image to generate a modified digital image 1900 with realistic light blur by boosting the (multiple) light sources detected in the digital image via brightness boost values ​​applied to pixels corresponding to the light sources in the digital image. More specifically, the lens blur system 102 uses the boost amount, highlight value, depth value, and blur radius to produce a lens blur with light source boost.

[0140] Additionally, in one or more embodiments, lens blur system 102 provides tools for correcting or refining depth values ​​of selected areas in a digital image. Fig. 20 A graphical user interface is shown for modifying depth values ​​of selected portions of a digital image 2000. In particular, the lens blur system 102 provides a focus blur modification tool 2002 for selecting one or more portions of a digital image to modify depth values ​​of the selected portions.

[0141] According to one or more embodiments, the lens blur system 102 determines the selected mode. For example, Fig. 20 The focus blur modification tool 2002 is shown to include a focus mode for focusing (or deblurring) the selected portion(s). In addition, the focus blur modification tool 2002 includes a blur mode for blurring the selected portion(s). In addition, the lens blur system 102 provides tools for indicating or selecting a portion 2004 of the digital image 2000. As shown, the tools for indicating the portion 2004 include a parameterized brush tool for applying a stroke or brushstroke to the digital image 2000.

[0142] In at least some embodiments, the tool includes customizable parameters, including amount (e.g., opacity), brush size, feathering, and / or flow. For example, the lens blur system 102 applies a brush stroke or brushstroke to the digital image 2000 using the selected parameters in conjunction with the position of the cursor 2006 relative to the digital image 2000. For illustration, the lens blur system 102 detects an interaction with the digital image 2000 via a graphical user interface to select a portion 2004 with the cursor 2006 by applying a brush stroke or brushstroke on the digital image 2000 using the parameterized brush parameters. Although Fig. 20Utilizing a parameterized brush to select a portion of a digital image is shown, but the lens blur system 102 alternatively provides tools for selecting a portion via a selection tool or other interface tools. In additional embodiments, the lens blur system 102 utilizes an automatic masking process that identifies edges of objects (e.g., utilizing color / object identification) and selects portions of the digital image with edge awareness for accurately selecting objects at a particular depth.

[0143] In response to determining that portion 2004 of digital image 2000 is to be modified, lens blur system 102 modifies depth values ​​of pixels in portion 2004. Specifically, as previously mentioned, lens blur system 102 determines a selected mode (e.g., focus mode or blur mode), and modifies the fused depth value and the layered depth value. For example, lens blur system 102 determines a focus range indicating a focus range of depth values, and determines whether to move depth values ​​of pixels in portion 2004 toward the foreground or the background. As previously mentioned, in some embodiments, lens blur system 102 modifies focus matte values ​​in a focus matte of portion 2004 according to the selected mode.

[0144] Fig.21 21. A graphical user interface for interacting with a depth map 2100 of a digital image is shown. For example, as shown, the lens blur system 102 determines a selected portion 2102 corresponding to a correction for modifying the depth values ​​of the depth map 2100. For illustration, the lens blur system 102 determines that the selected portion 2102 of the depth map 2100 corresponds to Fig. 20 2004 of digital image 2000. Additionally, as shown, in response to determining that the selected correction mode is the focus mode, lens blur system 102 modifies the depth value of selected portion 2102 to move toward a foreground of depth map 2100 having a depth value similar to object 2104 of the digital image.

[0145] Fig. 22 A graphical user interface is shown for interacting with a depth map 2200 of a digital image. In particular, as shown, the lens blur system 102 determines that a blur mode is selected for the focus blur modification tool 2202. Thus, in response to interaction with the depth map 2200 via the focus blur modification tool 2202 (e.g., by drawing a parameterized brush stroke or brush stroke on the depth map 2200), the lens blur system 102 blurs the selected portion of the digital image.

[0146] To illustrate, the lens blur system 102 determines a focus region (e.g., based on the position of the sliding window 2204) that defines a focus range of depth values ​​corresponding to foreground focus (e.g., background defocus) of a digital image. For example, Fig. 22A selected portion 2206 of the depth map 2200 is shown in response to a cursor drawing a stroke on the depth map 2200. As shown, in response to determining that the selected portion 2206 corresponds to a foreground object (e.g., a set of pixels in the foreground having depth values), the lens blur system 102 blurs the selected portion 2206 by moving the depth values ​​of the selected portion 2206 toward the background. For example, the lens blur system 102 modifies the pixels of the selected portion 2206 to have depth values ​​similar to pixels surrounding the selected portion.

[0147] Additionally, in one or more embodiments, lens blur system 102 provides one or more options for modifying the size and / or location of a focus area of ​​a digital image. Fig.23 A graphical user interface is shown for displaying a depth map 2300 including a visual representation of depth values ​​for a digital image. Specifically, the lens blur system 102 determines a focus range based on the size and position of the sliding window 2302. For illustration, as mentioned, the lens blur system 102 provides an option for moving the sliding window 2302 toward a foreground depth value or a background depth value (or between the two).

[0148] In addition, the lens blur system 102 provides an option for modifying the size of the sliding window 2302. For example, the client device displays a text input field to indicate a minimum and / or maximum value of the focus range of the depth value. For illustration, Fig.23 The sliding window 2302 has a position and size corresponding to a narrow range of depth values ​​in the background (e.g., from "56-65" on a 100-point depth scale). In additional embodiments, the lens blur system 102 provides an option for interacting with the left or right edge of the sliding window 2302 to change the size of the sliding window 2302. Thus, in various embodiments, the lens blur system 102 determines whether the in-focus range of depth values ​​is in the foreground, in the background, or somewhere in between based on the position and size of the sliding window 2302.

[0149] The lens blur system 102 utilizes the focal range of the depth values ​​to determine how to correct or refine the depth values ​​of the selected portion of the digital image. In particular, Fig.23As shown, the lens blur system 102 determines a correction-indicative blur mode for the selected portion 2304. Furthermore, as mentioned, the lens blur system 102 compares the depth value of the selected portion 2304 to the focus range of depth values, and modifies the depth value of the selected portion 2304 accordingly. To illustrate, in response to determining that the depth value of the selected portion 2304 is behind or near the back end of the focus range of depth values ​​(e.g., further in the background), the lens blur system 102 blurs the selected portion 2304 by modifying the depth value of the selected portion 2304 to be deeper into the background. Alternatively, in response to determining that the depth value of the selected portion 2304 is before or near the front end of the focus range of depth values ​​(e.g., further in the foreground), the lens blur system 102 blurs the selected portion 2304 by modifying the depth value of the selected portion 2304 to be deeper into the foreground.

[0150] In some embodiments, the lens blur system 102 modifies a previous correction in response to a change in the focus area to blur / deblur a portion of the digital image. For example, in response to determining that the focus area is shifted toward the background, the lens blur system 102 modifies a previous correction by shifting a portion of the digital image toward the foreground to blur the portion. In an additional example, the lens blur system 102 modifies a previous correction by shifting a portion of the digital image toward a new focus range of values ​​to keep the portion in focus to deblur the portion.

[0151] Fig.24 Detailed schematic diagram of an embodiment of the above-mentioned lens blur system 102 is shown. As shown, the lens blur system 102 is implemented on (multiple) computing devices 2400 (e.g., Figure 1 The client device and / or server device described in Fig.28 2404, a user interface manager 2406, a lens blur manager 2408, a highlight guide manager 2410, and a data storage manager 2412. In one or more embodiments, the lens blur system 102 is implemented on any number of computing devices. For example, the lens blur system 102 can be implemented in a distributed system of server devices for digital images. The lens blur system 102 can also be implemented in one or more additional systems. Alternatively, the lens blur system 102 can be implemented on a single computing device such as a single client device.

[0152] In one or more embodiments, each of the components of the lens blur system 102 communicates with the other components using any suitable communication technology. In addition, the components of the lens blur system 102 can communicate with one or more other devices, including other computing devices of the user, server devices (e.g., cloud storage devices), licensing servers, or other devices / systems. It will be appreciated that although Fig.24 The components of the lens blur system 102 are shown as separate, but any subcomponents may be combined into fewer components, such as into a single component, or divided into more components, as may be useful for a particular implementation. Fig.24 , but at least some of the components used to perform operations in conjunction with the lens blur system 102 described herein can be implemented on other devices within the environment.

[0153] In some embodiments, the components of the lens blur system 102 include software, hardware, or both. For example, the components of the lens blur system 102 include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., computing device(s) 2400). When executed by the one or more processors, the computer-executable instructions of the lens blur system 102 cause the computing device 2400 to perform the operations described herein. Alternatively, the components of the lens blur system 102 include hardware, such as a dedicated processing device for performing a particular function or group of functions. Additionally, or alternatively, the components of the lens blur system 102 include a combination of computer-executable instructions and hardware.

[0154] In addition, the components of the lens blur system 102 that perform the functions described herein with respect to the lens blur system 102 can be implemented, for example, as part of a standalone application, as a module of an application, as an application plug-in, as one or more library functions that can be called by other applications, and / or as a cloud computing model. Thus, the components of the lens blur system 102 can be implemented as part of a standalone application on a personal computing device or a mobile device. Alternatively, or in addition, the components of the lens blur system 102 can be implemented in any application that provides digital image editing, including but not limited to and CREATIVE software.

[0155] As shown, the lens blur system 102 includes an image manager 2402 for managing digital images. In particular, the image manager 2402 generates, accesses, or otherwise obtains digital images for display and interaction within a graphical user interface. For example, the image manager 2402 accesses digital images provided by a client device or from a digital image database. In addition, in some embodiments, the image manager 2402 manages one or more data sets including digital images for training one or more machine learning models related to generating lens blur effects in digital images.

[0156] The lens blur system 102 also includes a depth map manager 2404 to generate or obtain the depth of the digital image. For example, the depth map manager 2404 utilizes one or more machine learning models to generate an initial estimated depth map of the digital image. Alternatively, the depth map manager 2404 obtains the depth map of the digital image based on the captured depth data of the digital image. In addition, the depth map manager 2404 determines a hierarchical depth map of the digital image, such as by using one or more machine learning models to generate background and foreground depth data of the digital image.

[0157] In an additional embodiment, the lens blur system 102 includes a user interface manager 2406 to manage interactions with a digital image and / or a depth map of a digital image related to generating a lens blur effect. For example, the user interface manager 2406 detects interactions with one or more tools that indicate blur parameters related to lens blur parameters, a focus range of depth values, corrections to depth values, and / or light source boost. In addition, the user interface manager 2406 manages the display of the digital image and the rendering of the lens blur effect via a non-destructive processing pipeline.

[0158] In one or more embodiments, the lens blur system 102 includes a lens blur manager 2408 to manage the generation of lens blur effects for digital images. Specifically, the lens blur manager 2408 communicates with one or more other components (e.g., the depth map manager 2404 and the user interface manager 2406) to obtain depth data for generating lens blur effects. For example, the lens blur manager 2408 utilizes depth values ​​and blur parameters from a depth map for a digital image to modify depth values ​​of portions of the digital image associated with generating the lens blur effect.

[0159] According to one or more embodiments, the lens blur system 102 includes a highlight guide manager 2410 to manage light source detection and enhancement in a digital image. For example, the highlight guide manager 2410 generates a highlight guide image of a digital image that indicates light sources in the digital image. In addition, the highlight guide manager 2410 communicates with the lens blur manager 2408 to utilize the highlight guide image to generate a lens blur effect with realistic light source blurring / enhancement.

[0160] The lens blur system 102 also includes a data storage manager 2412 (which includes non-transitory computer memory) that stores and maintains data associated with editing digital images. For example, the data storage manager 2412 stores digital images, depth maps, correction masks, and highlight guide images. The data storage manager 2412 also stores data for one or more machine learning models and / or for training machine learning models for generating lens blur effects (e.g., via generating depth maps and / or highlight guide images).

[0161] Now go to Fig.25 , which shows a flow chart of a series of actions 2500 for generating a lens blur effect by rendering the focused edges of a digital image. Fig.25 Actions according to one embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify Fig.25 Any of the actions shown. Fig.25 The actions of are part of the method. Alternatively, the non-transitory computer readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform Fig.25 In yet a further embodiment, the system includes a Fig.25 The handler or server for the action.

[0162] As shown, the series of actions 2500 includes an action 2502 of generating a focus matte indicating a focus range of depth values ​​of the digital image. In addition, the series of actions 2500 includes an action 2504 of generating a layered depth map indicating foreground and background depth values ​​on the digital image. The series of actions 2500 also includes an action 2506 of rendering the digital image with a lens blur effect.

[0163] In one or more embodiments, act 2502 involves generating a focus matte indicating a focus range of depth values ​​of the digital image based on a focus area of ​​the digital image and a depth map. In one or more embodiments, act 2504 involves generating a layered depth map from the depth map and the focus matte, the layered depth map including foreground depth values ​​and background depth values ​​of pixels on the digital image. In additional embodiments, act 2506 involves rendering the digital image to include a lens blur effect by determining a combination of foreground depth values ​​and background depth values ​​using the focus matte and the layered depth map in conjunction with a splatter operation.

[0164] For example, the series of actions 2500 involves providing a focus range selection tool including a sliding window within a graphical user interface, the sliding window having an adjustable depth value range corresponding to a focus relative to a depth map of a digital image. In addition, the series of actions 2500 involves determining a focus area in response to a position of the sliding window of the focus range selection tool.

[0165] The series of actions 2500 further involves providing, within the graphical user interface, a focus range selection tool with one or more options for adjusting the size of the sliding window. The series of actions 2500 further involves determining the focus area further based on the size of the sliding window.

[0166] In one or more embodiments, the series of actions 2500 involves generating a focus mask for a digital image by thresholding depth values ​​in a depth map according to a focus region. In addition, the series of actions 2500 includes generating a ternary map segmentation for the digital image and based on the focus region, the ternary map segmentation including one or more regions corresponding to discontinuities in the depth map. The series of actions 2500 further includes generating a focus matte by utilizing a matte model to generate matte values ​​within unknown regions of the ternary map segmentation and blending the focus matte values ​​for the unknown regions with the focus mask.

[0167] In some embodiments, the series of actions 2500 also includes using the focus mask as a guiding mask for refining the machine learning model to generate foreground depth values ​​and background depth values ​​at all pixels of the digital image based on the focus range of the depth values.

[0168] In some embodiments, the series of actions 2500 further includes generating a fused depth map by utilizing the focus matte to generate a fused depth value by linearly interpolating between foreground depth values ​​and background depth values ​​corresponding to a particular pixel from the layered depth map, the fused depth map including fused depth values ​​for pixels in the digital image. Additionally, the series of actions 2500 includes rendering the digital image using the fused depth values ​​for the particular pixel in conjunction with a splatter operation.

[0169] In one or more embodiments, the series of actions 2500 includes determining a splash size of a splash operation for pixels of a digital image based on a fused depth value of the fused depth map. The series of actions 2500 further includes generating a lens blur effect by integrating the splash values ​​for pixels of the digital image according to the splash size. In some embodiments, the series of actions 2500 includes determining, for a particular pixel, that a focus matte value from a focus matte is greater than a threshold value above zero. The series of actions 2500 also includes modulating a splash weight for the particular pixel by the focus matte value.

[0170] In one or more embodiments, the series of actions 2500 includes providing, for display via a graphical user interface, a focus range selection tool, the focus range selection tool including an adjustable range of depth values ​​corresponding to a focus point relative to a depth map of the digital image. The series of actions 2500 also includes generating a focus matte indicating a focus range of depth values ​​for the digital image based on a focus area selected via the focus range selection tool and a ternary map segmentation corresponding to a discontinuity in the depth map. In addition, the series of actions 2500 includes generating a layered depth map from the depth map and the focus matte, the layered depth map including a set of foreground depth values ​​for pixels of the digital image and a set of background depth values ​​for pixels of the digital image. The series of actions 2500 further includes rendering the digital image with a lens blur effect by utilizing the focus matte to determine a fused depth value by interpolating between the set of foreground depth values ​​and the set of background depth values ​​of the layered depth map in conjunction with a splatter operation.

[0171] In one or more embodiments, the series of actions 2500 includes determining a position of a sliding window in response to user input via a focus range selection tool, the sliding window indicating an adjustable depth value range corresponding to a focus point relative to the depth map. In addition, the series of actions 2500 includes generating a focus matte by modifying a size of the adjustable depth value range in response to additional user input that changes a position of an edge of the sliding window.

[0172] According to one or more embodiments, the series of actions 2500 includes generating a focus mask by thresholding depth values ​​in the depth map according to the selected focus region. The series of actions 2500 also includes generating a ternary map segmentation based on the focus mask and the depth map, the ternary map segmentation indicating unknown regions corresponding to discontinuities in the focus mask. The series of actions 2500 further includes generating a focus matte by blending focus matte values ​​of the unknown regions of the ternary map segmentation with the focus mask.

[0173] In one or more embodiments, the series of actions 2500 includes utilizing a refined machine learning model guided by a focus matte to generate the set of foreground depth values ​​in the foreground image and the set of background depth values ​​in the background image, the foreground image overlapping the background image.

[0174] According to one or more embodiments, the series of actions 2500 includes, for a pixel of a digital image, determining a splash size of a splash operation based on a fused depth value corresponding to the pixel. In addition, the series of actions 2500 includes generating a lens blur effect by integrating the splash value of the pixel determined according to the splash size with the splash value of an adjacent pixel.

[0175] In some embodiments, the series of acts 2500 further includes, for a pixel of the digital image, determining that a focus matte value from the focus matte is greater than a threshold. The series of acts 2500 further includes modulating a splatter weight of the pixel by the focus matte value in conjunction with the splatter operation.

[0176] In one or more embodiments, the series of actions 2500 includes, in response to input via a graphical user interface, determining a focus mask including a focus range of depth values ​​based on a depth map of the digital image indicated by the focus area. The series of actions 2500 further includes generating a focus matte by refining an edge of the focus mask based on discontinuities in the depth map. In addition, the series of actions 2500 includes generating a layered depth map based on the depth map and the focus matte, the layered depth map including foreground depth values ​​and background depth values ​​for pixels of the digital image. The series of actions 2500 also includes rendering the digital image to include a lens blur effect by determining a combination of foreground depth values ​​and background depth values ​​using the focus matte and the layered depth map in conjunction with a splash operation.

[0177] In some embodiments, the series of actions 2500 includes providing, within a graphical user interface, a focus range selection tool including a sliding window having an adjustable position and an adjustable size for indicating an adjustable range of depth values ​​corresponding to a focus point relative to a depth map of a digital image. The series of actions 2500 also includes determining a focus area based on the position and size indicated by the sliding window of the focus range selection tool.

[0178] In addition, the series of actions 2500 includes generating, for the digital image, a focus mask by thresholding depth values ​​in the depth map according to the focus region. The series of actions 2500 also includes generating, for the digital image and based on the focus mask and the depth map, a ternary map segmentation including an unknown region representing a discontinuity in the depth map. In addition, the series of actions 2500 includes generating a focus matte by blending focus matte values ​​in the unknown region with the focus mask.

[0179] In some embodiments, the series of actions 2500 includes determining, for pixels of the digital image, a splash size based on a combination of a foreground depth value and a background depth value according to a focus matte. In addition, the series of actions 2500 includes modulating one or more splash weights for one or more pixels according to corresponding focus matte values ​​from the focus matte. The series of actions 2500 generates a lens blur effect by generating splash values ​​for pixels of the digital image according to splash sizes for the pixels of the digital image.

[0180] Now go to Fig.26 , which shows a flow chart of a series of actions 2600 for interactively correcting depth values ​​in a depth map in connection with generating a lens blur effect for a digital image. Fig.26 Actions according to one embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify Fig.26 Any of the actions shown. Fig.26 The actions of are part of the method. Alternatively, the non-transitory computer readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform Fig.26 In yet a further embodiment, the system includes a Fig.26 The handler or server for the action.

[0181] As shown, the series of actions 2600 includes an action 2602 of generating a fused depth map for a digital image. In addition, the series of actions 2600 includes an action 2604 of determining a selected portion of the digital image for modifying depth values. The series of actions 2600 also includes an action 2606 of generating modified depth values ​​according to the selected focus range and the selected correction mode. The series of actions 2600 further includes an action 2608 of rendering the digital image using the modified depth values ​​to include a lens blur effect.

[0182] In one or more embodiments, action 2602 involves generating a fused depth map from a layered depth map of pixels of a digital image and a focal spot indicating a focus range of depth values ​​of the digital image, the fused depth map including a combination of foreground depth values ​​and background depth values. Action 2604 involves determining one or more selected portions of the digital image for modifying the depth values ​​of the digital image in response to interaction with the digital image via a focus blur modification tool in a graphical user interface. Action 2606 involves generating modified depth values ​​for the one or more selected portions of the digital image by modifying the fused depth map, the foreground depth values, and the background depth values ​​according to a selected focus range and a selected correction mode. Further, action 2608 involves rendering the digital image to include a lens blur effect using the modified depth values ​​of the one or more selected portions of the digital image.

[0183] In one or more embodiments, the series of acts 2600 includes linearly interpolating between foreground depth values ​​and background depth values ​​corresponding to pixels from the layered depth map.

[0184] In some embodiments, the series of actions 2600 includes providing, for display within a graphical user interface, a focus blur modification tool for indicating one or more portions of a digital image for modifying a depth value of the digital image. The series of actions 2600 also includes determining one or more selected portions of the digital image in response to an interaction with the focus blur modification tool. For example, the series of actions 2600 includes determining one or more parameterized strokes indicating the one or more selected portions based on a stroke parameter of the focus blur modification tool. Furthermore, the series of actions 2600 includes generating a correction mask including the one or more selected portions in response to an interaction with the focus blur modification tool.

[0185] In one or more embodiments, the series of actions 2600 includes determining that the selected correction mode indicates a request to deblur the one or more selected portions. In addition, the series of actions 2600 includes modifying the foreground depth value, the background depth value, and the fused depth value of the fused depth map to the nearest depth value within the focus range of the depth value. For example, the series of actions 2600 includes determining, for a pixel of the digital image, that a focus matte value of a focus matte is greater than zero. In addition, the series of actions 2600 includes setting the focus matte value of the pixel to indicate a foreground region of the pixel.

[0186] In one or more embodiments, the series of actions 2600 includes determining that the selected correction mode indicates a request to blur the one or more selected portions. In addition, the series of actions 2600 includes comparing one or more fused depth values ​​for a subset of pixels in the one or more selected portions to a focus range of depth values. The series of actions 2600 further includes generating modified depth values ​​for pixels in the one or more selected portions by increasing or decreasing the depth values ​​for the pixels in the one or more selected portions based on the comparison of the fused depth values ​​for the pixels to the focus range of depth values.

[0187] In some embodiments, the series of actions 2600 includes determining that the selected correction mode indicates a foreground defocus of one or more selected portions based on a comparison of the fused depth value of the pixel with the focus range of the depth value. The series of actions 2600 also includes modifying the fused depth value of the pixel and the background depth value of the pixel to be equal to the foreground depth value of the pixel. For example, the series of actions 2600 includes determining that the selected correction mode indicates a background defocus of one or more selected portions based on a comparison of the fused depth value of the pixel with the focus range of the depth value. In addition, the series of actions 2600 includes modifying the fused depth value of the pixel and the foreground depth value of the pixel to be equal to the background depth value of the pixel.

[0188] In one or more embodiments, the series of actions 2600 includes determining a blur amount for the lens blur effect according to the selected correction mode. The series of actions 2600 also includes determining an interpolated depth value by interpolating between an initial depth value for one or more selected portions of the digital image and a modified depth value for the one or more selected portions according to the blur amount. In addition, the series of actions 2600 includes rendering the digital image in the one or more selected portions of the digital image using the interpolated depth value, thereby including a lens blur effect.

[0189] In one or more embodiments, the series of actions 2600 includes determining parameters of a focus blur modification tool. The series of actions 2600 also includes generating a correction mask including one or more selected portions in response to interacting with the digital image via the focus blur modification tool using the parameters of the focus blur modification tool.

[0190] In some embodiments, the series of actions 2600 includes determining that the selected correction mode indicates a request to deblur the one or more selected portions. The series of actions 2600 further includes modifying the foreground depth values, background depth values, and fused depth values ​​of the fused depth map to depth values ​​within the in-focus range of depth values.

[0191] In at least some embodiments, the series of actions 2600 includes determining that the selected correction mode indicates a request to blur the one or more selected portions. The series of actions 2600 further includes modifying the foreground depth value, the background depth value, and the fused depth map according to the selected correction mode, the selected correction mode indicating a request to blur or deblur the one or more selected portions.

[0192] According to some embodiments, the series of actions 2600 includes setting the foreground depth value, the background depth value, and the fused depth value of the fused depth map for the one or more selected portions to the foreground depth value in response to determining that the request to blur the one or more selected portions indicates a foreground defocus of the one or more selected portions. Alternatively, the series of actions 2600 includes setting the foreground depth value, the background depth value, and the fused depth value of the fused depth map for the one or more selected portions to the background depth value in response to determining that the request to blur the one or more selected portions indicates a background defocus of the one or more selected portions.

[0193] In one or more embodiments, the series of actions 2600 includes generating a fused depth map from a layered depth map of pixels of the digital image and a focus matte indicating a focus range of depth values ​​of the digital image, the fused depth map including a combination of foreground depth values ​​and background depth values. In addition, the series of actions 2600 includes generating modified depth values ​​for one or more selected portions of the digital image by modifying the fused depth map, the foreground depth values, and the background depth values ​​according to a selected focus range and a selected correction mode including a focus mode or a blur mode. In some embodiments, the series of actions 2600 includes rendering the digital image to include a lens blur effect by deblurring or blurring the one or more selected portions of the digital image using the modified depth values ​​according to the selected correction mode.

[0194] In one or more embodiments, the series of actions 2600 includes providing, for display within a graphical user interface, a focus blur modification tool including an option indicating a selected correction mode. The series of actions 2600 also includes determining, in response to the one or more brush strokes, one or more selected portions of the digital image according to the selected correction mode.

[0195] Furthermore, in some embodiments, the series of acts 2600 includes moving the depth values ​​of the one or more selected portions toward a foreground depth of the digital image or a background depth of the digital image according to the selected correction mode.

[0196] In some embodiments, the series of actions 2600 includes determining an interpolated depth value by linearly interpolating between an initial depth value and a modified depth value of the one or more selected portions according to the selected focus range. In addition, the series of actions 2600 includes rendering a digital image in the one or more selected portions using the interpolated depth value, thereby including a lens blur effect.

[0197] Now go to Fig. 27 , which shows a flow chart of a series of actions 2700 associated with generating and utilizing a highlight guide image to boost the brightness value of a light source for generating a lens blur effect for a digital image. Fig. 27 Actions according to one embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify Fig. 27 Any of the actions shown. Fig. 27 The actions of are part of the method. Alternatively, the non-transitory computer readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform Fig. 27 In yet a further embodiment, the system includes a Fig. 27 The handler or server for the action.

[0198] As shown, the series of actions 2700 includes an action 2702 of generating a highlight guide image indicating a light source in a digital image. In addition, the series of actions 2700 includes an action 2704 of determining a brightness boost value for a pixel based on the highlight guide image. The series of actions 2700 further includes an action 2706 of rendering the digital image with the modified brightness of the pixel to include a lens blur effect.

[0199] In one or more embodiments, action 2702 includes, in response to a request to apply a lens blur effect to a digital image, generating a highlight guide image including a visual indication of one or more light sources in the digital image. Further, action 2704 involves determining a brightness boost value for a pixel corresponding to the one or more light sources based on a brightness value of the pixel in the digital image, a highlight value of the pixel in the highlight guide image, and a blur value. Further, action 2706 involves rendering the digital image to include a lens blur effect by generating a modified brightness value for the pixel according to the brightness boost value.

[0200] In one or more embodiments, the series of actions 2700 includes determining a gradient mask by detecting edges of a brightness map including brightness values ​​of pixels in the digital image. In addition, in one or more embodiments, the series of actions 2700 includes determining a highlight mask by thresholding the brightness map to determine a subset of pixels whose brightness values ​​meet a threshold brightness. In addition, the series of actions 2700 further includes generating a gradient highlight mask including pixel values ​​from a combination of the gradient mask and the highlight mask. In some embodiments, the series of actions 2700 includes generating a highlight guide image based on the gradient highlight mask and the highlight mask, the highlight guide image including an indication of one or more light sources in the digital image.

[0201] In one or more embodiments, the series of actions 2700 includes generating a brightness map for the digital image by downsampling the digital image and determining brightness values ​​of the pixels. In some embodiments, the series of actions 2700 includes further determining a highlight mask by applying a tone curve filter to a subset of pixels having brightness values ​​above a threshold brightness. Additionally, in some embodiments, the series of actions 2700 includes generating a gradient highlight mask by multiplying the gradient mask with the highlight mask to create a gradient highlight mask.

[0202] In some embodiments, the series of actions 2700 includes generating gradients from the brightness map by applying a Laplacian filter to the brightness map.The series of actions 2700 also includes determining a gradient mask by selecting a subset of gradients that satisfy a threshold gradient value.

[0203] In one or more embodiments, the series of actions 2700 includes generating an inverse mask of a gradient highlight mask. The series of actions 2700 further includes generating a highlight guide image by combining the gradient highlight mask, the inverse mask of the gradient highlight mask, and the highlight mask. In at least some embodiments, the series of actions 2700 includes generating a highlight guide image by combining the gradient highlight mask, the inverse mask of the gradient highlight mask, and the highlight mask via a weighted sum of the gradient highlight mask, the inverse mask of the gradient highlight mask, and the highlight mask.

[0204] In one or more embodiments, the series of actions 2700 includes, in response to a request to apply a lens blur effect to a digital image, determining a brightness boost value for a pixel of one or more light sources based on a highlight value of the pixel in the highlight guide image and a brightness value of the pixel in the digital image. In some embodiments, the series of actions 2700 includes determining the brightness boost value for the pixel by further determining the brightness boost value based on a local blur radius associated with the lens blur effect. In addition, the series of actions 2700 includes rendering the digital image to include a lens blur effect based on the brightness boost value of the pixel.

[0205] In one or more embodiments, the series of actions 2700 includes generating a highlighted guide image using a machine learning model including parameters learned on the highlighted guide image and a corresponding digital image from a digital image library.

[0206] In one or more embodiments, the series of actions 2700 includes determining a gradient mask indicating an edge in a brightness map of the digital image. In addition, the series of actions 2700 includes determining a highlight mask including a subset of pixels having brightness values ​​above a threshold brightness. The series of actions 2700 also includes generating a gradient highlight mask by combining the gradient mask and the highlight mask. In addition, the series of actions 2700 includes generating a highlight guide image based on the gradient highlight mask and the highlight mask.

[0207] In one or more embodiments, the series of actions 2700 includes generating a mapping between a highlight guide image and a digital image. In addition, the series of actions 2700 includes generating the highlight guide image as a separate file associated with the digital image according to the mapping. In some embodiments, the series of actions 2700 includes opening a digital image within a digital image application. In addition, the series of actions 2700 includes, in response to opening the digital image, accessing a highlight guide image mapped to the digital image associated with editing the digital image in the digital image application.

[0208] In some embodiments, the series of actions 2700 includes generating an additional version of the digital image by modifying the resolution of the digital image. The series of actions 2700 also includes rendering the additional version of the digital image according to a highlight guide image associated with the digital image, thereby including an additional lens blur effect.

[0209] In one or more embodiments, the series of actions 2700 includes determining a boost amount in response to input of a boost setting within a graphical user interface displaying a digital image. In addition, the series of actions 2700 includes selecting a highlight value from the highlight guide image by modifying a brightness threshold based on the boost amount, the highlight value including a highlight value of the pixel. The series of actions 2700 also includes determining a scaling amount for the highlight value of the pixel based on the boost amount.

[0210] According to one or more embodiments, the series of actions 2700 includes generating a gradient highlight mask by detecting edges of a brightness map of the digital image; and determining a subset of pixels of the digital image having brightness values ​​above a brightness threshold. In addition, in one or more embodiments, the series of actions 2700 includes generating a highlight guide image by combining the gradient highlight mask, an inverse mask of the gradient highlight mask, and the edges of the brightness map, the highlight guide image including an indication of one or more light sources in the digital image. The series of actions 2700 also includes rendering the digital image to include a lens blur effect by modifying the brightness values ​​of one or more pixels of the digital image using the highlight guide image.

[0211] In one or more embodiments, the series of actions 2700 includes applying a gradient detection filter to the luminance map. The series of actions 2700 also includes selecting a subset of gradients that satisfy a threshold gradient value.

[0212] In one or more embodiments, the series of actions 2700 includes, in response to input via a boost setting in a request to apply a lens blur effect to a digital image, determining a brightness boost value for one or more pixels of the digital image based on a boost amount. In addition, the series of actions 2700 includes modifying brightness values ​​of one or more pixels of the digital image according to the brightness boost value.

[0213] In some embodiments, the series of actions 2700 includes generating the highlight guide image as metadata for the digital image. The series of actions 2700 further includes rendering multiple versions of the digital image with multiple different lens blur effects by accessing the highlight guide image for each of the multiple different lens blur effects.

[0214] Embodiments of the present disclosure may include or utilize a special purpose computer or a general purpose computer including computer hardware (e.g., one or more processors and system memory), as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory) and executes the instructions to perform one or more processes, including one or more of the processes described herein.

[0215] Computer-readable media can be any available media that can be accessed by a general-purpose computer system or a special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct computer-readable media: a non-transitory computer-readable storage medium (device) and a transmission medium.

[0216] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives (“SSD”) (e.g., RAM-based), flash memory, phase change memory (“PCM”), other types of memory, other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store the desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general purpose or special purpose computer.

[0217] "Network" is defined as one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (hard wiring, wireless, or a combination of hard wiring or wireless), the computer appropriately regards the connection as a transmission medium. The transmission medium may include a network and / or a data link, which may be used to carry desired program code means in the form of computer executable instructions or data structures and may be accessed by a general-purpose computer or a special-purpose computer. The combination of the above items should also be included in the scope of computer-readable media.

[0218] Furthermore, where various computer system components are involved, program code means in the form of computer executable instructions or data structures may be automatically transferred from a transmission medium to a non-transitory computer readable storage medium (device) (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., "NIC") and then ultimately transferred to computer system RAM and / or to a less volatile computer storage medium (device) at the computer system. Thus, it should be understood that a non-transitory computer readable storage medium (device) may be included in a computer system component that also (or even primarily) utilizes a transmission medium.

[0219] Computer executable instructions include, for example, instructions and data, which, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a certain function or group of functions. In some embodiments, computer executable instructions are executed by a general-purpose computer to turn a general-purpose computer into a special-purpose computer that implements an element of the present disclosure. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, or even source code. Although the subject matter has been described in a language specific to structural features and / or method behavior, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or the actions described above. On the contrary, the described features and actions are disclosed as example forms of implementing the claims.

[0220] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based electronic products or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment, where local computer systems and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or by a combination of a hardwired data link and a wireless data link) all perform tasks. In a distributed system environment, program modules can be located in local memory storage devices and remote memory storage devices.

[0221] Embodiments of the present disclosure may also be implemented in a cloud computing environment. As used herein, the term "cloud computing" refers to a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in a market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.

[0222] The cloud computing model can consist of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model can also expose various service models, such as software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In addition, as used herein, the term "cloud computing environment" refers to an environment that employs cloud computing.

[0223] Fig.28 A block diagram of an example computing device 2800 is shown, which can be configured to perform one or more of the above processes. It should be understood that one or more computing devices such as computing device 2800 can represent the above computing devices (e.g., server 104 and client device 106). In one or more embodiments, computing device 2800 can be a mobile device (e.g., a mobile phone, a smart phone, a PDA, a tablet computer, a laptop computer, a camera, a tracker, a watch, a wearable device, etc.). In some embodiments, computing device 2800 can be a non-mobile device (e.g., a desktop computer or another type of client device). In addition, computing device 2800 can be a server device including cloud-based processing and storage capabilities.

[0224] like Fig.28 As shown, computing device 2800 may include one or more processors 2802, memory 2804, storage 2806, input / output interface 2808 (or "I / O interface 2808"), and communication interface 2810, which may be communicatively coupled by way of a communication infrastructure (e.g., bus 2812). Fig.28 2800 is shown, but Fig.28 The components shown are not intended to be limiting. In other embodiments, additional or alternative components may be used. Furthermore, in some embodiments, the computing device 2800 includes more than one component. Fig.28 The components shown are fewer than those shown. Fig.28 Components of computing device 2800 are shown.

[0225] In certain embodiments, processor(s) 2802 include hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, to execute instructions, processor 2802 may retrieve (or fetch) instructions from internal registers, internal cache, memory 2804, or storage 2806, and decode and execute them.

[0226] The computing device 2800 includes a memory 2804 coupled to the processor 2802. The memory 2804 may be used to store data, metadata, and programs for execution by the processor. The memory 2804 may include one or more of a volatile memory and a non-volatile memory, such as a random access memory ("RAM"), a read-only memory ("ROM"), a solid state disk ("SSD"), a flash memory, a phase change memory ("PCM"), or other types of data storage devices. The memory 2804 may be an internal memory or a distributed memory.

[0227] The computing device 2800 includes a storage device 2806, which includes a storage device for storing data or instructions. By way of example and not limitation, the storage device 2806 may include the non-transitory storage medium described above. The storage device 2806 may include a hard disk drive (HDD), a flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0228] As shown, computing device 2800 includes one or more I / O interfaces 2808, which are provided to allow a user to provide input (e.g., user strokes) to, receive output from, and otherwise transfer data to and from computing device 2800. These I / O interfaces 2808 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of such I / O interfaces 2808. The touch screen may be activated with a stylus or a finger.

[0229] I / O interface 2808 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O interface 2808 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0230] The computing device 2800 may further include a communication interface 2810. The communication interface 2810 may include hardware, software, or both. The communication interface 2810 provides one or more interfaces for communication (e.g., data packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example and not limitation, the communication interface 2810 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (such as WI-FI). The computing device 2800 may further include a bus 2812. The bus 2812 may include hardware, software, or both that connect the components of the computing device 2800 to each other.

[0231] In the foregoing description, the present invention has been described with reference to specific example embodiments of the present invention. Various embodiments and aspects of the present invention are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and the accompanying drawings are illustrative of the present invention and should not be construed as limiting the present invention. Many specific details are described to provide a comprehensive understanding of the various embodiments of the present invention.

[0232] The present invention may be embodied in other specific forms without departing from the spirit or essential features of the present invention. The described embodiments are considered to be illustrative and non-restrictive in all respects. For example, the method described herein may be performed with fewer or more steps / actions, or these steps / actions may be performed in different orders. In addition, the steps / actions described herein may be repeated or performed in parallel with each other, or repeated and performed in parallel with different instances of the same or similar steps / actions. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the meaning and equivalent scope of the claims should be included within the scope of the claims.

Claims

1. A computer-implemented method comprising: generating, by at least one processor, a focus matte indicating a focus range of depth values ​​of the digital image based on a focus area of ​​the digital image and a depth map; generating, by the at least one processor, a layered depth map based on the depth map and the focus matte, the layered depth map comprising foreground depth values ​​and background depth values ​​across pixels of the digital image; as well as The digital image is rendered, by the at least one processor, to include a lens blur effect by utilizing the focus matte and the layered depth map in conjunction with a splatter operation to determine a combination of the foreground depth value and the background depth value.

2. The computer-implemented method of claim 1 , wherein generating the focus matte comprises: providing, within the graphical user interface, a focus range selection tool comprising a sliding window having an adjustable range of depth values ​​corresponding to a focus relative to a depth map of the digital image; as well as The focus area is determined in response to a position of the sliding window of the focus range selection tool.

3. The computer-implemented method of claim 2 , wherein generating the focus matte comprises: within the graphical user interface, providing the focus range selection tool with one or more options for adjusting a size of the sliding window; as well as The focus area is determined further based on the size of the sliding window.

4. The computer-implemented method of claim 1 , wherein generating the focus matte comprises: For the digital image, a focus mask is generated by thresholding depth values ​​in the depth map according to the focus area.

5. The computer-implemented method of claim 4, wherein generating the focus matte comprises: generating, for the digital image and based on the focus region, a ternary map segmentation, the ternary map segmentation comprising one or more regions corresponding to discontinuities in the depth map; as well as The focus matte is generated by: generating a focal matte value for an unknown region segmented by the ternary image using a matte model; as well as The focus matte values ​​of the unknown regions are blended with the focus mask.

6. The computer-implemented method of claim 1 , wherein generating the layered depth map comprises: The foreground depth value and the background depth value are generated on all pixels of the digital image according to the in-focus range of depth values ​​using the focus matte as a guiding mask for refining a machine learning model.

7. The computer-implemented method of claim 1 , wherein rendering the digital image to include the lens blur effect comprises: generating a fused depth map by utilizing the focal matte to generate fused depth values ​​by linearly interpolating between foreground depth values ​​and background depth values ​​corresponding to particular pixels from the layered depth map, the fused depth map comprising fused depth values ​​for pixels in the digital image; as well as The digital image is rendered using the fused depth value for the particular pixel in conjunction with the splatter operation.

8. The computer-implemented method of claim 7, wherein rendering the digital image to include the lens blur effect comprises: determining a splash size of the splash operation for the pixel of the digital image based on a fused depth value of the fused depth map; as well as The lens blur effect is generated by integrating the splash values ​​for the pixels of the digital image according to the splash size.

9. The computer-implemented method of claim 7, wherein rendering the digital image to include the lens blur effect comprises: determining, for the particular pixel, that a focus matte value from the focus matte is greater than a threshold value above zero; as well as The splatter weight for the particular pixel is modulated by the focal matte value.

10. A system comprising: Memory components; as well as One or more processing devices coupled to the memory component, the one or more processing devices configured to perform operations, the operations comprising: providing, for display via a graphical user interface, a focus range selection tool comprising an adjustable range of depth values ​​corresponding to a focus relative to a depth map of a digital image; generating a focus matte indicating an in-focus range of depth values ​​for a digital image based on a focus region selected via the focus range selection tool and a ternary map segmentation corresponding to a discontinuity in the depth map; generating a layered depth map based on the depth map and the focus matte, the layered depth map comprising a set of foreground depth values ​​for pixels of the digital image and a set of background depth values ​​for the pixels of the digital image; and The digital image is rendered with a lens blur effect by utilizing the focus matte to determine a fused depth value by interpolating between the set of foreground depth values ​​and the set of background depth values ​​of the layered depth map in conjunction with a splatter operation.

11. The system of claim 10, wherein generating the focus mask comprises: In response to user input via the focus range selection tool, a position of a sliding window is determined, the sliding window indicating the adjustable range of depth values ​​corresponding to the focus relative to the depth map.

12. The system of claim 11, wherein generating the focus matte comprises: The size of the adjustable depth value range is modified in response to additional user input that changes the position of an edge of the sliding window.

13. The system of claim 10, wherein generating the focus matte comprises: generating a focus mask by thresholding depth values ​​in the depth map according to the selected focus area; generating the ternary map segmentation based on the focus mask and the depth map, the ternary map segmentation indicating unknown regions corresponding to discontinuities in the focus mask; as well as The focus matte is generated by blending the focus matte values ​​of the unknown region segmented by the ternary map with the focus mask.

14. The system of claim 10, wherein generating the layered depth map comprises: The set of foreground depth values ​​in a foreground image and the set of background depth values ​​in a background image are generated using a refined machine learning model guided by the focus matte, the foreground image overlapping the background image.

15. The system of claim 10, wherein rendering the digital image to include the lens blur effect comprises: For a pixel of the digital image, determining a splash size of the splash operation based on a fusion depth value corresponding to the pixel; as well as The lens blur effect is generated by integrating a splash value for the pixel determined according to the splash size with splash values ​​of adjacent pixels.

16. The system of claim 10, wherein rendering the digital image to include the lens blur effect comprises: determining, for a pixel of the digital image, that a focus mask value from the focus mask is greater than a threshold; as well as A splatter weight for the pixel is modulated by the focal matte value in conjunction with the splatter operation.

17. A non-transitory computer-readable medium storing executable instructions, which, when executed by a processing device, cause the processing device to perform operations comprising: In response to input via the graphical user interface, determining a focus mask including a focus range of depth values ​​based on a depth map of the digital image indicated by the focus area; generating a focus matte by refining an edge of the focus mask according to discontinuities in the depth map; generating a layered depth map according to the depth map and the focus matte, the layered depth map comprising foreground depth values ​​and background depth values ​​of pixels of the digital image; as well as The digital image is rendered to include a lens blur effect by utilizing the focus matte and the layered depth map in conjunction with a splatter operation to determine a combination of the foreground depth value and the background depth value.

18. The non-transitory computer readable medium of claim 17, the operations further comprising: providing, within the graphical user interface, a focus range selection tool comprising a sliding window having an adjustable position and an adjustable size for indicating an adjustable range of depth values ​​corresponding to focus relative to the depth map of the digital image; as well as The focus area is determined based on a position and a size indicated by the sliding window of the focus range selection tool.

19. The non-transitory computer-readable medium of claim 17, wherein generating the focus matte comprises: For the digital image, generating the focus mask by thresholding the depth values ​​in the depth map according to the focus area; generating, for the digital image and based on the focus mask and the depth map, a ternary image segmentation, the ternary image segmentation including unknown regions representing discontinuities in the depth map; as well as The focus matte is generated by blending focus matte values ​​in the unknown region with the focus mask.

20. The non-transitory computer-readable medium of claim 17, wherein rendering the digital image comprises: determining, for the pixel of the digital image, a splash size based on the combination of the foreground depth value and the background depth value according to the focus matte; modulating one or more splatter weights for one or more pixels according to corresponding focal matte values ​​from the focal matte; as well as The lens blur effect is generated by generating a splash value for the pixel of the digital image according to the splash size of the pixel of the digital image.

Images