Generating shadows for digital objects within a digital image using heightmaps

Abstract

A method for generating shadows for digital objects within a digital image using a height map is disclosed. This disclosure relates to a system, method, and non-transient computer-readable medium for generating a height map for a digital object depicted in a digital image and further utilizing that height map to generate shadows for the digital object. In practice, in one or more embodiments, the disclosed system generates (e.g., using a neural network) a height map indicating the pixel height of pixels representing digital objects depicted in the digital image. The disclosed system uses the pixel heights and lighting information for the digital image to determine how the pixels of the digital image are projected to create shadows for the digital object. Furthermore, in some implementations, the disclosed system utilizes the determined shadow projection to generate (e.g., using another neural network) soft shadows for the digital object. Therefore, in some cases, the disclosed system modifies the digital image to include shadows.

Description

Background Technology

[0001] In recent years, hardware and software platforms for editing digital images have made significant progress. Specifically, many traditional systems implement various techniques to improve the aesthetics of digital images, such as increasing the realism depicted in the digital image. To illustrate, some traditional systems provide tools for creating shadows to depict the appearance of objects (e.g., people) depicted within a digital image under certain lighting conditions. In effect, these systems create shadows where they were not originally present (e.g., within a synthetic digital image where the object is manually inserted into a background image) to depict the object more realistically in the digital image.

[0002] However, despite these advancements, traditional image shadowing systems suffer from several technical limitations, leading to inflexibility, inaccuracy, and inefficiency. For example, many traditional systems rely on using a 3D model of the object depicted in a 2D image to generate shadows for that object. Specifically, these systems typically employ physically based shadow rendering algorithms to generate shadows using the object's 3D model. Therefore, if the corresponding 3D model is unavailable, such systems cannot produce shadows for the object. Some systems address this issue by using neural network-based image synthesis to create shadows. However, these systems lack features for controlling the properties of the generated shadows (e.g., orientation, perspective, softness).

[0003] Beyond the aforementioned flexibility issues, traditional image shadowing systems often fail to generate shadows that accurately depict objects under realistic lighting conditions. For example, many traditional systems employ flawed 3D models, and physically based shadow rendering algorithms are typically sensitive to these errors. Consequently, shadows produced by such models often include various artifacts—especially in areas of ground contact, as the 3D model typically does not fully contact the ground. While traditional systems utilizing neural network-based image synthesis avoid using 3D models, the neural networks employed often have limited generalization capabilities due to their failure to explicitly model the object's geometry and associated lighting. In practice, such systems often rely on the neural network to learn the mapping between light information and the generated shadows, without geometric constraints. Therefore, these systems also tend to produce unrealistic shadows.

[0004] Furthermore, traditional image shadowing systems suffer from inefficiency. For example, physically based shadow rendering algorithms used by many traditional systems typically require significant computational resources (e.g., memory and computer processing) to generate the resulting shadows. Moreover, these systems often require various user adjustments to align the 3D model on which the shadow is based with the corresponding object depicted in the image, which can be difficult and time-consuming. Therefore, these systems often require substantial user interaction to generate object-aligned shadows.

[0005] There are these and additional difficulties and problems with traditional image shadowing systems. Summary of the Invention

[0006] The one or more embodiments described herein utilize systems, methods, and non-transient computer-readable media that generate shadows for objects depicted in a digital image using a flexible height map corresponding to the object, providing benefits and / or solving one or more problems in the art. For example, in one or more embodiments, the system generates a height map for objects depicted in a two-dimensional digital image. In some cases, the height map indicates the height of the object's pixels within the image projection space of the digital image. The system generates shadows for the object using the height map, for example, by determining how the object's shadow is projected onto a surface depicted in the digital image based on certain lighting conditions. In some embodiments, the system provides a digital image with shadows within a graphical user interface. The system also modifies the shadows based on one or more user interactions with the digital image detected via the graphical user interface. In this way, the system generates accurate shadows for objects within the digital image while providing flexible control over the properties of these shadows.

[0007] Additional features and advantages of one or more embodiments of this disclosure are set forth in the description below, and will be apparent in part from the description, or may be learned by practice of such exemplary embodiments. Attached Figure Description

[0008] With reference to the accompanying drawings, this disclosure will describe one or more embodiments of the invention with additional specificity and detail. The following paragraphs briefly describe these drawings, in which:

[0009] Figure 1 An example environment for operating a height-based shadow system according to one or more embodiments is shown;

[0010] Figures 2A to 2B An overview diagram is shown of modifying a digital image to include object shadows of digital objects depicted therein, according to one or more embodiments of a height-based shading system.

[0011] Figure 3A The illustration shows an example of generating a height map of a digital object depicted in a digital image using a height prediction neural network, according to one or more embodiments.

[0012] Figure 3B An architecture for a height prediction neural network for generating height maps of digital objects depicted in a digital image is illustrated, according to one or more embodiments.

[0013] Figure 3CThe illustration shows a height prediction neural network for generating a height map according to one or more embodiments;

[0014] Figures 4A to 4B The illustration shows a method for determining the shadow projection of pixels for a digital object using height maps and light source information, according to one or more embodiments.

[0015] Figure 5 The illustration shows a method for generating soft object shadows using a shadow stylization neural network according to one or more embodiments;

[0016] Figures 6A to 6C Each illustrates, according to one or more embodiments, modifications to digital images to include object shadows for digital objects displayed therein;

[0017] Figure 7 A graphical user interface based on a height-based shading system, according to one or more embodiments, is shown for displaying digital images and allowing user interaction for further modifications.

[0018] Figure 8 A schematic diagram of a height-based shading system according to one or more embodiments is shown;

[0019] Figure 9 A flowchart illustrating a series of actions for generating object shadows for digital objects depicted in a digital image, according to one or more embodiments; and

[0020] Figure 10 A block diagram of an example computing device according to one or more embodiments is shown. Detailed Implementation

[0021] One or more embodiments described herein include a height-based shadowing system for flexibly generating shadows for objects within a digital image by accurately determining the projection of shadows using a height map of the object. In practice, in one or more embodiments, the height-based shadowing system generates a height map for an object depicted in a two-dimensional digital image, indicating the height of the object's pixels relative to a plane depicted in the digital image (e.g., relative to the ground). The height-based shadowing system uses the height map and the position of a light source for the digital image to project shadows in two-dimensional space based on three-dimensional geometric constraints. In other words, the height-based shadowing system determines how the shadow for the object is projected across the digital image and uses the determined projection to generate the shadow for the object.

[0022] For illustration, in one or more embodiments, a height-based shadowing system extracts an object mask corresponding to the digital object from a digital image depicting the digital object. Furthermore, the height-based shadowing system generates a height map from the object mask corresponding to the digital object, the height map including pixel heights of pixels for the digital object using a height prediction neural network. Using the pixel heights from the height map and light source information for the digital image, the height-based shadowing system determines shadow projections of the pixels for the digital object. Additionally, the height-based shadowing system uses the shadow projections of the pixels for the digital object to modify the digital image to include object shadows for the digital object.

[0023] As described above, in one or more embodiments, a height-based shading system generates a height map for digital objects depicted within a digital image. Specifically, the height-based shading system generates a height map that indicates the pixel height of the digital object within the image space. For illustration, in one or more embodiments, the height-based shading system extracts an object mask for the digital object and uses the object mask to generate the height map. In some cases, the height-based shading system also determines an object cutout for the digital object and a coordinate scheme corresponding to the digital image. Therefore, the height-based shading system further utilizes the object cutout and the coordinate scheme to generate the height map height.

[0024] In some implementations, height-based shadowing systems utilize a height prediction neural network to generate height maps. In other cases, height-based shadowing systems use synthetic training images and / or real training images to generate the height prediction neural network. Therefore, in some scenarios, height-based shadowing systems leverage a height prediction neural network to generate height maps for digital objects based on object masks, object matting, and / or coordinate schemes.

[0025] As described above, in one or more embodiments, a height-based shading system uses a height map to determine the shadow projection of a digital object. Specifically, in some cases, the height-based shading system uses the pixel heights provided by the height map to determine the shadow projection of a pixel for the digital object. In one or more embodiments, the height-based shading system determines the shadow projection of a pixel for the digital object by utilizing the corresponding pixel height from the height map to determine the position of the corresponding shadow pixel within the digital image.

[0026] In some embodiments, the height-based shadowing system also utilizes light source information associated with the digital image to determine shadow projection. In practice, in some cases, the height-based shadowing system uses the position of the light source relative to the digital image to determine shadow projection. In some cases, the height-based shadowing system also utilizes the position of the horizontal line associated with the digital image or the projection of the light source onto the horizontal line.

[0027] Furthermore, height-based shadow systems allow for shadow customization. For example, in some implementations, a height-based shadow system uses a determined projection to generate a hard shadow mask, and further generates a soft shadow for the digital object from that hard shadow mask. In some cases, in response to detecting user interaction with a graphical user interface displaying a digital image, a height-based shadow system modifies one or more properties of the generated shadow (e.g., the softness or orientation of the shadow).

[0028] More specifically, as described above, in one or more implementations, a height-based shadowing system generates object shadows for digital objects using shadow projection (e.g., using shadow pixels determined via shadow projection). For illustration, in some implementations, the height-based shadowing system uses shadow projection to generate a hard shadow mask for the digital object. Furthermore, the height-based shadowing system generates soft object shadows from the hard shadow mask. In some cases, the height-based shadowing system also utilizes the object mask to generate soft object shadows. Additionally, in some embodiments, the height-based shadowing system generates soft object shadows based on a softness value (e.g., a value indicating the softness providing the soft object shadow). In some implementations, the height-based shadowing system utilizes a shadow-styling neural network to generate soft object shadows from the hard shadow mask, the object mask, and / or the softness value.

[0029] In some embodiments, a height-based shading system modifies a digital image to include object shadows for digital objects. Specifically, the height-based shading system modifies the digital image such that the digital object appears to cast shadows (e.g., object shadows) across one or more surfaces depicted in the digital image. In some cases, the height-based shading system provides object shadows to the digital image for display within a graphical user interface of a client device. In some cases, the height-based shading system detects one or more user interactions via the graphical user interface. For example, in some embodiments, the height-based shading system detects user interactions with digital objects, object shadows, or otherwise with the digital image (e.g., directly or via one or more controls provided via the graphical user interface). Based on the detected user interactions(s), in some cases, the height-based shading system modifies one or more properties of the object shadows within the digital image. For example, in some implementations, the height-based shading system modifies the position, orientation, or softness of the object shadow.

[0030] In some cases, height-based shading systems generate object shadows for digital objects that do not touch the ground. Similarly, in one or more embodiments, in response to determining that a digital object has moved vertically within a digital image, the height-based shading system can modify the object shadow of the digital object. For example, in some implementations, the height-based shading system determines the vertical displacement of the digital object and adds a corresponding displacement value to the height map for the digital object. Specifically, the height-based shading system adds the displacement value to the pixel height included in the height map.

[0031] Furthermore, in some embodiments, height-based shadowing systems provide decorative shadows for digital image assets, such as images of products to be displayed on e-commerce sites, by generating object shadows. For example, in some cases, height-based shadowing systems run batch processing to generate object shadows for multiple digital images. In other cases, height-based shadowing systems utilize shared light source information for each digital image in the batch, making the digital objects depicted appear to have been captured under consistent lighting conditions.

[0032] Compared to traditional systems, height-based shading systems offer several advantages. For example, they are more flexible in operation. By generating object shadows for digital objects using height maps, height-based shading systems can flexibly generate object shadows without relying on the 3D models of those objects. Furthermore, by modifying object shadows in response to user interactions detected via a graphical user interface, height-based shading systems maintain flexible control over the properties of object shadows.

[0033] Furthermore, height-based shadowing systems generate more accurate object shadows compared to traditional systems. Specifically, height-based shadowing systems generate object shadows that more accurately depict the shadows of objects under certain lighting conditions within digital images. For example, by generating object shadows using a height map of the digital object instead of an error-prone 3D model, height-based shadowing systems reduce artifacts present in the resulting object shadows. Moreover, by using the height map and corresponding light source information, height-based shadowing systems improve the modeling of object geometry and lighting to create more realistic object shadows.

[0034] Furthermore, height-based shadow systems operate with improved efficiency. In fact, compared to conventional systems that employ physically based shadow rendering algorithms to create shadows from a 3D model of an object, height-based shadow systems reduce the computational resources required to generate object shadows. Moreover, height-based shadow systems utilize reduced user interaction to generate object shadows. Specifically, by generating and using a height map corresponding to the digital object, height-based shadow systems eliminate the need for user interaction to adjust the 3D model to correctly align the model with the digital object. Therefore, height-based shadow systems reduce the user interaction, time, and computational resources required to generate object shadows with the desired (e.g., realistic) visual appearance.

[0035] As discussed above, this disclosure utilizes various terms to describe the features and advantages of segmentation refinement systems. Additional details regarding the meaning of these terms are now provided. For example, as used herein, the term "digital image" refers to a digital visual representation (e.g., an image composed of digital data). Specifically, in some embodiments, a digital image refers to a digital file composed of digital image data and displayable via a user interface. For example, in some implementations, a digital image includes digital photographs, digital renderings (e.g., scans or other digital reproductions) of photographs or other documents, or frames of digital video or other animation sequences. In some implementations, a digital image includes digitally generated drawings, charts, maps, graphics, logos, or other graphics. Relatedly, as used herein, the term "image space" refers to a space defined by a digital image. Specifically, in some embodiments, image space refers to a space defined by the pixels of a digital image. For example, in some embodiments, a height-based shading system determines the distance between two points in a digital image within its corresponding image space as a distance in pixels.

[0036] Furthermore, as used herein, the term "digital object" refers to an item or object depicted in a digital image. Specifically, in one or more embodiments, a digital object refers to an organic or inorganic object depicted in a digital image. For illustration, in some embodiments, digital objects include, but are not limited to, people, animals, buildings, plants, vehicles, chairs, or handheld items.

[0037] Furthermore, as used herein, the term "pixel" refers to an element of a digital image. Specifically, in some embodiments, a pixel includes the smallest unit of a digital image that can be displayed or otherwise represented (e.g., within a graphical user interface). For illustration, in some implementations, a pixel includes elements of a digital image associated with one or more specific characteristics of a digital image. For example, in some cases, a pixel includes elements of a digital image that depict or contribute to depicting a digital object depicted in a digital image.

[0038] As used herein, the term "pixel height" refers to the height of an object in a digital image based on the pixels of the digital image. Specifically, in some embodiments, pixel height refers to a value or measurement indicating the vertical distance between a pixel of the digital image and another point, such as another pixel of the digital image. For illustration, in some implementations, pixel height includes a measurement indicating the vertical distance of a pixel within the digital image relative to the ground associated with (e.g., depicted therein) the digital image, measured in pixels.

[0039] Furthermore, as used herein, the term "object shadow" refers to a shadow associated with a digital object. Specifically, in one or more embodiments, an object shadow refers to a shadow cast by a digital object depicted in a digital image based on lighting conditions associated with the digital image. For example, in some cases, an object shadow includes a shadow cast from one or more surfaces depicted by a digital object in a corresponding digital image. In some implementations, object shadows include hard object shadows or soft object shadows. As used herein, the term "hard object shadow" refers to an object shadow with hard boundaries. Specifically, in some embodiments, a hard object shadow refers to an object shadow with clearly defined boundaries. For example, in some cases, there is a clear distinction between each portion (e.g., each edge) of a hard object shadow and the surrounding area. Conversely, as used herein, the term "soft object shadow" refers to an object shadow with one or more soft boundaries. Specifically, in some embodiments, a soft object shadow refers to an object shadow with one or more indistinctly defined boundaries (e.g., blending into the surrounding area). For example, in some cases, a soft object shadow includes an object shadow with at least a portion appearing to fade into the surrounding area.

[0040] As used herein, the term "softness" refers to a characteristic or property that distinguishes hard object shadows from soft object shadows. Specifically, in some embodiments, softness refers to a characteristic having one or more soft boundaries. Furthermore, in some cases, softness is associated with degree. In other words, in some implementations, soft object shadows may have different degrees or levels of softness, where a higher level of softness is associated with a stronger blending of the soft object shadow with the surrounding area. As used herein, the term "softness value" refers to a value or set of values ​​(e.g., a vector of values) that quantifies the softness of an object shadow, such as a soft object shadow. In some cases, a baseline softness value (e.g., a softness value of zero) is associated with hard shadow objects, softness values ​​above the baseline softness value are associated with soft object shadows, and relatively high softness values ​​are associated with higher softness for soft object shadows. In some cases, height-based shading systems generate soft object shadows with a certain degree of softness based on softness values. Relatedly, as used herein, the term "softness control" refers to a feature used to determine or modify softness values. Specifically, in some embodiments, a softness control refers to a graphical user interface element that indicates the softness value used to generate an object shadow and is interactive to modify the softness value.

[0041] As used herein, the term "shadow projection" refers to the projection of shadows from one part of a digital image onto another part of the digital image based on lighting conditions associated with the digital image. For example, in some cases, shadow projection refers to the shadow (i.e., object shadow) cast by a digital object depicted in a digital image on one or more surfaces of the digital image due to associated lighting conditions. In some cases, shadow projection includes the location of the object shadow corresponding to the digital object (e.g., the location of its individual shadow pixels). In some cases, shadow projection includes one or more other properties of the object shadow, such as its orientation, size, boundaries, etc. In some cases, height-based shadowing systems determine these other properties based on the location of the individual shadow pixels.

[0042] As used herein, the term "shadow pixel" refers to a pixel in a digital image associated with an object's shadow. Specifically, in some embodiments, a shadow pixel refers to a pixel that depicts or contributes to depicting an object's shadow. As will be discussed below, in one or more embodiments, a height-based shadow system identifies, generates, or otherwise determines shadow pixels of an object's shadow from the shadow projection of the corresponding digital object (e.g., from the shadow projection of the pixels of the digital object).

[0043] Furthermore, as used herein, the term "height map" refers to a mapping from pixels to their corresponding pixel heights. Specifically, in some embodiments, a height map includes a map indicating the pixel heights of pixels associated with one or more digital objects depicted in a digital image. For example, in some cases, a height map includes a single-channel map having a single value for each pixel of a digital object, indicating the pixel height of that pixel. In one or more implementations, a height map indicates the pixel heights of pixels of objects relative to a horizontal line, the ground, or other reference point in the digital image.

[0044] As used herein, the term "neural network" refers to a type of machine learning model that can be tuned (e.g., trained) based on inputs to approximate an unknown function used to generate a corresponding output. Specifically, in some embodiments, a neural network refers to a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to generate outputs based on multiple inputs provided to the model to approximate complex functions. In some cases, a neural network includes one or more machine learning algorithms. Furthermore, in some cases, a neural network includes algorithms (or a set of algorithms) that implement deep learning techniques that utilize the set of algorithms to model high-level abstractions in data. For illustration, in some embodiments, a neural network includes a convolutional neural network, a recurrent neural network (e.g., a long short-term memory neural network), a generative adversarial neural network, a graph neural network, or a multilayer perceptron. In some embodiments, a neural network includes a neural network or a combination of neural network components.

[0045] Furthermore, as used herein, the term "height prediction neural network" refers to a computer-implemented neural network that generates height maps for digital images. In practice, in some embodiments, a height prediction neural network refers to a neural network that analyzes digital images and generates height maps for those images. Specifically, in some cases, a height prediction neural network analyzes one or more digital objects depicted in a digital image and generates height maps (or multiple height maps) for those one or more digital objects. In some cases, a height prediction neural network analyzes a digital image (e.g., one or more digital objects) by analyzing one or more object masks, object matting, and / or one or more coordinate schemes associated with the digital image.

[0046] Furthermore, as used herein, the term "shadow-styled neural network" refers to a computer-implemented neural network that generates soft object shadows for digital objects depicted in a digital image. Specifically, in some embodiments, a shadow-styled neural network refers to a neural network that generates soft object shadows based on shadow projections determined for digital objects. For example, in some cases, a shadow-styled neural network includes a neural network that analyzes hard shadow masks and object masks for digital objects and generates soft object shadows for digital objects based on that analysis.

[0047] As used herein, the term "mask" refers to a graph that can be used to segment a digital image into separate portions. Specifically, in some cases, a mask refers to a graph corresponding to a digital image and identifying a set of pixels belonging to one portion of the digital image and another set of pixels belonging to another portion of the digital image. For example, in some embodiments, a mask comprises a graph of the digital image that, for each pixel, indicates whether that pixel corresponds to a particular portion of the digital image. In some implementations, this indication includes a binary indication (1 for pixels belonging to the portion and 0 for pixels not belonging to the portion). In alternative implementations, the indication includes a probability (e.g., a number between 1 and 0) indicating the likelihood that a pixel belongs to a particular portion. In such implementations, the closer the value is to 1, the more likely the pixel is to belong to the portion, and vice versa. As used herein, the term "object mask" refers to a mask that distinguishes one or more objects of a digital image from other portions of the digital image. Furthermore, as used herein, the term "hard shadow mask" refers to a mask that distinguishes hard object shadows for digital objects from other portions of the digital image.

[0048] Furthermore, as used herein, the term "light source information" refers to information about the lighting conditions associated with a digital image. Specifically, in some embodiments, light source information refers to information indicating how a light source associated with a digital image affects one or more shadow projections determined for one or more digital objects depicted in the digital image. For example, in some cases, light source information includes the position of a light source for the digital image. In some embodiments, light source information includes the position of a horizontal line associated with the digital image. Specifically, in some cases, light source information includes the position of the light source relative to the horizontal line or the projection of the light source onto the horizontal line.

[0049] As used herein, the term "object matting" refers to the depiction of a digital object separated from a digital image. Specifically, in some embodiments, object matting refers to extracting or copying a digital object by depicting it within a color palette (e.g., the RGB palette of the digital image) in which the digital image depicts the digital object. In some cases, object matting includes depicting an image of the digital object against a background different from the background of the corresponding digital image (e.g., a pure white, gray, or black background). In practice, in some implementations, object matting includes depicting an image of the foreground of the digital image. In some cases, object matting preserves a two-dimensional representation of the digital object from a two-dimensional digital image.

[0050] Furthermore, as used herein, the term "coordinate scheme" refers to a standard for coordinate points associated with a digital image. Specifically, in some embodiments, a coordinate scheme refers to a mapping from pixels to coordinate points in a digital image. In one or more embodiments, the coordinate scheme establishes an origin at a corner point of the digital image, the center of the digital image, or another portion of the digital image. In some cases, the coordinate scheme includes a two-dimensional coordinate scheme associated with a two-dimensional digital image. In some implementations, the coordinate scheme includes an indication of the y-axis associated with the digital image (e.g., the orientation of the y-axis and / or the direction of increasing / decreasing values ​​relative to the y-axis).

[0051] Additional details regarding the height-based shading system will now be provided with reference to the accompanying drawings. For example, Figure 1 A schematic diagram of an exemplary system 100 in which a height-based shading system 106 operates is shown. Figure 1 As shown, system 100 includes (multiple) servers 102, network 108, and client devices 110a to 110n.

[0052] although Figure 1 System 100 is shown as having a specific number of components, but system 100 is capable of having any number of additional or alternative components (e.g., any number of servers, client devices, or other components communicating with height-based shadowing system 106 via network 108). Similarly, although Figure 1 A specific arrangement of (multiple) servers 102, network 108, and client devices 110a to 110n is shown, but various additional arrangements are possible.

[0053] (Multiple) servers 102, network 108, and client devices 110a to 110n directly or indirectly (e.g., through the following combination) Figure 10 The network 108, discussed in more detail, is communicatively coupled to each other. Furthermore, the (multiple) servers 102 and client devices 110a to 110n include one of a variety of computing devices (including reference devices). Figure 10 (One or more computing devices discussed in more detail).

[0054] As described above, system 100 includes servers 102. In one or more embodiments, servers 102 generate, store, receive, and / or transmit data, including digital images and modified digital images (e.g., digital images modified to include object shadows for digital objects depicted in the digital images). For example, in some embodiments, servers 102 receive digital images from client devices (e.g., one of client devices 110a to 110n) and then transmit modified digital images to the client devices. In one or more embodiments, servers 102 include data servers. In some implementations, servers 102 include communication servers or web hosting servers.

[0055] like Figure 1 As shown, servers 102 include an image editing system 104. In one or more embodiments, the image editing system 104 provides client devices (e.g., one of client devices 110a to 110n) with the ability to generate, edit, manage, and / or store digital images. For example, in some cases, the client devices send digital images to the image editing system 104 hosted on servers 102 via network 108. The image editing system 104 then provides the client devices with multiple options for editing, storing, and subsequently searching, accessing, and viewing the digital images.

[0056] Furthermore, the (multiple) servers 102 include a height-based shading system 106. Specifically, in one or more embodiments, the height-based shading system 106 utilizes the (multiple) servers 102 to modify a digital image to include object shadows for digital objects depicted in the digital image. For example, in some cases, the height-based shading system 106 utilizes the (multiple) servers 102 to receive a digital image depicting a digital object and modifies the digital image to include object shadows for the digital object.

[0057] For illustration, in one or more embodiments, a height-based shadowing system 106 extracts an object mask for a digital object from a digital image depicting a digital object via servers(s)102. Furthermore, via servers(s), the height-based shadowing system 106 generates a height map from the object mask corresponding to the digital object using a height prediction neural network; this height map includes the pixel heights of the pixels for the digital object. Additionally, via servers(s), the height-based shadowing system 106 uses the pixel heights from the height map and light source information for the digital image to determine shadow projections for the pixels of the digital object. Furthermore, via servers(s), the height-based shadowing system 106 also uses the shadow projections for the pixels of the digital object to modify the digital image to include object shadows for the digital object.

[0058] In one or more embodiments, client devices 110a to 110n include computing devices for displaying and / or modifying digital images. For example, client devices 110a to 110n include one or more of a smartphone, tablet computer, desktop computer, laptop computer, head-mounted display device, and / or other electronic device. In some instances, client devices 110a to 110n include one or more applications (e.g., image editing application 112) that allow the display and / or modification of digital images. For example, in one or more embodiments, image editing application 112 includes a software application installed on client devices 110a to 110n. Additionally or alternatively, image editing application 112 includes a software application hosted on server(s) 102 (and supported by image editing system 104), which can be accessed by client devices 110a to 110n via another application such as a web browser.

[0059] Specifically, in some implementations, a height-based shadowing system 106 on server(s) 102 supports a height-based shadowing system 106 on client device 110n. For example, the height-based shadowing system 106 on server(s) 102 learns parameters for a height prediction neural network and / or a shadow stylization neural network. The height-based shadowing system 106 then provides the height prediction neural network and / or shadow stylization neural network to client device 110n via server(s) 102. In other words, client device 110n obtains (e.g., downloads) the height prediction neural network and / or shadow stylization neural network with the learned parameters from server(s) 102. Once downloaded, the height-based shadowing system 106 on client device 110n is able to generate object shadows for digital objects depicted in a digital image independently of server(s) 102 using the height prediction neural network and / or shadow stylization neural network.

[0060] In an alternative implementation, the height-based shadowing system 106 includes a web-hosted application that allows client device 110n to interact with content and services hosted on servers(s)102. For illustration, in one or more implementations, client device 110n accesses web pages supported by servers(s)102. Client device 110n provides a digital image to servers(s)102, and in response, the height-based shadowing system 106 on servers(s)102 modifies the digital image to include object shadows for digital objects depicted in the digital image. Server(s)102 then provides the modified digital image to client device 110n for display or further editing.

[0061] In fact, the height-based shadow system 106 can be implemented wholly or partially by the individual elements of system 100. In fact, although... Figure 1 A height-based shadowing system 106 implemented relative to server(s)102 is shown, but different components of the height-based shadowing system 106 can be implemented by various devices within system 100. For example, in one or more implementations, one or more (or all) components of the height-based shadowing system 106 are implemented by different computing devices (e.g., one of client devices 110a to 110n) or servers separate from the server(s)102 hosting the image editing system 104. In fact, as... Figure 1 As shown, client devices 110a to 110n include a height-based shadowing system 106. Reference will be made below. Figure 8 Describes an example component of a height-based shadow system 106.

[0062] As described above, in one or more embodiments, a height-based shadow system 106 modifies a digital image to include object shadows for digital objects depicted in the digital image. Figures 2A to 2B An overview diagram is shown of how a height-based shadow system 106 modifies a digital image to include object shadows, according to one or more embodiments.

[0063] In fact, such as Figure 2AAs shown, a height-based shadowing system 106 determines (e.g., identifies or otherwise obtains) a digital image 202. In one or more embodiments, the height-based shadowing system 106 obtains the digital image 202 by receiving it from a computing device (e.g., a server hosting a third-party system or client device). However, in some embodiments, the height-based shadowing system 106 determines the digital image 202 by accessing a database that stores digital images. For example, in at least one implementation, the height-based shadowing system 106 maintains a database and stores multiple digital images therein. In some cases, external devices or systems store digital images for access by the height-based shadowing system 106.

[0064] In some embodiments, the height-based shading system 106 receives the digital image 202 by receiving an instruction for the digital image 202. For example, in some cases, the height-based shading system 106 receives the storage location of the digital image 202, the filename of the digital image 202, or a selection of the digital image 202. Therefore, the height-based shading system 106 retrieves the digital image 202 based on the received instruction. For illustration, as... Figure 2A As shown, in some instances, the height-based shading system 106 is used in computing devices 200 (e.g., servers 102, or above). Figure 1 The discussion is conducted on one of the client devices 110a to 110n (or some other mobile computing device such as a smartphone or tablet computer). Therefore, in some embodiments, the height-based shadowing system 106 retrieves the digital image 202 by accessing it from local storage or from a remote storage location accessible to the computing device 200.

[0065] like Figure 2A As shown, digital image 202 depicts digital object 204 (e.g., a person). Furthermore, as illustrated, digital image 202 does not include object shadows for digital object 204. In one or more embodiments, the lack of object shadows is due to the nature of digital image 202. For example, in some cases, digital object 204 is edited into digital image 202 (e.g., via an image compositing process or other editing process); therefore, when digital object 204 is captured, digital object 204 does not appear under the lighting conditions initially associated with digital image 202. In some cases, digital image 202 consists of elements (including digital object 204) manually edited together from various pre-configured image elements; therefore, digital image 202 is created in the absence of any lighting conditions.

[0066] like Figure 2AAs further shown, the height-based shading system 106 modifies the digital image 202 to include an object shadow 208 for the digital object 204. Specifically, in some implementations, the height-based shading system 106 generates the object shadow 208 based on analysis of the digital image 202 (e.g., analysis of the digital object 204). Therefore, the height-based shading system 106 modifies the digital image 202 by inserting the object shadow 208.

[0067] As shown, the height-based shadow system 106 uses a height map 206 to modify the digital image 202 to include object shadows 208. For example, in one or more embodiments, the height-based shadow system 106 generates a height map 206 for a digital object 204 and utilizes the height map 206 to generate object shadows 208 for inclusion in the digital image 202. The generation and use of height maps will be discussed in more detail below. In one or more embodiments, the height-based shadow system 106 utilizes light source information associated with the digital image 202 and the height map 206 to generate object shadows 208 for the digital object 204.

[0068] As further shown, object shadow 208 includes a soft object shadow corresponding to digital object 204. However, in some embodiments, the height-based shadow system 106 modifies the digital image 202 to include a hard object shadow for digital object 204 or some other style of shadow.

[0069] Furthermore, despite Figure 2A The diagram illustrates the generation of a single object shadow for a single digital object depicted in a digital image. However, in some implementations, the height-based shadow system 106 generates an object shadow for each of multiple digital objects depicted in the digital image. In practice, in some cases, the height-based shadow system 106 generates a single heightmap corresponding to multiple digital objects within the digital image, or generates a heightmap for each digital object. Therefore, the height-based shadow system 106 uses the heightmap(s) to generate object shadows for multiple digital objects and modifies the digital image to include the object shadows.

[0070] Figure 2B An overview diagram is shown of how, according to one or more embodiments, a height-based shadowing system 106 employs various models with height maps to generate soft object shadows for digital images. In fact, as... Figure 2BAs shown, the height-based shading system 106 utilizes a height prediction neural network 224 to generate a height map 226 for a digital image 220. Specifically, the height prediction neural network 224 generates a height map 226 for a digital object 222 depicted in the digital image 220 (e.g., pixel heights of pixels indicating the digital object). In some cases, the height prediction neural network 224 generates the height map 226 by analyzing the digital image 220. For example, in some embodiments, the height prediction neural network 224 analyzes various components corresponding to the digital image 220 (e.g., components included in, derived from, or otherwise associated with the digital image 220) and generates the height map 226 based on this analysis.

[0071] like Figure 2B As further shown, the height-based shading system 106 utilizes a hard shading rendering model 228 to determine a shadow projection 230 for a digital object 222 depicted in the digital image 220 using a height map 226. Specifically, the height-based shading system 106 utilizes a hard shading rendering model 228 to determine the shadow projection of pixels for the digital object 222 using a height map 226. As used herein, the term "hard shading rendering model" refers to a computer-implemented model or algorithm for determining the shadow projection of a pixel in two-dimensional space. Specifically, in some embodiments, a hard shading rendering model refers to a computer-implemented algorithm for determining the corresponding shadow projection for a given pixel. In some cases, the hard shading rendering model determines the shadow projection of a pixel by determining the position of the corresponding shadow pixel.

[0072] In addition, such as Figure 2B As shown, the height-based shadow system 106 utilizes a hard shadow rendering model 228 to further generate shadow projections 230 based on light source information 232 for the digital image 220. In one or more embodiments, the height-based shadow system 106 determines the light source information 232 from user input. In practice, in some cases, the height-based shadow system 106 receives user input providing information such as the position of a light source or the position of a horizontal line. In some cases, the height-based shadow system 106 determines the light source information 232 based on analysis of the digital image 220. For example, the height-based shadow system 106 analyzes the digital image 220 to detect light sources or determine the position of light sources based on one or more shadows already present within the digital image 220. Figure 2B As shown, and will be explained in more detail below, the light source information includes light source position 238 and horizontal line position 240.

[0073] In one or more embodiments, when determining shadow projection 230, the height-based shading system 106 generates hard shadow objects for the digital objects 222 depicted in the digital image. For example, using the locations of shadow pixels identified via shadow projection 230, the height-based shading system 106 generates shadow pixels at those locations. In one or more embodiments, the height-based shading system 106 generates shadow pixels when the locations of shadow pixels are identified via determining shadow projection 230. In some cases, the height-based shading system 106 determines shadow projection 230 (e.g., for the location of each shadow pixel) and then generates shadow pixels at the corresponding locations.

[0074] like Figure 2B As further shown, the height-based shading system 106 utilizes a shading style neural network 234 to generate a soft object shadow 236 for the digital object 222 from a shading projection 230 (e.g., from a hard shadow object generated based on the shading projection 230). As shown, the height-based shading system 106 provides a softness value 242 to the shading style neural network 234, enabling control over the softness of the soft object shadow 236. Furthermore, the height-based shading system 106 modifies the digital image 220 to include the soft object shadow 236 for the digital object 222.

[0075] By utilizing height maps to generate object shadows for digital objects depicted in digital images, the height-based shadow system 106 operates more flexibly than many conventional systems. In fact, the height-based shadow system 106 generates object shadows for digital objects without using 3D models of these digital objects. Therefore, the height-based shadow system 106 flexibly generates object shadows for digital objects depicted in 2D digital images where corresponding 3D models are unavailable.

[0076] As previously described, in one or more embodiments, the height-based shading system 106 utilizes a height prediction neural network to generate height maps for digital objects depicted in a digital image. Figure 3A The illustration shows an example of using a height prediction neural network to generate a height map of a digital object depicted in a digital image, according to one or more embodiments.

[0077] like Figure 3A As shown, the height-based shading system 106 determines various components corresponding to the digital image 302 depicting the digital object 304 for generating a height map 314 for the digital object 304. For example, as Figure 3AAs shown, the height-based shadow system 106 extracts an object mask 306 corresponding to a digital object 304 from the digital image 302. For example, in one or more embodiments, the height-based shadow system 106 extracts the object mask 306 from the digital image 302 using a segmentation model described in U.S. Patent Application No. 16 / 988,408, filed August 7, 2020, entitled "GENERATING REFINED SEGMENTATION MASKS BASED ONUNCERTAIN PIXELS," or a segmentation model described in U.S. Patent Application No. 17 / 200,525, filed March 12, 2021, entitled "GENERATING REFINED SEGMENTATION MASKS VIA METICULOUS OBJECT SEGMENTATION," both of which are incorporated herein by reference in their entirety.

[0078] like Figure 3A As further shown, the height-based shading system 106 determines an object matting 308 for the digital object 304. In one or more embodiments, the height-based shading system 106 determines the object matting 308 by applying an object mask 306 to the digital image 302. In some cases, the height-based shading system 106 receives the object matting 308 via user input. For example, in at least one implementation, the height-based shading system 106 receives the object matting 308 from a client device that extracts the object matting 308 from the digital image 302 via user interaction with a cropping tool.

[0079] Furthermore, as shown in the figure, the height-based shading system 106 determines the coordinate scheme 310 for the digital image 302. In some cases, the height-based shading system 106 utilizes a default or predetermined coordinate scheme as the coordinate scheme 310. In some implementations, the height-based shading system 106 determines the coordinate scheme 310 based on user input (e.g., received from a client device) or pre-configured user preferences.

[0080] like Figure 3AAs shown, the height-based shading system 106 utilizes a height prediction neural network 312 to generate a height map 314. In one or more embodiments, the height prediction neural network 312 includes a neural network with an encoder-decoder neural network architecture. For illustration, in some embodiments, the height prediction neural network 312 uses an encoder to encode neural network inputs (e.g., object mask 306, object matting 308, and coordinate scheme 310) and uses a decoder to generate neural network outputs (e.g., height map 314) from the encoded inputs. In some implementations, the height prediction neural network 312 also includes one or more skip links, each skip link providing at least one layer of the encoder's output to at least one layer of the decoder.

[0081] Figure 3B The architecture of a height-based shading system 106 for generating a height prediction neural network 340 for height maps of digital objects depicted in a digital image is shown according to one or more embodiments. As previously described, and as... Figure 3B As shown, the height-based shading system 106 provides an object mask 342, an object matting 344, and a coordinate scheme 346 as inputs to the height prediction neural network 340. For example, in some cases, the height-based shading system 106 combines (e.g., cascades) the object mask 342, the object matting 344, and the coordinate scheme 346, and provides this combination to the height prediction neural network 340. In some cases, the height-based shading system 106 normalizes the coordinate scheme 346 by setting the lowest point in the object mask 342 to zero.

[0082] like Figure 3B As shown, the height prediction neural network 340 includes a converter backbone composed of converter blocks 348a to 348d. Although a specific number of converter blocks is shown, in other embodiments, the height prediction neural network 340 may include a variety of numbers of converter blocks. The height prediction neural network 340 encodes a concatenation of object mask 342, object matting 344, and coordinate scheme 346, and generates or extracts features from this concatenation. More specifically, as shown, the height prediction neural network 340 uses various converter blocks to generate features of different scales. As used herein, the term "feature" refers to a characteristic extracted from an input (e.g., an image or a concatenation of an image with other inputs) by a neural network. Features may include deep or hidden features that are not easily understood by humans or obviously visible proprietary features. A feature map is a collection of one or more features. Specifically, a feature map may refer to a grid, matrix, vector, or different multidimensional feature sets. Furthermore, a feature map may include the locations or places of the individual feature values ​​that make up the feature map. For example, a feature map may include feature values ​​corresponding to pixels of a digital image, such that the feature map represents the digital image in the form of feature values.

[0083] like Figure 3B As further shown, the height prediction neural network 340 includes a multi-level decoder 350. The height prediction neural network 340 utilizes the multi-level decoder to merge features from different scales. For example, in one or more implementations, each transformer block generates a feature map of a given scale (i.e., size). In one or more implementations, the feature maps progressively decrease in size and become more abstract or more hidden. The adapter rescales the feature maps to a common size to allow cascading. The multi-level decoder 350 then decodes the cascaded feature maps to generate a height map 352.

[0084] In one or more implementations, the use of object mask 342 allows height prediction neural network 340 to encode only the portion of the image containing the object for which a height map is to be generated. In other words, the use of object mask 342 allows height prediction neural network 340 to efficiently generate height maps by ignoring other parts of the image.

[0085] Therefore, the height-based shading system 106 generates a height map for digital objects depicted in a digital image. Specifically, in some embodiments, the height-based shading system 106 utilizes a height prediction neural network to generate the height map for the digital objects. (See reference...) Figure 3A The described algorithms and actions may include corresponding structures for performing steps to generate height maps of digital objects depicted in digital images. Furthermore, refer to... Figure 3B The described height prediction neural network architecture may include a corresponding structure for performing steps for generating height maps of digital objects depicted in a digital image.

[0086] In one or more embodiments, a height-based shading system 106 generates (e.g., trains) a height prediction neural network to generate a height map for digital objects depicted in a digital image. Figure 3C The illustration shows a height prediction neural network for generating a height map, according to one or more embodiments.

[0087] In fact, such as Figure 3C As shown, the height-based shadow system 106 utilizes trained digital images 320 to generate a height prediction neural network 326. Specifically, as Figure 3CAs shown, the training digital image 320 includes a synthetic training image 322 and a real training image 324. In one or more embodiments, the synthetic training image 322 includes computer-generated digital images. For example, in some cases, the synthetic training image 322 includes a three-dimensional model of a computer-generated digital object. In some cases, the synthetic training image 322 includes various poses of each model in the generated three-dimensional model. For each pose of the three-dimensional model, the synthetic training image 322 includes images captured from different angles. In some implementations, the real training image 324 includes photographs (e.g., digital photographs).

[0088] like Figure 3C As shown, the height-based shadow system 106 utilizes a height prediction neural network 326 to analyze training digital images from training digital images 320 (e.g., one of the synthetic training images 322 or the real training images 324), and generates a predicted height map 328 based on this analysis. Furthermore, the height-based shadow system 106 compares the predicted height map 328 with the ground truth 330 via a loss function 332.

[0089] In one or more embodiments, the ground truth value corresponding to the synthetic training image 322 includes the pixel height of a three-dimensional model of the digital object depicted in the synthetic training image 322. For example, in some cases, the ground truth value corresponding to the synthetic training image 322 includes an object mask corresponding to a two-dimensional image captured from the proposed three-dimensional model and the pixel height of the pixels represented within the object mask.

[0090] Furthermore, in some embodiments, the ground truth value corresponding to the real training image 324 includes one or more annotations of the digital objects depicted in the real training image 324. For illustration, in one or more embodiments, the ground truth value corresponding to each real training image includes one or more labels (e.g., annotations) provided via user interaction, wherein the labels include lines drawn from points of the digital objects depicted in the real training image to points on the ground depicted in the real training image. Specifically, in some cases, each label displays the projection of a point from the digital object to the ground. In one or more embodiments, the ground truth value corresponding to each real training image also includes a measurement of the length of each line (e.g., the pixel height of the object corresponding to the drawn line). In some cases, the ground truth value also includes a dense pixel height map of the digital objects determined via bilinear interpolation using annotations.

[0091] In one or more embodiments, the height-based shadowing system 106 compares the predicted height map 328 with the ground truth 330 to determine the loss (i.e., error) of the height prediction neural network 326. In some embodiments, where the ground truth 330 corresponds to a real training image, the height-based shadowing system 106 determines a loss corresponding to one or more labels showing the projection of a point of the depicted digital object onto the ground.

[0092] like Figure 3C As shown, the height-based shading system 106 backpropagates the determined loss to the height prediction neural network 326 (as shown by dashed line 334) to update the parameters of the height prediction neural network 326. Specifically, the height-based shading system 106 updates the parameters to minimize the error of the height prediction neural network 326 in generating height maps for digital objects depicted in a digital image.

[0093] although Figure 3C The diagram illustrates the use of a single refinement iteration to generate the height prediction neural network 326; however, it should be noted that in some embodiments, the height-based shading system 106 uses multiple refinement iterations to generate the height prediction neural network 326. In one or more embodiments, with each refinement iteration, the height-based shading system 106 gradually improves the accuracy of the height prediction neural network 326 in generating height maps for digital objects. Therefore, the height-based shading system 106 utilizes the learned network parameters to generate the height prediction neural network 336.

[0094] In fact, such as Figure 3C As shown, the height-based shadow system 106 uses both synthetic and real training images to generate a height prediction neural network. However, in some cases, the height-based shadow system 106 uses only synthetic or only real training images to generate the height prediction neural network. In some implementations, generating the height prediction neural network from both synthetic and real training images improves the ability of the height prediction neural network to generate height maps. For example, in some cases, using both synthetic and real training images reduces artifacts present in the height maps generated by the height prediction neural network.

[0095] As described above, in one or more embodiments, the height-based shadowing system 106 utilizes a height map of the digital object to determine the shadow projection of the digital object. Specifically, the height-based shadowing system 106 determines the shadow projection of pixels relative to the height map. As further discussed, in one or more embodiments, the height-based shadowing system 106 uses light source information associated with a digital image depicting the digital object to determine the shadow projection. Figures 4A to 4BThe illustration shows how, according to one or more embodiments, height maps and light source information are used to determine the shadow projection of pixels for a digital object.

[0096] Figure 4A The projection correlation according to one or more embodiments is illustrated. Specifically, Figure 4A This illustrates the correlation between objects that cast shadows in the physical world and how those objects and their shadows can be captured within the image space depicted in a digital image. For example... Figure 4A As shown, lines A'B' represent objects standing on the ground in the physical world. When light source P' illuminates the environment, the object casts a shadow represented by lines B'C'. When the scene is captured by a digital image, the light source, object, and shadow are projected onto P, AB, and BC, respectively. D' represents the vertical projection of light source P' onto the ground, and D' is projected onto D within the captured digital image. It can be noted that P', A', and C' are collinear, while C', B', and D' are also collinear.

[0097] according to Figure 4A The projection correlation shown in the figure illustrates that the height-based shadow system 106 utilizes a height map to generate object shadows for digital objects depicted within a digital image. In reality, as described above, various factors can cause a digital image to not depict object shadows for the digital objects depicted therein. For example, the digital image may be the result of image compositing, or it may have been otherwise edited to include digital objects, such that the scene captured by the digital image does not include real-world objects corresponding to the digital objects. Therefore, the height-based shadow system 106 can generate object shadows to depict the shadows that would be captured if real-world objects were present when the digital image was captured.

[0098] Figure 4B An illustration shows the use of a hard shading drawing model 402 to determine the shadow projection of pixel 408 (denoted as A) of a digital object. In one or more embodiments, a digital object including pixel 408 is depicted in a digital image having a boundary represented by box 404. Furthermore, a height-based shading system 106 utilizes a height map 406 including a pixel height for pixel 408 to determine the shadow projection for pixel 408.

[0099] like Figure 4B As shown, a height-based shading system 106 identifies the position of pixel 408 within a digital image depicting a corresponding digital object. In one or more embodiments, the height-based shading system 106 uses coordinates x, representing the horizontal position of pixel 408. a and the coordinates y representing the vertical position of pixel 408 a To define the position. In some embodiments, the height-based shadow system 106 uses a coordinate scheme associated with a digital image to define the position, such as the reference above. Figure 3A The coordinate scheme used to generate the heightmap 406 is discussed. However, in some cases, the height-based shading system 106 uses a different coordinate scheme. For example, for Figure 4B The digital image shown can be used by a height-based shadow system 106 with a coordinate scheme that positions the origin at the top left corner of the digital image and increases the values ​​of x and y as the x and y axes move horizontally to the right and vertically downward from the origin, respectively.

[0100] like Figure 4B As further shown, the height-based shadow system 106 identifies the light source 410 (denoted as P and located at point (x)) for the digital image. p ,y p The position of the light source 410 and / or the horizontal line 412 corresponding to the digital image. As described above, in some cases, the height-based shadow system 106 determines the position of the light source 410 and / or the horizontal line 412 via user input. In some cases, the height-based shadow system 106 determines the position of the light source 410 and / or the horizontal line 412 by analyzing one or more shadows already present in the digital image. For example, in at least one implementation, the height-based shadow system 106 determines the position by performing the inverse calculation described below using one or more shadows already present in the digital image. Furthermore, the height-based shadow system 106 determines the projection 420 of the light source 410 onto the horizontal line 412 (where the distance of the projection is denoted as H, and the resulting projection point is denoted as D).

[0101] like Figure 4B As shown, the light source 410 is located outside the digital image boundary. However, in some cases, the height-based shadow system 106 determines that the light source 410 is within the digital image boundary. In fact, as described above, in some embodiments, the height-based shadow system 106 uses an object detection model to identify the light source for the digital image.

[0102] like Figure 4B As further shown, the height-based shadow system 106 determines the shadow projection for pixel 408 by determining the position for the corresponding shadow pixel 414 (denoted as C). Specifically, the height-based shadow system 106 determines a coordinate x representing the horizontal position of the corresponding shadow pixel 414. c And represents the coordinates y of the vertical position corresponding to the shadow pixel 414. c The corresponding shadow pixel 414 is located at this position.

[0103] As shown in the figure, the height-based shadow system 106 determines the position of the corresponding shadow pixel 414 based on the position of pixel 408, the position of light source 410, and the position of horizontal line 412. For example, as Figure 4BAs shown, the height-based shading system 106 determines that the corresponding shadow pixel 414 is located at the intersection of two lines: a first line 416 including the light source 410 and pixel 408; and a second line 418 including the projection 420 of the light source 410 on the horizontal line 412 and the projection 422 of the pixel 408 on the ground 424 of the digital image (where the distance of the projection is denoted as h, and the resulting projection point is denoted as B). In other words, the height-based shading system 106 determines that the corresponding shadow pixel 414 is located at the intersection of the extensions of line PA and line DB. Figure 4B As shown, the distance h of the projection 422 of pixel 408 onto ground 424 corresponds to the pixel height of pixel 408 included in height map 406.

[0104] like Figure 4B As shown, in some embodiments, the height-based shading system 106 uses various values ​​determined from the digital image to determine the position of the corresponding shadow pixel 414. For example, as shown, the height-based shading system 106 determines the vertical distance 426 (denoted as Z) from the top of the digital image to the horizontal line 412 in the image space. Furthermore, the height-based shading system 106 determines the vertical distance 428 (e.g., in the image space) between the horizontal line 412 and the ground 424 (denoted as d1). Additionally, the height-based shading system 106 determines the vertical distance 430 (denoted as d2) between the ground 424 and the vertical position of the corresponding shadow pixel 414. In fact, as... Figure 4B As shown, the height-based shadow system 106 is defined as follows:

[0105] d1 = h + y a -Z (1)

[0106] d2=y c -y a -h (2)

[0107] In one or more embodiments, the height-based shading system 106 determines the position of the corresponding shadow pixel 414 based on the triangular similarity between ΔANC and ΔPMC. For example, in some cases, the height-based shading system 106 utilizes triangular similarity to determine the following relationship:

[0108]

[0109] Therefore, the height-based shadow system 106 uses Equation 3 to determine the position for the corresponding shadow pixel 414, as shown below:

[0110]

[0111] Therefore, using the above process, the height-based shading system 106 determines the shadow projection 432 for the pixels of the digital object depicted in the digital image. Specifically, the height-based shading system 106 determines the position of the shadow pixel corresponding to the pixel of the digital object. Furthermore, as shown, the height-based shading system 106 utilizes a height map 406 generated for the digital object when determining the shadow projection 432.

[0112] like Figure 4B As shown, the height-based shadow system 106 utilizes shadow projection 432 to generate a hard shadow mask 434 for a digital object. For illustration, in one or more embodiments, when determining the location of shadow pixels for a digital image, the height-based shadow system 106 determines the object shadow for the digital object. Specifically, the height-based shadow system 106 determines the hard object shadow for the digital image. For example, in some cases, the height-based shadow system 106 generates shadow pixels corresponding to the pixels of the digital object at the location indicated by shadow projection 432, thereby generating a hard shadow object for the digital object. Furthermore, the height-based shadow system 106 generates the hard shadow mask 434 from the hard shadow object.

[0113] Therefore, the height-based shading system 106 utilizes the shadow projection 432—based on the pixel height of these pixels and the light source information for the digital image, corresponding to the position of the shadow pixels of the digital object—to generate a hard shadow mask 434 corresponding to the hard object shadow of the digital object. In some cases, the height-based shading system 106 generates shadow pixels at the positions indicated by the shadow projection 432 and uses the generated shadow pixels to generate the hard shadow mask 434.

[0114] In one or more embodiments, the height-based shadow system 106 generates shadow pixels from shadow projection 432 using a forward-tracking projection model. Specifically, the height-based shadow system 106 determines the shadow projection of each pixel of a digital object depicted in the digital image and generates the corresponding shadow pixel. However, in some implementations, the height-based shadow system 106 determines shadow pixels from shadow projection 432 using a backward-tracking projection model, wherein the height-based shadow system 106 generates shadow pixels from shadow projection when it determines that the shadow pixel will be visible within the digital image (e.g., will not be occluded by some other object depicted in the digital image).

[0115] In some cases, the height-based shading system 106 determines that the shadow pixels corresponding to the pixels in the digital image are located outside the boundaries of the digital image. Therefore, in some cases, the height-based shading system 106 uses only those shadow pixels determined to be located within the boundaries of the digital image to generate a hard shadow mask (e.g., only generating shadow pixels determined to be located within the digital image).

[0116] As previously described, in one or more embodiments, the height-based shadow system 106 utilizes a shadow-stylizing neural network to generate soft object shadows for digital objects depicted in a digital image. Specifically, the height-based shadow system 106 utilizes a shadow-stylizing neural network to generate soft object shadows from a hard shadow mask generated using a height map for the digital object. Figure 5 The illustration shows an example of generating soft object shadows using a shadow-styling neural network, according to one or more embodiments.

[0117] In fact, such as Figure 5 As shown, the height-based shadow system 106 provides the shadow stylization neural network 506 with an object mask 502 and a hard shadow mask 504 corresponding to digital objects depicted in a digital image. In one or more embodiments, as referenced above... Figure 3A The discussed height-based shadow system 106 generates an object mask 502 for digital objects. Furthermore, in some cases, as mentioned above... Figure 5 The height-based shadow system 106 discussed generates a hard shadow mask 504 for digital objects. As shown, the height-based shadow system 106 utilizes a shadow stylization neural network 506 to generate soft object shadows 510 from the object mask 502 and the hard shadow mask 504.

[0118] In one or more embodiments, the shadow-styled neural network 506 includes a neural network with an encoder-decoder neural network architecture. For illustration, in some embodiments, the shadow-styled neural network 506 utilizes an encoder to encode neural network inputs (e.g., object mask 502 and hard shadow mask 504) and a decoder to generate neural network outputs (e.g., soft object shadow 510). In some implementations, the shadow-styled neural network 506 also includes one or more skip links, each skip link providing at least one layer of the encoder's output to at least one layer of the decoder.

[0119] like Figure 5As further shown, the height-based shadow system 106 provides a softness value 508 to the shadow style neural network 506. In fact, as illustrated, the height-based shadow system 106 utilizes the shadow style neural network 506 to further generate a soft object shadow 510 based on the softness value 508. In one or more embodiments, the height-based shadow system 106 determines the softness value 508 from user input. For example, in some cases, the height-based shadow system 106 receives the softness value 508 from a client device or otherwise determines the softness value 508 based on user input received from the client device (e.g., based on user interaction with a softness control provided within a graphical user interface displayed on the client device). In some implementations, the height-based shadow system 106 utilizes a predetermined softness value, a default softness value, or a softness value determined from user preferences.

[0120] In one or more embodiments, the height-based shading system 106 provides softness values ​​508 to the shading style neural network 506 by providing softness values ​​508 to one or more decoder layers. For illustration, in some implementations, the height-based shading system 106 defines the softness value 508 as a two-dimensional feature map (e.g., a two-dimensional vector indicating the softness of the object shadow for each shadow pixel associated with the object shadow). Furthermore, the height-based shading system 106 concatenates the two-dimensional feature map with the input (or output) of one or more layers of the decoder of the shading style neural network 506, such as one or more initial layers in the decoder. Therefore, in some cases, the decoder layers analyze softness information from the softness value 508 when generating the soft object shadow 510.

[0121] In one or more embodiments, a height-based shadow system 106 generates (e.g., trains) a shadow-styled neural network 506 to generate soft object shadows for digital objects. In some cases, the height-based shadow system 106 generates the shadow-styled neural network 506 by refining its parameters and improving its accuracy in generating soft object shadows through multiple refinement iterations. For illustration, in some cases, the height-based shadow system 106 utilizes the shadow-styled neural network 506 to analyze a training object mask and a training hard shadow mask for a given refinement iteration. Furthermore, the height-based shadow system 106 provides training softness values ​​to the shadow-styled neural network 506. Therefore, the height-based shadow system 106 uses the shadow-styled neural network 506 to generate predicted soft object shadows, compares the predicted soft object shadows with the true values ​​via a loss function, and backpropagates the determined loss to the shadow-styled neural network 506 to update its parameters.

[0122] In some cases, the height-based shadow system 106 utilizes synthetic images to generate a shadow-styled neural network 506. Specifically, the height-based shadow system 106 utilizes a training hard shadow mask and a training object mask generated from synthetic training images. In some cases, the height-based shadow system 106 utilizes real images or a combination of synthetic and real images to generate the shadow-styled neural network 506. For illustration, in some embodiments, the height-based shadow system 106 utilizes synthetic images composed of 3D models as training images and also utilizes a physically based shadow rendering model to generate soft object shadows from the 3D models to serve as corresponding ground truth values. For example, in some cases, the height-based shadow system 106 utilizes the physically based rendering model described below: PBR renderer, https: / / substance3d.adobe.com / documentation / sddoc / pbr-render-194216472.html, 2021, the entire contents of which are incorporated herein by reference.

[0123] In one or more embodiments, a height-based shading system 106 generates a shading-styled neural network 506, while simultaneously generating a height prediction neural network. However, in some cases, the height-based shading system 106 generates each neural network via a separate process.

[0124] Furthermore, as shown in the figure, the height-based shading system 106 modifies the digital image 512 using soft object shadows 510 generated from the shading style neural network 506. Specifically, the height-based shading system 106 modifies the digital image 512 to include soft object shadows 510. For example, as... Figure 5 As shown, the height-based shadow system 106 modifies the digital image so that the digital object 514 depicted in the digital image 512 appears to be casting a soft object shadow 510 according to the lighting conditions associated with the digital image 512.

[0125] One or more embodiments of the height-based shadow system 106 can generate object shadows (e.g., soft object shadows) for digital objects depicted in various digital images. Therefore, one or more embodiments of the height-based shadow system 106 can modify various digital images to include object shadows. Figures 6A to 6C Various digital images are shown, according to one or more embodiments, to include object shadows for digital objects displayed therein.

[0126] Specifically, Figure 6AA digital photograph 602 is shown modified to include object shadows 604a to 604b for digital objects 606a to 606b displayed therein. In one or more embodiments, the digital photograph 602 includes a digital image created by extracting digital objects 606a to 606b from another digital photograph and inserting digital objects 606a to 606b into the digital photograph 602 (e.g., via an image compositing process). Therefore, in some implementations, the digital photograph 602 does not initially include object shadows for digital objects 606a to 606b.

[0127] like Figure 6A As shown, the height-based shadow system 106 generates object shadows 604a to 604b for digital objects 606a to 606b. Furthermore, the height-based shadow system 106 modifies the digital photograph 602 to include object shadows 604a to 604b. As further shown, object shadows 604a to 604b are complementary to other object shadows already present in the digital photograph 602. In other words, the height-based shadow system 106 uses lighting conditions (e.g., light source information) that cause other object shadows to generate object shadows 604a to 604b. Therefore, in one or more embodiments, the height-based shadow system 106 generates object shadows to facilitate the accurate (e.g., realistic) incorporation of digital objects into the digital photograph.

[0128] Figure 6B A digital image 610 depicting multiple digital objects 612a to 612d is shown modified to include object shadows 614a to 614d for the digital objects 612a to 612d. In one or more embodiments, the digital objects 612a to 612d include products to be advertised for sale (e.g., via web pages of an e-commerce website). Figure 6B As shown, the height-based shading system 106 modifies the digital image 610 to include object shadows 614a to 614d. In some cases, the height-based shading system 106 generates object shadows 614a to 614d to provide a more realistic and marketable appearance to digital objects 612a to 612d. As shown, object shadows 614a to 614d are consistent. In other words, object shadows 614a to 614d provide the appearance of digital objects 612a to 612d under the same lighting conditions.

[0129] although Figure 6BWhile a single digital image is discussed as having multiple digital objects, in some cases, the depicted digital objects may be included within multiple digital objects. In fact, in one or more embodiments, the height-based shadow system 106 modifies multiple digital images—each depicting at least one digital object—to include object shadows. For example, as described above, the digital images may represent image assets of products to be advertised on an e-commerce website. In one or more embodiments, the height-based shadow system 106 modifies multiple digital images via batch processing. Furthermore, in some cases, the height-based shadow system 106 uses general (e.g., preset) light source information to generate object shadows for the depicted digital objects to provide a consistent appearance of lighting conditions across the digital images. Thus, the height-based shadow system can provide decorative shadows with a consistent appearance across a variety of digital images to be presented within a general framework such as a website.

[0130] Figure 6C A digital image 620 depicting a modified computer-generated digital object 622 is shown. Specifically, as... Figure 6C As shown, digital object 622 includes the digital objects shown, such as illustrations created using vector graphics. Figure 6C As shown, the height-based shading system 106 modifies the digital image 620 to include object shadows 624 for digital objects. Therefore, the height-based shading system 106 can generate and modify digital images to include object shadows (e.g., soft object shadows) for both real digital objects (e.g., digital objects existing in the physical world) and computer-generated digital objects. Specifically, the height-based shading system 106 can generate height maps for various types of digital objects, use these height maps to generate object shadows, and modify the digital image depicting the digital objects to include the object shadows.

[0131] Furthermore, in one or more embodiments, the height-based shadow system 106 generates object shadows for animated digital objects depicted in an animation sequence (e.g., digital video), and modifies the animation sequence to include the object shadows. To illustrate, in one or more embodiments, for each frame of the animation sequence depicting the digital object, the height-based shadow system 106 generates object shadows for the digital object (e.g., generates a height map for the digital object and uses the height map and corresponding light source information to generate the object shadows). Furthermore, the height-based shadow system 106 modifies the frames of the animation sequence depicting the digital object to include the object shadows. Therefore, the height-based shadow system 106 modifies the animation sequence to include object shadows that track the movement of the digital object. For example, the object shadows may change as the position of the digital object relative to the light source position changes throughout the animation sequence.

[0132] By generating object shadows using shadow projections determined from a height map, the height-based shadow system 106 operates more accurately than conventional systems. Specifically, the height-based shadow system 106 generates object shadows that more accurately depict the object's shadow under the lighting conditions of a digital image. For example, by using a height map instead of a 3D model to determine the shadow projection, the height-based shadow system 106 avoids errors caused by using a 3D model, such as artifacts appearing in the final shadow. Furthermore, by using the height map and corresponding light source information, the height-based shadow system improves the modeling of the object's geometry and lighting to create more realistic object shadows.

[0133] Furthermore, the height-based shading system 106 generates object shadows more efficiently than many conventional systems. Indeed, as previously mentioned, many conventional systems consume significant computational resources when generating object shadows from 3D models using physically based shading. By generating object shadows from height maps, the height-based shading system 106 reduces the computational resources consumed. Moreover, the height-based shading system utilizes reduced user interaction to generate object shadows. Specifically, by generating and using height maps corresponding to digital objects, the height-based shading system avoids the need for user interaction to adjust the 3D model to correctly align the model with the digital object. Therefore, the height-based shading system reduces the user interaction, time, and computational resources required to generate object shadows with the desired (e.g., realistic) visual appearance.

[0134] As previously described, in one or more embodiments, the height-based shadow system 106 provides a digital image modified to include object shadows for display within a graphical user interface of a client device. Furthermore, the height-based shadow system 106 modifies the object shadows within the digital image in response to interaction with one or more users of the graphical user interface. Figure 7 A graphical user interface is shown, according to one or more embodiments, of a height-based shading system 106 used to display digital images and allow user interaction for further modifications.

[0135] In fact, such as Figure 7 As shown, the height-based shadow system 106 provides a digital image 702 for display within the graphical user interface 708 of the client device 710. Specifically, the digital image 702 includes a digital image that has been modified to include an object shadow 704 for a digital object 706 depicted therein. In one or more embodiments, as described above, the height-based shadow system 106 uses a height map to generate the object shadow 704.

[0136] like Figure 7As shown, object shadow 704 covers portions of non-planar surfaces (e.g., object shadow 704 extends across a horizontal floor surface and a vertical wall surface depicted in digital image 702). In practice, in one or more embodiments, height-based shadowing system 106 generates object shadows on one or more non-planar surfaces. For example, in one or more embodiments, height-based shadowing system 106 utilizes a height map generated for the digital object to determine the shadow projection of the digital object on the non-planar surface (e.g., determining the shadow projection of the pixels of the digital object).

[0137] As further shown, the height-based shadow system 106 provides a softness control 712 within the graphical user interface 708. In one or more embodiments, the height-based shadow system 106 provides the softness control 712 to indicate the current softness of the object shadow 704 (e.g., via the position of the interactive element 714). Furthermore, in one or more embodiments, the height-based shadow system 106 detects user interaction with the softness control 712. Specifically, the height-based shadow system 106 detects user interaction that repositions the interactive element 714.

[0138] In some embodiments, in response to detecting user interaction with the softness control 712, the height-based shadow system 106 modifies the object shadow 704 within the digital image 702. For illustration, in some cases, the height-based shadow system 106 determines a softness value corresponding to the position of the interactive element 714 caused by the user interaction. Furthermore, the height-based shadow system 106 provides the softness value to a shadow styling neural network used to generate soft object shadows, and uses the shadow styling neural network to modify the object shadow 704 (e.g., by generating a modified object shadow). Therefore, the height-based shadow system 106 facilitates flexible control over the softness of the generated object shadow.

[0139] Furthermore, as shown, the height-based shadow system 106 provides a distortion control 716 within the graphical user interface 708. In one or more embodiments, the height-based shadow system 106 provides the distortion control 716 to indicate the position of a horizontal line corresponding to the digital image 702 (e.g., via the position of the interactive element 718). Additionally, in one or more embodiments, the height-based shadow system 106 detects user interaction with the distortion control 716. Specifically, the height-based shadow system 106 detects user interaction that repositions the interactive element 718.

[0140] In some embodiments, in response to detecting user interaction with distortion control 716, height-based shading system 106 modifies object shadow 704 within digital image 702. For illustration, in some cases, height-based shading system 106 determines a horizontal line position corresponding to the location of interactive element 718 caused by user interaction. Furthermore, height-based shading system 106 utilizes a hard shadow drawing model to determine a modified shadow projection for pixels of digital object 706 based on a previously generated height map for digital object 706, the position of a light source corresponding to digital image 702, and the horizontal line position caused by user interaction. Height-based shading system 106 uses the modified shadow projection to modify object shadow 704 (e.g., by generating a modified object shadow). Therefore, height-based shading system 106 facilitates flexible control over the distortion of the generated object shadow.

[0141] In some embodiments, the height-based shading system 106 detects user interaction with digital object 706 (e.g., via cursor 720) and modifies object shadow 704 based on the detected user interaction. For example, in some cases, the height-based shading system 106 determines that the user interaction vertically moves digital object 706 within digital image 702. The height-based shading system 106 determines the vertical shift of digital object 706 (e.g., vertical shift of monitored pixels of digital object 706) and adds a shift value corresponding to the vertical shift to the pixel height represented in the height map of digital object 706. Using the shifted height value, the height-based shading system 106 utilizes the aforementioned hard shading rendering model to determine a modified shadow projection for the pixels of digital object 706. The height-based shading system 106 uses the modified shadow projection to modify object shadow 704 (e.g., by generating a modified object shadow). Therefore, the height-based shading system 106 provides control over other characteristics of the generated object shadow (e.g., those characteristics corresponding to the vertical shift of the corresponding digital object). In some cases where the user interacts with the horizontal movement of the digital object 706, the height-based shadow system 106 determines a modified shadow projection and uses the modified horizontal position to generate a modified object shadow.

[0142] In one or more embodiments, the height-based shadow system 106 detects user interaction with the object shadow 704 itself (e.g., via cursor 720) and modifies the object shadow 704 based on the detected user interaction. Specifically, in some embodiments, the height-based shadow system 106 determines that the user interaction is for repositioning the object shadow 704 within the digital image 702, such as moving the object shadow 704 around the digital object 706. In some implementations, the height-based shadow system 106 determines a change in the position of a light source corresponding to the digital image 702 based on the user interaction and modifies the object shadow 704 based on the changed light source position.

[0143] For illustration, in some cases, the height-based shadow system 106 tracks the cursor 720 as it selects the object shadow 704 and moves around the digital object 706. When it is determined that the cursor 720 has stopped moving, the height-based shadow system 106 determines the position of the cursor 720 and the modified position of the light source, as if the position of the cursor 720 corresponds to a shadow pixel. Specifically, the height-based shadow system 106 uses the above references... Figure 4B The process described in Formulas 1 to 4 is used to reverse-calculate the modified position of the light source. Furthermore, the height-based shadow system 106 determines the modified shadow projection of the pixels for the digital object 706 based on the modified position of the light source, and uses the modified shadow projection to modify the object shadow 704 (e.g., to generate a modified object shadow).

[0144] In one or more embodiments, the height-based shadow system 106 does not track the cursor 720, but rather tracks the shadow pixels selected via the cursor 720, and modifies the object shadow 704 based on the tracked shadow pixels. In effect, the height-based shadow system 106 determines the modified position of the shadow pixels caused by user interaction via the cursor 720 by reversing the above reference... Figure 4B The described calculations determine the modified light position, and the modified light position is used to determine the modified shadow projection of other shadow pixels corresponding to object shadow 704. Therefore, since there is an explicit pixel-level correspondence between digital object 706 and object shadow 704 (e.g., based on the determined shadow projection), the height-based shading system 106 can efficiently and accurately modify the position of object shadow 704.

[0145] In some embodiments, the height-based shadow system 106 modifies the object shadows of multiple digital objects depicted in a digital image based on one or more user interactions with one of the object shadows. For example, in response to detecting a user interaction for repositioning the object shadow of a digital object, the height-based shadow system 106 modifies the object shadows of other digital objects depicted in the same digital image. In fact, as described above, the height-based shadow system 106 determines changes in the light source position based on user interactions for repositioning object shadows. Therefore, the height-based shadow system 106 uses the modified light source position to modify the object shadows of other digital objects. Thus, the height-based shadow system 106 is capable of simultaneously and efficiently modifying multiple object shadows.

[0146] By modifying object shadows within a digital image in response to one or more user interactions with a graphical user interface displaying the digital image, the height-based shadow system 106 operates more flexibly than conventional systems. Indeed, as described above, many conventional systems (such as those employing neural networks to generate object shadows) lack control over the characteristics of the generated object shadows. By modifying the generated object shadows as described above, the height-based shadow system 106 provides flexible control over various characteristics of the object shadows, such as their position, angle, distortion, and / or softness.

[0147] Now go to Figure 8 Additional details regarding the various components and capabilities of the height-based shadowing system 106 will now be provided. Specifically, Figure 8 This illustrates a computing device 800 (e.g., the above reference). Figure 1 The height-based shadow system 106 is implemented by one of the discussed server(s) 102 and / or client devices 110a to 110n. Furthermore, the height-based shadow system 106 is also part of the image editing system 104. As shown, in one or more embodiments, the height-based shadow system 106 includes, but is not limited to, a neural network training engine 802, a neural network application manager 804, an object mask generator 806, an object matting generator 808, a coordinate scheme manager 810, a shadow projection manager 812, a digital image modification engine 814, a graphical user interface manager 816, and a data repository 818 (which includes a height prediction neural network 820, a shadow stylization neural network 822, and training digital images 824).

[0148] As stated above, and as Figure 8As shown, the height-based shadow system 106 includes a neural network training engine 802. In one or more embodiments, the neural network training engine 802 generates (e.g., trains) a neural network to generate object shadows (e.g., soft object shadows) for digital objects depicted in a digital image. For example, in some cases, the neural network training engine 802 generates a height prediction neural network for generating height maps of digital objects depicted in a digital image. Furthermore, in some implementations, the neural network training engine 802 generates a shadow-styling neural network for generating soft object shadows for digital objects.

[0149] like Figure 8 As further shown, the height-based shadow system 106 includes a neural network application manager 804. In one or more embodiments, the neural network application manager 804 implements a neural network generated by the neural network training engine 802. For example, in some cases, the neural network application manager 804 utilizes a height prediction neural network generated by the neural network training engine 802 to generate a height map for digital objects depicted in a digital image. Furthermore, in some implementations, the neural network application manager 804 utilizes a shadow styling neural network generated by the neural network training engine 802 to generate soft object shadows for digital objects.

[0150] In addition, such as Figure 8 As shown, the height-based shading system 106 includes an object mask generator 806. In one or more embodiments, the object mask generator 806 generates an object mask for a digital object depicted in a digital image. For illustration, in some cases, the object mask generator 806 generates an object mask for a digital object depicted in a digital image and provides this object mask to a height prediction neural network employed by a neural network application manager 804 to generate a height map for the digital object. Furthermore, the object mask generator 806 provides the object mask to a shading style neural network employed by the neural network application manager 804 to generate soft object shadows for the digital object.

[0151] like Figure 8 As shown, the height-based shading system 106 also includes an object matting generator 808. In one or more embodiments, the object matting generator 808 generates object mattings for digital objects depicted in a digital image. For example, in some cases, the object matting generator 808 generates object mattings for digital objects depicted in a digital image and provides the object mattings to a height prediction neural network employed by the neural network application manager 804 to generate a height map for the digital objects. In some implementations, the object matting generator 808 utilizes an object mask generated by the object mask generator 806 to generate the object mattings.

[0152] In addition, such as Figure 8 As shown, the height-based shading system 106 includes a coordinate scheme manager 810. In one or more embodiments, the coordinate scheme manager 810 determines and / or manages the coordinate scheme used to generate the height map. For example, in some cases, the coordinate scheme manager 810 determines the coordinate scheme corresponding to a digital image via user input, user settings, or default settings. Furthermore, the coordinate scheme manager 810 provides the coordinate scheme to a height prediction neural network employed by the neural network application manager 804 to generate a height map for the digital object.

[0153] like Figure 8 As shown, the height-based shading system 106 also includes a shadow projection manager 812. In one or more embodiments, the shadow projection manager 812 determines shadow projections for pixels of a digital object depicted in a digital image. For example, in some cases, the shadow projection manager 812 uses a height map generated for the digital object by a height prediction neural network used by the neural network application manager 804 to determine the shadow projection. In some implementations, the shadow projection manager 812 also uses light source information corresponding to the digital image to determine the shadow projection. In some cases, the shadow projection manager 812 utilizes the determined shadow projection to generate shadow pixels corresponding to the pixels of the digital object. Furthermore, in some embodiments, the shadow projection manager 812 uses the determined shadow projection (e.g., using the generated shadow pixels) to generate a hard shadow mask for the digital image.

[0154] In addition, such as Figure 8 As shown, the height-based shadow system 106 includes a digital image modification engine 814. In one or more embodiments, the digital image modification engine 814 modifies a digital image depicting a digital object to include object shadows generated for the digital object. For example, in some cases, the height-based shadow system 106 modifies the digital image to include soft object shadows generated by a neural network application manager 804 using a shadow-styling neural network. In some cases, the digital image modification engine 814 modifies object shadows within the digital image in response to one or more user interactions with the digital object, object shadows, or other user interactions with the digital image.

[0155] In addition, such as Figure 8As shown, the height-based shadow system 106 includes a graphical user interface manager 816. In one or more embodiments, the graphical user interface manager 816 provides a digital image that has been modified to include object shadows generated for digital objects depicted therein, for display within a graphical user interface. Furthermore, in some implementations, the graphical user interface manager 816 provides one or more interactive elements, such as soft controls or distortion controls, to facilitate interaction with the displayed digital image. In some cases, the graphical user interface manager 816 also detects user interaction with the graphical user interface displaying the digital image.

[0156] Furthermore, as shown in the figure, the height-based shading system 106 includes a data repository 818. Specifically, the data repository 818 includes a height prediction neural network 820, a shading style neural network 822, and training digital images 824. In one or more embodiments, the height prediction neural network 820 stores a height prediction neural network generated by a neural network training engine 802 and implemented by a neural network application manager 804 to generate height maps for digital objects depicted in digital images. Similarly, in one or more embodiments, the shading style neural network 822 stores a shading style neural network generated by a neural network training engine 802 and implemented by a neural network application manager 804 to generate soft object shading for digital objects depicted in digital images. Additionally, in some embodiments, the training digital images 824 store training digital images used by the neural network training engine 802 to generate the height prediction neural network and / or the shading style neural network. For example, in some cases, the training digital images 824 store real training images and / or synthetic training images. In some implementations, the training digital images 824 also store corresponding real values ​​used to generate the neural network.

[0157] Each of components 802 to 824 of the height-based shadow system 106 may include software, hardware, or both. For example, components 802 to 824 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices, such as a client device or a server device. When executed by one or more processors, the computer-executable instructions of the height-based shadow system 106 may cause the computing device(s) to perform the methods described herein. Alternatively, components 802 to 824 may include hardware, such as a dedicated processing device for performing a particular function or group of functions. Alternatively, components 802 to 824 of the height-based shadow system 106 may include a combination of computer-executable instructions and hardware.

[0158] Furthermore, components 802 to 824 of the height-based shadow system 106 can be implemented, for example, as one or more operating systems, one or more standalone applications, one or more modules of an application, one or more plugins, one or more library functions or functions that can be called by other applications, and / or cloud computing models. Therefore, components 802 to 824 of the height-based shadow system 106 can be implemented as standalone applications such as desktop or mobile applications. Furthermore, components 802 to 824 of the height-based shadow system 106 can be implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, components 802 to 824 of the height-based shadow system 106 can be implemented in a set of mobile device applications or "applications". For example, in one or more embodiments, the height-based shadow system 106 may include, for example,... AFTER or Digital software applications or those used in conjunction with it. “ADOBE”, “AFTEREFFECTS”, “ILLUSTRATOR”, and “PHOTOSHOP” are registered trademarks or trademarks of Adobe Systems Incorporated in the U.S. and / or other countries.

[0159] Figures 1 to 8 The corresponding text and examples provide a variety of different methods, systems, devices, and non-transient computer-readable media for a height-based shading system 106. In addition to the foregoing, one or more embodiments may be described according to flowcharts including actions for achieving specific results, such as... Figure 9 As shown. Figure 9 It can be performed with more or fewer actions. Furthermore, actions can be performed in different orders. Additionally, the actions described herein can be repeated or performed in parallel with each other, or performed in parallel with different instances of the same or similar actions.

[0160] Figure 9 A flowchart is shown illustrating a series of actions 900 for generating object shadows for digital objects depicted in a digital image, according to one or more embodiments. Although Figure 9 Actions according to one embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify them. Figure 9 Any of the actions shown. In some implementations, Figure 9 The action is performed as part of the method. For example, in some embodiments, Figure 9 The actions are performed within a digital medium environment used for editing digital images as part of a computer-implemented method for generating shadows for the depicted image objects. Alternatively, a non-transitory computer-readable medium may store instructions thereon that, when executed by at least one processor, cause the computing device to perform... Figure 9 In some embodiments, the system performs the following actions. Figure 9 The system includes, for example, in one or more embodiments, at least one storage device comprising a digital image depicting a digital object and a highly predictive neural network. The system also includes components configured to cause the system to perform... Figure 9 At least one server device for the action.

[0161] This series of actions 900 includes action 902 of extracting an object mask from a digital image. For example, in one or more embodiments, action 902 relates to extracting an object mask corresponding to a digital object from a digital image depicting a digital object.

[0162] The series of actions 900 also includes action 904 of generating a height map from an object mask. For example, in some embodiments, action 904 involves using a height prediction neural network to generate a height map from an object mask corresponding to a digital object, the height map including pixel heights for pixels of the digital object.

[0163] In one or more embodiments, the height-based shading system 106 further determines object matting for the digital object and a coordinate scheme corresponding to the digital image. Therefore, in some cases, the height-based shading system 106 generates a height map from the object mask, object matting, and coordinate scheme by utilizing a height prediction neural network, thereby generating a height map from the object mask corresponding to the digital object.

[0164] Furthermore, this series of actions 900 includes action 906, which uses a shadow height map to determine the shadow projection. For illustration, in some implementations, action 906 involves using pixel heights from a height map and light source information for the digital image to determine the shadow projection of pixels for a digital object. In fact, in one or more embodiments, the height-based shadow system 106 uses a height map to determine the shadow projection of digital objects depicted in a digital image. In some cases, the height-based shadow system 106 determines the shadow projection by determining the shadow projection of pixels for the digital object.

[0165] In some embodiments, the height-based shading system 106 determines the position of a corresponding shadow pixel within a digital image for each pixel of a digital object by utilizing the corresponding pixel height from a height map and light source information for the digital image, thereby using the pixel height from the height map and light source information for the digital image to determine the shadow projection of the pixel for the digital object. In one or more embodiments, determining the position of a corresponding shadow pixel within a digital image for each pixel of a digital object using the corresponding pixel height from a height map and light source information includes: determining the vertical position of the corresponding shadow pixel using the corresponding pixel height, the position of the pixel within the digital image, the position of the light source for the digital image, and the position of a horizontal line associated with the digital image; and determining the horizontal position of the corresponding shadow pixel using the position of the pixel within the digital image and the position of the light source for the digital image.

[0166] In some embodiments, the height-based shading system 106 determines the shading projection of pixels for a digital object by utilizing one of a forward tracking projection model or a backward tracking projection model based on a height map.

[0167] In one or more embodiments, using a height map to determine the shadow projection of a digital object depicted in a digital image (e.g., determining the shadow projection for pixels of the digital object) includes determining the shadow projection on a non-planar surface depicted in the digital image (e.g., shadow projection).

[0168] Furthermore, this series of actions 900 includes action 908, which modifies a digital image to include object shadows using shadow projection. For example, in some cases, action 908 involves modifying a digital image to include object shadows for digital objects using shadow projection of pixels for digital objects. In fact, in one or more embodiments, the height-based shading system 106 uses shadow projection to modify a digital image to include object shadows for digital objects. Therefore, in some cases, the height-based shading system 106 uses shadow projection for digital objects by using shadow projection of pixels for digital objects.

[0169] In one or more embodiments, modifying a digital image to include object shadows for digital objects using shadow projection (e.g., using shadow projection for pixels of a digital object) includes: modifying the digital image to include object shadows projected from the digital object and across a non-planar surface.

[0170] In some implementations, modifying a digital image to include object shadows for digital objects includes modifying the digital image to include soft object shadows associated with the digital objects. In fact, in one or more embodiments, the height-based shadowing system 106 uses shadow projections of pixels for the digital object to generate soft object shadows for the digital object. Therefore, in some cases, the height-based shadowing system 106 modifies the digital image to include object shadows for the digital object by modifying the digital image to include soft object shadows for the digital object. In one or more embodiments, the height-based shadowing system 106 generates soft object shadows for the digital object by utilizing a shadow-stylizing neural network to generate soft object shadows.

[0171] In some cases, the height-based shading system 106 uses shadow projection of pixels for the digital object to generate a hard shadow mask for the digital object. Therefore, in one or more embodiments, the height-based shading system 106 generates a soft object shadow for the digital object by using a hard shadow mask corresponding to the digital object and an object mask, thereby generating a soft object shadow for the digital object using shadow projection of pixels for the digital object.

[0172] In one or more embodiments, the series of actions 900 further includes actions for providing a digital image for display within a graphical user interface and / or modifying object shadows generated for a digital object based on one or more user interactions detected via the graphical user interface. For example, in one or more embodiments, these actions involve providing a digital image with object shadows for display within a graphical user interface of a client device; detecting one or more user interactions via the graphical user interface to reposition a digital object within the digital image; and modifying the object shadows within the digital image based on one or more user interactions to reposition the digital object. In some cases, these actions include providing a digital image with soft object shadows for display within a graphical user interface of a client device; detecting one or more user interactions via the graphical user interface with a softness control corresponding to the soft object shadow; and modifying the softness of the soft object shadow based on one or more user interactions with the softness control.

[0173] In some embodiments, these actions include detecting user interaction of vertically moving a digital object within a digital image via a graphical user interface of a client device displaying a digital image with object shadows; generating a modified pixel height for a height map by adding a shift value to the pixel height based on the user interaction of vertically moving the digital object within the digital image; and modifying the object shadow within the digital image using the modified pixel height. In some implementations, these actions include providing a digital image with object shadows for display within a graphical user interface of a client device; detecting one or more user interactions via the graphical user interface for repositioning the object shadow within the digital image; and modifying the object shadow within the digital image based on one or more user interactions for repositioning the object shadow. In some cases, the height-based shadow system 106 modifies an additional object shadow corresponding to an additional digital object depicted in the digital image based on one or more user interactions for repositioning the object shadow corresponding to the digital object.

[0174] For illustration, in one or more embodiments, a height-based shading system 106 extracts an object mask corresponding to a digital object from a digital image; generates a height map from the object mask corresponding to the digital object using a height prediction neural network, the height map including pixel heights for pixels of the digital object within the image space; determines the position of a corresponding shadow pixel within the digital image for the pixel of the digital object by utilizing the corresponding pixel height from the height map, the position of a light source for the digital image, and the projection of the light source onto a horizontal line associated with the digital image, thereby determining a shadow projection for the pixel of the digital object; and modifies the digital image to include object shadows for the digital object using the shadow projection for the pixel of the digital object.

[0175] In some cases, the height-based shading system 106 modifies a digital image to include object shadows for digital objects by using shadow projections of pixels for digital objects: generating soft object shadows for digital objects based on shadow projections of pixels for digital objects using a shadow stylization neural network; and modifying the digital image to include soft object shadows for digital objects. In some embodiments, generating soft object shadows for digital objects based on shadow projections using a shadow stylization neural network includes generating soft object shadows for digital objects based on shadow projections and softness values ​​determined via user input using a shadow stylization neural network.

[0176] Furthermore, in one or more embodiments, a height-based shadowing system 106 determines a two-dimensional object matting for a digital object and a two-dimensional coordinate scheme corresponding to the digital image; and generates a height map from the object mask, the two-dimensional object matting, and the two-dimensional coordinate scheme by utilizing a height prediction neural network, thereby generating a height map from the object mask corresponding to the digital object using the height prediction neural network.

[0177] Embodiments of this disclosure may include or utilize a dedicated or general-purpose computer including computer hardware (such as one or more processors and system memory), as discussed in more detail below. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Specifically, one or more processes described herein may be implemented at least in part as instructions implemented in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Generally, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory) and executes those instructions to perform one or more processes, including one or more processes described herein.

[0178] Computer-readable media can be any available medium accessible by a general-purpose or special-purpose computer system. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinct types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0179] Non-transient computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code components in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0180] "Network" is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately considers that connection as a transmission medium. Transmission media may include networks and / or data links, which may be used to carry desired program code components in the form of computer-executable instructions or data structures, and may be accessible by general-purpose or special-purpose computers. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0181] Furthermore, upon arrival at various computer system components, program code components in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to a non-transitory computer-readable storage medium (device) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or a less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0182] Computer-executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. In some embodiments, executing the computer-executable instructions on a general-purpose computer transforms the general-purpose computer into a special-purpose computer that implements the elements of this disclosure. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0183] Those skilled in the art will understand that this disclosure can be implemented in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. This disclosure can also be implemented in distributed system environments, where both local and remote computer systems, linked via a network (via a hardwired data link, a wireless data link, or a combination of hardwired and wireless data links), perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0184] Embodiments of this disclosure can also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization, released with minimal management effort or service provider interaction, and then scaled accordingly.

[0185] Cloud computing models can be composed of various characteristics, such as on-demand self-service, widespread network access, resource pooling, rapid elasticity, and measurable services. Cloud computing models can also exhibit various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Cloud computing models can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, and hybrid clouds. In this specification and claims, a “cloud computing environment” means an environment in which cloud computing is employed.

[0186] Figure 10 A block diagram of an example computing device 1000 is shown, which can be configured to perform one or more of the processes described above. It is understood that one or more computing devices, such as computing device 1000, can represent the aforementioned computing devices (e.g., servers 102 and / or client devices 110a to 110n). In one or more embodiments, computing device 1000 can be a mobile device (e.g., mobile phone, smartphone, PDA, tablet computer, laptop computer, camera, tracker, watch, wearable device). In some embodiments, computing device 1000 can be a non-mobile device (e.g., desktop computer or another type of client device). Furthermore, computing device 1000 can be a server device including cloud-based processing and storage capabilities.

[0187] like Figure 10 As shown, computing device 1000 may include one or more processors 1002, memory 1004, storage device 1006, input / output interface 1008 (or "I / O interface 1008"), and communication interface 1010, which may be communicatively coupled via a communication infrastructure (e.g., bus 1012). Although Figure 10 The image shows a computing device 1000, but... Figure 10 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in some embodiments, the computing device 1000 includes more than Figure 10 The components shown are fewer than those in the diagram. A more detailed description will follow. Figure 10 The components of the computing device 1000 shown.

[0188] In a specific embodiment, processor(s) 1002 includes hardware for executing instructions, such as those constituting a computer program. By way of example and not limitation, in order to execute instructions, processor(s) 1002 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1004, or storage device 1006, and decode and execute them.

[0189] Computing device 1000 includes memory 1004 coupled to processor(s) 1002. Memory 1004 can be used to store data, metadata, and programs for execution by processor(s). Memory 1004 may include one or more of volatile and non-volatile memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state drive (“SSD”), flash memory, phase-change memory (“PCM”), or other types of data storage. Memory 1004 may be internal memory or distributed memory.

[0190] Computing device 1000 includes storage device 1006, which includes a storage device for storing data or instructions. By way of example and not limitation, storage device 1006 may include the aforementioned non-transient storage media. Storage device 1006 may include hard disk drives (HDDs), flash memory, universal serial bus (USB) drives, or combinations of these or other storage devices.

[0191] As shown in the figure, computing device 1000 includes one or more I / O interfaces 1008 provided to allow a user to provide input (such as user strokes) to computing device 1000, receive output from computing device 1000, and otherwise transmit data to and from computing device 1000. These I / O interfaces 1008 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O interfaces 1008. Touchscreens can be activated with a stylus or finger.

[0192] I / O interface 1008 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O interface 1008 is configured to provide graphical data to the display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0193] The computing device 1000 may also include a communication interface 1010. The communication interface 1010 may include hardware, software, or both. The communication interface 1010 provides one or more interfaces for communication (e.g., packet-based communication) between the computing device and one or more other computing devices or one or more networks. By way of example and not limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi. The computing device 1000 may also include a bus 1012. The bus 1012 may include hardware, software, or both for connecting components of the computing device 1000 to each other.

[0194] In the foregoing description, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention have been described with reference to the details discussed herein, and various embodiments are illustrated in the accompanying drawings. The foregoing description and drawings are illustrative of the invention and should not be construed as limiting the invention. Numerous specific details have been described to provide a thorough understanding of various embodiments of the invention.

[0195] This invention may be practiced in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered illustrative rather than restrictive in all respects. For example, the methods described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in a different order. Furthermore, the steps / actions described herein may be repeated or performed in parallel with each other, or performed in parallel with different instances of the same or similar steps / actions. Therefore, the scope of the invention is indicated by the appended claims, not by the foregoing description. All modifications within the meaning and equivalent scope of the claims should be included within its scope.

Claims

1. A method for generating shadows, comprising: Extract the object mask corresponding to the digital object from the digital image depicting the digital object; A height map is generated from an object mask corresponding to the digital object using a height prediction neural network, the height map including the pixel heights of the pixels for the digital object; Using the pixel height for the pixel in the height map, determine the vertical projection of the pixel of the digital object onto the surface depicted in the digital image; By determining the intersection point between the first line and the second line as the position of the shadow pixel corresponding to the pixel of the digital object, the shadow projection of the pixel of the digital object is determined using the pixel height from the height map and the light source for the digital image. The first line includes the light source and the pixel, and the second line includes the vertical projection of the light source on the horizontal line of the digital image and the vertical projection of the pixel on the surface determined using the height map. as well as The digital image is modified using the shadow projection of the pixels of the digital object to include object shadows for the digital object.

2. The method of claim 1, wherein modifying the digital image to include the object shadow for the digital object comprises: Modify the digital image to include soft object shadows associated with the digital objects.

3. The method according to claim 1, further comprising: The digital image with the shadow of the object is provided for display within the graphical user interface of the client device; One or more user interactions that reposition the digital object within the digital image are detected via the graphical user interface. as well as The shadow of the object within the digital image is modified based on one or more user interactions that reposition the digital object.

4. The method according to claim 1, wherein: Determining the shadow projection of the pixel depicted in the digital image for the digital object using the pixel height from the height map includes: determining the shadow projection on a non-planar surface depicted in the digital image; and Modifying the digital image to include the object shadow for the digital object using the shadow projection of the pixels of the digital object includes: modifying the digital image to include the object shadow projected from the digital object and across the non-planar surface.

5. A non-transitory computer-readable medium having executable instructions stored thereon, the instructions, when executed by a processing device, causing the processing device to perform an operation, the operation comprising: Extract the object mask corresponding to the digital object from the digital image depicting the digital object; A height map is generated from an object mask corresponding to the digital object using a height prediction neural network, the height map including pixel heights for pixels of the digital object; Using the pixel height for the pixel in the height map, determine the vertical projection of the pixel of the digital object onto the surface depicted in the digital image; By determining the intersection point between the first line and the second line as the position of the shadow pixel corresponding to the pixel of the digital object, the shadow projection of the pixel of the digital object is determined using the pixel height from the height map and the light source for the digital image. The first line includes the light source and the pixel, and the second line includes the vertical projection of the light source on the horizontal line of the digital image and the vertical projection of the pixel on the surface determined using the height map. as well as The digital image is modified to include object shadows for the digital object using the shadow projection of the pixels of the digital object.

6. The non-transient computer-readable medium according to claim 5, wherein: The operation further includes: generating a soft object shadow for the digital object using the shadow projection for the pixels of the digital object, and Modifying the digital image to include the object shadow for the digital object includes: modifying the digital image to include the soft object shadow for the digital object.

7. The non-transient computer-readable medium of claim 6, wherein generating the soft object shadow for the digital object comprises: The shadows of the soft objects are generated using a shadow stylization neural network.

8. The non-transient computer-readable medium according to claim 6, wherein: The operation further includes: generating a hard shadow mask for the digital object using the shadow projection of the pixels of the digital object; and Generating a soft object shadow for a digital object using the shadow projection of the pixels of the digital object includes: generating the soft object shadow for the digital object using a hard shadow mask corresponding to the digital object and the object mask.

9. The non-transient computer-readable medium of claim 6, wherein the operation further comprises: A digital image with the soft object shadow is provided for display within the graphical user interface of the client device; Detect one or more user interactions with soft controls corresponding to the soft object shadow via the graphical user interface; as well as The softness of the soft object shadow is modified based on one or more user interactions with the softness control.

10. The non-transient computer-readable medium of claim 5, wherein the operation further comprises: User interaction involving vertical movement of the digital object within the digital image is detected via the graphical user interface of a client device displaying the digital image with the object's shadow. By using the user interaction based on vertically moving the digital object within the digital image, a shift value is added to the pixel height, thereby generating a modified pixel height for the height map; as well as The modified pixel height is used to modify the object shadow within the digital image.

11. The non-transient computer-readable medium according to claim 5, wherein: The operation further includes: determining object matting for the digital object and a coordinate scheme corresponding to the digital image; and Generating the height map from the object mask corresponding to the digital object using the height prediction neural network includes: generating the height map from the object mask, the object matting, and the coordinate scheme using the height prediction neural network.

12. The non-transient computer-readable medium of claim 5, wherein the operation further comprises: The position of the horizontal line in the digital image is determined by receiving user input indicating the position of the horizontal line via a client device displaying the digital image.

13. The non-transient computer-readable medium of claim 5, wherein the operation further comprises: The position of the horizontal line in the digital image is determined based on one or more shadows that already exist within the digital image.

14. A system for generating shadows, comprising: At least one memory component, including: At least one processing device coupled to the at least one memory component, the at least one processing device being configured to perform operations including: Extract the object mask corresponding to the digital object depicted in the digital image from the digital image; A height map is generated from an object mask corresponding to the digital object using a height prediction neural network, the height map including the pixel height of the pixels of the digital object in image space; Using the pixel height for the pixel in the height map, determine the vertical projection of the pixel of the digital object onto the surface depicted in the digital image; The shadow projection of the pixel of the digital object is determined by determining the position of the corresponding shadow pixel within the digital image based on the intersection of a first line and a second line. The first line includes a light source and the pixel for the digital image, and the second line includes the vertical projection of the light source onto a horizontal line of the digital image and the vertical projection of the pixel onto a surface determined using the height map. The digital image is modified to include object shadows for the digital object using the shadow projection of the pixels of the digital object.

15. The system of claim 14, wherein determining the shadow projection of the pixel for the digital object comprises: The shadow projection of the pixel for the digital object is determined based on the height map using either a forward tracking projection model or a backward tracking projection model.

16. The system of claim 14, wherein the operation further comprises: The digital image with the shadow of the object is provided for display within the graphical user interface of the client device; One or more user interactions for repositioning the object shadow within the digital image are detected via the graphical user interface. as well as The object shadow within the digital image is modified based on one or more user interactions that reposition the object shadow.

17. The system of claim 16, wherein the operation further comprises: The system modifies the additional object shadow corresponding to the additional digital object depicted in the digital image based on one or more user interactions that reposition the object shadow corresponding to the digital object.

18. The system of claim 14, wherein modifying the digital image to include the object shadow for the digital object using the shadow projection of the pixels for the digital object comprises: Using a shadow stylization neural network, a soft object shadow is generated for the digital object based on the shadow projection for the pixel of the digital object; as well as Modify the digital image to include the soft object shadow for the digital object.

19. The system of claim 18, wherein generating the soft object shadow for the digital object based on the shadow projection using the shadow stylization neural network comprises: Using the shadow stylization neural network, a soft object shadow is generated for the digital object based on the shadow projection and a softness value determined via user input.

20. The system of claim 14, wherein: The operation also includes determining a two-dimensional object matting for the digital object and a two-dimensional coordinate scheme corresponding to the digital image; as well as Generating the height map from the object mask corresponding to the digital object using the height prediction neural network includes: generating the height map from the object mask, the two-dimensional object matting, and the two-dimensional coordinate scheme using the height prediction neural network.

Images