Modifying two-dimensional images utilizing iterative three-dimensional meshes of the two-dimensional images

Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for generating three-dimensional meshes representing two-dimensional images for editing the two dimensional images. The disclosed system utilizes a first neural network to determine density values of pixels of a two-dimensional image based on estimated disparity. The disclosed system samples points in the two-dimensional image according to the density values and generates a tessellation based on the sampled points. The disclosed system utilizes a second neural network to estimate camera parameters and modify the three-dimensional mesh based on the estimated camera parameters of the pixels of the two-dimensional image. In one or more additional embodiments, the disclosed system generates a three-dimensional mesh to modify a two dimensional image according to a displacement input. Specifically, the disclosed system maps the three-dimensional mesh to the two-dimensional image, modifies the three-dimensional mesh in response to a displacement input, and updates the two-dimensional image. 2023229591 15 Sep 2023

Description

[0120] In alternative embodiments, instead of changing the position of the displacement to a new position and reverting the displacement at the original position, the depth displacement system 102, the depth displacement system 102 applies the displacement to all areas of the threedimensional mesh 1008b and the two-dimensional image 1004b along a path of movement of the displacement input. To illustrate, the depth displacement system 102 determines whether to move the displacement or apply the displacement along a path based on a selected setting associated with the displacement tool 1006. Accordingly, the depth displacement system 102 applies the displacement similar to a brushstroke along the path based on a selected setting of the displacement tool 1006.

[0121] As mentioned, the depth displacement system 102 also provides a displacement tool with additional attributes that allow for greater control over the shape and / or displacement associated with a displacement input. FIG. 10G illustrates the client device 1000 displaying a twodimensional image 1004c in response to an additional attribute associated with the displacement input. For instance, the depth displacement system 102 determines that the displacement input moves in an additional direction to modify a selected portion 1028 of the two-dimensional image 1004c. To illustrate, in response to a horizontal movement of the displacement input (e.g., a perpendicular movement to the original movement), the depth displacement system 102 modifies a radius or shape associated with the displacement input.

[0122] In connection with the additional attribute associated with the displacement input, FIG. 10H illustrates a three-dimensional mesh 1008c corresponding to the two-dimensional image 1004c. In particular, based on the additional attribute of the displacement input, the depth 2023229591   15 Sep 2023 displacement system 102 determines a selected portion 1030 of the three-dimensional mesh 1008c corresponding to the selected portion 1028 of the two-dimensional image 1004c. In one or more embodiments, the depth displacement system 102 modifies a radius, shape, or other attribute of the displaced portion according to the additional attribute of the displacement input. To illustrate, the depth displacement system 102 flattens the displaced portion while displacing a greater selection of vertices of the three-dimensional mesh 1008c. As shown in FIG. 10G, the client device 1000 displays the two-dimensional image 1004c with modifications based on the displaced portion of the three-dimensional mesh 1008c.

[0123] In one or more embodiments, the depth displacement system 102 also provides tools for generating animations via a displacement tool. FIGS. 11A-1 IC illustrate graphical user interfaces for generating an animation of displacement operations in connection with a two-dimensional image. Specifically, as illustrated in FIGS. 11A-11C, the depth displacement system 102 determines one or more displacement inputs to displace one or more portions of a two-dimensional image and generates an animation from the one or more displacement inputs.

[0124] FIG. 11A illustrates a graphical user interface of a client device 1100 including an image editing application 1102. For example, as illustrated, the client device 1100 displays a first twodimensional image 1104 including a modified portion based on a displacement input. In particular, the depth displacement system 102 displaces / warps a portion of a bridge within the first twodimensional image 1104 based on displacement settings associated with a displacement input. FIG. 11B illustrates the client device 1100 displaying a second two-dimensional image 1106 including a different modified portion based on the same displacement input or a different displacement input. 2023229591   15 Sep 2023

[0125] In one or more embodiments, the depth displacement system 102 generates an animation based on the first two-dimensional image 1104 and the second two-dimensional image 1106. For instance, the depth displacement system 102 generates an animation including a plurality of video frames to show how the first two-dimensional image 1104 changes into the second twodimensional image 1106. According to one or more embodiments, the depth displacement system 102 records (e.g., as individual video frames) movement of a displaced portion from the position of the first two-dimensional image 1104 to the position of the second two-dimensional image 1106, resulting in a digital video that shows a ripple effect across the bridge in the images.

[0126] In alternative embodiments, the depth displacement system 102 utilizes the first twodimensional image 1104 and the second two-dimensional image 1106 to predict a plurality of images between the two images. Specifically, rather than recording a plurality of displacement modifications, the depth displacement system 102 utilizes the underlying three-dimensional mesh to interpolate a plurality of displaced portions. For example, the depth displacement system 102 utilizes a first displaced portion and a second displaced portion of a three-dimensional mesh to generate estimated positions of a plurality of displaced portions between the first displaced portion and the second displaced portion.

[0127] As an example, the depth displacement system 102 utilizes the first displaced portion of the first two-dimensional image 1104 and the second displaced portion of the second twodimensional image 1106 to interpolate a plurality of displaced portions between the first displaced portion and the second displaced portion. To illustrate, the depth displacement system 102 determines a plurality of displaced portions of the three-dimensional mesh corresponding to the bridge based on the settings of the displacement input and the positions of the first and second portions. Accordingly, the depth displacement system 102 automatically generates an animation 2023229591   15 Sep 2023 including the indicated displaced portions and the estimated displaced portions. In some instances, the depth displacement system 102 also provides options to modify the animation by automatically changing attributes of the indicated portions and the estimated portions without requiring additional displacement inputs (e.g., by computationally generating the displacements based on the selected settings and the three-dimensional mesh).

[0128] FIG. 11C illustrates that the client device 1100 displays an animation generated based on the first two-dimensional image 1104 and the second two-dimensional image 1106. For instance, the client device 1100 displays a window 1108 including a digital video generated by the depth displacement system 102 based on the animation. Additionally, as illustrated in FIG. 1 IC, the client device 1100 provides a save option 1110 for saving the digital video including the animation.

[0129] In one or more embodiments, the depth displacement system 102 determines a displacement direction of a displacement input according to one or more settings of a displacement tool. FIGS. 12A-12C illustrate embodiments in which the depth displacement system 102 determines a displacement direction based on one or more attributes of a displacement input and / or based on a geometry of a three-dimensional mesh representing a two-dimensional image. Specifically, FIG. 12A illustrates a graphical user interface in connection with automatically determining a displacement direction. FIGS. 12B-12C illustrate determining one or more displacement directions relative to vertices of a three-dimensional mesh.

[0130] As mentioned, in one or more embodiments, the depth displacement system 102 determines a displacement direction based on a selected setting associated with a displacement tool. For example, FIG. 12A illustrates a graphical user interface of a client device 1200 including an image editing application 1202 for modifying a two-dimensional image 1204. In particular, the 2023229591   15 Sep 2023 client device 1200 displays a direction setting 1206 for selecting a predetermined direction of a displacement operation. As illustrated in FIG. 12A, the depth displacement system 102 dynamically determines a displacement direction based on the direction setting 1206 according to a selected surface.

[0131] More specifically, the depth displacement system 102 determines a direction to displace a selected portion of a two-dimensional image based on a surface associated with the selected portion. In one or more embodiments, the depth displacement system 102 determines a displacement direction based on a normal of a selected surface. For example, as FIG. 12A illustrates, the depth displacement system 102 determines a displacement direction based on a surface of a selected portion 1208 of the two-dimensional image 1204. To illustrate, the depth displacement system 102 determines that the selected portion 1208 includes a surface with an overall normal pointing in a particular direction (e.g., based on the selected portion 1208 including a plurality of building faces). For example, in one or more embodiments, the depth displacement system 102 determines the overall normal of the surface based on an average normal position of polygons of the corresponding three-dimensional mesh pointing in a specific direction.

[0132] As mentioned, in one or more embodiments, the depth displacement system 102 determines displacement directions according to a single predetermined direction. FIG. 12B illustrates a plurality of connected vertices in a selected portion of a three-dimensional mesh. As illustrated in FIG. 12B, the depth displacement system 102 determines a displacement direction 1210 for each of the vertices based on a setting indicating the displacement direction 1210 and / or based on an overall surface to which each vertex belongs (e.g., as in FIG. 12A). Accordingly, the depth displacement system 102 displaces each of the vertices (e.g., a first vertex 1212a and a second vertex 1212b) in the displacement direction 1210 according to a displacement input. 2023229591   15 Sep 2023 Furthermore, in one or more embodiments, the depth displacement system 102 determines a displacement distance to modify the vertices according to a selected height fdter, shape, radius, and / or height of the displacement input and / or based on a distance of each vertex to an initial displacement input point.

[0133] In one or more alternative embodiments, the depth displacement system 102 dynamically determines a specific displacement direction for each vertex a selected portion of a three-dimensional mesh. For example, FIG. 12C illustrates that the depth displacement system 102 determines a plurality of different displacement directions for a plurality of vertices in a selected portion of a three-dimensional mesh. To illustrate, the depth displacement system 102 determines a normal for each vertex in the selected portion and determines the displacement direction for the vertex based on the normal. In particular, FIG. 12C illustrates that the selected portion includes a first vertex 1214a and a second vertex 1214b. The depth displacement system 102 determines the displacement directions based on a normal for each vertex, resulting in a first displacement direction 1216a for the first vertex 1214a and a second displacement direction 1216b for the second vertex 1214b.

[0134] By determining displacement directions based on predetermined directions, vertex normals, or surface directions, the depth displacement system 102 provides flexible and dynamic displacement of two-dimensional images. For example, the depth displacement system 102 provides displacement of selected portions of two-dimensional images by raising, lowering, flattening, inflating / deflating, or otherwise modifying objects within two-dimensional images by displacing individual vertices of the corresponding three-dimensional mesh. Additionally, the depth displacement system 102 thus provides a variety of different ways to interact with / modify 2023229591   15 Sep 2023 objects within a two-dimensional image in a way that is consistent with a three-dimensional representation of the two-dimensional image.

[0135] In one or more embodiments, the depth displacement system 102 also provides segmentation of a three-dimensional mesh corresponding to a two-dimensional image based on objects in the two-dimensional image. In particular, the depth displacement system 102 detects one or more objects in a two-dimensional image and segments a three-dimensional mesh based on the detected object(s). FIG. 13 illustrates an overview of the depth displacement system 102 generating and segmenting a three-dimensional mesh representing a two-dimensional image based on objects in the two-dimensional image.

[0136] As illustrated in FIG. 13, the depth displacement system 102 determines a twodimensional image 1300. For example, the two-dimensional image 1300 includes a plurality of objects—e.g., one or more objects in a foreground region and / or one or more objects in a background region. In one or more embodiments, the depth displacement system 102 also generates an initial three-dimensional mesh 1302 for the two-dimensional image 1300 utilizing the adaptive tessellation operations described above. Accordingly, the initial three-dimensional mesh 1302 includes a connected tessellation of polygons representing the objects in the two-dimensional image 1300.

[0137] In one or more embodiments, the depth displacement system 102 generates a segmented three-dimensional mesh 1304. Specifically, the depth displacement system 102 utilizes information about the objects to segment the initial three-dimensional mesh 1302 into a plurality of separate three-dimensional object meshes corresponding to the objects. FIGS. 14-15 and the corresponding description provide additional detail with respect to segmenting a three-dimensional mesh representing a two-dimensional image. Additionally, FIGS. 16A-16B and the corresponding 2023229591   15 Sep 2023 description provide additional detail with respect to modifying / displacing portions of a twodimensional image based on a segmented three-dimensional mesh.

[0138] According to one or more embodiments, the depth displacement system 102 detects objects or object boundaries in a two-dimensional image for generating a three-dimensional mesh. FIG. 14 illustrates an embodiment in which the depth displacement system 102 utilizes a semantic map to segment a three-dimensional mesh. In particular, the depth displacement system 102 determines the semantic map to indicate separate objects within a two-dimensional image for generating separate three-dimensional object meshes in a corresponding three-dimensional mesh.

[0139] As illustrated in FIG. 14, the depth displacement system 102 determines a twodimensional image 1400 including a plurality of separate objects. To illustrate, the twodimensional image 1400 includes the image of a car parked on a road against a scenic overlook. In one or more embodiments, the depth displacement system 102 generates a semantic map 1402 based on the two-dimensional image 1400. For example, the depth displacement system 102 utilizes a semantic segmentation neural network (e.g., an object detection model, a deep learning model) to automatically label pixels of the two-dimensional image 1400 into object classifications based on the detected objects in the two-dimensional image 1400. The depth displacement system 102 can utilize a variety of models or architectures to determine object classifications and image segmentations. To illustrate, the depth displacement system 102 utilizes a segmentation neural network as described in U.S. Patent No. 10,460,214, filed October 31, 2017, titled “Deep salient content neural networks for efficient digital object segmentation,” which is herein incorporated by reference in its entirety. Additionally, the depth displacement system 102 generates the semantic map 1402 including the object classifications of the pixels of the two-dimensional image 1400. 2023229591   15 Sep 2023

[0140] In one or more embodiments, the depth displacement system 102 utilizes the semantic map 1402 to generate a segmented three-dimensional mesh 1404. Specifically, the depth displacement system 102 utilizes the object classifications of the pixels in the two-dimensional image 1400 to determine portions of a three-dimensional mesh that correspond to the objects in the two-dimensional image 1400. For example, the depth displacement system 102 utilizes a mapping between the two-dimensional image 1400 and the three-dimensional mesh representing the two-dimensional image 1400 to determine object classifications of portions of the threedimensional mesh. To illustrate, the depth displacement system 102 determines specific vertices of the three-dimensional mesh that correspond to a specific object detected in the two-dimensional image 1400 based on the mapping between the two-dimensional image 1400 and the twodimensional image.

[0141] In one or more embodiments, in response to determining that different portions of a three-dimensional mesh associated with a two-dimensional image correspond to different objects, the depth displacement system 102 segments the three-dimensional mesh. In particular, the depth displacement system 102 utilizes the object classification information associated with portions of the three-dimensional mesh to separate the three-dimensional mesh into a plurality of separate three-dimensional object meshes. For instance, the depth displacement system 102 determines that a portion of the three-dimensional mesh corresponds to the car in the two-dimensional image 1400 and separates the portion of the three-dimensional mesh corresponding to the car from the rest of the three-dimensional mesh.

[0142] Accordingly, in one or more embodiments, the depth displacement system 102 segments a three-dimensional mesh into two or more separate meshes corresponding to a two-dimensional image. To illustrate, the depth displacement system 102 generates the segmented three- 2023229591   15 Sep 2023 dimensional mesh 1404 by separating the two-dimensional image 1400 into a plurality of separate three-dimensional object meshes in the scene. For example, the depth displacement system 102 generates a three-dimensional object mesh corresponding to the car, a three-dimensional object mesh corresponding to the road, one or more three-dimensional object meshes corresponding to the one or more groups of trees, etc.

[0143] In additional embodiments, the depth displacement system 102 segments a threedimensional mesh based on a subset of objects in a two-dimensional image. To illustrate, the depth displacement system 102 determines one or more objects in the two-dimensional image 1400 for segmenting the three-dimensional mesh. For example, the depth displacement system 102 determines one or more objects in a foreground of the two-dimensional image 1400 for generating separate three-dimensional object meshes. In some embodiments, the depth displacement system 102 determines a prominence (e.g., proportional size) of the objects for generating separate threedimensional object meshes. In one or more embodiments, the depth displacement system 102 determines one or more objects in response to a selection of one or more objects (e.g., a manual selection of the car in the two-dimensional image 1400 via a graphical user interface displaying the two-dimensional image).

[0144] According to one or more embodiments, the depth displacement system 102 segments a three-dimensional mesh based on discontinuities of depth in a two-dimensional image and / or in the three-dimensional mesh. FIG. 15 illustrates that the depth displacement system 102 determines a two-dimensional image 1500 including a plurality of separate objects. In connection with the two-dimensional image 1500, the depth displacement system 102 generates a three-dimensional mesh representing the objects in the two-dimensional image 1500. 2023229591   15 Sep 2023

[0145] Furthermore, in one or more embodiments, the depth displacement system 102 determines depth discontinuities 1502 based on differences in depth in the two-dimensional image 1500 and / or the three-dimensional mesh. Specifically, the depth displacement system 102 determines one or more portions of a three-dimensional mesh that indicate sharp changes in depth. For instance, the depth displacement system 102 determines that edges of the car in the threedimensional mesh have depth discontinuities relative to the sky, road, and / or other background elements.

[0146] In one or more embodiments, the depth displacement system 102 generates a segmented three-dimensional mesh 1504 based on the depth discontinuities 1502. In response to detecting the depth discontinuities 1502, the depth displacement system 102 determines that the depth discontinuities 1502 indicate separate objects. To illustrate, the depth displacement system 102 detects separate objects based on depth discontinuities that exceed a specific threshold. More specifically, the depth displacement system 102 generates the segmented three-dimensional mesh 1504 by separating / slicing the three-dimensional mesh at the locations with the depth discontinuities 1502 into a plurality of separate three-dimensional object meshes.

[0147] In one or more embodiments, in addition to generating separate three-dimensional object meshes for separate objects in a three-dimensional mesh that represents a two-dimensional image, the depth displacement system 102 utilizes a neural network to fill in portions of the threedimensional mesh created by slicing the three-dimensional mesh. For example, in response to generating the segmented three-dimensional mesh 1504 by separating the portion of the threedimensional mesh corresponding to the car from the rest of the three-dimensional mesh, the depth displacement system 102 fills in a portion of the rest of the three-dimensional mesh resulting from segmenting the three-dimensional mesh. To illustrate, the depth displacement system 102 inserts 2023229591   15 Sep 2023 a plurality of vertices to connect the missing portions, such as by interpolating or otherwise generating surfaces in the missing portions (e.g., via content-aware filling).

[0148] In additional embodiments, the depth displacement system 102 utilizes information about object classes corresponding to segmented portions to fill connect one or more portions of a three-dimensional object mesh. For example, the depth displacement system 102 determines that a three-dimensional object mesh segmented from a three-dimensional mesh corresponds to a human or a body part of a human. The depth displacement system 102 utilizes information associated with the detected class of object (e.g., human or arm) to complete the three-dimensional object mesh for areas of the three-dimensional object mesh not visible in the two-dimensional image (e.g., by connecting a front portion of a mesh representing an arm through a backside of an arm not visible in the two-dimensional image).

[0149] FIGS. 16A-16B illustrate a plurality of graphical user interfaces of a client device 1600 including an image editing application 1602. As illustrated in FIGS. 16A-16B, the depth displacement system 102 provides tools for modifying individual objects in a two-dimensional image 1604 according to a displacement three-dimensional mesh representing the two-dimensional image 1604. More specifically, the depth displacement system 102 automatically segments the three-dimensional mesh based on objects in the two-dimensional image 1604. Accordingly, the depth displacement system 102 provides tools for modifying individual objects by segmenting the three-dimensional mesh into a plurality of three-dimensional object meshes.

[0150] For example, FIG. 16A illustrates that the client device 1600 displays the twodimensional image 1604 including a plurality of objects for editing via a displacement tool. In one or more embodiments, the depth displacement system 102 generates a three-dimensional mesh including one or more three-dimensional object meshes corresponding to one or more of the 2023229591   15 Sep 2023 objects in the two-dimensional image 1604. For example, the depth displacement system 102 segments the three-dimensional mesh corresponding to the two-dimensional image 1604 by generating a separate three-dimensional object mesh for the rock 1606 in the two-dimensional image 1604. To illustrate, the depth displacement system 102 generates a three-dimensional object mesh representing the rock in three-dimensional space by separating the corresponding portion from the initial three-dimensional mesh generated for the two-dimensional image 1604.

[0151] In one or more embodiments, the client device 1600 detects a displacement input to displace the rock 1606 (or a portion of the rock 1606) without modifying the other portions of the two-dimensional image 1604. As illustrated in FIG. 16B, the depth displacement system 102 determines attributes of the displacement input (e.g., based on the position and / or other settings associated with the displacement input). In response to modifying the selected portion, the client device 1600 displays a modified two-dimensional image 1604a including a modified rock 1606a based on the displacement input. For example, the depth displacement system 102 displaces portions of the modified rock 1606a in one or more directions associated with the displacement input and / or based on vertices of the three-dimensional object mesh corresponding to the modified rock 1606a. Thus, the client device 1600 displays that the depth displacement system 102 generates the modified rock 1606a in the modified two-dimensional image 1604a according to modifications of the three-dimensional object mesh representing the rock.

[0152] In additional embodiments, as mentioned, the depth displacement system 102 provides iterative modification and tessellation of a two-dimensional image. FIG. 17 illustrates an embodiment of the depth displacement system 102 iteratively modifying and tessellating a twodimensional image. Specifically, the depth displacement system 102 generates a new tessellation representing a two-dimensional image in response to modifying the two-dimensional image. 2023229591   15 Sep 2023

[0153] In one or more embodiments, as illustrated in FIG. 17, the depth displacement system 102 determines a two-dimensional image 1700 including one or more objects for modifying via a displacement tool. For example, in response to (or otherwise in connection with) detecting a displacement input to displace a portion of the two-dimensional image 1700, the depth displacement system generates a three-dimensional mesh 1702 including depth data associated with the objects in the two-dimensional image 1700. In one or more embodiments, the depth displacement system 102 generates the three-dimensional mesh 1702 in an initial tessellation operation prior to applying any modifications to the two-dimensional image 1700.

[0154] According to one or more embodiments, in response to a displacement input to modify the two-dimensional image 1700 utilizing the three-dimensional mesh 1702, the depth displacement system 102 generates a modified two-dimensional image 1704. For example, the depth displacement system 102 determines a selected portion of the three-dimensional mesh 1702 based on a selected portion of the two-dimensional image 1700. The depth displacement system 102 also modifies the selected portion according to the attributes of the displacement input. Furthermore, the depth displacement system 102 generates the modified two-dimensional image 1704 according to the modified portion of the three-dimensional mesh 1702 given a mapping between the two-dimensional image 1700 and the three-dimensional mesh 1702.

[0155] In one or more embodiments, as illustrated in FIG. 17, the depth displacement system 102 generates a new three-dimensional mesh 1706 representing the modified two-dimensional image 1704. Specifically, the depth displacement system 102 generates the new three-dimensional mesh 1706 by performing the tessellation process again. For instance, the depth displacement system 102 determines density values of pixels of the modified two-dimensional image 1704, generates a tessellation based on the pixel density values, and generates the new three-dimensional 2023229591   15 Sep 2023 mesh 1706 by modifying the tessellation to include displacement according to pixel depth values and estimated camera parameters for the modified two-dimensional image 1704. Accordingly, the depth displacement system 102 generates the new three-dimensional mesh 1706 to accurately represent the modified two-dimensional image 1704.

[0156] In one or more embodiments, by generating a new three-dimensional mesh in response to modifying a corresponding two-dimensional image, the depth displacement system 102 provides an updated tessellation that reduces artifacts in connection with further modifying the two-dimensional image. For example, displacement operations that introduce sharp transitions between vertices of a three-dimensional mesh result in elongated polygons. Applying further modifications to the two-dimensional image involving previously modified portions of a threedimensional mesh (e.g., the portions including the elongated polygons) may result in artifacts and incorrect distortions of the geometry corresponding to the portions of the two-dimensional image. Thus, by iteratively updating a three-dimensional mesh representing a two-dimensional image after one or more displacement operations, the depth displacement system 102 improves the tessellations in modified regions to reduce or eliminate artifacts in future displacement operations.

[0157] FIGS. 18A-18D illustrate a plurality of graphical user interfaces and three-dimensional meshes in connection with an iterative tessellation process based on a displacement input modifying a two-dimensional image. Specifically, FIG. 18A illustrates a graphical user interface of a client device 1800 including an image editing application 1802 for editing digital images. For example, as illustrated in FIG. 18 A, the client device 1800 displays a two-dimensional image 1804 including a plurality of objects in a scene (e.g., a desert scene).

[0158] In one or more embodiments, in connection with a displacement operation to modify the two-dimensional image 1804, the depth displacement system 102 generates a three-dimensional 2023229591   15 Sep 2023 mesh. FIG. 18B illustrates a first three-dimensional mesh 1806 that the depth displacement system 102 generates for the two-dimensional image 1804. To illustrate, the first three-dimensional mesh 1806 includes a tessellation of the objects in the two-dimensional image 1804 including displacement information of objects based for vertices in the tessellation based on the depth values according to a viewpoint associated with the two-dimensional image 1804.

[0159] In response to a displacement input to modify a portion of the two-dimensional image 1804, the depth displacement system 102 modifies a corresponding portion of the first threedimensional mesh 1806. For example, FIG. 18C illustrates that the client device 1800 displays a modified two-dimensional image 1804a based on a displacement input. To illustrate, the depth displacement system 102 generates a modified portion of the modified two-dimensional image 1804a based on the modified portion of the first three-dimensional mesh 1806.

[0160] In additional embodiments, the depth displacement system 102 generates an updated tessellation based on the modified two-dimensional image 1804a. In particular, FIG. 18D illustrates a second three-dimensional mesh 1806a that the depth displacement system 102 generates for the modified two-dimensional image 1804a. For instance, the second threedimensional mesh 1806a includes a tessellation of the objects in the modified two-dimensional image 1804a including displacement information for objects in the modified two-dimensional image 1804a based on new depth values according to the viewpoint associated with the twodimensional image 1804 and the modified two-dimensional image 1804a.

[0161] In one or more embodiments, the depth displacement system 102 generates the second three-dimensional mesh 1806a by utilizing a new tessellation process (e.g., as described above). To illustrate, the depth displacement system 102 determines new density values for pixels of the modified two-dimensional image 1804a and samples points based on the new density values. 2023229591   15 Sep 2023 Furthermore, the depth displacement system 102 generates a new tessellation based on the sampled points according to the new density values and modifies the new tessellation to include depth information according to the viewpoint of the modified two-dimensional image 1804a.

[0162] In one or more alternative embodiments, the depth displacement system 102 generates the second three-dimensional mesh 1806a by interpolating data based on the first threedimensional mesh 1806. For example, the depth displacement system 102 determines one or more regions of the first three-dimensional mesh 1806 that include elongated polygons or other artifacts (e.g., texture artifacts) introduced based on the displacement to the two-dimensional image 1804. The depth displacement system 102 utilizes the new positions of vertices in the first threedimensional mesh 1806 to insert a plurality of vertices and reduce the size of polygons in the tessellation by interpolating one or more surfaces of the first three-dimensional mesh 1806. By inserting the new vertices into the tessellation, the depth displacement system 102 generates the second three-dimensional mesh 1806a to include more accurate geometry and prevent artifacts in further modifications while also retaining information in the second three-dimensional mesh 1806a that become obscured relative to the viewpoint of the modified two-dimensional image 1804a in response to a displacement of a portion of the first three-dimensional mesh 1806.

[0163] FIGS. 19A-19B illustrate images in connection with a displacement operation to modify a digital image. Specifically, FIG. 19A illustrates a two-dimensional image 1900 including a scene with a plurality of objects (e.g., a ground, trees). In one or more embodiments, the depth displacement system 102 generates a three-dimensional mesh representing the two-dimensional image 1900 for modifying the two-dimensional image 1900 via a displacement tool. FIG. 19B illustrates a modified two-dimensional image 1902 including a plurality of hills created by displacing one or more portions of the two-dimensional image 1900 including a displaced portion 2023229591   15 Sep 2023 1904. To illustrate, the displaced portion 1904 includes a displacement of a ground to introduce details that were not previously included in the two-dimensional image 1900.

[0164] In one or more embodiments, in response to a displacement input to generate the displaced portion 1904, the depth displacement system 102 generates a new three-dimensional mesh representing the modified two-dimensional image 1902. For example, the depth displacement system 102 generates the new three-dimensional mesh to insert additional vertices into a tessellation based on the original three-dimensional mesh including elongated polygons. In additional embodiments, the depth displacement system 102 generates the new three-dimensional mesh in response to determining that the displacement input introduced artifacts and / or additional image detail not previously in the two-dimensional image 1900 (e.g., the cliff faces).

[0165] According to one or more embodiments, the depth displacement system 102 also utilizes content-aware filling to modify texture data associated with the new three-dimensional mesh and / or the modified two-dimensional image 1902. For instance, the depth displacement system 102 utilizes an inpainting neural network to inpaint a portion of the modified two-dimensional image 1902. The depth displacement system 102 can utilize a variety of models or architectures to inpaint pixels of a digital image. For example, in one or more embodiments, the depth displacement system 102 utilizes an inpainting neural network as described in U.S. Patent Application No. 17 / 663,317, filed May 13, 2022, titled OBJECT CLASS INPAINTING IN DIGITAL IMAGES UTILIZING CLASS-SPECIFIC INPAINTING NEURAL NETWORKS or as described in U.S. Patent Application No. 17 / 815,409, filed July 27, 2022, titled “GENERATING NEURAL NETWORK BASED PERCEPTUAL ARTIFACT SEGMENTATIONS INMODIFIED PORTIONS OF A DIGITAL IMAGE,” which are herein incorporated by reference in their entirety. To illustrate, the depth displacement system 102 2023229591   15 Sep 2023 utilizes the inpainting neural network to generate new image details for a new surface within the displaced portion 1904. Accordingly, the depth displacement system 102 inpaints cliff textures onto the cliff faces generated within the displaced portion 1904 according to the contextual information surrounding the displaced portion 1904.

[0166] In one or more embodiments, the depth displacement system 102 also provides iterative tessellation in response to modifications made to a two-dimensional image via two-dimensional editing tools. For example, after generating a three-dimensional mesh corresponding to a twodimensional image, the depth displacement system 102 detects an additional input to modify the two-dimensional image via a two-dimensional editing tool (e.g., a two-dimensional image filter / warping tool). In response to determining that the two-dimensional image is modified via the two-dimensional editing tool, the depth displacement system 102 performs an additional mesh generation process to update the three-dimensional mesh corresponding to the modified twodimensional image. Thus, the depth displacement system 102 provides iterative updating of twodimensional images and corresponding three-dimensional meshes based on modifications made in three-dimensional space and / or in two-dimensional space.

[0167] FIG. 20 illustrates a detailed schematic diagram of an embodiment of the depth displacement system 102 described above. As shown, the depth displacement system 102 is implemented in an image editing system 110 on computing device(s) 2000 (e.g., a client device and / or server device as described in FIG. 1, and as further described below in relation to FIG. 25). Additionally, the depth displacement system 102 includes, but is not limited to, a mesh generator 2002 (which includes neural network(s) 2004), a user interface manager 2006, a mesh displacement manager 2008, an image modification manager 2010, and a storage manager 2012. The depth displacement system 102 can be implemented on any number of computing devices. 2023229591   15 Sep 2023 For example, the depth displacement system 102 can be implemented in a distributed system of server devices for digital images. The depth displacement system 102 can also be implemented within one or more additional systems. Alternatively, the depth displacement system 102 can be implemented on a single computing device such as a single client device.

[0168] In one or more embodiments, each of the components of the depth displacement system 102 is in communication with other components using any suitable communication technologies. Additionally, the components of the depth displacement system 102 are capable of being in communication with one or more other devices including other computing devices of a user, server devices (e.g., cloud storage devices), licensing servers, or other devices / systems. It will be recognized that although the components of the depth displacement system 102 are shown to be separate in FIG. 20, any of the subcomponents may be combined into fewer components, such as into a single component, or divided into more components as may serve a particular implementation. Furthermore, although the components of FIG. 20 are described in connection with the depth displacement system 102, at least some of the components for performing operations in conjunction with the depth displacement system 102 described herein may be implemented on other devices within the environment.

[0169] In some embodiments, the components of the depth displacement system 102 include software, hardware, or both. For example, the components of the depth displacement system 102 include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices (e.g., the computing device(s) 2000). When executed by the one or more processors, the computer-executable instructions of the depth displacement system 102 cause the computing device(s) 2000 to perform the operations described herein. Alternatively, the components of the depth displacement system 102 include hardware, 2023229591   15 Sep 2023 such as a special purpose processing device to perform a certain function or group of functions. Additionally, or alternatively, the components of the depth displacement system 102 include a combination of computer-executable instructions and hardware.

[0170] Furthermore, the components of the depth displacement system 102 performing the functions described herein with respect to the depth displacement system 102 may, for example, be implemented as part of a stand-alone application, as a module of an application, as a plug-in for applications, as a library function or functions that may be called by other applications, and / or as a cloud-computing model. Thus, the components of the depth displacement system 102 may be implemented as part of a stand-alone application on a personal computing device or a mobile device. Alternatively, or additionally, the components of the depth displacement system 102 may be implemented in any application that provides digital image modification.

[0171] As illustrated in FIG. 20, the depth displacement system 102 includes the mesh generator 2002 to generate three-dimensional meshes from two-dimensional images. For example, the mesh generator 2002 utilizes the neural network(s) 2004 to estimate depth values for pixels of a twodimensional image and one or more filters to determine a density map based on the estimated depth values. Additionally, the mesh generator 2002 samples points based on the density map and generates a tessellation based on the sampled points. The mesh generator 2002 further generates (e.g., utilizing the neural network(s) 2004) a displacement three-dimensional mesh by modifying positions of vertices in the tessellation to incorporate depth and displacement information into a three-dimensional mesh representing the two-dimensional image.

[0172] The depth displacement system 102 also includes the user interface manager 2006 to manage user interactions in connection with modifying two-dimensional images via a displacement tool. For example, the user interface manager 2006 detects positions of displacement 2023229591   15 Sep 2023 inputs relative to a two-dimensional image and translates the positions into a three-dimensional space associated with a corresponding three-dimensional mesh. The user interface manager 2006 also converts changes made to a three-dimensional mesh back to a corresponding two-dimensional image for display within a graphical user interface.

[0173] According to one or more embodiments, the depth displacement system 102 includes the mesh displacement manager 2008 to modify a three-dimensional mesh based on a displacement input in connection with a displacement tool. Specifically, the mesh displacement manager 2008 determines a displacement of a selected portion of a three-dimensional mesh corresponding to a displacement input. To illustrate, the mesh displacement manager 2008 utilizes settings associated with the displacement input to determine which vertices to displace and how to displace the vertices.

[0174] The depth displacement system 102 also includes the image modification manager 2010 to modify two-dimensional images. For instance, the image modification manager 2010 generates an updated two-dimensional image in response to detecting modifications to a corresponding three-dimensional mesh. To illustrate, the image modification manager 2010 utilizes a mapping between the two-dimensional image and the three-dimensional mesh to re-render the twodimensional image (e.g., based on a texture mapping between the two-dimensional image and the three-dimensional mesh) according to one or more displaced portions of the three-dimensional mesh.

[0175] The depth displacement system 102 also includes a storage manager 2012 (that comprises a non-transitory computer memory / one or more memory devices) that stores and maintains data associated with modifying two-dimensional images utilizing three-dimensional meshes. For example, the storage manager 2012 stores data associated with neural networks that 2023229591   15 Sep 2023 generate three-dimensional meshes based on depth information associated with corresponding two-dimensional images for modifying the two-dimensional images. To illustrate, the storage manager 2012 stores two-dimensional images, three-dimensional meshes, and mappings between two-dimensional images and three-dimensional meshes.

[0176] Turning now to FIG. 21, this figure shows a flowchart of a series of acts 2100 of generating an adaptive displacement three-dimensional mesh for a two-dimensional image. While FIG. 21 illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and / or modify any of the acts shown in FIG. 21. The acts of FIG. 21 are part of a method. Alternatively, a non-transitory computer readable medium comprises instructions, that when executed by one or more processors, cause the one or more processors to perform the acts of FIG. 21. In still further embodiments, a system includes a processor or server configured to perform the acts of FIG. 21.

[0177] As shown, the series of acts 2100 includes an act 2102 of determining density values for pixels of a two-dimensional image. For example, act 2102 involves determining density values corresponding to pixels of a two-dimensional image based on disparity estimation values, wherein the disparity estimation values are generated utilizing a first neural network. Additionally, in one or more embodiments, act 2102 involves determining density values corresponding to pixels of a two-dimensional image based on disparity estimation values generated utilizing a first neural network according to relative positions of objects in the two-dimensional image.

[0178] In one or more embodiments, act 2102 involves determining, utilizing a plurality of image filters, a second order derivative change in depth of the pixels of the two-dimensional image based on the disparity estimation values. For example, act 2102 involves determining absolute values of a matrix corresponding to the disparity estimation values of the two-dimensional image. 2023229591   15 Sep 2023 Additionally, act 2102 involves determining the density values corresponding to the pixels of the two-dimensional image based on the absolute values of the matrix.

[0179] In one or more embodiments, act 2102 involves determining, utilizing a convolution operation, smoothed values from the absolute values of the matrix. Act 2102 also involves generating a density map by truncating the smoothed values according to a set of processing parameters.

[0180] Act 2102 can involve generating, utilizing the first neural network, the disparity estimation values indicating estimated values inversely related to distances between corresponding points in a scene of the two-dimensional image and a viewpoint of the two-dimensional image. In one or more embodiments, act 2102 involves generating, utilizing the first neural network, the disparity estimation values indicating estimated values inversely related to distances between corresponding points in a scene of the two-dimensional image and a viewpoint of the twodimensional image. Additionally, act 2102 involves determining, utilizing one or more image filters, a second order derivative change in depth of the pixels of the two-dimensional image based on the disparity estimation values.

[0181] Furthermore, act 2102 involves determining the density values corresponding to the pixels of the two-dimensional image based on absolute values of a matrix corresponding to the disparity estimation values of the two-dimensional image. Act 2102 also involves generating a density map comprising the density values corresponding to the pixels of the two-dimensional image by smoothing and truncating the absolute values of the matrix according to a set of processing parameters.

[0182] In one or more embodiments, act 2102 involves determining a matrix representing a second-order derivative of depth values of the two-dimensional image, determining absolute 2023229591   15 Sep 2023 values of diagonals of the matrix, generating, utilizing a convolution operation, smoothed values based on the absolute values, and truncating the smoothed values according to a predetermined threshold.

[0183] The series of acts 2100 also includes an act 2104 of sampling points according to the density values. For example, act 2104 involves sampling a plurality of points in the twodimensional image according to the density values corresponding to the pixels of the twodimensional image. In one or more embodiments, act 2104 involves sampling a plurality of points in the two-dimensional image according to a probability distribution indicated by the density values corresponding to the pixels of the two-dimensional image. In one or more embodiments, act 2104 involves selecting points from the two-dimensional image utilizing the density values as a probability distribution. Act 2104 can involve selecting the plurality of points utilizing the density values as the probability distribution across the two-dimensional image.

[0184] Additionally, the series of acts 2100 includes an act 2106 of generating a threedimensional mesh based on the sampled points. For example, act 2106 involves generating a threedimensional mesh based on the plurality of points sampled in the two-dimensional image. To illustrate, act 2106 involves generating an initial tessellation based on the plurality of points sampled in the two-dimensional image. Act 2106 can involve generating a tessellation representing content of the two-dimensional image based on the plurality of points. In one or more embodiments, act 2106 involves generating a tessellation representing content of the twodimensional image by utilizing a relaxation model in connection with sampling points from the two-dimensional image. For example, act 2106 involves sampling the plurality of points in a plurality of iterations according to a relaxation algorithm that iteratively moves sampled points 2023229591   15 Sep 2023 towards centers of tessellation cells based on the density values. Act 2106 can also involve generating, utilizing Delaunay triangulation, a tessellation according to the sampled points.

[0185] The series of acts 2100 further includes an act 2108 of generating a displacement threedimensional mesh from the three-dimensional mesh based on estimated camera parameters. For example, act 2108 involves generating, utilizing a second neural network, a displacement threedimensional mesh from the three-dimensional mesh based on estimated camera parameters of the two-dimensional image. To illustrate, act 2108 involves generating, utilizing a second neural network, a displacement three-dimensional mesh comprising an updated tessellation according to estimated camera parameters of the two-dimensional image.

[0186] In one or more embodiments, act 2108 involves determining, based on one or more inputs via one or more user-interface elements, one or more processing parameters comprising a sampling budget or a tessellation budget in connection with generating the displacement threedimensional mesh. Act 2108 also involves generating the three-dimensional mesh based on the plurality of points sampled in the two-dimensional image according to the one or more processing parameters.

[0187] In one or more embodiments, act 2108 involves determining, utilizing the second neural network, the estimated camera parameters corresponding to a viewpoint of the two-dimensional image. Additionally, act 2108 involves determining displacement of vertices in the threedimensional mesh based on the estimated camera parameters and pixel depth values of the twodimensional image. Act 2108 also involves generating the updated tessellation by modifying positions of the vertices of the three-dimensional mesh according to the determined displacement of the vertices in the three-dimensional mesh. For example, act 2108 involves modifying positions 2023229591   15 Sep 2023 of the vertices of the three-dimensional mesh to include depth displacement according to the estimated camera parameters of the three-dimensional mesh.

[0188] In one or more embodiments, the series of acts 2100 includes modifying the displacement three-dimensional mesh in response to a request to modify the two-dimensional image. Additionally, the series of acts 2100 includes generating a modified two-dimensional image in response to modifying the displacement three-dimensional mesh.

[0189] Turning now to FIG. 22, this figure shows a flowchart of a series of acts 2200 of modifying a two-dimensional image utilizing a displacement three-dimensional mesh. While FIG. 22 illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and / or modify any of the acts shown in FIG. 22. The acts of FIG. 22 are part of a method. Alternatively, a non-transitory computer readable medium comprises instructions, that when executed by one or more processors, cause the one or more processors to perform the acts of FIG. 22. In still further embodiments, a system includes a processor or server configured to perform the acts of FIG. 22.

[0190] As shown, the series of acts 2200 includes an act 2202 of generating a three-dimensional mesh based on pixel depth values of a two-dimensional image. For example, act 2202 involves generating, utilizing one or more neural networks, a three-dimensional mesh based on pixel depth values corresponding to objects of a two-dimensional image.

[0191] In one or more embodiments, act 2202 involves generating the three-dimensional mesh by determining displacement of vertices of a tessellation of the two-dimensional image based on the pixel depth values and estimated camera parameters. Alternatively, act 2202 involves generating the three-dimensional mesh based on a plurality of points sampled in the two- 2023229591   15 Sep 2023 dimensional image according to density values determined from the pixel depth values of the twodimensional image.

[0192] The series of acts 2200 also includes an act 2204 of determining a position of the threedimensional mesh based on a displacement input. For example, act 2204 involves determining a position of the three-dimensional mesh based on a corresponding position of a displacement input within the two-dimensional image.

[0193] In one or more embodiments, act 2204 involves determining the corresponding position of the displacement input comprising a coordinate within the two-dimensional image. Act 2204 also involves determining the position of the three-dimensional mesh based on the coordinate within the two-dimensional image and a projection from the two-dimensional image onto the threedimensional mesh.

[0194] In one or more embodiments, act 2204 involves determining a projection from the twodimensional image onto the three-dimensional mesh. Act 2204 also involves determining, according to the projection from the two-dimensional image onto the three-dimensional mesh, a three-dimensional position of the three-dimensional mesh corresponding to the displacement input based on a position of the two-dimensional position of the two-dimensional image corresponding to the displacement input.

[0195] Additionally, the series of acts 2200 includes an act 2206 of modifying the threedimensional mesh by determining a displaced portion of the three-dimensional mesh at the position of the three-dimensional mesh. For example, act 2206 involves modifying, in response to the displacement input, the three-dimensional mesh by determining a displaced portion of the threedimensional mesh at the position of the three-dimensional mesh. 2023229591   15 Sep 2023

[0196] Act 2206 involves determining a two-dimensional position of the displacement input within the two-dimensional image. Act 2206 also involves determining a three-dimensional position of the displacement input on the three-dimensional mesh based on the two-dimensional position of the displacement input within the two-dimensional image.

[0197] Act 2206 involves determining, based on an attribute of the displacement input, that the displacement input indicates a displacement direction for a portion of the three-dimensional mesh. For instance, act 2206 involves determining a displacement direction based on a selected portion of the three-dimensional mesh. Act 2206 further involves displacing the portion of the threedimensional mesh in the displacement direction.

[0198] In one or more embodiments, act 2206 involves determining, based on an additional attribute of the displacement input, that the displacement input indicates an additional displacement direction of the portion of the three-dimensional mesh. Act 2206 also involves displacing the portion of the three-dimensional mesh according to the additional displacement direction.

[0199] Furthermore, act 2206 involves selecting, in response to an additional input in connection with the displacement input, a new portion of the three-dimensional mesh to displace. Act 2206 also involves displacing the new portion of the three-dimensional mesh according to the displacement input in the displacement direction. Additionally, in one or more embodiments, act 2206 involves displacing, based on movement of the displacement input, one or more additional portions of the three-dimensional mesh from the portion of the three-dimensional mesh to the new portion of the three-dimensional mesh in the displacement direction.

[0200] In one or more embodiments, act 2206 involves determining a displacement filter indicating a shape associated with the displacement input. Act 2206 also involves displacing a 2023229591   15 Sep 2023 portion of the three-dimensional mesh according to the shape of the displacement input in a direction of the displacement input.

[0201] Act 2206 can involve determining that the displacement input indicates a displacement of a portion of the three-dimensional mesh according to the position of the three-dimensional mesh. Act 2206 can also involve determining the displaced portion of the three-dimensional mesh in response to the displacement input.

[0202] In one or more embodiments, act 2206 involves determining a direction of movement of the displacement input within a graphical user interface displaying the two-dimensional image. Act 2206 also involves determining a displacement height and a displacement radius based on the direction of movement of the displacement. Act 2206 further involves determining the displaced portion of the three-dimensional mesh based on the displacement height and the displacement radius.

[0203] In one or more embodiments, act 2206 involves determining, based on the projection from the two-dimensional image onto the three-dimensional mesh, movement of the displacement input relative to the two-dimensional image and a corresponding movement of the displacement input relative to the three-dimensional mesh. Act 2206 also involves determining the displaced portion of the three-dimensional mesh based on the corresponding movement of the displacement input relative to the three-dimensional mesh.

[0204] In one or more embodiments, act 2206 involves determining one or more normal values corresponding to one or more vertices or one or more faces at the position of the three-dimensional mesh. Act 2206 also involves determining, in response to the displacement input, the displaced portion of the three-dimensional mesh in one or more directions corresponding to the one or more normal values corresponding to the one or more vertices or the one or more faces. 2023229591   15 Sep 2023

[0205] The series of acts 2200 further includes an act 2208 of generating a modified twodimensional image based on the displaced portion of the three-dimensional mesh. For example, act 2208 involves generating a modified two-dimensional image comprising at least one modified portion according to the displaced portion of the three-dimensional mesh.

[0206] In one or more embodiments, act 2208 involves determining a two-dimensional position of the two-dimensional image corresponding to a three-dimensional position of the displaced portion of the three-dimensional mesh based on a mapping between the three-dimensional mesh and the two-dimensional image. Act 2208 also involves generating the modified two-dimensional image comprising the at least one modified portion at the two-dimensional position based on the three-dimensional position of the displaced portion of the three-dimensional mesh.

[0207] In one or more embodiments, act 2208 involves providing a preview two-dimensional image comprising the at least one modified portion according to the displaced portion of the threedimensional mesh in response to the displacement input. Act 2208 also involves generating the modified two-dimensional image comprising the at least one modified portion in response to detecting an action to commit the displaced portion of the three-dimensional mesh.

[0208] In one or more embodiments, act 2208 involves determining, based on the projection from the two-dimensional image onto the three-dimensional mesh, a two-dimensional position of the two-dimensional image corresponding to the displaced portion of the three-dimensional mesh. Act 2208 can also involve generating, based on the two-dimensional position of the twodimensional image, the modified two-dimensional image comprising the at least one modified portion according to the displaced portion of the three-dimensional mesh.

[0209] Turning now to FIG. 23, this figure shows a flowchart of a series of acts 2300 of modifying a two-dimensional image utilizing segmented three-dimensional object meshes. While 2023229591   15 Sep 2023 FIG. 23 illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and / or modify any of the acts shown in FIG. 23. The acts of FIG. 23 are part of a method. Alternatively, a non-transitory computer readable medium comprises instructions, that when executed by one or more processors, cause the one or more processors to perform the acts of FIG. 23. In still further embodiments, a system includes a processor or server configured to perform the acts of FIG. 23.

[0210] As shown, the series of acts 2300 includes an act 2302 of generating a three-dimensional mesh based on pixel depth values of a two-dimensional image. For example, act 2302 involves generating, utilizing one or more neural networks, a three-dimensional mesh based on pixel depth values of a two-dimensional image.

[0211] In one or more embodiments, act 2302 involves generating the three-dimensional mesh by determining displacement of vertices of a tessellation of the two-dimensional image based on the pixel depth values and estimated camera parameters. Alternatively, act 2302 involves generating the three-dimensional mesh based on a plurality of points sampled in the twodimensional image according to density values determined from the pixel depth values of the twodimensional image.

[0212] The series of acts 2300 further includes an act 2304 of segmenting the three-dimensional mesh into three-dimensional object meshes. For example, act 2304 involves segmenting, utilizing the one or more neural networks, the three-dimensional mesh into a plurality of three-dimensional object meshes corresponding to objects of the two-dimensional image.

[0213] In one or more embodiments, act 2304 involves detecting, utilizing one or more object detection models, a plurality of objects of the two-dimensional image. Act 2304 can further involve segmenting, in response to detecting the plurality of objects, the three-dimensional mesh 2023229591   15 Sep 2023 into a plurality of three-dimensional object meshes corresponding to the plurality of objects of the two-dimensional image.

[0214] For example, act 2304 involves detecting one or more objects in the two-dimensional image or in the three-dimensional mesh. Act 2304 also involves separating a first portion of the three-dimensional mesh from a second portion of the three-dimensional mesh based on the one or more objects detected in the two-dimensional image or in the three-dimensional mesh.

[0215] Act 2304 can involve generating, utilizing the one or more object detection models, a semantic map comprising labels indicating object classifications of pixels in the two-dimensional image. Act 2304 can also involve detecting the plurality of objects of the two-dimensional image based on the labels of the semantic map.

[0216] In one or more embodiments, act 2304 involves determining, based on the pixel depth values of the two-dimensional image, a portion of the two-dimensional image comprising a depth discontinuity between adjacent regions of the two-dimensional image. Act 2304 involves determining that a first region of the adjacent regions corresponds to a first object and a second region of the adjacent regions corresponds to a second object.

[0217] In one or more embodiments, act 2304 involves determining a semantic map comprising labels indicating object classifications of pixels in the two-dimensional image. Act 2304 also involves detecting the one or more objects in the two-dimensional image based on the labels of the semantic map.

[0218] In one or more embodiments, act 2304 involves determining a depth discontinuity at a portion of the three-dimensional mesh based on corresponding pixel depth values of the twodimensional image. Act 2304 involves detecting the one or more objects in the three-dimensional mesh based on the depth discontinuity at the portion of the three-dimensional mesh. 2023229591   15 Sep 2023

[0219] According to one or more embodiments, act 2304 involves detecting the objects of the two-dimensional image according to: a semantic map corresponding to the two-dimensional image; or depth discontinuities based on the pixel depth values of the two-dimensional image. Act 2304 also involves separating the three-dimensional mesh into the plurality of three-dimensional object meshes in response to detecting the objects of the two-dimensional image.

[0220] The series of acts 2300 also includes an act 2306 of modifying a selected threedimensional object mesh in response to a displacement input. For example, act 2306 involves modifying, in response to a displacement input within a graphical user interface displaying the two-dimensional image, a selected three-dimensional object mesh of the plurality of threedimensional object meshes based on a displaced portion of the selected three-dimensional object mesh.

[0221] Act 2306 can involve determining a projection from the two-dimensional image onto the plurality of three-dimensional object meshes in a three-dimensional environment. Act 2306 also involves determining the selected three-dimensional object mesh based on a two-dimensional position of the displacement input relative to the two-dimensional image and the projection from the two-dimensional image onto the plurality of three-dimensional object meshes. For example, act 2306 involves determining a two-dimensional position of the displacement input relative to the two-dimensional image. Act 2306 also involves determining a three-dimensional position corresponding to a three-dimensional object mesh of the plurality of three-dimensional object meshes based on a mapping between the two-dimensional image and a three-dimensional environment comprising the plurality of three-dimensional object meshes.

[0222] In one or more embodiments, act 2306 involve determining that the displacement input indicates a displacement direction for a portion of the selected three-dimensional object mesh. Act 2023229591   15 Sep 2023 2306 also involves modifying a portion of the selected three-dimensional object mesh by displacing the portion of the selected three-dimensional object mesh according to the displacement direction.

[0223] In one or more embodiments, act 2306 involves modifying the selected threedimensional object mesh according to the displacement input without modifying one or more additional three-dimensional object meshes adjacent to the selected three-dimensional object mesh within a three-dimensional environment.

[0224] In one or more embodiments, act 2306 involves determining, based on an attribute of the displacement input, that the displacement input indicates one or more displacement directions for a portion of the selected three-dimensional object mesh. Act 2306 also involves displacing the portion of the selected three-dimensional object mesh in the one or more displacement directions.

[0225] Additionally, the series of acts 2300 includes an act 2308 of generating a modified twodimensional image in response to modifying the selected three-dimensional object mesh. For example, act 2308 involves generating a modified two-dimensional image comprising at least one modified portion according to the displaced portion of the selected three-dimensional object mesh.

[0226] In one or more embodiments, act 2308 involves determining a two-dimensional position of the two-dimensional image corresponding to a three-dimensional position of the displaced portion of the selected three-dimensional object mesh based on a mapping between the plurality of three-dimensional object meshes and the two-dimensional image. Act 2308 also involves generating the modified two-dimensional image comprising the at least one modified portion at the two-dimensional position based on the three-dimensional position of the displaced portion of the selected three-dimensional object mesh. 2023229591   15 Sep 2023

[0227] In one or more embodiments, act 2308 involves determining that the displacement input indicates a displacement direction for the selected three-dimensional object mesh, the selected three-dimensional object mesh being adjacent an additional three-dimensional object mesh. Act 2308 also involves displacing a portion of the selected three-dimensional object mesh according to the displacement direction without modifying the additional three-dimensional object mesh.

[0228] According to one or more embodiments, act 2308 involves determining, based on a mapping between the two-dimensional image and the three-dimensional mesh, a two-dimensional position of the two-dimensional image corresponding to a three-dimensional position of the displaced portion of the selected three-dimensional object mesh. Act 2308 also involves generating the modified two-dimensional image comprising the at least one modified portion at the two-dimensional position of the two-dimensional image according to the displaced portion of the selected three-dimensional object mesh.

[0229] For example, act 2308 involves determining a mapping between the two-dimensional image and the three-dimensional mesh. Act 2308 involves determining a three-dimensional position of the displaced portion of the selected three-dimensional object mesh. Act 2308 further involves generating, based on the mapping between the two-dimensional image and the threedimensional mesh, the modified two-dimensional image comprising the at least one modified portion at a two-dimensional position of the two-dimensional image corresponding to the threedimensional position of the displaced portion of the selected three-dimensional object mesh.

[0230] Turning now to FIG. 24, this figure shows a flowchart of a series of acts 2400 of modifying a two-dimensional image in an iterative tessellation process. While FIG. 24 illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and / or modify any of the acts shown in FIG. 24. The acts of FIG. 24 are part of a method. Alternatively, 2023229591   15 Sep 2023 a non-transitory computer readable medium comprises instructions, that when executed by one or more processors, cause the one or more processors to perform the acts of FIG. 24. In still further embodiments, a system includes a processor or server configured to perform the acts of FIG. 24.

[0231] As shown, the series of acts 2400 includes an act 2402 of generating a three-dimensional mesh based on pixel depth values of a two-dimensional image. For example, act 2402 involves generating, utilizing one or more neural networks, a three-dimensional mesh based on pixel depth values of a two-dimensional image. For example, act 2402 involves generating, utilizing one or more neural networks, a first three-dimensional mesh based on first pixel depth values of a twodimensional image.

[0232] In one or more embodiments, act 2402 involves generating the three-dimensional mesh by determining displacement of vertices of a tessellation of the two-dimensional image based on the pixel depth values and estimated camera parameters. Act 2402 alternatively involves generating the three-dimensional mesh based on a plurality of points sampled in the twodimensional image according to density values determined from the pixel depth values of the twodimensional image.

[0233] Act 2402 can involve generating a first tessellation based on a first set of sampled points of the two-dimensional image. Act 2402 can involve determining, utilizing the one or more neural networks, the first pixel depth values according to estimated camera parameters corresponding to a viewpoint of the two-dimensional image.

[0234] The series of acts 2400 also includes an act 2404 of determining a modified twodimensional image based on a displacement input. For example, act 2404 involves determining a modified two-dimensional image comprising at least one modified portion of the two-dimensional image based on a displacement input within a graphical user interface displaying the two- 2023229591   15 Sep 2023 dimensional image. In one or more embodiments, act 2404 involves determining, based on a displacement input within a graphical user interface displaying the two-dimensional image, a modified two-dimensional image comprising at least one modified portion of the two-dimensional image according to a corresponding modified portion of the first three-dimensional mesh.

[0235] F or example, act 2404 involves determining a displaced portion of the three-dimensional mesh based on the displacement input. Act 2404 also involves generating the modified twodimensional image according to the displaced portion of the three-dimensional mesh.

[0236] Act 2404 can involve determining a two-dimensional position of the displacement input within the two-dimensional image. Act 2404 also involves determining a three-dimensional position of the displacement input on the first three-dimensional mesh based on the twodimensional position of the displacement input within the two-dimensional image. Act 2404 further involves determining the at least one modified portion of the two-dimensional image at the two-dimensional position based on a modified portion of the first three-dimensional mesh at the three-dimensional position according to the displacement input.

[0237] Act 2404 can involve determining a displacement direction of the displacement input relative to the first three-dimensional mesh. Act 2404 can further involve determining the modified portion of the first three-dimensional mesh according to the displacement direction of the displacement input. To illustrate, act 2404 involves determining a displaced portion of the three-dimensional mesh based on one or more displacement directions of the displacement input, and determining the at least one modified portion of the two-dimensional image based on the displaced portion of the three-dimensional mesh.

[0238] Act 2404 can involve determining that the at least one modified portion of the twodimensional image comprises an image artifact. Act 2404 also involves generating, utilizing an 2023229591   15 Sep 2023 inpainting neural network, inpainted image content correcting the image artifact within the at least one modified portion. Act 2404 can also involve generating, utilizing an inpainting neural network, inpainted image content for the at least one modified portion of the two-dimensional image in response to detecting an artifact in the at least one modified portion of the twodimensional image.

[0239] The series of acts 2400 further includes an act 2406 of generating an updated threedimensional mesh based on new pixel depth values of the modified two-dimensional image. For example, act 2406 involves generating, utilizing the one or more neural networks, an updated three-dimensional mesh based on new pixel depth values for the modified two-dimensional image according to the at least one modified portion. For example, act 2406 involves generating, utilizing the one or more neural networks, a second three-dimensional mesh based on second pixel depth values for the modified two-dimensional image according to the at least one modified portion.

[0240] In one or more embodiments, act 2406 involves determining the new pixel depth values for the modified two-dimensional image in response to detecting an action to commit the at least one modified portion.

[0241] Additionally, act 2406 can involve generating the second three-dimensional mesh comprises generating a second tessellation based on a second set of sampled points of the modified two-dimensional image. Act 2406 can involve determining, utilizing the one or more neural networks, the second pixel depth values according to the estimated camera parameters corresponding to the viewpoint of the two-dimensional image.

[0242] Act 2406 involves sampling a plurality of points of the modified two-dimensional image according to density values corresponding to pixels of the modified two-dimensional image. Act 2023229591   15 Sep 2023 2406 further involves generating the updated three-dimensional mesh based on the plurality of points sampled in the modified two-dimensional image.

[0243] In one or more embodiments, act 2406 involves generating the second three-dimensional mesh in response to request to commit the at least one modified portion of the two-dimensional image. For example, act 2406 involves detecting an action to generate the modified twodimensional image by committing a displacement of the at least one modified portion to the twodimensional image. Act 2406 also involves generating the updated three-dimensional mesh in response to committing the displacement of the at least one modified portion to the twodimensional image.

[0244] Act 2406 also involves determining that an initial position of the at least one modified portion of the two-dimensional image obscures an additional portion of the two-dimensional image. Act 2406 involves generating the updated three-dimensional mesh by interpolating vertex positions in a portion of the three-dimensional mesh corresponding to the additional portion of the two-dimensional image obscured by the initial position of the at least one modified portion of the two-dimensional image.

[0245] In one or more embodiments, act 2406 involves interpolating, in connection with the at least one modified portion of the two-dimensional image, vertex positions of a plurality of vertices in a portion of the second three-dimensional mesh corresponding to an obscured portion of the two-dimensional image.

[0246] The series of acts 2400 can also include generating, in a plurality of displacement iterations comprising a plurality of displacement inputs within the graphical user interface, a plurality of updated three-dimensional meshes corresponding to a plurality of modified twodimensional images in connection with the modified two-dimensional image. 2023229591   15 Sep 2023

[0247] In one or more embodiments, act 2402 involves sampling a first set of points of the twodimensional image according to first density values corresponding to pixels of the twodimensional image. Act 2402 further involves generating the three-dimensional mesh based on the first set of points sampled in the two-dimensional image. Accordingly, act 2406 involves sampling a second set of points of the modified two-dimensional image according to second density values corresponding to pixels of the modified two-dimensional image. Act 2406 also involves sampling a second set of points of the modified two-dimensional image according to second density values corresponding to pixels of the modified two-dimensional image. Act 2406 further involves generating the updated three-dimensional mesh based on the second set of points sampled in the modified two-dimensional image.

[0248] Embodiments of the present disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

[0249] Computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer- 2023229591   15 Sep 2023 executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0250] Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phasechange memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0251] A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and / or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0252] Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices) (or vice 2023229591   15 Sep 2023 versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and eventually transferred to computer system RAM and / or to less volatile computer storage media (devices) at a computer system. Thus, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0253] Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. In some embodiments, computer-executable instructions are executed on a general-purpose computer to turn the general-purpose computer into a special purpose computer implementing elements of the disclosure. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0254] Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be practiced in distributed system environments 2023229591   15 Sep 2023 where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0255] Embodiments of the present disclosure can also be implemented in cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction and scaled accordingly.

[0256] A cloud-computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud-computing model can also expose various service models, such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“laaS”). A cloud-computing model can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud-computing environment” is an environment in which cloud computing is employed.

[0257] FIG. 25 illustrates a block diagram of exemplary computing device 2500 that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices such as the computing device 2500 may implement the system(s) of FIG. 1. As shown by FIG. 25, the computing device 2500 can comprise a processor 2502, a 2023229591   15 Sep 2023 memory 2504, a storage device 2506, an I / O interface 2508, and a communication interface 2510, which may be communicatively coupled by way of a communication infrastructure 2512. In certain embodiments, the computing device 2500 can include fewer or more components than those shown in FIG. 25. Components of the computing device 2500 shown in FIG. 25 will now be described in additional detail.

[0258] In one or more embodiments, the processor 2502 includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions for dynamically modifying workflows, the processor 2502 may retrieve (or fetch) the instructions from an internal register, an internal cache, the memory 2504, or the storage device 2506 and decode and execute them. The memory 2504 may be a volatile or non-volatile memory used for storing data, metadata, and programs for execution by the processor(s). The storage device 2506 includes storage, such as a hard disk, flash disk drive, or other digital storage device, for storing data or instructions for performing the methods described herein.

[0259] The I / O interface 2508 allows a user to provide input to, receive output from, and otherwise transfer data to and receive data from computing device 2500. The I / O interface 2508 may include a mouse, a keypad or a keyboard, a touch screen, a camera, an optical scanner, network interface, modem, other known I / O devices or a combination of such I / O interfaces. The I / O interface 2508 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O interface 2508 is configured to provide graphical data to a display for 2023229591   15 Sep 2023 presentation to a user. The graphical data may be representative of one or more graphical user interfaces and / or any other graphical content as may serve a particular implementation.

[0260] The communication interface 2510 can include hardware, software, or both. In any event, the communication interface 2510 can provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device 2500 and one or more other computing devices or networks. As an example, and not by way of limitation, the communication interface 2510 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI.

[0261] Additionally, the communication interface 2510 may facilitate communications with various types of wired or wireless networks. The communication interface 2510 may also facilitate communications using various communication protocols. The communication infrastructure 2512 may also include hardware, software, or both that couples components of the computing device 2500 to each other. For example, the communication interface 2510 may use one or more networks and / or protocols to enable a plurality of computing devices connected by a particular infrastructure to communicate with each other to perform one or more aspects of the processes described herein. To illustrate, the digital content campaign management process can allow a plurality of devices (e.g., a client device and server devices) to exchange information using various communication networks and protocols for sharing information such as electronic messages, user interaction information, engagement metrics, or campaign management resources.

[0262] In the foregoing specification, the present disclosure has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the present disclosure(s) are described with reference to details discussed herein, and the accompanying 2023229591   15 Sep 2023 drawings illustrate the various embodiments. The description above and drawings are illustrative of the disclosure and are not to be construed as limiting the disclosure. Numerous specific details are described to provide a thorough understanding of various embodiments of the present disclosure.

[0263] The present disclosure may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with less or more steps / acts or the steps / acts may be performed in differing orders. Additionally, the steps / acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar steps / acts. The scope of the present application is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. A system for generating a modified two-dimensional image comprising:a memory component; anda processing device coupled to the memory component, the processing device to perform operations comprising:determining, utilizing one or more neural networks, first pixel depth values representing first estimated depths at pixels of a two-dimensional image;generating, utilizing the one or more neural networks, a first three-dimensional mesh by generating a first tessellation incorporating depth information from the first pixel depth values of the two-dimensional image;determining, in response to a displacement input to a portion of the two-dimensional image within a graphical user interface displaying the two-dimensional image, that the twodimensional image is modified to create a modified two-dimensional image comprising at least one modified portion of the two-dimensional image according to a corresponding modified portion of the first three-dimensional mesh;determining, utilizing the one or more neural networks, second pixel depth values representing second estimated depths at pixels of the modified two-dimensional image;generating, utilizing the one or more neural networks in response to determining that the two-dimensional image is modified to create the modified two-dimensional image, a second three-dimensional mesh by generating a second tessellation incorporating depth information of the modified two-dimensional image based on the second pixel depth values for the modified two-dimensional image according to the at least one modified portion of the modified twodimensional image; andutilizing the second three-dimensional mesh for further modifications to the modified two-dimensional image.

2. The system of claim 1, wherein:generating the first three-dimensional mesh comprises generating the first tessellation based on a first set of sampled points of the two-dimensional image; andgenerating the second three-dimensional mesh comprises generating the second tessellation based on a second set of sampled points of the modified two-dimensional image.

3. The system of claim 2, wherein:2023229591   29 May 2026generating the first three-dimensional mesh comprises determining, utilizing the one or more neural networks, the first pixel depth values according to estimated camera parameters corresponding to a viewpoint of the two-dimensional image; andgenerating the second three-dimensional mesh comprises determining, utilizing the one or more neural networks, the second pixel depth values according to the estimated camera parameters corresponding to the viewpoint of the two-dimensional image.

4. The system of claim 1, wherein generating the second three-dimensional meshcomprises generating the second three-dimensional mesh in response to a request to commit the at least one modified portion of the two-dimensional image.

5. The system of claim 1, wherein determining the modified two-dimensional imagecomprises:determining a two-dimensional position of the displacement input within the twodimensional image;determining a three-dimensional position of the displacement input on the first threedimensional mesh based on the two-dimensional position of the displacement input within the two-dimensional image; anddetermining the at least one modified portion of the two-dimensional image at the twodimensional position based on a modified portion of the first three-dimensional mesh at the three-dimensional position according to the displacement input.

6. The system of claim 5, wherein determining the modified two-dimensional imagecomprises:determining a displacement direction of the displacement input relative to the first three-dimensional mesh; anddetermining the modified portion of the first three-dimensional mesh according to the displacement direction of the displacement input.

7. The system of claim 1, wherein generating the second three-dimensional meshcomprises interpolating, in connection with the at least one modified portion of the twodimensional image, vertex positions of a plurality of vertices in a portion of the second threedimensional mesh corresponding to an obscured portion of the two-dimensional image.2023229591   29 May 20268.     The system of claim 1, wherein determining the modified two-dimensional imagecomprises generating, utilizing an inpainting neural network, inpainted image content for the at least one modified portion of the two-dimensional image in response to detecting an artifact in the at least one modified portion of the two-dimensional image.

9. A non-transitory computer readable medium comprising instructions, which whenexecuted by a processing device, cause the processing device to perform operations to generate a modified two-dimensional image comprising:determining, utilizing one or more neural networks, pixel depth values representing estimated depths at pixels of a two-dimensional image;generating, utilizing the one or more neural networks, a three-dimensional mesh by generating a tessellation incorporating depth information from the pixel depth values of the two-dimensional image;determining that the two-dimensional image is modified to create a modified twodimensional image comprising at least one modified portion of the two-dimensional image in response to a displacement input to a portion of the two-dimensional image within a graphical user interface displaying the two-dimensional image;determining, utilizing the one or more neural networks, new pixel depth values representing new estimated depths at pixels of the modified two-dimensional image;generating, utilizing the one or more neural networks in response to determining that the two-dimensional image is modified to create the modified two-dimensional image, an updated three-dimensional mesh by generating an updated tessellation incorporating depth information of the modified two-dimensional image based on the new pixel depth values for the modified two-dimensional image according to the at least one modified portion of the modified two-dimensional image; andutilizing the second three-dimensional mesh for further modifications to the modified two-dimensional image.

10. The non-transitory computer readable medium of claim 9, wherein generating the threedimensional mesh comprises:generating the three-dimensional mesh comprises:sampling a first set of points of the two-dimensional image according to first density values corresponding to pixels of the two-dimensional image; and2023229591   29 May 2026generating the three-dimensional mesh based on the first set of points sampled in the two-dimensional image; andgenerating the updated three-dimensional mesh comprises:sampling a second set of points of the modified two-dimensional image according to second density values corresponding to pixels of the modified two-dimensional image; andgenerating the updated three-dimensional mesh based on the second set of points sampled in the modified two-dimensional image.

11. The non-transitory computer readable medium of claim 9, wherein determining the modified two-dimensional image comprises:determining a displaced portion of the three-dimensional mesh based on one or more displacement directions of the displacement input; anddetermining the at least one modified portion of the two-dimensional image based on the displaced portion of the three-dimensional mesh.

12. The non-transitory computer readable medium of claim 9, wherein generating the updated three-dimensional mesh comprises determining the new pixel depth values for the modified two-dimensional image in response to detecting an action to commit the at least one modified portion.

13. A method for generating a modified two-dimensional image comprising: determining, by at least one processor utilizing one or more neural networks, pixel depth values representing estimated depths at pixels of a two-dimensional image;generating, by the at least one processor utilizing the one or more neural networks, a three-dimensional mesh by generating a tessellation incorporating depth information from the pixel depth values of the two-dimensional image;determining, by the at least one processor, that the two-dimensional image is modified to create a modified two-dimensional image comprising at least one modified portion of the two-dimensional image in response to a displacement input to a portion of the two-dimensional image within a graphical user interface displaying the two-dimensional image;determining, by the at least one processor utilizing the one or more neural networks, new pixel depth values representing new estimated depths at pixels of the modified twodimensional image;2023229591   29 May 2026generating, by the at least one processor utilizing the one or more neural networks in response to determining that the two-dimensional image is modified to create the modified two-dimensional image, an updated three-dimensional mesh by generating an updated tessellation incorporating depth information of the modified two-dimensional image based on the new pixel depth values for the modified two-dimensional image according to the at least one modified portion of the modified two-dimensional image; andutilizing the second three-dimensional mesh for further modifications to the modified two-dimensional image.

14. The method of claim 13, wherein generating the three-dimensional mesh comprises:generating the three-dimensional mesh by determining displacement of vertices of the tessellation from the depth information of the two-dimensional image based on the pixel depth values and estimated camera parameters; orgenerating the three-dimensional mesh based on a plurality of points sampled in the two-dimensional image according to density values determined from the pixel depth values of the two-dimensional image.

15. The method of claim 13, wherein determining the modified two-dimensional image comprises:determining a displaced portion of the three-dimensional mesh based on the displacement input to the portion of the two-dimensional image; andgenerating the modified two-dimensional image according to the displaced portion of the three-dimensional mesh.

16. The method of claim 15, wherein generating the updated three-dimensional mesh comprises:sampling a plurality of points of the modified two-dimensional image according to density values determined from the new pixel depth values corresponding to the pixels of the modified two-dimensional image; andgenerating the updated three-dimensional mesh based on the plurality of points sampled in the modified two-dimensional image.

17. The method of claim 13, wherein generating the updated three-dimensional mesh comprises:2023229591   29 May 2026detecting an action to generate the modified two-dimensional image by committing a displacement of the at least one modified portion to the two-dimensional image; andgenerating the updated three-dimensional mesh in response to committing the displacement of the at least one modified portion to the two-dimensional image.

18. The method of claim 13, wherein generating the updated three-dimensional mesh comprises:determining that an initial position of the at least one modified portion of the twodimensional image obscures an additional portion of the two-dimensional image; andgenerating the updated three-dimensional mesh by interpolating vertex positions in a portion of the three-dimensional mesh corresponding to the additional portion of the two-dimensional image obscured by the initial position of the at least one modified portion of the two-dimensional image.

19. The method of claim 13, wherein generating the modified two-dimensional image comprises:determining that the at least one modified portion of the two-dimensional image comprises an image artifact; andgenerating, utilizing an inpainting neural network, inpainted image content correcting the image artifact within the at least one modified portion.

20. The method of claim 13, further comprising generating, in a plurality of displacement iterations comprising a plurality of displacement inputs within the graphical user interface, a plurality of updated three-dimensional meshes corresponding to a plurality of modified twodimensional images in connection with the modified two-dimensional image.

Images