Image modification technology

Abstract

Systems and techniques for modifying the scale and / or position of an object in an image are described herein. For example, the system obtains a two-dimensional (2D) input image and a three-dimensional (3D) representation of the 2D input image from a camera. The system may also determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image. The system may adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object. The system may also generate a 2D output image with a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object for output on a display.

Description

Technical Field

[0001] In general, the present disclosure relates to image adjustment or modification. For example, aspects of the present disclosure relate to systems and techniques for modifying the scale and / or position of objects in an image. Background Art

[0002] Many devices and systems allow for capturing a scene by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera or a device including a camera can capture a sequence of frames of a scene (e.g., a video of the scene). In some cases, the sequence of frames can be processed for performing one or more functions, can be output for display, can be output for processing and / or consumption by other devices, and other uses.

[0003] When using a device to capture an image, a user may have to choose between different available fields of view provided by different image capture devices. In some cases, a user may choose whether to use a telephoto lens or a telephoto lens that results in larger objects being depicted in an image with a narrow field of view, or to use a wide-angle lens with a larger field of view. For example, an image captured using a wide-angle lens may capture a larger portion of a scene (due to the wide field of view), but the objects in the image may be relatively small.

[0004] The user may provide an input (e.g., a zoom input, such as a pinch-to-zoom input) to increase the size of a portion of the image (e.g., to enlarge the two people in the lower left of the image above). However, the entire image will also be enlarged, resulting in most of the background being outside the field of view of the zoomed image. Summary of the invention

[0005] In some examples, systems and techniques for adjusting one or more objects or features in an image are described. According to at least one example, a method for processing one or more images is provided. The method includes: obtaining a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjusting a pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; generating a 2D output image having a modified version of the target object for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object.

[0006] In another example, an apparatus for processing one or more images is provided, comprising: a memory (e.g., configured to store data, such as virtual content data, one or more images, etc.); and one or more processors (e.g., implemented in a circuit system) coupled to the memory. The one or more processors are configured and can perform the following operations: obtain a two-dimensional (2D) input image from a camera; obtain a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; generate a 2D output image with a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object for output on a display.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to perform the following operations: obtain a two-dimensional (2D) input image from a camera; obtain a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image; adjust a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object; and generate a 2D output image having a modified version of the target object for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object.

[0008] In another example, an apparatus for processing one or more images is provided. The apparatus includes: a unit for obtaining a two-dimensional (2D) input image from a camera; a unit for obtaining a three-dimensional (3D) representation of the 2D input image; a unit for determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; a unit for adjusting a pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and a unit for generating a 2D output image having a modified version of the target object for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object.

[0009] In some aspects, at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-portion of the target object in the 2D input image.

[0010] In some aspects, adjusting the pose of the first portion of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.

[0011] In some aspects, depth values ​​of first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object are reduced.

[0012] In some aspects, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further comprises increasing depth values ​​of second one or more vertices of the first portion of the 3D representation of the 2D input image.

[0013] In some aspects, the above-mentioned methods, apparatus and computer-readable media also include: obtaining user input identifying the target object; the posture of the first part of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

[0014] In some aspects, the above-mentioned methods, apparatus and computer-readable media also include: obtaining an indication of an adjustment amount for the target object; and adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the adjustment amount for the target object.

[0015] In some aspects, the above-mentioned methods, apparatus and computer-readable media also include: determining the adjustment amount of the target object based on one or more of an adjustment range, an apparent size change and a maximum distortion amount; adjusting the posture of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.

[0016] In some aspects, the above-mentioned methods, apparatuses, and computer-readable media also include: obtaining an object mask associated with the target object in the 2D input image; determining the first portion of the 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask; and combining the first portion of the 3D representation having an adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.

[0017] In some aspects, generating the 2D output image includes converting the combined 3D mesh to the 2D output image.

[0018] In some aspects, the above-mentioned methods, apparatuses, and computer-readable media also include: obtaining object masks associated with one or more additional objects in the 2D input image that are different from the target object; determining the first portion of the 3D representation of the 2D input image corresponding to the target object based on the object masks; determining a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object masks; adjusting the posture of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.

[0019] In some aspects, adjusting the posture of the first portion of the 3D representation of the 2D input image corresponding to the target object includes: reducing the depth values ​​of the vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object so that the vertices have a predetermined numerical correspondence relative to one or more fixed vertices of the second portion of the 3D representation.

[0020] In some aspects, generating the 2D output image includes converting the adjusted 3D mesh to the 2D output image.

[0021] In some aspects, determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.

[0022] In some aspects, determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

[0023] In some aspects, the above methods, apparatus, and computer-readable media further include adjusting the pose of the first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.

[0024] In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on user input.

[0025] In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration.

[0026] In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted based on machine learning of preferences and past behavior of a user.

[0027] In some aspects, the above-mentioned methods, apparatus and computer-readable media also include: determining a second part of the 3D representation of the 2D input image corresponding to a second target object in the 2D input image; adjusting the posture of the second part of the 3D representation of the 2D input image corresponding to the second target object; and further generating the 2D output image based on the adjusted posture of the second part of the 3D representation of the 2D input image corresponding to the second target object.

[0028] In some aspects, the adjustment to the pose of the second portion is different than the adjustment to the pose of the first portion.

[0029] In some aspects, the modification of at least a portion of the modified version of the target object comprises one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.

[0030] In some aspects, the above methods, apparatus, and computer-readable media further include: obtaining a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image; obtaining a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.

[0031] In some aspects, one or more of the above devices are, are part of, or include a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a computing device of a vehicle), or other device. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors may be used to determine the position and / or posture of the device, the state of the device, and / or for other purposes.

[0032] This summary is neither intended to identify key or essential features of the claimed subject matter nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0033] The foregoing and other features and embodiments will become more fully apparent after reference to the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Illustrative embodiments of the present application are described in detail below with reference to the following drawings:

[0035] Figures 1A to 1C are images depicting different fields of view in example images according to some examples of the present disclosure;

[0036] FIG. 2A to FIG. 2C Example image adjustment techniques according to some examples are shown;

[0037] Figure 3 is a block diagram illustrating the architecture of an image capture and processing device according to some examples of the present disclosure;

[0038] Figure 4 is a block diagram illustrating an example image adjustment system according to some examples of the present disclosure;

[0039] FIG. 5A to FIG. 5K are images illustrating example stages of an image adjustment technique according to some examples of the present disclosure;

[0040] Figure 6 is a flow chart illustrating an example image adjustment process according to some examples of the present disclosure;

[0041] 7A to 7E are images showing example image adjustments according to some examples of the present disclosure;

[0042] Figure 8 is a flow chart illustrating an example of a process for processing one or more images according to some examples;

[0043] Fig. 9 is a block diagram illustrating an example of a deep learning network according to some examples;

[0044] Fig.10 is a block diagram illustrating an example of a convolutional neural network according to some examples;

[0045] Fig.11 is a schematic diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0046] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments and embodiments can be applied independently, and some of them and embodiments can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0047] The following description only provides exemplary embodiments, and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description of the exemplary embodiments will provide a feasible description for implementing the exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of the present application as set forth in the appended claims.

[0048] When capturing an image of a scene, the photographer is often required to choose between capturing a wide field of view (FOV) that includes background objects and information that gives context to the photograph, and capturing a narrow field of view (e.g., using a telephoto lens) to capture foreground objects more prominently and / or in greater detail. Figure 1A-Figure 1C are images showing different scenes based on different selected FOVs. Figure 1A is a photograph of two persons 102 posing in front of a large building 104 captured using a wide FOV. Figure 1A , image 100 depicts building 104 occupying nearly the entire height of image 100 , while person 102 occupies less than a quarter of the height of image 100 . Figure 1B and Figure 1C Two images of the event captured with different FOVs are shown. Figure 1B In , image 110 is captured using a narrow FOV, and a person 112 near the camera (e.g., in the foreground) and a performer 114 are prominently included in image 110, while only a limited portion of background 116 is included in image 110. Figure 1C , image 120 is captured using a wide FOV, and person 112 continues to be prominently displayed due to being close to the camera. In image 120, performer 114 appears smaller relative to person 112, and a larger portion of background 116 provides further context that the event depicted in image 120 is occurring in the stadium.

[0049] Additionally, in some cases, the front-facing camera of the device may not have a zoom function to allow the user to zoom in on a portion of an image (e.g., an object in the background of the image). For example, when capturing a "selfie" image using the front-facing camera of a mobile phone, the user may not be able to provide input to cause the mobile phone to zoom in on an object in the background of the selfie image.

[0050] Image adjustment techniques may be used to adjust a region of interest (ROI) or target object in the captured image to combine the benefits of different FOVs. For example, image adjustment techniques may be used to make performer 114 appear more prominently in image 120 (e.g., by scaling and / or increasing the size of the singer) while maintaining the proportions of background 116.

[0051] FIG. 2A to FIG. 2C Example image adjustment techniques are shown. Figure 2A An example inpainting technique 200 that can be used to adjust a ROI or target object in a 2D image is shown. An input image 202 shows an initial state of a 2D image captured by a camera. In the input image 202, a person 203 is in the foreground of the image, and many other objects (such as trees, buildings, etc.) are in the background 205 of the input image 202. Figure 2A In the example restoration technique 200 shown in FIG. 2 , the target object selected for adjustment is a person 203 , and the adjustment selected is to enlarge (or enlarge) the person 203 .

[0052] Segmented image 204 provides an example of segmentation that may be performed on input image 202 to separate a ROI or target object from other objects in input image 202. In segmented image 204, a region of input image 202 associated with person 203 (represented by a white region shaped like person 203) is shown as being segmented from the rest of input image 202 (represented by a black background surrounding person 203). In some cases, the segmentation shown in segmented image 204 may be performed in whole or in part by a machine learning model. For example, a machine learning model may be trained to detect features in input image 202. In some cases, based on the features detected in input image 202, the machine learning model may classify objects in input image 202. Figure 4 4. The feature segmentation machine learning model 404 shown in FIG. 4 describes further examples of segmentation in more detail. Example classifications for person 203 may include “person,” “foreground object,” etc. In some cases, a user may select which objects (or classifications) in input image 202 are to be the target of adjustment (e.g., enlargement) using restoration technique 200.

[0053] Based on the segmentation between the person 203 and the background 205 of the input image 202 shown in the segmented image 204, the restoration technique 200 can generate a target object image 206 containing pixels associated with the target object (e.g., person 203) in the input image 202 and a non-adjusted image 208 containing pixels associated with the background 205 in the input image 202. In the target object image 206, the pixels associated with the background 205 have been removed, as shown by the blank area 207, while conversely, in the non-adjusted image 208, the pixels associated with the target object (e.g., person 203) have been removed, as shown by the blank area 209. The adjusted image 210 shows an enlargement of the person 203. As shown, the enlargement includes enlarging the size of the person 203 into an enlarged person 211. In some aspects, the technique for enlarging the target object (e.g., person 203) may include interpolating the pixels associated with the target object over a larger portion when compared to the portion occupied by the target object in the input image 202. In some cases, after the enlargement, the pixels of the enlarged person 211 may not completely overlap with the blank area 209 corresponding to the original size of the person 203. Similarly, the blank area 209 may occupy the pixels that the enlarged person 211 does not cover after the enlargement. The inpainted image 212 shows the inpainting process, which attempts to fill the blank area 209 in the non-adjusted image 208 with the color of the portion of the scene captured in the input image 202 that is obscured by the person 203. In one illustrative example, the inpainting can be performed by interpolating and / or blending the colors from the pixels adjacent to the blank area 209 in the input image 202 and applying the interpolated color to the blank area 209. As shown in the inpainted image 212, the outline of the scaled object (e.g., the person 203) in the inpainted portion 213 can be retained after the inpainting is completed. The inpainting technique 200 can combine the adjusted image 210 and the inpainted image 212. In some cases, restoration technique 200 may perform blending and smoothing around the edges of enlarged person 211 and restored portion 213. In another illustrative example, a second input image (not shown) of the same scene or portion of the same scene as input image 202 may include portions of the scene that were occluded in input image 202 by person 203. In some examples, restoration technique 200 may combine pixels from the second input image with adjusted image 210, restored image 212, one or more additional images, or any combination thereof. Image 216 shows the final result of restoration technique 200. Figure 2A The described inpainting techniques may be computationally intensive and may result in artifacts where the inpainting must infer pixel color information for parts of the scene that are not visible in the input image.

[0054] Figure 2B and Figure 2C Another example image adjustment technique 220 is shown for adjusting the appearance of a ROI and / or target object in an image. Figure 2B An example of a 2D image 240 and a corresponding 3D mesh 250 is shown. The 2D image 240 includes a 2D representation of a target object 243 (eg, a vehicle), and the 3D mesh 250 includes a 3D representation of a target object 253. Figure 2B As shown, 2D image 240 can be considered as a projection of 3D mesh 250 onto image plane 248. In some aspects, 3D mesh 250 can also be considered as a back-projection of the 2D image located at image plane 248 into the 3D space of 3D mesh 250. The position of image plane 248 relative to camera 244 can be determined based on parameters of the camera, such as focal length, field of view, skew (e.g., misalignment between the camera and image plane 248), etc. Figure 2B and Figure 2C In the example shown in FIG. 2 , the field of view 270 of the camera 244 is shown as a pyramidal region in the 3D space of the 3D grid. An example pixel 242 on an object in the 2D image 240 and a ray originating from the projection center 245 of the camera 244 The location in the physical world corresponding to the projection center 245 of the camera 244 may be represented as a point in the 3D grid 250. In some cases, if the distance of the target object 243 is at a known distance d from the projection center 245, then the point 246 (e.g., a 3D vertex) corresponding to the example pixel 242 may be located at the ray At a distance d from the projection center 245 in the direction.

[0055] In some cases, the distance information corresponding to each of the pixels of the 2D image may be known when the distance information is included in the 2D image 240. In one illustrative example, the 2D image 240 may include an RGB-D image that includes color information and depth information for each pixel position in the 2D image 240. In some cases, the distance information corresponding to each pixel of the 2D image may be estimated from the 2D image that does not include explicit distance information.

[0056] In one illustrative example, depth information can be inferred by a machine learning system trained to determine depth (or the relative depth of an object in a 2D image). In some cases, the training data set may include a 2D image with depth information for each pixel in the 2D image (e.g., RGB-D) image. In some cases, the training data set may include images with labeled features, objects, and / or classifications (e.g., cars, people, trees, buildings, foreground objects, background objects) and depth information corresponding to the labeled features, objects, and / or classifications. During inference (e.g., after the machine learning model has been trained), the machine learning model may estimate distances corresponding to pixels, features, objects, labels, and / or classifications associated with the 2D image.

[0057] Figure 2C An example technique for generating an adjusted 2D image 260 corresponding to 2D image 240 is shown, wherein Figure 2B 2D representation of target object 253 increases in size and / or has the appearance of being magnified compared to the 2D representation of target object 243 shown in . The magnification effect can be achieved by moving projection center 245 of camera 244 to a new position in 3D grid 250 (e.g., with different vertex coordinates). For example, projection center 245 of camera 244 can be moved closer to the 3D representation of target object 253. As shown, a ray passing through point 246 of the 3D representation of target object 253 The intersection point between intersects image plane 268 at pixel position 262. In some aspects, adjusted 2D image 260 can be constructed by projecting a ray from camera projection center 245 through each of the vertices of 3D mesh 250 (including the vertices of the 3D representation of target object 253) and determining the corresponding pixel position that intersects the projected ray. However, this technique can be computationally intensive.

[0058] Another example implementation for generating adjusted 2D image 260 is to use a perspective projection technique. In one illustrative example, the projection from 3D mesh 250 into adjusted 2D image 260 may include a perspective projection as shown in equation (1).

[0059]

[0060] Where x and y are projected pixel coordinates in the 2D image, X is the X component of the vertex in the 3D mesh (e.g., horizontal axis position), Y is the Y component of the vertex in the 3D mesh (e.g., vertical axis position), and Z is the Z component of the vertex in the 3D mesh (e.g., depth). In the example of equation (1), the origin of the coordinate system of the 3D mesh can be located at the camera projection center 245, which is also referred to as the camera coordinate system. In some cases, if the coordinates of the vertices of the 3D mesh 250 are relative to a coordinate system other than the camera coordinate system, the image adjustment technique 220 can perform a coordinate transformation on the camera coordinate system before projecting the 3D mesh 250 onto the adjusted 2D image 260. In addition to or as an alternative to the perspective projection technique, other projection techniques from 3D to 2D can be used without departing from the scope of the present disclosure, including but not limited to orthogonal projection and oblique projection. In some implementations, a projection technique corresponding to the projection technique of the physical camera that captures the 2D image can be used. In some cases, using a projection technology that corresponds to the projection technology of the physical camera can reduce distortion in the adjusted 2D image, which may be introduced by a mismatch between the physical camera projection technology and the projection technology used to project 3D mesh 250 onto adjusted 2D image 260.

[0061] Compared to Figure 2A The repair technique shown in 200, in Figure 2B and Figure 2C The image adjustment technique 220 shown in FIG. 2 may have advantages because the 3D representation of the target object 253 and the rest of the 3D mesh remain fixed. As a result, no image adjustment techniques such as those in FIG. 2 are created. Figure 2A 209 shown in the figure, and image adjustment technique 220 may not require repair. One disadvantage of the illustrated image adjustment technique 220 is that moving the position of camera 244, thereby moving projection center 245, simultaneously adjusts (e.g., enlarges) all vertices in the 3D mesh, including the 3D representation of target object 253 and vertices in the 3D mesh that are not associated with the target object (not shown), such as other objects (e.g., cars, people, buildings, trees, etc.). As a result, while image adjustment technique 220 achieves the desired effect of adjusting (e.g., enlarging) the target object, image adjustment technique 220 does not provide the ability to adjust only the ROI or target object while leaving other portions of original 2D image 240 unadjusted.

[0062] Systems and techniques are needed for accurately and efficiently adjusting an ROI or target object in an image without affecting other portions of the image. For example, adjusting the appearance of an ROI or target object in an image while keeping other objects and / or portions of the image unadjusted can combine the advantages of both wide FOV and narrow FOV. For example, an ROI that appears small in an input image due to wide FOV image capture can be adjusted in size (e.g., enlarged) to appear larger and / or closer to the camera in the adjusted image.

[0063] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to as "systems and techniques") for selectively adjusting one or more regions of interest (ROIs) of an image without affecting other portions of the image. The systems and techniques allow a user to provide user input to adjust the appearance (e.g., increase its size and / or change its position) of an ROI or target object in an image (e.g., selected by a user, a machine learning model, etc.) without affecting the appearance (e.g., size, position, etc.) of other portions of the image (e.g., foreground objects, background objects, etc.). In one illustrative example, the systems and techniques may enhance a user experience by allowing a user to provide input to selectively zoom in on one or more target objects or ROIs in an image while maintaining a FOV (e.g., a wide FOV provided by a wide-angle lens) such that objects other than the ROI / target object and background in the image are not scaled or are scaled to a lesser degree (compared to the target object).

[0064] In some examples, the image adjustment system can process one or more depth images and one or more color images (e.g., an image with red (R), green (G), and blue (B) color components per pixel, referred to as an RGB image). The depth image can indicate the depth of each pixel in the color image from the camera used to capture the color image. In some aspects, one or more color images are captured using a wide-angle lens. In some cases, the image adjustment system can process one or more monochrome images instead of or in addition to one or more color images. In some examples, a single image can include color information (e.g., RGB) and depth information for each pixel, which is referred to as an RGB depth (RGB-D) image.

[0065] The image adjustment system can use the color image and the depth image corresponding to the color image to generate a three-dimensional (3D) representation (e.g., a 3D mesh) of the two-dimensional (2D) color image. For example, based on the depth information for each pixel indicated by the depth image and the known FOV of the camera (e.g., based on camera parameters, such as intrinsic parameters and / or extrinsic parameters), the system can determine the 3D position of a given pixel of the color image in a 3D space (e.g., a 3D mesh). In an illustrative example, at a pixel distance of 10 meters (from the camera) as indicated by the depth image, the pixel can represent the intersection between rays originating from the camera projection center pointing to the direction corresponding to the 3D position of the pixel. The 3D position corresponding to the pixel in the 2D color image can be located 10 meters away from the camera center in the direction of the ray. Once each pixel of the color image is back-projected, the system can connect the corresponding 3D position (e.g., vertex) to the mesh. The system can use the color information from the color image to texture the mesh (using color). Such a technique is called perspective back-projection (from 2D RGB-D to 3D). In some cases, a 3D mesh may be a triangular or polygonal mesh representing the structure of an item in an image (e.g., an object such as a person or other physical object in an image). For example, a 3D mesh may include a collection of vertices, edges, and faces that define the shape of an object in the 3D space of a scene.

[0066] The image adjustment system may also perform object detection (e.g., face detection) on the color image to detect one or more objects (e.g., faces) in the image. The user may provide user input (e.g., via a touch screen interface, using voice input, using gesture input, gaze input, etc.) to indicate or select one or more ROIs (which may be associated with a target object or multiple target objects from the detected one or more objects) in the color image that the user would like to adjust (e.g., make it larger, change rotation, etc.).

[0067] In some cases, the most corrected object (e.g., most aligned with the vertical axis or horizontal axis of the 2D image) can be used to indicate or select one or more ROIs. In an illustrative example, the posture of the camera and / or the device incorporating the camera can be used to determine the most corrected object. In some cases, the scene or part of the scene captured by the first camera in the 2D color image can also be captured by the second camera. In some cases, an object or part of the scene in the scene captured by both the first camera and the second camera can be used to determine one or more ROIs. In some cases, the first camera and the second camera can be included in a single device (e.g., a mobile device with multiple cameras). In some cases, the first camera and the second camera can be included in separate devices.

[0068] In some examples, the image adjustment system can perform segmentation (e.g., binary segmentation, semantic segmentation, instance segmentation, etc.) on the color image to generate an object mask that identifies one or more objects in the color image. In one example, such as when capturing a selfie image, the object mask can indicate pixels in the image that belong to the object closest to the camera and pixels in the image that do not belong to the object closest to the camera. The image adjustment system can use the object mask to prevent one or more objects identified in the object mask from being adjusted (e.g., by preventing adjustment or fixing vertices associated with the masked object).

[0069] Based on the 3D mesh, object masking, and selection of an ROI (corresponding to a target object), an image adjustment system may determine an adjustment portion of the 3D mesh corresponding to the ROI (or target object) and a non-adjustment portion of the 3D mesh (e.g., a portion of the 3D mesh that is prevented from adjustment). The image adjustment system may deform the geometry (e.g., vertices, edges, and faces) of the adjustment portion of the 3D mesh corresponding to the ROI to effectively increase the size, change the rotation, or otherwise adjust the corresponding scale of a two-dimensional (2D) output image to which the deformed 3D mesh will be reprojected. The deformation of the adjustment portion of the 3D mesh corresponding to the ROI may leave the non-adjustment portion of the 3D mesh unchanged. In some cases, the deformation of the adjustment portion of the 3D mesh corresponding to the ROI may also leave a sub-portion of the adjustment portion that is not affected by the deformation of the geometry of the ROI unchanged.

[0070] In one illustrative example, to deform the geometry of the adjusted portion of the 3D mesh to cause the ROI in the 2D image to be enlarged, the system and technology can reduce the depth value (e.g., z value) of the vertex (and therefore the edge and face) within the adjusted portion of the 3D mesh, so that the vertex is closer to the camera. In addition or alternatively, to deform the geometry of the adjusted portion of the 3D mesh, the system can translate the geometry of the adjusted portion along a translation vector that maintains the position of the sub-portion of the ROI in the 2D output image. For example, the translation vector can be selected to maintain a fixed 2D pixel position of a sub-portion of the ROI / target object (e.g., the center of the face). In some cases, the direction of the translation vector can correspond to the direction of a ray from the center of the camera through the fixed 2D pixel position and the corresponding 3D vertex. As used herein, deforming the geometry of the adjusted portion can also be referred to as adjusting the pose of the geometry of the first portion of the 3D mesh. In some implementations, after deforming the adjusted portion, the image adjustment system can fuse or combine the deformed (or pose-adjusted) adjusted portion of the 3D mesh with the non-adjusted portion of the 3D mesh. In some implementations, the adjusted and non-adjusted portions can be maintained in a single 3D mesh, where 3D vertices corresponding to the non-adjusted portions are fixed and excluded from any deformations to the adjusted portions that may occur proximate to the non-adjusted portions.

[0071] The image adjustment system can then convert or translate the 3D mesh (including the deformed adjusted portion corresponding to the target object) to the 2D image domain, resulting in an output 2D image with a modified version of the ROI / target object. In one illustrative example, after the portion of the 3D mesh corresponding to the ROI / target object is deformed, the ROI / target object in 2D can be enlarged (compared to the original image). For example, a perspective projection technique can be performed to convert the 3D mesh from the 3D domain to the 2D image domain. As a result, the output image includes the wide field of view of the wide-angle lens and an enlarged version of the ROI / target object.

[0072] Although examples are described herein for adjusting one or more ROIs in an image obtained from a handheld camera, the image adjustment systems and techniques described herein can be used to perform image adjustments for images obtained by other devices, such as cameras in vehicles, robotic equipment in a factory environment, and / or any other system that includes a camera.

[0073] Various aspects of the technology described herein are discussed below with respect to the accompanying figures. Figure 3is a block diagram illustrating the architecture of an image capture and processing system 300. The image capture and processing system 300 includes various components for capturing and processing an image of a scene (e.g., an image of a scene 310). The image capture and processing system 300 can capture an independent image (or photograph) and / or can capture a video including multiple images (or video frames) in a specific order. A lens 315 of the image capture and processing system 300 faces the scene 310 and receives light from the scene 310. The lens 315 bends the light toward the image sensor 330. The light received by the lens 315 passes through an aperture controlled by one or more control mechanisms 320 and is received by the image sensor 330.

[0074] The one or more control mechanisms 320 may control exposure, focus, and / or zoom based on information from the image sensor 330 and / or based on information from the image processor 350. The one or more control mechanisms 320 may include multiple mechanisms and components; for example, the control mechanisms 320 may include one or more exposure control mechanisms 325A, one or more focus control mechanisms 325B, and / or one or more zoom control mechanisms 325C. The one or more control mechanisms 320 may also include additional control mechanisms beyond those shown, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0075] A focus control mechanism 325B in the control mechanism 320 may obtain a focus setting. In some examples, the focus control mechanism 325B stores the focus setting in a storage register. Based on the focus setting, the focus control mechanism 325B may adjust the position of the lens 315 relative to the position of the image sensor 330. For example, based on the focus setting, the focus control mechanism 325B may move the lens 315 closer to the image sensor 330 or further away from the image sensor 330 by driving a motor or servo device (or other lens mechanism), thereby adjusting the focus. In some cases, an additional lens may be included in the image capture and processing system 300, such as one or more micro lenses on each photodiode of the image sensor 330, each of which bends the light received from the lens 315 toward the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid focus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 320, the image sensor 330, and / or the image processor 350. Focus settings may be referred to as image capture settings and / or image processing settings.

[0076] An exposure control mechanism 325A in control mechanism 320 may obtain an exposure setting. In some cases, exposure control mechanism 325A stores the exposure setting in a storage register. Based on this exposure setting, exposure control mechanism 325A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 330 (e.g., ISO speed or film speed), the analog gain applied by image sensor 330, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0077] A zoom control mechanism 325C in the control mechanism 320 may obtain a zoom setting. In some examples, the zoom control mechanism 325C stores the zoom setting in a storage register. Based on the zoom setting, the zoom control mechanism 325C may control the focal length of a lens element assembly (lens assembly), which includes the lens 315 and one or more additional lenses. For example, the zoom control mechanism 325C may control the focal length of the lens assembly by driving one or more motors or servos (or other lens mechanisms) to move one or more of the lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a zoom zoom lens. In some examples, the lens assembly may include a focusing lens (which may be a lens 315 in some cases), which first receives light from the scene 310, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., lens 315) and the image sensor 330 before the light reaches the image sensor 330. In some cases, the afocal zoom system can include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other) with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 325C moves one or more of the lenses in the afocal zoom system, such as a negative lens and one or both of the positive lenses.

[0078] Image sensor 330 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 330. In some cases, different photodiodes may be covered by different color filters, and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, wherein each pixel of an image is generated based on red light data from at least one photodiode covered in a red color filter, blue light data from at least one photodiode covered in a blue color filter, and green light data from at least one photodiode covered in a green color filter. Instead of red, blue, and / or green color filters, or in addition to red, blue, and / or green color filters, other types of color filters may use yellow, magenta, and / or cyan (also referred to as "emerald") color filters. Some image sensors (e.g., image sensor 330) may lack color filters entirely, and may instead use different photodiodes (vertically stacked in some cases) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, responding to different wavelengths of light.A monochrome image sensor may also lack color filters, and therefore lack color depth.

[0079] In some cases, the image sensor 330 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). The image sensor 330 may also include an analog gain amplifier for amplifying the analog signal output by the photodiode and / or an analog-to-digital converter (ADC) for converting the analog signal output of the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms in the control mechanism 320 may alternatively or additionally be included in the image sensor 330. The image sensor 330 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0080] The image processor 350 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 354), one or more host processors (including host processor 352), and / or one or more of any other types of processors 1110 discussed with respect to the computing system 1100. The host processor 352 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 350 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 352 and the ISP 354. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 356), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM , global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 356 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an inter-integrated circuit 2 (I2C) interface, an inter-integrated circuit 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a mobile industrial processor interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an advanced high-performance bus (AHB) bus, any combination thereof), and / or other input / output ports. In an illustrative example, the host processor 352 may communicate with the image sensor 330 using an I2C port, and the ISP 354 may communicate with the image sensor 330 using a MIPI port.

[0081] The image processor 350 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 350 may store image frames and / or processed images in a random access memory (RAM) 140 / 3225, a read-only memory (ROM) 145 / 1120, a cache, a memory unit, another storage device, or some combination thereof.

[0082] Various input / output (I / O) devices 360 may be connected to the image processor 350. The I / O devices 360 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1135, any other input device 1145, or some combination thereof. In some cases, subtitles may be input into the image processing device 305B through a physical keyboard or keypad of the I / O device 360 ​​or through a virtual keyboard or keypad of a touch screen of the I / O device 360. The I / O 360 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 300 and one or more peripheral devices, through which the image capture and processing system 300 can receive data from one or more peripheral devices and / or send data to one or more peripheral devices. The I / O 360 may include one or more wireless transceivers that enable a wireless connection between the image capture and processing system 300 and one or more peripheral devices, through which the image capture and processing system 300 can receive data from one or more peripheral devices and / or send data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 360 discussed previously, and once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 360.

[0083] In some cases, the image capture and processing system 300 may be a single device. In some cases, the image capture and processing system 300 may be two or more separate devices, including an image capture device 305A (e.g., a camera) and an image processing device 305B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 305A and the image processing device 305B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 305A and the image processing device 305B may be disconnected from each other.

[0084] like Figure 3 As shown, the vertical dashed line will Figure 3The image capture and processing system 300 is divided into two parts, which are respectively represented by the image capture device 305A and the image processing device 305B. The image capture device 305A includes a lens 315, a control mechanism 320, and an image sensor 330. The image processing device 305B includes an image processor 350 (including an ISP 354 and a host processor 352), a RAM 340, a ROM 345, and an I / O 360. In some cases, some components shown in the image capture device 305A (such as the ISP 354 and / or the host processor 352) can be included in the image capture device 305A.

[0085] The image capture and processing system 300 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 300 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, or some combination thereof). In some implementations, the image capture device 305A and the image processing device 305B may be different devices. For example, the image capture device 305A may include a camera device, and the image processing device 305B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.

[0086] Although the image capture and processing system 300 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 300 may include more than Figure 3Components shown in . The components of the image capture and processing system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 300 may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or the components of the image capture and processing system 300 may include computer software, firmware, or any combination thereof and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 300. In some examples, the image adjustment system 400 may include the image capture and processing system 300, the image capture device 305A, the image processing device 305B, or a combination thereof.

[0087] Figure 4 is a block diagram illustrating an example of an image adjustment system 400. The image adjustment system 400 includes various components for processing one or more images, such as changing the size and / or apparent position of features or objects in one or more images. The image adjustment system 400 can generate a 3D mesh based on an input image, adjust the ROI or object in the 3D mesh to generate an adjusted 3D mesh, and project the adjusted 3D mesh back to a 2D image to generate an adjusted 2D image. As shown, the components of the image adjustment system 400 include one or more image capture devices 402, a feature segmentation machine learning model 404, a 3D mesh engine 406, and a projection engine 408.

[0088] In the following Figure 4 and the description of the image adjustment system 400, refer to FIG. 5A to FIG. 5K The example image shown in FIG. 5A to FIG. 5K Different aspects of image adjustment performed by image adjustment system 400 are shown. FIG. 5A to FIG. 5K , second person 504 is selected for adjustment (e.g., second person 504 is the ROI or selected object). In the example shown, the adjustment applied to second person 504 is selected to bring second person 504 closer to first person 502. In one illustrative example, vertices associated with second person 504 may be adjusted by bringing the vertices closer to the camera position in the 3D coordinate system.

[0089] Return to Figure 4, the image adjustment system 400 may include or be part of any of the following: a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a server computer (e.g., in communication with a vehicle computing system), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the image adjustment system 400 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications (such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, Bluetooth or other short-range communications, any combination thereof, and / or other communications). In some implementations, the components of the image adjustment system 400 (e.g., one or more image capture devices 402, feature segmentation machine learning models 404, 3D mesh engine 406, and projection engine 408) may be part of the same computing device. In some implementations, the components of the image adjustment system 400 may be part of two or more separate computing devices. In some cases, the image adjustment system 400 may be implemented as a Fig.11 A portion of a computing system 1100 is shown in FIG.

[0090] Although image adjustment system 400 is shown as including certain components, one of ordinary skill will appreciate that image adjustment system 400 may include more than the components shown in FIG. Figure 3 In some cases, the additional components of the image adjustment system 400 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, the image adjustment system 400 may include Figure 3One or more other sensors not shown in the image adjustment system 400 (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components. In some implementations, the additional components of the image adjustment system 400 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuits), and / or the additional components of the image adjustment system 400 may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image adjustment system 400.

[0091] One or more images may be captured by one or more image capture devices 402. One or more image capture devices 402 (e.g., cameras or other image sensors) may be included in a mobile device and may be oriented toward a user of the device (e.g., with one or more front-facing cameras) or away from a user of the device (e.g., with one or more rear-facing cameras).

[0092] Each of the one or more image capture devices 402 may include a camera or other type of image sensor. In some examples, the one or more image capture devices 402 may include an IR camera configured to capture infrared (IR) images and / or near infrared (NIR) images. For example, an IR camera or sensor may capture an IR signal. The IR signal has a wavelength and frequency that falls in the IR electromagnetic spectrum. The IR electromagnetic spectrum includes wavelengths in the range of 2,500 nanometers (nm) to 1 millimeter (mm), which corresponds to frequencies ranging from 430 terahertz (THz) to 400 gigahertz (GHz). The infrared spectrum includes an NIR spectrum, which includes wavelengths in the range of 780nm to 2,500nm. In some cases, the image adjustment system 400 may include an IR sensor configured to capture IR and NIR signals. In some cases, separate IR and NIR sensors may be included in the image adjustment system 400. In some examples, the one or more image capture devices 402 may include a camera configured to capture color images and / or monochrome images. Color images may include: red-green-blue (RGB) images; luminance, chrominance-blue, chrominance-red (YCbCr or Y'CbCr) images; and / or any other suitable type of image. In one illustrative example, image adjustment system 400 may include one RGB camera or multiple RGB cameras. In some cases, one or more image capture devices 402 may include one or more IR cameras and one or more RGB cameras.

[0093] In some embodiments, one or more image capture devices 402 may include one or more depth sensors. One or more depth sensors may obtain measurements corresponding to the distance of objects in the captured scene. In an illustrative example, the depth sensor may take the form of a light source that projects a structured or textured light pattern onto one or more objects in the scene, the structured or textured light pattern may include one or more narrow light bands. Then, depth information may be obtained by utilizing the geometric distortion of the projected pattern caused by the surface shape of the object. In some cases, one or more depth sensors may generate a depth image that includes depth values ​​corresponding to pixel positions in one or more images captured by one or more image capture devices 402. In some cases, the depth sensor may be located in the same general position as other sensors of one or more image capture devices 402. In some cases, the depth sensor may capture a depth image simultaneously with images captured by one or more other sensors included in one or more image capture devices 402. In some implementations, the systems and techniques described herein may also be used when inferring depth information from one or more images.

[0094] In one illustrative example, one or more image capture devices 402 and, in some cases, one or more depth sensors may capture one or more RGB-D images. In some cases, one or more image capture devices 402 may capture other image types that include depth information, such as monochrome depth, NIR depth, etc. For purposes of illustration, examples of the present disclosure discuss performing image adjustments on RGB-D images, but the systems and techniques described herein may also be used with other image types that include depth information without departing from the scope of the present disclosure.

[0095] Figure 5A An example 2D input image 500 (eg, an RGB image) is shown, and Figure 5B An example depth image 510 is shown. In some cases, the 2D input image 500 and the depth image 510 may together comprise an RGB-D image. Both the 2D input image 500 and the depth image 510 include representations of a first person 502 positioned closer to the camera and a second person 504 positioned at a greater distance from the camera. In the depth image 510, the brightness of each pixel may represent the distance of the object or feature from the camera. In the example depth image 510, closer objects appear brighter (e.g., the first person 502), while farther objects have lower brightness (e.g., the second person 504 or the background 506).

[0096] Return to Figure 4 , one or more images captured by one or more image capture devices 402 may be provided as input to a feature segmentation machine learning model 404. The feature segmentation machine learning model 404 may be trained to detect spatial information (e.g., features) associated with the one or more images. In some cases, the feature segmentation machine learning model 404 may be further trained to provide one or more classifications to objects in the one or more images based on the detected features. The feature segmentation machine learning model 404 may then use the classifications to segment the images into different parts associated with the one or more classifications. For example, the feature segmentation machine learning model 404 may segment the one or more images into different parts associated with people, buildings, cars, furniture, plants, etc.

[0097] The feature segmentation machine learning model 404 may also be trained to classify objects corresponding to features extracted from one or more images using one or more classifications. A training dataset including example images and classification labels may be used to train the feature segmentation machine learning model 404, such as using the method described below with respect to Fig. 9 and Fig.10In one illustrative example, during inference (e.g., after the feature segmentation machine learning model 404 has been trained), the feature segmentation machine learning model 404 can use the feature vectors to classify one or more input images. Example classifications can include "person," "face," "building," "tree," and any other classifications that the feature segmentation machine learning model 404 is trained to classify.

[0098] In one illustrative example, the feature segmentation machine learning model 404 can perform semantic segmentation on the image. In semantic segmentation, the feature segmentation machine learning model 404 can associate each object in the scene with one or more classifications (also referred to as labels in this article). In some cases, if more than one object has the same label, the semantic segmentation does not distinguish between two objects. In such an example, depth information in one or more captured images (e.g., RGB-D images) can be used to distinguish between objects and further segment between objects. For example, a portion of an input image containing two people can be classified using a single classifier for "person". Figure 5C An example image 520 is shown depicting a first classifier 508 (shown in white) associated with a first person 502 and a second classifier 512 associated with a second person 504. In some examples, the first classifier 508 and / or the second classifier 512 can represent a classification such as “person” or “face.” In some aspects, using semantic segmentation, the feature segmentation machine learning model 404 may not be able to identify the first person 502 and the second person 504 as different objects based on the first classifier 508 and the second classifier 512 alone.

[0099] In another illustrative example, feature segmentation machine learning model 404 can perform instance segmentation. In instance segmentation, feature segmentation machine learning model 404 can utilize the same classifier in different ways to separately identify multiple instances of an object. Figure 5D The image 530 shown in FIG. 5 provides an illustrative example of instance segmentation, where the first instance 514 associated with the second person 504 is shown in gray and the second instance 516 associated with the person 502 is shown in white. In some cases, using instance segmentation, the feature segmentation machine learning model 404 is able to separately identify the first person 502 and the second person 504 as different objects based on the first instance 514 and the second instance 516.

[0100] In some cases, 3D mesh engine 406 can utilize depth information as well as camera parameters (e.g., focal length, field of view, etc.) to generate a 3D mesh corresponding to the RGB-D image. In some cases, for each pixel in the input image, 3D mesh engine 406 can generate a corresponding 3D vertex in a 3D model of the scene depicted in the input image. In one illustrative example, the 3D mesh engine can use a captured 2D input image (e.g., Figure 5A The 3D mesh is generated by using the depth value and field of view of the camera of the input image 500). For example, the 3D mesh engine 406 can generate the 3D mesh using a back-projection technique. Figure 2B and Figure 2C As discussed, the 3D vertex V for a pixel at coordinate x, y in a 3D mesh is determined. x,y An example technique for the location of may include casting a ray from the center of the camera through coordinates x, y, and based on the distance d from the camera located on the ray x,y The 3D coordinates at the corresponding 3D vertex V x,y , where d x,y is the distance value corresponding to the coordinate x, y in the depth image. The 3D mesh engine 406 may perform a back-projection technique for each pixel coordinate x, y in the 2D input image. Any suitable back-projection technique may be used without departing from the scope of the present disclosure. Figure 5E An example image 540 of a 3D mesh 542 generated from a 2D input image is shown. As shown, the 3D mesh 542 may include 3D representations of a first person 502 and a second person 504.

[0101] In some cases, the features, objects, and / or classifications determined by the feature segmentation machine learning model 404 may be provided as input to the 3D mesh engine 406. In some cases, based on the classifications, the 3D mesh engine 406 may associate vertices in the 3D mesh to corresponding features, objects, and / or classifications associated with corresponding pixel locations in the 2D input image. In some cases, such as when the feature segmentation machine learning model 404 provides instance segmentation, the 3D mesh engine may distinguish between different features or objects associated with the same classification based on the output from the feature segmentation machine learning model. In some cases, when the feature segmentation machine learning model 404 may only provide semantic segmentation, the 3D mesh engine 406 may distinguish between different objects or features in the 3D mesh based on the depth values ​​associated with each object. For example, the 3D mesh engine 406 may determine that two objects sharing the same classifier are different objects based on different portions of the 3D mesh associated with the same classifier having different depth values. In one illustrative example, the feature segmentation machine learning model 404 may provide a classification "face" that applies to both the first person 502 and the second person 504. In such an example, 3D mesh engine 406 may determine a depth value at a location in the 3D mesh associated with a “face” classifier (e.g., the center of a bounding box, the tip of the nose, between the eyes, etc.), and determine that other vertices in the 3D mesh that are within a threshold distance from the determined depth are part of the same object.

[0102] In some cases, the 3D mesh engine 406 and / or the feature segmentation machine learning model 404 may also apply one or more masks to regions (e.g., pixels) of the 2D input image and corresponding portions (e.g., vertices) of the 3D mesh. Fig. 5F An image 550 including an example mask is shown, where pixels associated with a person 502 are shown as a masked region 552. In some examples, the 3D mesh engine 406 and / or the feature segmentation machine learning model 404 can select the foreground object closest to the camera to include in the masked region 552. In the example shown, the first person 502 in the foreground of the 2D input image 500 is selected for masking because any incorrect adjustment of vertices associated with the first person 502 may produce noticeable visual artifacts in the adjusted 2D image after the image adjustment is completed. In some cases, other masks and / or multiple masks may be used without departing from the scope of the present disclosure.

[0103] In some cases, 3D mesh engine 406 may determine a non-adjustment portion of a 3D mesh (e.g., 3D mesh 542) based on masked area 552 in the 2D image. In some cases, any portion of the 3D mesh outside of the non-adjustment portion may be referred to as an adjustment region. Including a vertex in an adjustment portion does not necessarily mean that the included vertex will be adjusted by 3D mesh engine 406. In some implementations, based on the determination of the adjustment portion and the non-adjustment portion in the 3D mesh, 3D mesh engine 406 may separate the 3D mesh into an adjustment 3D mesh and a separate non-adjustment 3D mesh.

[0104] Figure 5G An image 560 is shown, which shows an example non-adjusted 3D mesh 562 that includes a 3D representation of person 502 and may be based on Fig. 5F Generated by the masked area 552 shown in . Figure 5H An example image 570 is shown showing an adjusted 3D grid 572 that includes the second person 504 as well as other objects (such as a sofa, a wall, etc.) that are not included in the masked area 552 .

[0105] Back to Figure 4 In some implementations, rather than splitting the 3D mesh into separate non-adjusted portions of the 3D mesh and adjusted portions of the 3D mesh, the 3D mesh engine 406 may instead fix the coordinates of vertices within the non-adjusted portions of the 3D mesh or otherwise prevent adjustment of vertices within the non-adjusted portions of the 3D mesh.

[0106] Once 3D mesh engine 406 has generated a 3D mesh (or multiple 3D meshes) based on a 2D input image, determined one or more masks, and / or performed further segmentation of objects in the 3D mesh based on depth values, 3D mesh engine 406 may adjust one or more portions of the 3D mesh. In some cases, 3D mesh engine 406 may adjust one or more portions of the 3D mesh associated with one or more of the features detected and / or classified by feature segmentation machine learning model 404 as being associated with 3D vertices in the 3D mesh. For example, 3D mesh engine 406 may obtain an indication of one or more selected objects (or ROIs) in the input image to be adjusted. In some cases, the input image (e.g., Figure 5A The indication of one or more selected objects to be adjusted in the 2D input image 500 of the image adjustment system 400 can be based on user input. In one illustrative example, the user can indicate the one or more selected objects using physical contact (e.g., tapping the screen, swiping, etc.), gestures (e.g., detected by a camera or other sensor), using an input device, or by any other means that allows the user to interact with the image adjustment system 400.

[0107] Based on the indication of one or more selected objects in the input image to be adjusted, the 3D mesh engine 406 may adjust the 3D mesh (e.g., Figure 5E 542 shown in ). In some cases, the 3D mesh engine 406 can adjust the vertices associated with one or more ROIs or objects in the 3D mesh. Adjustment of the ROI or object is also referred to as changing the ROI or object. The change in the posture of the ROI or object may include changes in depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, roll rotation, and any combination thereof. In an illustrative example, mesh adjustment may include bringing one or more selected objects closer to the camera in the 3D model. Fig.5I An image 580 is shown showing vertices associated with the second person 504 in the 3D mesh displaced in the direction of arrow 582. In the example shown, the direction of arrow 582 is selected to maintain a fixed position of the sub-portion of the second person 504 in the adjusted 2D image near the center of the face (e.g., nose, between the eyes, etc.).

[0108] In one illustrative example, a specific anchor point of an object can be selected to remain in a fixed position in the adjusted 2D image. In such an example, the posture change can include adjusting the vertices of one or more selected objects along the direction of a ray passing through the center of the camera and the initial position of the 3D vertex corresponding to the anchor point. Another illustrative example of adjusting the posture can include reducing the depth value of a portion or all of the vertices associated with the selected object. In some cases, reducing the depth value of the vertices of the selected object can result in expanding the size of the selected object in the resulting 2D image. Another illustrative example of adjusting the posture can include (simultaneously) increasing and decreasing the depth value of a portion or all of the vertices associated with the selected object. In some cases, increasing and decreasing the depth value of the vertices of the selected object can result in a pitch rotation, a yaw rotation, and / or a roll rotation of the selected object in the resulting 2D image.

[0109] In some cases, the image adjustment system 400 may obtain multiple images of a scene or a portion of a scene captured in a 2D input image. For example, the electronic device may include a first image capture device and a second image capture device (e.g., included in the image capture device 402) that simultaneously capture at least a portion of a common portion of the scene. In some cases, a single image sensor may capture multiple images of a scene to provide, for example, zero shutter lag capture, preview images, etc. In some examples, the image adjustment system may generate multiple 3D meshes based on multiple images of a scene or a portion of a scene. In some cases, the 3D mesh engine 406 may incorporate (e.g., merge, synthesize, repair, etc.) portions from multiple 3D meshes to generate a combined 3D mesh. In an illustrative example, the 3D mesh engine 406 may adjust 3D vertices associated with a selected object in a first 2D image from a first 3D mesh, and also include 3D vertices from a second 3D mesh associated with a second 2D image. In an illustrative example, the 3D vertices associated with the selected object may be horizontally displaced, and the portion of the 3D mesh previously occupied by the selected object may be incorporated into the portion from the second 2D mesh.

[0110] After the 3D mesh is adjusted by the 3D mesh engine 406, the projection engine 408 can project the 3D mesh onto the adjusted 2D image. The projection engine 408 can utilize the above-mentioned Figure 2B and Figure 2C In one illustrative example, the projection engine 408 can utilize the perspective projection of equation (1). In some aspects, without departing from the scope of the present disclosure, the projection engine 408 can use any other projection technology (including but not limited to orthogonal projection or oblique projection) to generate the adjusted 2D image. Figure 5K An adjusted 2D image 595 that can be produced by the image adjustment system 400 is shown. In some implementations, the projection engine 408 can utilize a projection technology that corresponds to the projection technology of the physical camera that captured the 2D image. In some cases, using a projection technology that corresponds to the projection technology of the physical camera can reduce distortion in the adjusted 2D image, which may be introduced by a mismatch between the physical camera projection technology and the projection technology utilized by the projection engine 408.

[0111] Figure 6 An example process 600 for performing image adjustments according to an example of the present disclosure is shown. At block 602, process 600 may (e.g., from Figure 44 ) obtains a 2D input image. At block 604, process 600 may perform segmentation of features or objects included in the input image. For example, process 600 may perform one or more of feature detection, semantic segmentation, instance segmentation, object classification, etc., such as with respect to Figure 4 The feature segmentation machine learning model 404 is described.

[0112] At block 606, process 600 may include generating a 3D mesh based on the depth information and / or color information included in the RGB-D image obtained at block 602. In some cases, for each pixel in the 2D image, the 3D mesh may be generated using the depth information and / or color information included in the RGB-D image obtained at block 602. Figure 2B , Figure 2C and Figure 4 3D mesh engine 406 may generate corresponding 3D vertices in a 3D model of the scene using the back-projection technique described herein or using any other suitable technique for converting a 2D image into a 3D mesh.

[0113] At block 608, process 600 may perform masking of the 3D mesh. In some cases, masking may include creating non-adjusted portions and adjusted portions in the 3D mesh. In some cases, the non-adjusted portions and adjusted portions may be separated into two separate 3D meshes to prevent adjustment of the non-adjusted portions. In some cases, a single mesh may include both the non-adjusted portions and the adjusted portions, and the vertices of the non-adjusted portions may be fixed or otherwise prevented from being adjusted. In some aspects, masking may include vertices in the 3D mesh associated with one or more ROIs or objects.

[0114] At block 610, process 600 may perform mesh adjustment on the 3D mesh generated at block 608. In some cases, process 600 may obtain input indicating that one or more selected objects (or ROIs) in the 2D image have been selected for adjustment. For example, process 600 may adjust the pose of one or more selected objects by changing the coordinates of vertices corresponding to the one or more selected objects. Figure 4 As mentioned in the 3D mesh engine 406 of the process 600, each 2D pixel position in the input image obtained at block 602 can have a corresponding 3D vertex in the 3D mesh generated at block 608. In some cases, the process 600 can associate the 3D vertices generated at block 608 with one or more objects (or features). For example, the features, objects, and / or classifications determined at block 604 and associated with each of the 2D pixel positions of the input image can also be associated with the corresponding 3D vertices. Once the 3D vertices corresponding to the selected objects are identified, the process 600 can adjust the vertices to provide the desired effect (e.g., enlargement, translation, rotation, etc.) in the adjusted 2D image.

[0115] At block 612, process 600 may perform a fusion between two or more portions of the 3D mesh. For example, if process 600 separates the 3D mesh into a non-adjusted 3D mesh and an adjusted 3D mesh at block 608, process 600 may combine the non-adjusted 3D mesh and the adjusted 3D mesh into a single combined 3D mesh (e.g., Figure 5J In some cases, process 600 may not be required to perform any merging of 3D meshes, where process 600 does not generate a separate 3D mesh but instead locks vertices in non-adjusted portions of the 3D meshes.

[0116] At block 614, process 600 may use the information in this document regarding Figure 2B , Figure 2C and Figure 4 The projection technology described by the projection engine 408 of the process 600 is used to generate the adjusted 2D image. An illustrative example of the projection that can be performed by the process 600 is the perspective projection shown in equation (1). However, without departing from the scope of the present disclosure, the process 600 can use any projection or rendering technology that can convert a 3D mesh into a 2D image. However, without departing from the scope of the present disclosure, the process 600 can use any projection or rendering technology that can convert a 3D mesh into a 2D image. In some implementations, the process 600 can use a projection technology corresponding to the physical projection of the camera that captures the 2D image. In some cases, using a projection technology corresponding to the projection technology in the physical camera can reduce distortion in the adjusted 2D image, which may be introduced by a mismatch between the physical camera projection and the projection technology utilized by the process 600 at box 614.

[0117] exist Figure 6 The process 600 shown in FIG. 6 provides an illustrative example of an image adjustment process according to the present disclosure. However, the order of operations of the process 600 may be changed without departing from the scope of the present disclosure. In addition, one or more steps of the process 600 may be omitted or additional steps may be performed without departing from the scope of the present disclosure.

[0118] In some cases, the process 600 may also obtain an input corresponding to an adjustment amount to be applied to one or more objects or features. In some cases, the process 600 may dynamically determine the adjustment amount to be applied to one or more selected objects. In one illustrative example, the process 600 may determine the approximate size of the foreground object (e.g., using a bounding box, calculating the number of image pixels occupied by the foreground object, etc.) and the approximate size of each of the one or more objects. In such an example, the process 600 may determine the adjustment amount based on the relative size of the foreground object and the one or more objects. In one illustrative example, the one or more objects may be adjusted to enlarge the size of the one or more objects in the 2D image so that it has a size corresponding to a specific numerical correspondence (e.g., a specific percentage value, ratio, proportion, function, relationship, etc.) with the size of the foreground object. In one illustrative example, one or more selected objects (or ROIs) may be adjusted to have a size corresponding to a predetermined ratio in a range between 7 / 10 (e.g., 70 percent) and 9 / 10 (e.g., 90 percent) of the size of the foreground object. In some cases, the numerical correspondence (e.g., percentage value) and / or the range for the numerical correspondence (e.g., percentage value range) can be limited based on the distance between the foreground object and the background object in the 3D grid. In another illustrative example, if the distance (e.g., average distance) between the foreground object and the first object in one or more selected objects is small (e.g., less than half a meter (m), less than 1m, less than 3m, etc.), then the first object in one or more selected objects can be adjusted up to a ratio of 1 of the size of the foreground object (e.g., one hundred percent). In another illustrative example, if the distance (e.g., average distance) between the second object in one or more selected objects and the foreground object is large (e.g., greater than 5m, greater than 10m, or any other selected distance), then the second object in one or more selected objects can be adjusted up to a ratio of 4 / 5 (e.g., eighty percent) of the size of the foreground object. Other techniques for determining the amount of adjustment for each object in one or more selected objects can also be used without departing from the scope of the present disclosure. For example, the total change in the pose of the object can be limited to within the adjustment range. In some cases, the adjustment range for changing the pose of the object can be determined based on the maximum amount of distortion in the 2D image that would be caused by performing the pose change. In another example, the maximum adjustment amount for each of the one or more selected objects can be based on a ratio of the apparent size of the object in the adjusted image compared to the original image. For example, the maximum adjustment amount can be limited to allow the apparent size of each object to increase by fifty percent, one hundred percent, two hundred percent, or any other value.

[0119] As mentioned above, the image adjustment system 400 and related techniques described herein can allow the system to adjust an ROI or object in an image without adjusting other objects in the image or scene. For example, the image adjustment system 400 can adjust an image in which two people are sitting apart from each other (e.g., Figure 5A and Figure 5B The image adjustment system may adjust the appearance of an image (shown in FIG. 1 ) to bring two people closer together. In some cases, the adjustment may include reducing the depth of the target object, vertical shifting, pitch rotation, roll rotation, yaw rotation, etc. By performing the adjustment on the 3D grid, the image adjustment system may maintain continuity between the adjusted target object and the remaining portion of the image that has not been adjusted (e.g., continuity of the 3D grid).

[0120] 7A to 7E Additional illustrative examples of image adjustments that can be performed using the systems and techniques of this disclosure are provided. Fig. 7A A 2D input image 700 is shown, which includes a first person 702 in the foreground and a second person 704 , a third person 706 , a fourth person 708 , and a fifth person 710 at different distances behind the first person 702 . Figure 7B An example instance segmentation 720 is shown performed on a 2D input image 700, where each of a first person 702, a second person 704, a third person 706, a fourth person 708, and a fifth person 710 in the image is individually identified. In one illustrative example, each person 702 to 710 is classified as a separate instance of the class "person", such as Figure 7B The different shades shown in Figure 7C In the first adjusted image 740 shown in , each of the second, third, fourth, and fifth persons 704 to 710 has been adjusted using image adjustment techniques as described herein. For example, a 3D mesh corresponding to the 2D input image 700 is generated using depth information included in the RGB-D input image. Figure 7C In the example shown, the positions (eg, vertices) corresponding to the second, third, fourth, and fifth persons 704 to 710 are adjusted in the 3D mesh to be closer to the camera, and the adjusted 3D mesh is reprojected into 2D to generate a first adjusted image 740. Similarly, Fig.7D As shown, the second adjusted image 760 shows an example in which only the third person 706 has his position adjusted closer to the camera, and Fig. 7E The third adjusted image 780 shown in shows an example in which only the fifth person has his position adjusted closer to the camera. In some cases, the positions of the second, third, fourth, and fifth persons 704 to 710 may be dynamically moved back and forth in the adjusted image based on user input. Fig. 7A and FIG. 7C to FIG. 7EIn all the examples shown in , the first person 702 in the foreground remains fixed. In the example 3D mesh generated based on the 2D input image 700, the vertices associated with the person 702 may have been included in the non-adjusted portion and / or the non-adjusted mesh, as described above with respect to Figure 4 The 3D mesh engine 406 is described. 7A to 7E The flexibility of the systems and techniques described herein for image adjustment is shown. For example, the systems and techniques allow adjustment of ROIs or objects in a captured image while maintaining other ROIs or objects in the image fixed.

[0121] Figure 8 8 is a flow chart illustrating an example of a process 800 for processing one or more frames. At block 802, the process 800 includes obtaining a 2D input image from a camera. For example, the 2D input image may be obtained from the image capture and processing system 300, the image capture device 305A, and / or the image processing device 305B.

[0122] At block 804, process 800 includes obtaining a 3D representation of a 2D input image. Figure 4 The 3D mesh engine 406 shown in derives the 3D representation.

[0123] At block 806, process 800 includes determining a first portion of the 3D representation of the 2D input image that corresponds to a first target object in the 2D input image. For example, process 800 may obtain user input indicating a target object and / or ROI. In some cases, based on the user input, process 800 may determine vertices in the 3D mesh associated with the target object and / or ROI. In some cases, determining the first portion of the 3D representation of the 2D input image includes at least one of feature detection and object detection. In some examples, determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

[0124] At box 808, process 800 includes adjusting the pose of a first portion of the 3D representation of the 2D input image corresponding to the first target object. In some examples, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object includes reducing the depth value of the first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object. In some cases, adjusting the pose of the first portion of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation. For example, the target object can be a person, and the modification of the 3D representation of the person can include changing the orientation of the person's head (e.g., rotation around a pitch axis, a yaw axis, and / or a roll axis). In some examples, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of the second one or more vertices of the first portion of the 3D representation of the 2D input image.

[0125] At block 810, process 800 includes generating a 2D output image having a modified version of the target object for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object. At least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-portion of the target object in the 2D input image.

[0126] In some examples, process 800 includes obtaining user input identifying a target object, wherein a pose of a first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

[0127] In some examples, process 800 includes obtaining an indication of an amount of adjustment to the target object, and adjusting a pose of a 3D representation of the 2D input image by an amount corresponding to the indication of the amount of adjustment to the target object.

[0128] In some examples, process 800 includes obtaining an object mask associated with a target object in a 2D input image. In some cases, process 800 includes determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask. In some examples, process 800 includes determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask. In some aspects, process 800 includes combining the first portion of the 3D representation having an adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh. In some cases, generating a 2D output image includes converting the combined 3D mesh into the 2D output image.

[0129] In some examples, process 800 includes obtaining an object mask associated with one or more additional objects in the 2D input image that are different from the target object. In some aspects, process 800 includes determining a first portion of the 3D representation of the 2D input image corresponding to the target object based on the object mask. In some cases, process 800 includes determining a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask. In some examples, process 800 includes adjusting the pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh. In some cases, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object includes reducing the depth values ​​of the vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object, so that these vertices have a predetermined numerical correspondence relative to the fixed one or more vertices of the second portion of the 3D representation. In some aspects, generating a 2D output image includes converting the adjusted 3D mesh into a 2D output image.

[0130] In some cases, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on the user input. In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration. In some examples, the pose of the first portion of the 3D representation of the 2D input image is adjusted based on machine learning of the user's preferences and past behavior. In some implementations, the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.

[0131] In some examples, process 800 includes adjusting a pose of a first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.

[0132] In some examples, process 800 includes determining a second portion of the 3D representation of the 2D input image corresponding to a second target object in the 2D input image. In some cases, process 800 includes adjusting a pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object. In some aspects, process 800 includes generating a 2D output image further based on the adjusted pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object. In some implementations, the adjustment of the pose of the second portion is different from the adjustment of the pose of the first portion.

[0133] In some examples, process 800 includes obtaining a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image. In some aspects, process 800 includes obtaining a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.

[0134] In some examples, the processes described herein (e.g., process 800 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, one or more of the processes described can be performed by Figure 4 In another example, one or more of the processes described may be performed by the image adjustment system 400. Fig.11 For example, with Fig.11 The computing device of the computing system 1100 shown in FIG. 1 may include components of the image adjustment system 400 and may implement Figure 6 The process of 600 Figure 8 Operations of process 800 and / or other processes described herein.

[0135] The computing device may include any suitable device, such as a vehicle or a computing device of a vehicle (e.g., a driver monitoring system (DMS) of the vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch or other wearable device), a server computer, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein (including process 600, process 800, and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0136] Components of a computing device may be implemented in circuitry. For example, a component may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or a component may include and / or be implemented using computer software, firmware, or a combination thereof for performing the various operations described herein.

[0137] Process 600 and process 800 are illustrated as a logical flow diagram, the operations of which represent a series of operations that can be implemented using hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

[0138] In addition, process 600, process 800, and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0139] As mentioned above, various aspects of the present disclosure may utilize a machine learning model or system. Fig. 9It is an illustrative example of a deep learning neural network 900 that can be used to implement the above-mentioned machine learning-based feature segmentation, instance segmentation, depth estimation and / or classification. Input layer 920 includes input data. In an illustrative example, input layer 920 may include data representing pixels of an input image. Neural network 900 includes multiple hidden layers 922a, 922b to 922n. Hidden layers 922a, 922b to 922n include "n" hidden layers, where "n" is an integer greater than or equal to 1. This number of hidden layers can include as many layers as required for a given application. Neural network 900 also includes an output layer 921, which provides an output obtained from the processing performed by hidden layers 922a, 922b to 922n. In an illustrative example, output layer 921 can provide a classification for an object in a 2D input image. Classification may include a category that identifies an activity type (e.g., looking up, looking down, closing eyes, yawning, etc.).

[0140] Neural network 900 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back to itself. In some cases, neural network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read in.

[0141] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 920 can activate the node set in the first hidden layer 922a. For example, as shown in the figure, each input node in the input node of input layer 920 is connected to each node in the node of the first hidden layer 922a. The nodes of the first hidden layer 922a can transform the information of each input node by applying an activation function to the input node information. The information derived from this transformation can then be transferred to the nodes of the next hidden layer 922b and the nodes of the next hidden layer 922b can be activated, and the nodes of the next hidden layer 922b can perform their own specified functions. Example functions include convolution, upsampling, data conversion and / or any other suitable function. Then, the output of hidden layer 922b can activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 922n can activate one or more nodes of the output layer 921, and output is provided at the output layer 921. In some cases, although a node in neural network 900 (e.g., node 926) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0142] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from the training of neural network 900. Once neural network 900 is trained, it can be referred to as a trained neural network, which can be used to classify one or more activities. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnections can have adjustable digital weights that can be adjusted (e.g., based on a training data set), allowing neural network 900 to adapt to the input and be able to learn as more and more data is processed.

[0143] The neural network 900 is pre-trained to process features from the data in the input layer 920 using different hidden layers 922a, 922b to 922n to provide an output through the output layer 921. In an example where the neural network 900 is used to identify features in an image, the neural network 900 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training image has a label indicating a feature in the image (for a feature segmentation machine learning system) or a label indicating a class of activities in each image. In one example using object classification for illustrative purposes, the training images can include images of the number 2, in which case the labels for the images can be [0 0 1 0 0 0 0 0 0 0].

[0144] In some cases, the neural network 900 can use a training process called back propagation to adjust the weights of the nodes. As mentioned above, the back propagation process can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, the loss function, the backward pass, and the parameter update are performed for one training iteration. For each set of training images, this process can be repeated a certain number of iterations until the neural network 900 is trained well enough so that the weights of each layer are accurately adjusted.

[0145] For the example of recognizing an object in an image, a forward pass may include passing a training image through the neural network 900. Prior to training the neural network 900, the weights are initially randomized. As an illustrative example, an image may include an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that position in the array. In one example, the array may include a 28 x 28 x 3 array of numbers having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or luminance and two chrominance components, etc.).

[0146] As mentioned above, for the first training iteration of the neural network 900, the output will likely include values ​​that do not prioritize any particular class due to the random selection of weights at initialization. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 900 is unable to determine low-level features and therefore cannot make an accurate determination about what the classification of the object may be. A loss function may be used to analyze the errors in the output. Any suitable loss function definition may be used, such as a cross entropy loss. Another example of a loss function includes the mean squared error (MSE), which is defined as The loss can be set equal to the value E total .

[0147] For the first training images, the loss (or error) will be high because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 900 can perform a backward pass by determining which inputs (weights) contribute the most to the loss of the network, and the weights can be adjusted so that the loss decreases and is ultimately minimized. The derivative of the loss with respect to the weights (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weights that contribute the most to the loss of the network. After calculating the derivatives, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Among them, w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, while a lower value indicates a smaller weight update.

[0148] The neural network 900 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer, wherein there are multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 900 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), etc.

[0149] Fig.10is an illustrative example of a convolutional neural network (CNN) 1000. The input layer 1020 of the CNN 1000 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of the image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at that position in the array. Using the previous example from above, the array may include a 28 x 28 x 3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a convolutional hidden layer 1022a, an optional non-linear activation layer, a pooling hidden layer 1022b, and a fully connected hidden layer 1022c to obtain an output at an output layer 1024. Although in Fig.10 Only one hidden layer in each hidden layer is shown in , but one of ordinary skill will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1000. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0150] The first layer of CNN 1000 is a convolutional hidden layer 1022a. The convolutional hidden layer 1022a analyzes the image data of the input layer 1020. Each node of the convolutional hidden layer 1022a is connected to a node (pixel) area of ​​the input image called a receptive field. The convolutional hidden layer 1022a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of the convolutional hidden layer 1022a. For example, the area covered by the filter in the input image at each convolution iteration will be the receptive field for the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 1022a. Each connection between a node and the receptive field for that node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of hidden layer 1022a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the image frame example, the filter will have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, which corresponds to the size of the node's receptive field.

[0151] The convolutional nature of the convolutional hidden layer 1022a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1022a can start at the upper left corner of the input image array and can be convolved around the input image. As mentioned above, each convolution iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 1022a. In each convolution iteration, the value of the filter is multiplied by the original pixel value of the corresponding number of the image (for example, a 5x5 filter array multiplied by a 5x5 input pixel value array at the upper left corner of the input image array). The multiplications from each convolution iteration can be added together to obtain the sum for that iteration or node. Next, the process is continued at the next position in the input image according to the receptive field of the next node in the convolutional hidden layer 1022a. For example, the filter can move a step amount (called stride) to the next receptive field. The stride can be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result for that location, resulting in a summed value being determined for each node of the convolutional hidden layer 1022a.

[0152] The mapping from the input layer to the convolutional hidden layer 1022a is called an activation map (or feature map). The activation map includes a value for each node that represents the result of the filter at each location in the input volume. The activation map may include an array that includes the various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24 x 24 array. The convolutional hidden layer 1022a may include several activation maps in order to identify multiple features in the image. Fig.10 The example shown in includes three activation maps. Using these three activation maps, the convolutional hidden layer 1022a can detect three different kinds of features, each of which is detectable over the entire image.

[0153] In some examples, a nonlinear hidden layer can be applied after the convolutional hidden layer 1022a. A nonlinear layer can be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU can increase the nonlinear properties of the CNN 1000 without affecting the receptive field of the convolutional hidden layer 1022a.

[0154] The pooling hidden layer 1022b may be applied after the convolutional hidden layer 1022a (and after the nonlinear hidden layer when used). The pooling hidden layer 1022b is used to simplify the information in the output from the convolutional hidden layer 1022a. For example, the pooling hidden layer 1022b may take each activation map output from the convolutional hidden layer 1022a and generate a condensed activation map (or feature map) using a pooling function. Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 1022a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1022a. In Fig.10 In the example shown in , three pooling filters are used for the three activation maps in the convolutional hidden layer 1022a.

[0155] In some examples, max pooling can be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 1022a. The output from the max pooling filter includes the maximum number in each sub-region around which the filter is convolved. Using a 2x2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (where each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2x2 max pooling filter at each iteration of the filter, where the maximum value among the four values ​​is output as the "maximum" value. If such a max pooling filter is applied to an activation filter with a dimension of 24x24 nodes from the convolutional hidden layer 1022a, the output from the pooling hidden layer 1022b will be an array of 12x12 nodes.

[0156] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0157] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in a region of an image, and discards the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: far fewer features are pooled, thereby reducing the number of parameters required in subsequent layers of CNN 1000.

[0158] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 1022b to each of the output nodes in the output layer 1024. Using the above example, the input layer includes 28x28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 1022a includes 3x24x24 hidden feature nodes based on applying a 5x5 local receptive field (for the filter) to three activation maps, and the pooling hidden layer 1022b includes a layer of 3x12x12 hidden feature nodes based on applying a max pooling filter to 2x2 regions across each of the three feature maps. Extending this example, the output layer 1024 may include ten output nodes. In such an example, each node of the 3x12x12 pooling hidden layer 1022b is connected to each node of the output layer 1024.

[0159] The fully connected layer 1022c can obtain the output of the previous pooled hidden layer 1022b (which should represent an activation map of high-level features) and determine the features most relevant to a particular category. For example, the fully connected layer 1022c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1022c and the pooled hidden layer 1022b can be calculated to obtain probabilities for different categories. For example, if CNN 1000 is used to predict that an object in an image is a person, there will be high values ​​in the activation map representing high-level features of a person (e.g., there are two legs, a face at the top of the object, two eyes at the upper left and upper right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to people).

[0160] In some examples, the output from the output layer 1024 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of categories that the CNN 1000 must choose when classifying an object in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents that objects of ten different categories are [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a 5% probability that the image is an object of the third category (e.g., dog), an 80% probability that the image is an object of the fourth category (e.g., person), and a 15% probability that the image is an object of the sixth category (e.g., kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.

[0161] Fig.11 is a diagram showing an example of a system for implementing certain aspects of the present technology. Specifically, Fig.11 An example of a computing system 1100 is shown, which may be, for example, any computing device constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection to the processor 1110 (e.g., in a chipset architecture). The connection 1105 may also be a virtual connection, a networked connection, or a logical connection.

[0162] In some embodiments, computing system 1100 is a distributed system, wherein the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the functionality of the component is described. In some embodiments, a component may be a physical or virtual device.

[0163] The example computing system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115 (e.g., read only memory (ROM) 1120 and random access memory (RAM) 1125) to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to, immediately adjacent to, or integrated as part of the processor 1110.

[0164] Processor 1110 may include any general purpose processor and hardware or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, which are configured to control processor 1110 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0165] To enable user interaction, the computing system 1100 includes an input device 1145, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 can also include an output device 1135, which can be one or more of a variety of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which can generally govern and manage user input and system output. The communication interface can use a wired and / or wireless transceiver to perform or facilitate the reception and / or transmission of wired or wireless communications, including a transceiver using the following: an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Bluetooth Wireless signal transmission, Bluetooth Low power (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1100 based on the reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction on operation on any particular hardware arrangement, and thus the basic features herein may be easily replaced with improved hardware or firmware arrangements as they are developed.

[0166] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a micro / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or cartridge memory, and / or a combination thereof.

[0167] Storage device 1130 may include software services, servers, services, etc., which cause the system to perform functions when processor 1110 executes code defining such software. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium that interface with the necessary hardware components (e.g., processor 1110, connection 1105, output device 1135, etc.) to perform the function.

[0168] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include the following: carrier waves and / or temporary electronic signals propagated wirelessly or on a wired connection. Examples of non-transitory media may include, but are not limited to: disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may have codes and / or machine-executable instructions stored thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Any appropriate means including memory sharing, message passing, token passing, network transmission, etc. may be used to pass, forward, or send information, independent variables, parameters, data, etc.

[0169] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0170] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those of ordinary skill in the art that these embodiments can be implemented without these specific details. For clarity of explanation, in some cases, the technology herein can be presented as a separate functional block including the following functional blocks, which include equipment, device components, steps or routines in the method embodied in software, or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, additional components can also be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form, so as not to obscure these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown as not having unnecessary details, so as to avoid obscuring these embodiments.

[0171] Each embodiment may be described as a process or method above, and the process or method is depicted as a flow chart, a schematic flow chart, a data flow chart, a structure diagram or a block diagram. Although the flow chart can describe the operation as a sequential process, many operations in these operations can be performed in parallel or simultaneously. In addition, the order of the operation can be rearranged. The process is terminated after its operation is completed, but may have additional steps that are not included in the figure. A process (process) may correspond to a method, a function, a procedure (procedure), a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0172] The processes and methods according to the above examples can be implemented using computer executable instructions, which are stored in or otherwise available from computer readable media. Such instructions may include, for example, instructions or data that cause a general-purpose computer, a special-purpose computer, or a processing device to execute or otherwise be configured to execute a specific function or a specific set of functions. Parts of the computer resources used can be accessed over a network. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0173] The equipment implementing the process and method according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may adopt any of a variety of form factors. When implemented with software, firmware, middleware or microcode, the program code or code segment (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc., of laptop computers, smart phones, mobile phones, tablet devices, or other small form factors. The functions described herein may also be embodied in peripheral devices or plug-in cards. By further example, this function may also be implemented on circuit boards between different chips or different processes executed in a single device.

[0174] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0175] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the inventive concept can be embodied and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for variations limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or collectively. In addition, without departing from the scope of this specification, the embodiments can be used in any number of environments and applications other than the environments and applications described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in alternative embodiments, the method can be performed in an order different from the described order.

[0176] It will be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0177] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing circuits or other hardware to perform the operation, by programming a programmable circuit (e.g., a microprocessor or other appropriate circuit) to perform the operation, or any combination thereof.

[0178] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other appropriate communication interface).

[0179] Claim language or other language in the present disclosure that recites "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0180] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits and steps have been generally described above around their functions. Whether this function is implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the application.

[0181] The technology described herein can also be implemented with electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device mobile phone, or an integrated circuit device with multiple uses (including applications in wireless communication devices, mobile phones, and other devices). Any features described as modules or components can be implemented together in an integrated logic device, or as separate but interoperable logic devices. If implemented with software, the technology can be implemented at least in part by a computer-readable data storage medium, which includes a program code, and the program code includes instructions for executing one or more of the above methods when executed. The computer-readable data storage medium can form a part of a computer program product, and the computer program product can include packaging materials. The computer-readable medium can include a memory or a data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium (such as a propagated signal or wave) that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0182] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any technology described in this disclosure. A general-purpose processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein.

[0183] Illustrative aspects of the disclosure include:

[0184] Aspect 1: A method for processing one or more images, comprising: obtaining a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjusting a pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generating a 2D output image having a modified version of the target object for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object.

[0185] Aspect 2: The method according to aspect 1, wherein at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-portion of the target object in the 2D input image.

[0186] Aspect 3: A method according to any one of Aspects 1 to 2, wherein adjusting the posture of the first part of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

[0187] Aspect 4: A method according to any one of Aspects 1 to 3, wherein adjusting the posture of the first part of the 3D representation corresponding to the 2D input image includes: reducing the depth value of the first one or more vertices of the first part of the 3D representation of the 2D input image corresponding to the target object.

[0188] Aspect 5: The method according to Aspect 4, wherein adjusting the posture of the first part of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of the second one or more vertices of the first part of the 3D representation of the 2D input image.

[0189] Aspect 6: The method according to any one of Aspects 1 to 5 further includes: obtaining user input identifying the target object; wherein the posture of the first part of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

[0190] Aspect 7: The method according to Aspect 6 further includes: obtaining an indication of an adjustment amount for the target object; and adjusting the posture of the 3D representation of the 2D input image by an amount corresponding to the indication of the adjustment amount for the target object.

[0191] Aspect 8: The method according to any one of Aspects 6 to 7 further includes: determining an adjustment amount of the target object based on one or more of an adjustment range, an apparent size change, and a maximum distortion amount; and adjusting the posture of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.

[0192] Aspect 9: The method according to any one of Aspects 6 to 8 further includes: obtaining an object mask associated with the target object in the 2D input image; determining the first part of the 3D representation of the 2D input image corresponding to the target object based on the object mask; determining the second part of the 3D representation corresponding to one or more additional objects based on the object mask; and combining the first part of the 3D representation with the adjusted posture and the second part of the 3D representation to generate a combined 3D mesh.

[0193] Aspect 10: The method according to aspect 9, wherein generating the 2D output image comprises converting the combined 3D mesh into the 2D output image.

[0194] Aspect 11: The method according to any one of Aspects 1 to 10 further includes: obtaining an object mask associated with one or more additional objects in the 2D input image that are different from the target object; determining the first part of the 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second part of the 3D representation corresponding to at least one additional object among the one or more additional objects based on the object mask; and adjusting the posture of the first part of the 3D representation and fixing one or more vertices of the second part of the 3D representation to generate an adjusted 3D mesh.

[0195] Aspect 12: A method according to Aspect 11, wherein adjusting the posture of the first part of the 3D representation of the 2D input image corresponding to the target object includes: reducing the depth value of the vertices of the first part of the 3D representation of the 2D input image corresponding to the target object, so that the vertices have a predetermined numerical correspondence relative to one or more fixed vertices of the second part of the 3D representation.

[0196] Aspect 13: The method according to any one of aspects 1 to 12, wherein generating the 2D output image comprises: converting the adjusted 3D mesh into the 2D output image.

[0197] Aspect 14: The method according to any one of aspects 1 to 13, wherein determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.

[0198] Aspect 15: A method according to any one of aspects 1 to 14, wherein determining the first part of the 3D representation of the 2D input image is based on depth information corresponding to the first part of the 3D representation of the 2D input image.

[0199] Aspect 16: The method according to any one of Aspects 1 to 15 further includes: adjusting the posture of the first part of the 3D representation of the 2D input image without adjusting the second part of the 3D representation of the 2D input image that does not correspond to the target object.

[0200] Aspect 17: The method according to any one of aspects 1 to 16, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on user input.

[0201] Aspect 18: The method according to any one of aspects 1 to 17, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration.

[0202] Aspect 19: The method according to any one of aspects 1 to 18, wherein the pose of the first part of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior.

[0203] Aspect 20: The method according to any one of Aspects 1 to 19, further comprising: determining a second part of the 3D representation of the 2D input image corresponding to a second target object in the 2D input image; adjusting the posture of the second part of the 3D representation of the 2D input image corresponding to the second target object; and further generating the 2D output image based on the adjusted posture of the second part of the 3D representation of the 2D input image corresponding to the second target object.

[0204] Aspect 21: The method according to any one of Aspect 20, wherein the adjustment to the posture of the second part is different from the adjustment to the posture of the first part.

[0205] Aspect 22: A method according to any one of Aspects 1 to 21, wherein the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.

[0206] Aspect 23: The method according to any one of Aspects 1 to 22 further includes: obtaining a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image; and obtaining a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.

[0207] Aspect 24: A device for processing one or more images, comprising: a memory; and one or more processors coupled to the memory and configured to: obtain a two-dimensional (2D) input image from a camera; obtain a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generate a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object for output on a display.

[0208] Aspect 25: The apparatus according to aspect 24, wherein at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-portion of the target object in the 2D input image.

[0209] Aspect 26: An apparatus according to any one of Aspects 24 to 25, wherein adjusting the posture of the first part of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

[0210] Aspect 27: An apparatus according to any one of Aspects 24 to 26, wherein the one or more processors are configured to: reduce the depth values ​​of the first one or more vertices of the first part of the 3D representation of the 2D input image corresponding to the target object.

[0211] Aspect 28: An apparatus according to Aspect 27, wherein adjusting the posture of the first part of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth values ​​of the second one or more vertices of the first part of the 3D representation of the 2D input image.

[0212] Aspect 29: An apparatus according to any one of Aspects 24 to 28, wherein the one or more processors are configured to: obtain user input identifying the target object; and the posture of the first part of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

[0213] Aspect 30: An apparatus according to Aspect 29, wherein the one or more processors are configured to: obtain an indication of an adjustment amount for the target object; and adjust the posture of the 3D representation of the 2D input image by an amount corresponding to the indication of the adjustment amount for the target object.

[0214] Aspect 31: An apparatus according to any one of Aspects 29 to 30, wherein the one or more processors are configured to: determine an adjustment amount of the target object based on one or more of an adjustment range, an apparent size change, and a maximum distortion amount; and adjust the posture of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.

[0215] Aspect 32: An apparatus according to any one of Aspects 29 to 31, wherein the one or more processors are configured to: obtain an object mask associated with the target object in the 2D input image; determine the first portion of the 3D representation of the 2D input image corresponding to the target object based on the object mask; determine the second portion of the 3D representation corresponding to one or more additional objects based on the object mask; and combine the first portion of the 3D representation with the adjusted posture and the second portion of the 3D representation to generate a combined 3D mesh.

[0216] Aspect 33: The apparatus according to any one of aspects 24 to 32, wherein generating the 2D output image comprises converting the combined 3D mesh into the 2D output image.

[0217] Aspect 34: An apparatus according to any one of Aspects 24 to 33, wherein the one or more processors are configured to: obtain an object mask associated with one or more additional objects in the 2D input image that are different from the target object; determine the first portion of the 3D representation of the 2D input image corresponding to the target object based on the object mask; determine a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask; and adjust the posture of the first portion of the 3D representation and fix one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.

[0218] Aspect 35: An apparatus according to Aspect 34, wherein, in order to adjust the posture of the first part of the 3D representation of the 2D input image corresponding to the target object, the one or more processors are configured to: reduce the depth values ​​of the vertices of the first part of the 3D representation of the 2D input image corresponding to the 2D input image, so that the vertices have a predetermined numerical correspondence relative to one or more fixed vertices of the second part of the 3D representation.

[0219] Aspect 36: The apparatus according to any one of aspects 34 to 35, wherein generating the 2D output image comprises: converting the adjusted 3D mesh into the 2D output image.

[0220] Clause 37: The apparatus according to any one of Clauses 24 to 36, wherein determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.

[0221] Aspect 38: An apparatus according to any one of aspects 24 to 37, wherein determining the first part of the 3D representation of the 2D input image is based on depth information corresponding to the first part of the 3D representation of the 2D input image.

[0222] Aspect 39: An apparatus according to any one of Aspects 24 to 38, wherein the one or more processors are configured to: adjust the posture of the first part of the 3D representation of the 2D input image without adjusting the second part of the 3D representation of the 2D input image that does not correspond to the target object.

[0223] Clause 40: The apparatus of any one of clauses 24 to 39, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on user input.

[0224] Clause 41: The apparatus of any one of Clauses 24 to 40, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration.

[0225] Clause 42: The apparatus according to any one of Clauses 24 to 41, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior.

[0226] Aspect 43: An apparatus according to any one of Aspects 24 to 42, wherein the one or more processors are configured to: determine a second part of the 3D representation of the 2D input image corresponding to a second target object in the 2D input image; adjust the posture of the second part of the 3D representation of the 2D input image corresponding to the second target object; and further generate the 2D output image based on the adjusted posture of the second part of the 3D representation of the 2D input image corresponding to the second target object.

[0227] Aspect 44: The apparatus according to Aspect 43, wherein the adjustment to the posture of the second part is different from the adjustment to the posture of the first part.

[0228] Aspect 45: An apparatus according to any one of Aspects 24 to 44, wherein the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.

[0229] Aspect 46: An apparatus according to any one of Aspects 24 to 45, wherein the one or more processors are configured to: obtain a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image; and obtain a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.

[0230] Aspect 47: A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform any of the operations of Aspects 1 to 46.

[0231] Aspect 48: An apparatus comprising means for performing any of the operations of Aspects 1 to 46.

Claims

1. A method for processing one or more images, comprising: Obtain a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; obtaining object masks associated with one or more additional objects in the 2D input image other than the target object, and wherein said determining the first portion of the 3D representation is based on the object masks; determining, based on the object occlusion, a second portion of the 3D representation corresponding to at least one additional object of the one or more additional objects; adjusting a pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh; and A 2D output image having a modified version of the target object is generated for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object.

2. The method according to claim 1, wherein: At least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-portion of the target object in the 2D input image.

3. The method according to claim 1, wherein: Adjusting the pose of the first portion of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.

4. The method according to claim 1, wherein: Adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object comprises: Depth values ​​of first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object are reduced.

5. The method according to claim 4, wherein: Adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further comprises increasing depth values ​​of second one or more vertices of the first portion of the 3D representation of the 2D input image.

6. The method according to claim 1, further comprising: User input identifying the target object is obtained, wherein the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

7. The method according to claim 6, further comprising: obtaining an indication of an amount of adjustment to the target object; as well as The pose of the 3D representation of the 2D input image is adjusted by an amount corresponding to the indication of an amount to adjust the target object.

8. The method according to claim 6, further comprising: determining an adjustment amount for the target object based on one or more of an adjustment range, an apparent size change, and a maximum amount of distortion; as well as The pose of the 3D representation of the 2D input image is adjusted by an amount corresponding to the adjustment amount of the target object.

9. The method according to claim 1, further comprising: Obtaining an object mask associated with the target object in the 2D input image; determining, based on the object mask, the first portion of the 3D representation of the 2D input image corresponding to the target object; determining, based on the object occlusion, a second portion of the 3D representation corresponding to one or more additional objects; as well as The first portion of the 3D representation having the adjusted pose and the second portion of the 3D representation are combined to generate a combined 3D mesh.

10. The method according to claim 9, wherein: Generating the 2D output image includes converting the combined 3D mesh into the 2D output image.

11. The method according to claim 1, wherein: Adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object comprises: The depth values ​​of vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object are reduced so that the vertices have a predetermined numerical correspondence with respect to the fixed one or more vertices of the second portion of the 3D representation.

12. The method according to claim 1, wherein: Generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.

13. The method according to claim 1, wherein: Determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.

14. The method according to claim 1, wherein: Determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

15. An apparatus for processing one or more images, comprising: Memory; as well as one or more processors coupled to the memory and configured to: Obtain a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; obtaining object masks associated with one or more additional objects in the 2D input image that are different from the target object, wherein the first portion of the 3D representation is determined based on the object masks; determining, based on the object occlusion, a second portion of the 3D representation corresponding to at least one additional object of the one or more additional objects; adjusting a pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh; and A 2D output image having a modified version of the target object is generated for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object.

16. The device according to claim 15, wherein: At least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-portion of the target object in the 2D input image.

17. The device according to claim 15, wherein: Adjusting the pose of the first portion of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.

18. The device according to claim 15, wherein: The one or more processors are configured to: Depth values ​​of first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object are reduced.

19. The device according to claim 18, wherein: Adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further comprises increasing depth values ​​of second one or more vertices of the first portion of the 3D representation of the 2D input image.

20. The device according to claim 15, wherein: The one or more processors are configured to obtain user input identifying the target object, wherein the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

21. The device according to claim 20, wherein: The one or more processors are configured to: obtaining an indication of an amount of adjustment to the target object; and The pose of the 3D representation of the 2D input image is adjusted by an amount corresponding to the indication of an amount to adjust the target object.

22. The device according to claim 20, wherein: The one or more processors are configured to: determining an adjustment amount for the target object based on one or more of an adjustment range, an apparent size change, and a maximum amount of distortion; as well as The pose of the 3D representation of the 2D input image is adjusted by an amount corresponding to the adjustment amount of the target object.

23. The device according to claim 15, wherein: The one or more processors are configured to: Obtaining an object mask associated with the target object in the 2D input image; determining, based on the object mask, the first portion of the 3D representation of the 2D input image corresponding to the target object; determining, based on the object occlusion, a second portion of the 3D representation corresponding to one or more additional objects; as well as The first portion of the 3D representation having the adjusted pose and the second portion of the 3D representation are combined to generate a combined 3D mesh.

24. The device according to claim 23, wherein: Generating the 2D output image includes converting the combined 3D mesh into the 2D output image.

25. The device according to claim 16, wherein: To adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object, the one or more processors are configured to: Depth values ​​of vertices of the first portion of the 3D representation of the 2D input image corresponding to the 2D input image are reduced so that the vertices have a predetermined numerical correspondence with respect to the fixed one or more vertices of the second portion of the 3D representation.

26. The device according to claim 16, wherein: Generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.

27. The device according to claim 15, wherein: Determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.

28. The device according to claim 15, wherein: Determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

29. A method of processing one or more images, comprising: obtaining a two-dimensional (2D) input image from an image capture device; obtaining a three-dimensional (3D) representation of the 2D input image, the 3D representation of the 2D input image comprising a first portion corresponding to a target object in the 2D input image and a second portion corresponding to at least one or more additional objects in the 2D input image; obtaining an object mask associated with one or more additional objects, and determining the first portion and the second portion of the 3D representation based on the object mask; adjusting a pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh; as well as A 2D output image having a modified version of the target object is generated for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image.

30. A method of processing one or more images, comprising: generating a three-dimensional (3D) representation of a 2D input image, the 3D representation of the 2D input image comprising a first portion corresponding to a target object in the 2D input image and a second portion corresponding to at least one or more additional objects in the 2D input image; determining the first portion and the second portion of the 3D representation based on object occlusions associated with one or more additional objects; adjusting a pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh; as well as A 2D output image having a modified version of the target object is generated for output on a display based on the adjusted pose of the first portion of the 3D representation of the 2D input image.

Images