Image retouching techniques
The method of creating a 3D representation of images and adjusting object poses addresses the challenge of zooming in on specific objects within a wider field of view, enhancing user experience by maintaining background context and object control.
Patent Information
- Application Number
- JP2024527213
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-11
- Filing Date
- 2022-08-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-08-30
AI Technical Summary
Existing image capture devices require users to choose between a wide-angle lens for a larger field of view and a telephoto lens for larger objects, leading to challenges in zooming in on specific objects without losing background context, and many devices lack zoom functionality for background objects.
A method and apparatus for processing images by obtaining a 2D input image, creating a 3D representation, adjusting the pose of target objects within this representation, and generating a modified 2D output image to enhance the size and position of objects, using techniques such as inpainting and perspective projection to maintain the background context.
Enables users to selectively zoom in on objects within an image while preserving the wider field of view, improving user experience by allowing precise control over object appearance and position without affecting other parts of the image.
Smart Images

Figure 2024543837000001_ABST
Abstract
Description
[Technical field]
[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to adjusting or correcting images. For example, aspects of the present disclosure relate to systems and techniques for correcting the scale and / or position of objects in an image. [Background technology]
[0002] Many devices and systems can capture a scene by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera or a device having a camera can capture a sequence of frames of a scene (e.g., a video of the scene). In some cases, the sequence of frames can be processed to perform one or more functions, output for display, output for processing and / or consumption by other devices, among other uses.
[0003] When using a device to capture an image, a user may have to choose between the different available fields of view offered by different image capture devices. In some cases, a user may choose to use a tele-angle or telephoto lens, which has a narrow field of view resulting in larger objects being depicted in the image, or a wide-angle lens, which has a larger field of view. For example, an image captured using a wide-angle lens may capture a larger portion of a scene (due to the wide field of view), but the objects in the image may be relatively small.
[0004] A user can provide an input (e.g., a zoom input such as a pinch-to-zoom input) to increase the size of a portion of the image (e.g., to enlarge the two people in the lower left of the image above), however, the entire image is also zoomed in and much of the background is outside the field of view of the zoomed image. Summary of the Invention
[0005] In some examples, systems and techniques for adjusting one or more objects or features in an image are described. According to at least one example, a method for processing one or more images is provided. The method includes acquiring a two-dimensional (2D) input image from a camera, acquiring a three-dimensional (3D) representation of the 2D input image, determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image, adjusting a pose of the first portion of the 3D representation of the 2D input image corresponding to the target object, and generating a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object to be output on a display.
[0006] In another example, an apparatus for processing one or more images is provided, the apparatus including a memory (e.g., configured to store data such as virtual content data, one or more images, etc.) and one or more processors coupled to the memory (e.g., implemented in circuitry). The one or more processors are configured and capable of: acquiring a two-dimensional (2D) input image from a camera, acquiring a three-dimensional (3D) representation of the 2D input image, determining a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image, adjusting a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object, and generating a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object to be output on a display.
[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to acquire a two-dimensional (2D) input image from a camera, acquire a three-dimensional (3D) representation of the 2D input image, determine a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image, adjust a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object, and generate a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object to be output on a display.
[0008] In another example, an apparatus for processing one or more images is provided, the apparatus including: means for acquiring a two-dimensional (2D) input image from a camera, means for acquiring a three-dimensional (3D) representation of the 2D input image, means for determining a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image, means for adjusting a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object, and means for generating a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object to be output on a display.
[0009] In some aspects, at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image compared to a corresponding sub-portion of the target object in the 2D input image.
[0010] In some aspects, adjusting the pose of the first portion of the 3D representation of the 2D input image includes one or more of adjusting depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.
[0011] In some aspects, decreasing a depth value of a first one or more vertices of a first portion of the 3D representation of the 2D input image that corresponds to the target object.
[0012] In some aspects, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further includes increasing a depth value of a second one or more vertices of the first portion of the 3D representation of the 2D input image.
[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include obtaining user input identifying a target object, and a pose of a first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.
[0014] In some aspects, the methods, apparatus, and computer-readable media described above further include obtaining an indication of an amount of adjustment of the target object, and adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the amount of adjustment of the target object.
[0015] In some aspects, the methods, apparatus, and computer-readable media described above further include determining an adjustment amount for the target object based on one or more of the adjustment range, the apparent size change, and the maximum amount of distortion, and adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.
[0016] In some aspects, the methods, apparatus, and computer-readable media described above further include obtaining an object mask associated with a target object in the 2D input image, determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask, determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask, and combining the first portion of the 3D representation with the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.
[0017] In some embodiments, generating the 2D output image includes converting the combined 3D mesh into the 2D output image.
[0018] In some aspects, the methods, apparatus, and computer-readable media described above further include obtaining an object mask associated with one or more additional objects in the 2D input image, distinct from the target object; determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask; and adjusting the pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.
[0019] In some aspects, adjusting the pose of a first portion of the 3D representation of the 2D input image corresponding to the target object includes reducing depth values of vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object such that the vertices have a predetermined numerical correspondence relative to one or more fixed vertices of the second portion of the 3D representation.
[0020] In some embodiments, generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.
[0021] In some aspects, determining the first portion of the 3D representation of the 2D input image includes at least one of feature detection and object detection.
[0022] In some aspects, determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.
[0023] In some aspects, the methods, apparatus, and computer-readable media described above further include adjusting a pose of a first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.
[0024] In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on the user input.
[0025] In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration.
[0026] In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior.
[0027] In some aspects, the methods, apparatus, and computer-readable media described above further include determining a second portion of the 3D representation of the 2D input image that corresponds to a second target object in the 2D input image, adjusting a pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object, and generating a 2D output image further based on the adjusted pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object.
[0028] In some embodiments, the adjustments to the pose of the second portion are different than the adjustments to the pose of the first portion.
[0029] In some embodiments, the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.
[0030] In some aspects, the methods, apparatus, and computer-readable media described above further include acquiring a second 2D input image from the camera, the second 2D input image including at least a portion of the scene contained in the 2D input image, and acquiring a second 3D representation of the second 2D input image, where at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.
[0031] In some aspects, one or more of the devices described above are part of or include a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a vehicular computing device), or other device. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors can be used to determine the location and / or pose of the device, the state of the device, and / or for other purposes.
[0032] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.
[0033] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.
[0034] Illustrative embodiments of the present application are described in detail below with reference to the following figures: [Brief description of the drawings]
[0035] [Figure 1A] 1 is an image illustrating different fields of view within an example image, according to some examples of the present disclosure. [Figure 1B] 1 is an image illustrating different fields of view within an example image, according to some examples of the present disclosure. [Figure 1C] 1 is an image illustrating different fields of view within an example image, according to some examples of the present disclosure. [Figure 2A] 1 illustrates an exemplary image adjustment technique, according to some examples. [Figure 2B] 1 illustrates an exemplary image adjustment technique, according to some examples. [Figure 2C] 1 illustrates an exemplary image adjustment technique, according to some examples. [Diagram 3] FIG. 2 is a block diagram illustrating an architecture of an image capture and processing device, according to some examples of the present disclosure. [Figure 4] 1 is a block diagram illustrating an example image adjustment system, according to some examples of the present disclosure. [Figure 5A] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5B]1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5C] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5D] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5E] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5F] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5G] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5H] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5I] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5J] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 5K] 1 is an image illustrating exemplary stages of an image adjustment technique, according to some examples of the present disclosure. [Figure 6] FIG. 4 is a flow diagram illustrating an exemplary image adjustment process, according to some examples of the present disclosure. [Figure 7A] 1 is an image illustrating an example image adjustment, according to some examples of the present disclosure. [Figure 7B] 1 is an image illustrating an example image adjustment, according to some examples of the present disclosure. [Figure 7C] 1 is an image illustrating an example image adjustment, according to some examples of the present disclosure. [Figure 7D] 1 is an image illustrating an example image adjustment, according to some examples of the present disclosure. [Figure 7E] 1 is an image illustrating an example image adjustment, according to some examples of the present disclosure. [Figure 8]FIG. 1 is a flow diagram illustrating an example of a process for processing one or more images, according to some examples. [Figure 9] FIG. 1 is a block diagram illustrating an example of a deep learning network, in accordance with some examples. [Figure 10] FIG. 1 is a block diagram illustrating an example of a convolutional neural network, in accordance with some examples. [Figure 11] FIG. 1 illustrates an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0036] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.
[0037] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the present application as set forth in the appended claims.
[0038] When capturing an image of a scene, a photographer must often choose between capturing a wide field of view (FOV) that includes background objects and information that gives the photograph context, or a narrower field of view (e.g., with a telephoto lens) to capture foreground objects more prominently and / or in more detail. FIGS. 1A-1C are images showing different scenes based on different selected FOVs. FIG. 1A is a photograph of two people 102 posing in front of a large building 104 captured using a wide FOV. In FIG. 1A, image 100 shows that the building 104 occupies almost the entire height of image 100, while the people 102 occupy less than a quarter of the height of image 100. FIGS. 1B and 1C show two images of an event captured with different FOVs. In Figure 1B, image 110 was captured with a narrow FOV, and people 112 near the camera (e.g., in the foreground) as well as actor 114 are prominently included in image 110, while only a limited portion of background 116 is included in image 110. In Figure 1C, image 120 was captured with a wide FOV, and person 112 continues to appear prominently due to his proximity to the camera. In image 120, actor 114 appears small relative to person 112, and a larger portion of background 116 provides further context that the event shown in image 120 is taking place in a stadium.
[0039] Further, in some cases, the device's front camera may not have zoom capabilities to allow a user to zoom in on a portion of an image (e.g., an object in the background of the image). For example, when capturing a "selfie" image using a mobile phone's front camera, the user may not be able to provide input to cause the mobile phone to zoom in on an object in the background of the selfie image.
[0040] Image adjustment techniques can be used to adjust regions of interest (ROI) or target objects within a captured image to combine the benefits of different FOVs. For example, image adjustment techniques can be used to make a performer 114 appear more prominent in an image 120 (e.g., by zooming in on the singer and / or increasing the size of the singer) while maintaining the scale of the background 116.
[0041] 2A-2C illustrate an exemplary image adjustment technique. FIG. 2A illustrates an exemplary inpainting technique 200 that can be used to adjust an ROI or target object in a 2D image. An input image 202 illustrates an initial state of a 2D image captured by a camera. In the input image 202, a person 203 is in the foreground of the image, and many other objects (such as trees, buildings, etc.) are in the background 205 of the input image 202. In the exemplary inpainting technique 200 illustrated in FIG. 2A, the target object selected for adjustment is the person 203, and the selected adjustment is to zoom in (or enlarge) the person 203.
[0042] Segmented image 204 provides one example of segmentation that may be performed on input image 202 to separate an ROI or target object from other objects in input image 202. In segmented image 204, a region of input image 202 associated with person 203 (represented by a white region shaped like person 203) is shown as being segmented from the remainder of input image 202 (represented by a black background surrounding person 203). In some cases, the segmentation shown in segmented image 204 may be performed in whole or in part by a machine learning model. For example, the machine learning model may be trained to detect features in input image 202. In some cases, based on the detected features in input image 202, the machine learning model may classify objects in input image 202. Further examples of segmentation are described in more detail below with respect to feature segmentation machine learning model 404 shown in FIG. 4. Exemplary classifications of person 203 may include "person", "foreground object", and the like. In some cases, the user may be able to select which objects (or classifications) in the input image 202 are targeted for adjustment (eg, zooming in) by the inpainting technique 200.
[0043] Based on the segmentation between the person 203 and the background 205 of the input image 202 shown in the segmented image 204, the inpainting technique 200 can generate a target object image 206 including pixels of the input image 202 associated with the target object (e.g., the person 203) and an unadjusted image 208 including pixels of the input image 202 associated with the background 205. In the target object image 206, pixels associated with the background 205 have been removed as indicated by the blank area 207, and conversely, in the unadjusted image 208, pixels associated with the target object (e.g., the person 203) have been removed as indicated by the blank area 209. The adjusted image 210 illustrates a zoom-in of the person 203. As shown, the zoom-in includes increasing the size of the person 203 to an enlarged person 211. In some aspects, the technique for zooming in on the target object (e.g., the person 203) can include an interpolation of pixels associated with the target object over a larger portion as compared to the portion occupied by the target object in the input image 202. In some cases, after zooming in, the pixels of the enlarged person 211 may not completely overlap with the blank area 209 corresponding to the original size of the person 203. Similarly, the blank area 209 may occupy pixels that are not covered by the enlarged person 211 after zooming in. The inpainted image 212 shows an inpainting process that attempts to fill the blank area 209 in the unadjusted image 208 with a color that approximates (or estimates) the portion of the scene captured in the input image 202 obscured by the person 203. In one illustrative example, the inpainting may be performed by interpolating and / or blending colors from pixels in the input image 202 adjacent to the blank area 209 and applying the interpolated colors to the blank area 209. As shown in the inpainted image 212, the outline of the zoomed object (e.g., the person 203) in the inpainted portion 213 may remain after the inpainting is completed.The inpainting technique 200 can combine the adjusted image 210 and the inpainted image 212. In some cases, the inpainting technique 200 can perform blending and smoothing around the edges of the enlarged figure 211 and the inpainted portion 213. In another illustrative example, a second input image (not shown) of the same scene or portion of the same scene as the input image 202 may include portions of the scene occluded by the figure 203 in the input image 202. In some examples, the inpainting technique 200 can combine pixels from the second input image with the adjusted image 210, the inpainted image 212, one or more additional images, or any combination thereof. Image 216 shows the final result of the inpainting technique 200. The inpainting technique described with respect to FIG. 2A can be computationally intensive and can result in artifacts where the inpainting must infer pixel color information for portions of the scene that were not visible in the input image.
[0044] 2B and 2C illustrate another exemplary image adjustment technique 220 for adjusting the appearance of a ROI and / or a target object in an image. FIG. 2B illustrates an example of a 2D image 240 and a corresponding 3D mesh 250. The 2D image 240 includes a 2D representation of a target object 243 (e.g., a vehicle), and the 3D mesh 250 includes a 3D representation of the target object 253. As illustrated in FIG. 2B, the 2D image 240 can be considered as a projection of the 3D mesh 250 onto the image plane 248. In some aspects, the 3D mesh 250 can also be considered as a back projection into 3D space of the 3D mesh 250 of the 2D image located in the image plane 248. The position of the image plane 248 relative to the camera 244 can be determined based on camera parameters such as focal length, field of view, skew (e.g., misalignment between the camera and the image plane 248). 2B and 2C, the field of view 270 of the camera 244 is shown as a pyramid-shaped region in the 3D space of the 3D mesh. An exemplary pixel 242 in the 2D image 240 on the object is represented by a ray emanating from the center of projection 245 of the camera 244.
number
number
[0045] In some cases, distance information corresponding to each of the pixels of the 2D image may be known when distance information is included with the 2D image 240. In one illustrative example, the 2D image 240 may include an RGB-D image that includes color and depth information for each pixel location in the 2D image 240. In some cases, distance information corresponding to each of the pixels of the 2D image may be estimated from a 2D image that does not include explicit distance information.
[0046] In one illustrative example, depth information may be inferred by a machine learning system trained to determine depth (or relative depth of an object in a 2D image). In some cases, the training dataset may include 2D images along with depth information for each pixel in the 2D (e.g., RGB-D) image. In some cases, the training dataset may include images including labeled features, objects, and / or classifications (e.g., cars, people, trees, buildings, foreground objects, background objects) and depth information corresponding to the labeled features, objects, and / or classifications. During inference (e.g., after the machine learning model is trained), the machine learning model may estimate distances corresponding to pixels, features, objects, labels, and / or classifications associated with the 2D image.
[0047] 2C illustrates an exemplary technique for generating an adjusted 2D image 260 corresponding to the 2D image 240, where the 2D representation of the target object 263 has an increased size and / or zoomed-in appearance compared to the 2D representation of the target object 243 illustrated in FIG. 2B. The zoom-in effect can be achieved by moving the center of projection 245 of the camera 244 to a new location (e.g., having different vertex coordinates) within the 3D mesh 250. For example, the camera's center of projection 245 can be moved closer to the 3D representation of the target object 253. As illustrated, a ray passing through a point 246 of the 3D representation of the target object 253
number
[0048] Another example implementation for generating the rectified 2D image 260 is to use a perspective projection technique. In one illustrative example, the projection from the 3D mesh 250 to the rectified 2D image 260 may include a perspective projection as shown in equation (1).
number
[0049] The image adjustment technique 220 shown in Figures 2B and 2C may be advantageous over the inpainting technique 200 shown in Figure 2A because the 3D representation of the target object 253 and the remainder of the 3D mesh remain fixed. As a result, no blank areas such as the blank area 209 shown in Figure 2A are generated and the image adjustment technique 220 may not require inpainting. One disadvantage of the illustrated image adjustment technique 220 is that moving the position of the camera 244, and thereby moving the projection center 245, simultaneously adjusts (e.g., zooms in on) all of the vertices in the 3D mesh, including both the 3D representation of the target object 253 and vertices in the 3D mesh that are not associated with the target object (not shown), such as other objects (e.g., cars, people, buildings, trees, etc.). As a result, while the image adjustment technique 220 achieves the desired effect of adjustment (e.g., zooming in) on the target object, the image adjustment technique 220 does not provide the ability to adjust only the ROI or the target object while leaving other portions of the original 2D image 240 unadjusted.
[0050] Systems and techniques are needed to accurately and efficiently adjust an ROI or target object in an image without affecting other parts of the image. For example, adjusting the appearance of an ROI or target object in an image while leaving other objects and / or parts of the image unadjusted can combine the benefits of both wide and narrow FOV. For example, an ROI that appears small in an input image due to wide FOV image capture can be adjusted in size (e.g., zoomed in) to appear larger and / or closer to the camera in the adjusted image.
[0051] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to as "systems and techniques") for selectively adjusting one or more regions of interest (ROIs) of an image without affecting other portions of the image. The systems and techniques enable a user to provide user input to adjust (e.g., increase size and / or change location) the appearance of a ROI or target object (e.g., selected by a user, a machine learning model, etc.) in an image without affecting the appearance (e.g., size, location, etc.) of other portions of the image (e.g., foreground objects, background objects, etc.). In one illustrative example, the systems and techniques can enhance the user experience by allowing a user to provide input to selectively zoom in on one or more target objects or ROIs in an image while maintaining an FOV (e.g., a wide FOV provided by a wide-angle lens) such that objects other than the ROI / target object and the background of the image are not zoomed in or are zoomed to a lesser extent compared to the target object.
[0052] In some examples, the image adjustment system may process one or more depth images and one or more color images (e.g., an image having red (R), green (G), and blue (B) color components per pixel, referred to as an RGB image). The depth image may indicate the depth of each pixel in the color image from a camera used to capture the color image. In some aspects, the one or more color images are captured using a wide-angle lens. In some cases, the image adjustment system may process one or more monochrome images instead of or in addition to the one or more color images. In some examples, a single image may include color information (e.g., RGB) and depth information per pixel, referred to as an RGB-depth (RGB-D) image.
[0053] The image adjustment system can generate a three-dimensional (3D) representation (e.g., a 3D mesh) of the two-dimensional (2D) color image using the color image and a depth image corresponding to the color image. For example, based on the depth information of each pixel indicated by the depth image and the known FOV of the camera (e.g., based on camera parameters such as intrinsic and / or extrinsic parameters), the system can determine the 3D location of a given pixel of the color image in the 3D space (e.g., the 3D mesh). In an illustrative example, a pixel distance of 10 meters (from the camera) indicated by the depth image can represent an intersection between rays emanating from the projection center of the camera pointing in the direction of the 3D location corresponding to the pixel. The 3D location corresponding to the pixel in the 2D color image can be located 10 meters from the camera center in the direction of the rays. Once each pixel of the color image is backprojected, the system can connect the corresponding 3D location (e.g., a vertex) to the mesh. The system can texture the mesh (with color) using color information from the color image. Such a technique is called perspective backprojection (2D RGB-D to 3D). In some cases, the 3D mesh may be a triangle or polygon mesh that represents the structure of items in an image (e.g., objects such as people and other physical objects in an image). For example, a 3D mesh may include a collection of vertices, edges, and faces that define the shape of an object in the 3D space of a scene.
[0054] The image adjustment system can also perform object detection (e.g., face detection) on the color image to detect one or more objects (e.g., faces) in the image. A user can provide user input (e.g., via a touch screen interface, using voice input, using gesture input, gaze input, etc.) to indicate or select one or more ROIs (which may be associated with a target object or multiple target objects from the detected object or objects) in the color image that the user wants to adjust (e.g., make larger, change rotation, etc.).
[0055] In some cases, the most retouched object (e.g., most aligned with a vertical or horizontal axis of the 2D image) can be used to indicate or select one or more ROIs. In one illustrative example, the pose of the camera and / or the device incorporating the camera can be used to determine the most retouched object. In some cases, the scene or a portion of the scene captured in the 2D color image by the first camera can also be captured by the second camera. In some cases, objects within the scene or a portion of the scene captured by both the first and second cameras can be used to determine one or more ROIs. In some cases, the first camera and the second camera can be included in a single device (e.g., a mobile device with multiple cameras). In some cases, the first camera and the second camera can be included in separate devices.
[0056] In some examples, the image adjustment system may perform segmentation (e.g., binary segmentation, semantic segmentation, instance segmentation, etc.) on the color image to generate an object mask that identifies one or more objects in the color image. In one example, such as when capturing a selfie image, the object mask may indicate pixels that belong to an object closest to the camera in the image and pixels that do not belong to an object closest to the camera in the image. The image adjustment system may use the object mask to prevent one or more objects identified in the object mask from being adjusted (e.g., by preventing the adjustment or by fixing vertices associated with the masked objects).
[0057] Based on the selection of the 3D mesh, the object mask, and the ROI (corresponding to the target object), the image adjustment system can determine an adjusted portion of the 3D mesh that corresponds to the ROI (or the target object) and a non-adjusted portion of the 3D mesh (e.g., a portion of the 3D mesh that is prevented from being adjusted). The image adjustment system can morph the geometry (e.g., vertices, edges, and faces) of the adjusted portion of the 3D mesh that corresponds to the ROI to effectively increase the size, change the rotation, or otherwise adjust the corresponding zoom for a two-dimensional (2D) output image onto which the morphed 3D mesh is reprojected. The morphing of the adjusted portion of the 3D mesh that corresponds to the ROI can leave the non-adjusted portion of the 3D mesh unchanged. In some cases, the morphing of the adjusted portion of the 3D mesh that corresponds to the ROI can also leave a sub-portion of the adjusted portion that is not affected by the morphing of the geometry of the ROI unchanged.
[0058] In one illustrative example, to morph the geometry of the adjusted portion of the 3D mesh to zoom in on the ROI in the 2D image, the system and techniques can reduce the depth values (e.g., z-values) of the vertices (and thus the edges and faces) in the adjusted portion of the 3D mesh, thus moving the vertices closer to the camera. Additionally or alternatively, to morph the geometry of the adjusted portion of the 3D mesh, the system can translate the geometry of the adjusted portion along a translation vector that maintains the position of the sub-portion of the ROI in the 2D output image. For example, the translation vector can be selected to maintain a fixed 2D pixel location of the sub-portion of the ROI / target object (e.g., the center of the face). In some cases, the direction of the translation vector can correspond to the direction of a ray that passes through the fixed 2D pixel location and the corresponding 3D vertex from the camera center. As used herein, morphing the geometry of the adjusted portion can also be referred to as adjusting the pose of the geometry of the first portion of the 3D mesh. In some implementations, after morphing the adjusted portion, the image adjustment system can merge or combine the morphed (or pose adjusted) adjusted portion of the 3D mesh with the non-adjusted portion of the 3D mesh. In some implementations, the adjusted portion and the non-adjusted portion can be maintained within a single 3D mesh, and the 3D vertices corresponding to the non-adjusted portion are fixed and excluded from any morphing of the adjusted portion that may occur in close proximity to the non-adjusted portion.
[0059] The image adjustment system can then convert or translate the 3D mesh (including the morphed adjusted portion corresponding to the target object) into a 2D image domain, resulting in an output 2D image with a modified version of the ROI / target object. In an illustrative example, the ROI / target object in 2D can be zoomed in (compared to the original image) after the portion of the 3D mesh corresponding to the ROI / target object is morphed. For example, a perspective projection technique can be performed to convert the 3D mesh from the 3D domain to the 2D image domain. As a result, the output image includes the wide field of view of the wide-angle lens and a magnified version of the ROI / target object.
[0060] Although examples are described herein for adjusting one or more ROIs in images acquired from a handheld camera, the image adjustment systems and techniques described herein may be used to perform image adjustments for images acquired by other devices, such as cameras in vehicles, robotic devices in a factory environment, and / or any other system that includes a camera.
[0061] Various aspects of the techniques described herein are described below with reference to the figures. FIG. 3 is a block diagram illustrating the architecture of an image capture and processing system 300. The image capture and processing system 300 includes various components used to capture and process an image of a scene (e.g., an image of a scene 310). The image capture and processing system 300 can capture a standalone image (or photo) and / or can capture a video that includes multiple images (or video frames) in a particular order. A lens 315 of the image capture and processing system 300 faces the scene 310 and receives light from the scene 310. The lens 315 bends the light toward the image sensor 330. The light received by the lens 315 passes through an aperture controlled by one or more control mechanisms 320 and is received by the image sensor 330.
[0062] The one or more controls 320 may control exposure, focus, and / or zoom based on information from the image sensor 330 and / or based on information from the image processor 350. The one or more controls 320 may include multiple mechanisms and components. For example, the controls 320 may include one or more exposure controls 325A, one or more focus controls 325B, and / or one or more zoom controls 325C. The one or more controls 320 may include additional controls beyond those shown, such as controls that control analog gain, flash, HDR, depth of field, and / or other image capture characteristics.
[0063] The focus control mechanism 325B of the control mechanism 320 can obtain the focus setting. In some examples, the focus control mechanism 325B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 325B can adjust the position of the lens 315 relative to the position of the image sensor 330. For example, based on the focus setting, the focus control mechanism 325B can move the lens 315 closer to or farther from the image sensor 330 by actuating a motor or servo (or other lens mechanism), thereby adjusting the focus. In some cases, additional lenses, such as one or more microlenses above each photodiode of the image sensor 330, may be included in the image capture and processing system 300, each of which bends light received from the lens 315 toward a corresponding photodiode before the light reaches the photodiode. The focus setting value can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus settings may be determined using the control mechanism 320, the image sensor 330, and / or the image processor 350. The focus settings may be referred to as image capture settings and / or image processing settings.
[0064] The exposure control 325A of the control mechanism 320 can obtain an exposure setting. In some cases, the exposure control 325A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control 325A can control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 330 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 330, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.
[0065] The zoom control 325C of the control mechanism 320 can obtain the zoom setting. In some examples, the zoom control 325C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control 325C can control the focal length of an assembly of lens elements (lens assembly) including the lens 315 and one or more additional lenses. For example, the zoom control 325C can control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more of the lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 315 in some cases) that first receives light from the scene 310, and then the light passes through an afocal zoom system between the focusing lens (e.g., the lens 315) and the image sensor 330 before the light reaches the image sensor 330. An afocal zoom system may include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference of each other), possibly with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control 325C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.
[0066] The image sensor 330 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 330. In some cases, different photodiodes may be covered by different color filters and may therefore measure light that matches the color of the filter covering the photodiode. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, and each pixel of the image is generated based on red light data from at least one photodiode covered by a red color filter, blue light data from at least one photodiode covered by a blue color filter, and green light data from at least one photodiode covered by a green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green filters. Some image sensors (e.g., the image sensor 330) may not have color filters at all and may instead use different photodiodes across the (possibly vertically stacked) pixel array. Different photodiodes across the pixel array can have different spectral sensitivity curves and therefore respond to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore no color depth.
[0067] In some cases, the image sensor 330 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching some photodiodes or portions of some photodiodes at some times and / or from some angles, which may be used for phase detection autofocus (PDAF). The image sensor 330 may also include an analog gain amplifier for amplifying an analog signal output by the photodiode and / or an analog to digital converter (ADC) for converting an analog signal output from the photodiode (and / or amplified by the analog gain amplifier) to a digital signal. In some cases, instead or in addition, some components or functions discussed with respect to one or more of the control mechanisms 320 may be included within the image sensor 330. The image sensor 330 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complimentary metal-oxide semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.
[0068] Image processor 350 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 354), one or more host processors (including host processor 352), and / or one or more of any other types of processors 1110 discussed with respect to computing system 1100. Host processor 352 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 350 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes host processor 352 and ISP 354. In some cases, the chip may include one or more input / output ports (e.g., input / output (I / O) ports 356), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components.The I / O ports 356 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (e.g., a MIPI CSI-2 physical (PHY) layer port or interface, etc.), an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 352 may communicate with the image sensor 330 using an I2C port and the ISP 354 may communicate with the image sensor 330 using a MIPI port.
[0069] Image processor 350 may perform several tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging images to form HDR images, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 350 may store image frames and / or processed images in random access memory (RAM) 140 / 3225, read-only memory (ROM) 145 / 1120, a cache, a memory unit, another storage device, or some combination thereof.
[0070] Various input / output (I / O) devices 360 may be connected to the image processor 350. The I / O devices 360 may include a display screen, a keyboard, a keypad, a touch screen, a track pad, a touch sensitive screen, a printer, any other output device 1135, any other input device 1145, or some combination thereof. In some cases, captions may be entered into the image processing device 305B through a physical keyboard or keypad of the I / O device 360 or through a virtual keyboard or keypad of a touch screen of the I / O device 360. The I / O 360 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 300 and one or more peripheral devices, through which the image capture and processing system 300 may receive data from and / or send data to the one or more peripheral devices. I / O 360 may include one or more wireless transceivers that enable a wireless connection between image capture and processing system 300 and one or more peripheral devices, through which image capture and processing system 300 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 360 previously described, and may themselves be considered I / O devices 360 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.
[0071] In some cases, the image capture and processing system 300 may be a single device. In some cases, the image capture and processing system 300 may be two or more separate devices, including an image capture device 305A (e.g., a camera) and an image processing device 305B (e.g., a computing device coupled to a camera). In some implementations, the image capture device 305A and the image processing device 305B may be coupled, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 305A and the image processing device 305B may be separate from each other.
[0072] As shown in Figure 3, a vertical dashed line divides the image capture and processing system 300 of Figure 3 into two portions representing an image capture device 305A and an image processing device 305B, respectively. The image capture device 305A includes a lens 315, a control mechanism 320, and an image sensor 330. The image processing device 305B includes an image processor 350 (including an ISP 354 and a host processor 352), a RAM 340, a ROM 345, and an I / O 360. In some cases, some of the components shown in the image capture device 305A, such as the ISP 354 and / or the host processor 352, may be included within the image capture device 305A.
[0073] The image capture and processing system 300 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smartphone, a mobile phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 300 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 305A and the image processing device 305B may be different devices. For example, the image capture device 305A may include a camera device, and the image processing device 305B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.
[0074] Although the image capture and processing system 300 is shown as including several components, one skilled in the art will appreciate that the image capture and processing system 300 may include many more components than those shown in FIG. 3. The components of the image capture and processing system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 300 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 300. In some examples, the image adjustment system 400 may include the image capture and processing system 300, the image capture device 305A, the image processing device 305B, or a combination thereof.
[0075] 4 is a block diagram illustrating an example of an image adjustment system 400. Image adjustment system 400 includes various components used to process one or more images, such as altering the size and / or apparent position of features or objects in one or more images. Image adjustment system 400 can generate a 3D mesh based on an input image, adjust ROIs or objects in the 3D mesh to generate an adjusted 3D mesh, and project the adjusted 3D mesh back into a 2D image to generate an adjusted 2D image. As shown, components of image adjustment system 400 include one or more image capture devices 402, a feature segmentation machine learning model 404, a 3D mesh engine 406, and a projection engine 408.
[0076] 4 and the description of image adjustment system 400 below refer to exemplary images shown in FIGS. 5A-5K illustrating different aspects of image adjustment performed by image adjustment system 400. In the example shown in FIGS. 5A-5K, a second person 504 is selected for adjustment (e.g., second person 504 is an ROI or selected object). In the illustrated example, the adjustment applied to second person 504 is selected to move second person 504 closer to first person 502. In one illustrative example, vertices associated with second person 504 may be adjusted by moving the vertices closer to a camera position in a 3D coordinate system.
[0077] 4, image adjustment system 400 may include or be part of a mobile or fixed telephone handset (e.g., a smartphone, a cell phone, etc.), a server computer (e.g., in communication with a vehicle computing system), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, image adjustment system 400 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, Bluetooth or other short-range communication, any combination thereof, and / or other communication. In some implementations, the components of image adjustment system 400 (e.g., one or more image capture devices 402, feature segmentation machine learning model 404, 3D mesh engine 406, and projection engine 408) may be part of the same computing device. In some implementations, the components of image adjustment system 400 may be part of two or more separate computing devices. In some cases, image adjustment system 400 may be implemented as part of a computing system 1100 shown in FIG. 11.
[0078] Although image adjustment system 400 is shown as including several components, one skilled in the art will understand that image adjustment system 400 can include more or fewer components than those shown in FIG. 3. In some cases, the additional components of image adjustment system 400 can include software, hardware, or one or more combinations of software and hardware. For example, in some cases, image adjustment system 400 can include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. In some implementations, the additional components of image adjustment system 400 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuitry (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing image adjustment system 400.
[0079] One or more image capture devices 402 can capture one or more images. The one or more image capture devices 402 (e.g., cameras or other image sensors) may be included in a mobile device and may be pointed toward a user of the device (e.g., with one or more front-facing cameras) or pointed away from a user of the device (e.g., with one or more rear-facing cameras).
[0080] Each of the one or more image capture devices 402 may include a camera or other type of image sensor. In some examples, the one or more image capture devices 402 may include an IR camera configured to capture infrared (IR) images and / or near-infrared (NIR) images. For example, the IR camera or sensor may capture an IR signal. The IR signal has a wavelength and frequency included in the IR electromagnetic spectrum. The IR electromagnetic spectrum includes wavelengths ranging from 2500 nanometers (nm) to 1 millimeter (mm), which corresponds to frequencies ranging from 430 terahertz (THz) to 400 gigahertz (GHz). The infrared spectrum includes the NIR spectrum, which includes wavelengths within a range of 780 nm to 2500 nm. In some cases, the image adjustment system 400 may include an IR sensor configured to capture the IR signal and the NIR signal. In some cases, separate IR and NIR sensors may be included in the image adjustment system 400. In some examples, the one or more image capture devices 402 may include cameras configured to capture color and / or monochrome images. The color images may include red-green-blue (RGB) images, luma, chroma-blue, chroma-red (YCbCr or Y'CbCr) images, and / or any other suitable type of image. In one illustrative example, the image adjustment system 400 may include an RGB camera or multiple RGB cameras. In some cases, the one or more image capture devices 402 may include one or more IR cameras and one or more RGB cameras.
[0081] In some embodiments, one or more image capture devices 402 may include one or more depth sensors. The one or more depth sensors may obtain distance measurements corresponding to objects in a captured scene. In one illustrative example, the depth sensor may take the form of a light source that may project a structured or textured light pattern, which may include one or more narrow bands of light, onto one or more objects in the scene. Depth information may then be obtained by exploiting geometric distortions of the projected pattern caused by the surface shape of the object. In some cases, the one or more depth sensors may generate a depth image that includes depth values that correspond to pixel locations in one or more images captured by the one or more image capture devices 402. In some cases, the depth sensor may be located in the same general location as other sensors of the one or more image capture devices 402. In some cases, the depth sensor may capture a depth image simultaneously with an image captured by one or more other sensors included in the one or more image capture devices 402. In some implementations, the systems and techniques described herein may also be used when depth information is inferred from one or more images.
[0082] In one illustrative example, the one or more image capture devices 402, and possibly one or more depth sensors, may capture one or more RGB-D images. In some cases, the one or more image capture devices 402 may capture other image types that include depth information, such as monochrome depth, NIR depth, etc. For illustrative purposes, the examples of this disclosure describe performing image adjustments on RGB-D images, although the systems and techniques described herein may also be used with other image types that include depth information without departing from the scope of this disclosure.
[0083] FIG. 5A illustrates an exemplary 2D input image 500 (e.g., an RGB image), and FIG. 5B illustrates an exemplary depth image 510. In some cases, the 2D input image 500 and the depth image 510 may collectively comprise an RGB-D image. Both the 2D input image 500 and the depth image 510 include a representation of a first person 502 located closer to the camera and a second person 504 located at a greater distance from the camera. In the depth image 510, the brightness of each pixel may represent the distance of the object or feature from the camera. In the exemplary depth image 510, closer objects appear brighter (e.g., the first person 502) and more distant objects have a lower brightness (e.g., the second person 504, or the background 506).
[0084] Returning to FIG. 4, one or more images captured by one or more image capture devices 402 may be provided as input to a feature segmentation machine learning model 404. The feature segmentation machine learning model 404 may be trained to detect spatial information (e.g., features) associated with the one or more images. In some cases, the feature segmentation machine learning model 404 may be further trained to provide one or more classifications to objects in the one or more images based on the detected features. The feature segmentation machine learning model 404 may then use the classifications to segment the image into different parts associated with the one or more classifications. For example, the feature segmentation machine learning model 404 may segment the one or more images into different parts associated with people, buildings, cars, furniture, plants, etc.
[0085] The feature segmentation machine learning model 404 may also be trained to classify objects corresponding to features extracted from one or more images with one or more classifications. A training dataset including exemplary images and classification labels may be used to train the feature segmentation machine learning model 404, such as using techniques described with respect to Figures 9 and 10. In one illustrative example, during inference (e.g., after the feature segmentation machine learning model 404 is trained), the feature segmentation machine learning model 404 may classify one or more input images using the feature vectors. Exemplary classifications may include "person," "face," "building," "tree," and any other classification that the feature segmentation machine learning model 404 is trained to classify.
[0086] In one illustrative example, the feature segmentation machine learning model 404 may perform semantic segmentation of the image. In semantic segmentation, the feature segmentation machine learning model 404 may associate each object in the scene with one or more classifications (also referred to herein as labels). In some cases, if two or more objects have the same label, the semantic segmentation does not distinguish between the two objects. In such an example, depth information in one or more captured images (e.g., RGB-D images) may be used to distinguish between the objects and further segment between the objects. For example, a portion of the input image that includes two people may be classified with a single classifier, "person." FIG. 5C shows an example image 520 illustrating a first classifier 508 (represented as white) associated with a first person 502 and a second classifier 512 associated with a second person 504. In some examples, the first classifier 508 and / or the second classifier 512 may represent a classification such as "person" or "face." In some aspects, with semantic segmentation, the feature segmentation machine learning model 404 may not be able to separately identify the first person 502 and the second person 504 as different objects based on the first classifier 508 and the second classifier 512.
[0087] In another illustrative example, the feature segmentation machine learning model 404 can perform instance segmentation. In instance segmentation, the feature segmentation machine learning model 404 can separately identify multiple instances of an object using the same classification. For example, the image 530 shown in FIG. 5D provides an illustrative example of instance segmentation, where a first instance 514 associated with a second person 504 is illustrated in gray and a second instance 516 associated with a person 502 is illustrated as white. In some cases, with instance segmentation, the feature segmentation machine learning model 404 can separately identify the first person 502 and the second person 504 as different objects based on the first instance 514 and the second instance 516.
[0088] In some cases, the 3D mesh engine 406 can utilize the depth information as well as the camera parameters (e.g., focal length, field of view, etc.) to generate a 3D mesh corresponding to the RGB-D image. In some cases, for each pixel in the input image, the 3D mesh engine 406 can generate a corresponding 3D vertex in a 3D model of the scene depicted in the input image. In one illustrative example, the 3D mesh engine can generate a 3D mesh using depth values and the field of view of the camera that captured the 2D input image (e.g., input image 500 of FIG. 5A). For example, the 3D mesh engine 406 can utilize backprojection techniques to generate the 3D mesh. As described above with respect to FIGS. 2B and 2C, a 3D vertex V of the 3D mesh for a pixel at coordinates x,y is generated. x,y An exemplary technique for determining the location of is to project a ray from the camera center through coordinates x,y, and a distance d from the camera. x,y Based on the 3D coordinates on the ray at x,y where d x,yis a distance value in the depth image corresponding to coordinates x,y. The 3D mesh engine 406 may perform a backprojection technique for each pixel coordinate x,y in the 2D input image. Any suitable backprojection technique may be used without departing from the scope of this disclosure. FIG. 5E illustrates an example image 540 of a 3D mesh 542 generated from the 2D input image. As illustrated, the 3D mesh 542 may include a 3D representation of a first person 502 and a second person 504.
[0089] In some cases, the features, objects, and / or classifications determined by the feature segmentation machine learning model 404 may be provided as inputs to the 3D mesh engine 406. In some cases, based on the classification, the 3D mesh engine 406 may associate vertices in the 3D mesh with corresponding features, objects, and / or classifications associated with corresponding pixel locations in the 2D input image. In some cases, such as when the feature segmentation machine learning model 404 provides instance segmentation, the 3D mesh engine may distinguish different features or objects associated with the same class based on the output from the feature segmentation machine learning model. In some cases, when the feature segmentation machine learning model 404 can only provide semantic segmentation, the 3D mesh engine 406 may distinguish different objects or features in the 3D mesh based on depth values associated with each object. For example, the 3D mesh engine 406 may determine that two objects sharing the same classifier are separate objects based on different portions of the 3D mesh associated with the classifiers having different depth values. In one illustrative example, the feature segmentation machine learning model 404 may provide a classification of “face” that is applied to both the first person 502 and the second person 504. In such an example, the 3D mesh engine 406 may determine a depth value at some location within the 3D mesh associated with the “face” classifier (e.g., the center of a bounding box, the tip of the nose, between the eyes, etc.) and determine that other vertices within the 3D mesh within a threshold distance from the determined depth are part of the same object.
[0090] In some cases, the 3D mesh engine 406 and / or the feature segmentation machine learning model 404 may also apply one or more masks to regions (e.g., pixels) of the 2D input image and corresponding portions (e.g., vertices) of the 3D mesh. FIG. 5F illustrates an image 550 including one exemplary mask, with pixels associated with a person 502 shown as a masking region 552. In some examples, a foreground object closest to the camera may be selected for inclusion in the masking region 552 by the 3D mesh engine 406 and / or the feature segmentation machine learning model 404. In the illustrated example, a first person 502 in the foreground of the 2D input image 500 is selected for masking because any erroneous adjustment of vertices associated with the first person 502 may generate noticeable visual artifacts in the adjusted 2D image after image adjustment is completed. In some cases, other masks and / or multiple masks may be used without departing from the scope of this disclosure.
[0091] In some cases, the 3D mesh engine 406 can determine a non-adjusted portion of the 3D mesh (e.g., the 3D mesh 542) based on the masking region 552 in the 2D image. In some cases, any portion of the 3D mesh outside the non-adjusted portion can be referred to as an adjusted region. The inclusion of vertices in an adjusted portion does not necessarily mean that the included vertices are adjusted by the 3D mesh engine 406. In some implementations, based on the determination of the adjusted and non-adjusted portions in the 3D mesh, the 3D mesh engine 406 can separate the 3D mesh into an adjusted 3D mesh and a separate non-adjusted 3D mesh.
[0092] Figure 5G shows an image 560 illustrating an exemplary unadjusted 3D mesh 562 that may be generated based on the masking region 552 shown in Figure 5F that includes a 3D representation of the person 502. Figure 5H shows an exemplary image 570 illustrating an adjusted 3D mesh 572 that includes a second person 504, as well as other objects not included in the masking region 552, such as a sofa, a wall, etc.
[0093] Returning to FIG. 4, in some implementations, the 3D mesh engine 406 may not separate the 3D mesh into separate non-adjusted and adjusted portions of the 3D mesh, but instead may fix coordinates of vertices of the 3D mesh within the non-adjusted portions or possibly prevent adjustments to vertices of the 3D mesh within the non-adjusted portions.
[0094] Once the 3D mesh engine 406 generates a 3D mesh (or multiple 3D meshes) based on the 2D input image, determines one or more masks, and / or performs further segmentation of the object in the 3D mesh based on the depth value, the 3D mesh engine 406 may adjust one or more portions of the 3D mesh. In some cases, the 3D mesh engine 406 may adjust one or more portions of the 3D mesh associated with one or more of the features detected and / or classified by the feature segmentation machine learning model 404 as associated with 3D vertices in the 3D mesh. For example, the 3D mesh engine 406 may obtain an indication of one or more selected objects (or ROIs) in the input image to be adjusted. In some cases, the indication of one or more selected objects in the input image (e.g., the 2D input image 500 of FIG. 5A) to be adjusted may be based on user input. In one illustrative example, a user may be able to indicate one or more selected objects using physical contact (e.g., tapping, swiping the screen, etc.), a gesture (e.g., detected by a camera or other sensor), using an input device, or through any other means that enables the user to interact with image adjustment system 400.
[0095] Based on an indication of one or more selected objects in the input image to be adjusted, the 3D mesh engine 406 can adjust a 3D mesh (e.g., mesh 542 shown in FIG. 5E). In some cases, the 3D mesh engine 406 can adjust vertices in a 3D mesh associated with one or more ROIs or objects. Adjustments to an ROI or object are also referred to as modifying the ROI or object. Modifying the pose of an ROI or object can include modifying the depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, roll rotation, and any combination thereof. In an illustrative example, the mesh adjustment can include moving one or more selected objects in the 3D model closer to the camera. FIG. 5I shows an image 580 showing vertices in a 3D mesh associated with the second person 504 shifted in the direction of an arrow 582. In the illustrated example, the direction of the arrow 582 was selected to maintain a fixed position of a sub-portion near the center of the face of the second person 504 in the adjusted 2D image (e.g., nose, between the eyes, etc.).
[0096] In one illustrative example, a particular anchor point of an object may be selected to remain at a fixed position in the adjusted 2D image. In such an example, the pose change may include adjusting one or more selected object vertices along a direction of a ray passing through the center of the camera and the initial position of the 3D vertices corresponding to the anchor point. Another illustrative example of adjusting a pose may include reducing depth values of some or all of the vertices associated with the selected object. In some cases, reducing depth values of the vertices of the selected object may increase the size of the selected object in the resulting 2D image. A further illustrative example of adjusting a pose may include (simultaneously) increasing and decreasing depth values of some or all of the vertices associated with the selected object. In some cases, increasing and decreasing depth values of the vertices of the selected object may result in a pitch rotation, a yaw rotation, and / or a roll rotation of the selected object in the resulting 2D image.
[0097] In some cases, the image adjustment system 400 may obtain multiple images of a captured scene or portion of a scene in a 2D input image. For example, the electronic device may include a first image capture device and a second image capture device (e.g., included in the image capture device 402) that simultaneously capture at least a partial common portion of the scene. In some cases, a single image sensor may capture multiple images of a scene to provide, for example, zero shutter lag capture, preview images, etc. In some examples, the image adjustment system may generate multiple 3D meshes based on the multiple images of the scene or portion of a scene. In some cases, the 3D mesh engine 406 may incorporate (e.g., merge, composite, inpaint, etc.) portions from the multiple 3D meshes to generate a combined 3D mesh. In one illustrative example, the 3D mesh engine 406 may adjust 3D vertices from a first 3D mesh associated with a selected object in the first 2D image and may also include 3D vertices from a second 3D mesh associated with the second 2D image. In one illustrative example, 3D vertices associated with the selected object may be shifted horizontally, and the portion of the 3D mesh previously occupied by the selected object may incorporate portions from the second 2D mesh.
[0098] After the 3D mesh is adjusted by the 3D mesh engine 406, the projection engine 408 can project the 3D mesh onto the adjusted 2D image. The projection engine 408 can utilize any of the projection techniques described above with respect to FIG. 2B and FIG. 2C. In an illustrative example, the projection engine 408 can utilize the perspective projection of Equation (1). In some aspects, the projection engine 408 can use any other projection technique, including but not limited to orthogonal projection or oblique projection, to generate the adjusted 2D image without departing from the scope of this disclosure. FIG. 5K illustrates an adjusted 2D image 595 that can be generated by the image adjustment system 400. In some implementations, the projection engine 408 can utilize a projection technique that corresponds to the projection technique of the physical camera that captured the 2D image. In some cases, using a projection technique that corresponds to the projection technique of the physical camera can reduce distortion in the adjusted 2D image that can be introduced by a mismatch between the physical camera projection technique and the projection technique utilized by the projection engine 408.
[0099] 6 illustrates an example process 600 for performing image adjustment according to an example of the present disclosure. At block 602, the process 600 may obtain a 2D input image (e.g., from the image capture device 402 shown in FIG. 4). At block 604, the process 600 may perform segmentation of features or objects included in the input image. For example, the process 600 may perform one or more of feature detection, semantic segmentation, instance segmentation, object classification, etc., as described with respect to the feature segmentation machine learning model 404 of FIG. 4.
[0100] At block 606, process 600 may include generating a 3D mesh based on the depth and / or color information included in the RGB-D image acquired at block 602. In some cases, for each pixel in the 2D image, a corresponding 3D vertex in the 3D model of the scene may be generated using the backprojection techniques described above with respect to 3D mesh engine 406 of Figures 2B, 2C, and 4, or any other suitable technique for converting a 2D image into a 3D mesh.
[0101] At block 608, the process 600 may perform masking of the 3D mesh. In some cases, the masking may include creating non-adjusted and adjusted portions within the 3D mesh. In some cases, the non-adjusted and adjusted portions may be separated into two separate 3D meshes to prevent adjustment of the non-adjusted portion. In some cases, a single mesh may include both non-adjusted and adjusted portions, and vertices of the non-adjusted portion may be fixed or otherwise prevented from being adjusted. In some aspects, the masking may include vertices within the 3D mesh associated with one or more ROIs or objects.
[0102] At block 610, process 600 may perform mesh adjustment on the 3D mesh generated at block 608. In some cases, process 600 may obtain an input indicating that one or more selected objects (or ROIs) in the 2D image have been selected for adjustment. For example, process 600 may adjust the pose of one or more selected objects by changing the coordinates of the vertices corresponding to the one or more selected objects. As described above with respect to 3D mesh engine 406 of FIG. 4, each 2D pixel location in the input image obtained at block 602 may have a corresponding 3D vertex in the 3D mesh generated at block 608. In some cases, process 600 may associate the 3D vertices generated at block 608 with one or more objects (or features). For example, the features, objects, and / or classes determined at block 604 and associated with each of the 2D pixel locations of the input image may also be associated with the corresponding 3D vertices. Once the 3D vertices corresponding to the selected objects are identified, process 600 may adjust the vertices to provide a desired effect (e.g., magnification, translation, rotation, etc.) in the adjusted 2D image.
[0103] At block 612, process 600 may perform fusion between two or more portions of the 3D mesh. For example, if process 600 separated the 3D mesh into an unadjusted 3D mesh and an adjusted 3D mesh at block 608, process 600 may combine the unadjusted 3D mesh and the adjusted 3D mesh into a single combined 3D mesh (e.g., as shown in image 590 of FIG. 5J). In some cases, if process 600 did not generate separate 3D meshes but instead locked vertices in the unadjusted portions of the 3D mesh, process 600 may not need to perform any fusion of the 3D meshes.
[0104] At block 614, the process 600 may generate the adjusted 2D image using a projection technique described herein with respect to the projection engine 408 of FIG. 2B, FIG. 2C, and FIG. 4. One illustrative example of a projection that may be performed by the process 600 is the perspective projection shown in equation (1). However, any projection or rendering technique that can convert a 3D mesh into a 2D image may be used by the process 600 without departing from the scope of this disclosure. However, any projection or rendering technique that can convert a 3D mesh into a 2D image may be used by the process 600 without departing from the scope of this disclosure. In some implementations, the process 600 may use a projection technique that corresponds to the physical projection of the camera that captured the 2D image. In some cases, using a projection technique that corresponds to the projection technique of the physical camera may reduce distortion in the adjusted 2D image that may be introduced by a mismatch between the physical camera projection technique and the projection technique utilized by the process 600 at block 614.
[0105] 6 provides an illustrative example of an image adjustment process according to the present disclosure. However, the order of operations of process 600 may be changed without departing from the scope of the present disclosure. Additionally, one or more steps of process 600 may be omitted or additional steps may be performed without departing from the scope of the present disclosure.
[0106] In some cases, the process 600 may also obtain input corresponding to an amount of adjustment to be applied to one or more objects or features. In some cases, the process 600 may dynamically determine an amount of adjustment to be applied to one or more selected objects. In one illustrative example, the process 600 may determine an approximate size of the foreground object and an approximate size of each of the one or more objects (e.g., using a bounding box to calculate the number of image pixels occupied by the foreground object, etc.). In such an example, the process 600 may determine the amount of adjustment based on the relative size of the foreground object and the one or more objects. In one illustrative example, the one or more objects may be adjusted to increase the size of the one or more objects in the 2D image, such that the one or more objects have a size that corresponds to a particular numerical correspondence (e.g., a particular percentage value, ratio, proportion, function, relationship, etc.) with the size of the foreground object. In one illustrative example, the one or more selected objects (or ROIs) may be adjusted to have a size that corresponds to a predetermined ratio within a range between 7 / 10 (e.g., 70 percent) and 9 / 10 (e.g., 90 percent) of the size of the foreground object. In some cases, the numerical correspondence (e.g., percentage value) and / or the range of the numerical correspondence (e.g., percentage value range) may be limited based on the distance in the 3D mesh between the foreground object and the background object. In another illustrative example, if the distance (e.g., average distance) between the foreground object and a first object of the one or more selected objects is small (e.g., less than half a meter (m), less than 1 m, less than 3 m, etc.), the first object of the one or more selected objects may be adjusted to a ratio of 1 (e.g., 100 percent) of the size of the foreground object.In another illustrative example, if the distance (e.g., average distance) between the second one of the one or more selected objects and the foreground object is large (e.g., greater than 5 m, greater than 10 m, or any other selected distance), the second one of the one or more selected objects may be adjusted up to a ratio of 4 / 5 (e.g., 80 percent) of the size of the foreground object. Other techniques for determining the amount of adjustment for each of the one or more selected objects may also be used without departing from the scope of this disclosure. For example, the total change in the pose of the object may be limited within an adjustment range. In some cases, the adjustment range for changing the pose of the object may be determined based on a maximum amount of distortion in the 2D image resulting from performing the pose change. In another example, the maximum amount of adjustment for each of the one or more selected objects may be based on a ratio of the apparent size of the object in the adjusted image compared to the original image. For example, the maximum amount of adjustment may be limited to allow the apparent size of each object to increase by 50 percent, 100 percent, 200 percent, or any other value.
[0107] As mentioned above, the image adjustment system 400 and related techniques described herein can enable the system to adjust an ROI or object in an image without adjusting other objects in the image or scene. For example, the image adjustment system 400 can adjust the appearance of an image in which two people are sitting far from each other (e.g., as shown in FIG. 5A and FIG. 5B) to bring the two people closer together. In some cases, the adjustments can include reducing the depth of the target object, shifting vertically, pitch rotation, roll rotation, yaw rotation, etc. By performing the adjustments on the 3D mesh, the image adjustment system can maintain continuity (e.g., continuity of the 3D mesh between the adjusted target object and the remainder of the unadjusted image).
[0108] 7A-7E provide additional illustrative examples of image adjustments that may be performed using the systems and techniques of the present disclosure. FIG. 7A shows a 2D input image 700 including a first person 702 in the foreground and a second person 704, a third person 706, a fourth person 708, and a fifth person 710 behind the first person 702 at different distances. FIG. 7B shows an exemplary instance segmentation 720 performed on the 2D input image 700, where each of the first person 702, the second person 704, the third person 706, the fourth person 708, and the fifth person 710 in the image are separately recognized. In an illustrative example, each person 702-710 is classified as a separate instance of the class "person", as indicated by the different shading shown in FIG. 7B. In the first adjusted image 740 shown in FIG. 7C, each of the second, third, fourth, and fifth persons 704-710 has been adjusted using image adjustment techniques as described herein. For example, a 3D mesh corresponding to the 2D input image 700 was generated using depth information contained in the RGB-D input image. In the illustrated example of FIG. 7C, the locations (e.g., vertices) corresponding to the second, third, fourth, and fifth persons 704-710 were adjusted in the 3D mesh to be closer to the camera, and the adjusted 3D mesh was reprojected to 2D to generate the first adjusted image 740. Similarly, as shown in FIG. 7D, the second adjusted image 760 shows an example where only the third person 706 has been adjusted closer to the camera, and the third adjusted image 780 shown in FIG. 7E shows an example where only the fifth person has been adjusted closer to the camera. In some cases, the positions of the second, third, fourth, and fifth people 704-710 may be dynamically moved back and forth within the adjusted image based on user input. In all of the examples shown in Figures 7A and 7C-7E, the first person 702 in the foreground remains fixed. In the exemplary 3D mesh generated based on the 2D input image 700, the vertices associated with the person 702 may be included in the unadjusted portion and / or unadjusted mesh, as described above with respect to the 3D mesh engine 406 of Figure 4. Figures 7A-7E illustrate the flexibility of the systems and techniques for image adjustment described herein.For example, the present systems and techniques allow for adjustment of a ROI or object in a captured image while keeping other ROIs or objects in the image fixed.
[0109] 8 is a flow diagram illustrating an example of a process 800 for processing one or more frames. At block 802, the process 800 includes acquiring a 2D input image from a camera. For example, the 2D input image may be acquired from the image capture and processing system 300, the image capture device 305A, and / or the image processing device 305B.
[0110] At block 804, the process 800 includes obtaining a 3D representation of the 2D input image. For example, the process 800 may obtain the 3D representation from the 3D mesh engine 406 shown in FIG.
[0111] At block 806, the process 800 includes determining a first portion of the 3D representation of the 2D input image that corresponds to a first target object in the 2D input image. For example, the process 800 may obtain a user input indicating the target object and / or ROI. In some cases, based on the user input, the process 800 may determine vertices in a 3D mesh associated with the target object and / or ROI. In some cases, determining the first portion of the 3D representation of the 2D input image includes at least one of feature detection and object detection. In some examples, determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.
[0112] At block 808, the process 800 includes adjusting a pose of a first portion of the 3D representation of the 2D input image corresponding to the first target object. In some examples, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object includes increasing a depth value of a first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object. In some cases, adjusting the pose of the first portion of the 3D representation of the 2D input image includes one or more of adjusting a depth, a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, and a roll rotation. For example, the target object may be a person, and modifying the 3D representation of the person may include changing an orientation of the person's head (e.g., a rotation about a pitch axis, a yaw axis, and / or a roll axis). In some examples, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further includes increasing a depth value of a second one or more vertices of the first portion of the 3D representation of the 2D input image.
[0113] At block 810, the process 800 includes generating a 2D output image having a modified version of the target object based on the adjusted pose of a first portion of the 3D representation of the 2D input image that corresponds to the target object to be output on the display, where at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image compared to a corresponding sub-portion of the target object in the 2D input image.
[0114] In some examples, the process 800 includes obtaining user input identifying a target object, and a pose of a first portion of the 3D representation of the 2D input image that corresponds to the target object is adjusted based on the user input.
[0115] In some examples, the process 800 includes obtaining an indication of an adjustment amount of the target object and adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the adjustment amount of the target object.
[0116] In some examples, the process 800 includes obtaining an object mask associated with a target object in the 2D input image. In some cases, the process 800 includes determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask. In some examples, the process 800 includes determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask. In some aspects, the process 800 includes combining the first portion of the 3D representation with the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh. In some cases, generating the 2D output image includes converting the combined 3D mesh to the 2D output image.
[0117] In some examples, the process 800 includes obtaining an object mask associated with one or more additional objects in the 2D input image that are different from the target object. In some aspects, the process 800 includes determining a first portion of the 3D representation of the 2D input image that corresponds to the target object based on the object mask. In some cases, the process 800 includes determining a second portion of the 3D representation that corresponds to at least one of the one or more additional objects based on the object mask. In some examples, the process 800 includes adjusting a pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh. In some cases, adjusting the pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object includes reducing depth values of vertices of the first portion of the 3D representation of the 2D input image that corresponds to the target object such that the vertices have a predetermined numerical correspondence relative to the fixed one or more vertices of the second portion of the 3D representation. In some aspects, generating the 2D output image includes converting the adjusted 3D mesh to the 2D output image.
[0118] In some cases, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on user input. In some aspects, the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration. In some cases, the pose of the first portion of the 3D representation of the 2D input image is adjusted based on user preferences and machine learning of past behavior. In some implementations, the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.
[0119] In some examples, the process 800 includes adjusting the pose of a first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.
[0120] In some examples, the process 800 includes determining a second portion of the 3D representation of the 2D input image that corresponds to a second target object in the 2D input image. In some cases, the process 800 includes adjusting a pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object. In some aspects, the process 800 includes generating the 2D output image further based on the adjusted pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object. In some implementations, the adjustment to the pose of the second portion is different from the adjustment to the pose of the first portion.
[0121] In some examples, the process 800 includes acquiring a second 2D input image from a camera, the second 2D input image including at least a portion of a scene included in the 2D input image. In some aspects, the process 800 includes acquiring a second 3D representation of the second 2D input image, and at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.
[0122] In some examples, the processes described herein (e.g., process 800 and / or other processes described herein) may be performed by a computing device or apparatus. In one example, one or more of the processes may be performed by image adjustment system 400 of FIG. 4. In another example, one or more of the processes may be performed by computing system 1100 shown in FIG. 11. For example, a computing device with computing system 1100 shown in FIG. 11 may include components of image adjustment system 400 and may implement operations of process 600 of FIG. 6, process 800 of FIG. 8, and / or other processes described herein.
[0123] The computing device may include any suitable device, such as a vehicle or a vehicular computing device (e.g., a vehicle's driver monitoring system (DMS)), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein, including process 600, process 800, and / or other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.
[0124] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.
[0125] Processes 600 and 800 are illustrated as logical flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.
[0126] Additionally, process 600, process 800, and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
[0127] As mentioned above, various aspects of the present disclosure may use machine learning models or systems. FIG. 9 is an illustrative example of a deep learning neural network 900 that may be used to implement the machine learning based feature segmentation, instance segmentation, depth estimation, and / or classification described above. The input layer 920 includes input data. In one illustrative example, the input layer 920 may include data representing pixels of an input image. The neural network 900 includes multiple hidden layers 922a, 922b-922n. The hidden layers 922a, 922b-922n include "n" hidden layers, where "n" is an integer equal to or greater than 1. The number of hidden layers may be as many layers as required for a given application. The neural network 900 further includes an output layer 921 that provides an output resulting from the processing performed by the hidden layers 922a, 922b-922n. In one illustrative example, the output layer 921 may provide a classification for objects in a 2D input image. The classification may include classes that identify types of activities (eg, looking up, looking down, closing eyes, yawning, etc.).
[0128] Neural network 900 is a multi-layered neural network of interconnected nodes. Each node can represent information. The information associated with a node is shared between different layers, with each layer retaining the information as it is processed. In some cases, neural network 900 can include a feed-forward network, in which there are no feedback connections where the output of the network is fed back to itself. In some cases, neural network 900 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading at the input.
[0129] Through node-to-node interconnections between the various layers, information may be exchanged between the nodes. The nodes of the input layer 920 may activate a set of nodes in the first hidden layer 922a. For example, as shown, each of the input nodes of the input layer 920 is connected to each of the nodes of the first hidden layer 922a. The nodes of the first hidden layer 922a may transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation may then be passed to the nodes of the next hidden layer 922b, which may activate such nodes, which may perform their own designated functions. Exemplary functions include convolution, upsampling, data transformation, and / or any other suitable functions. The output of the hidden layer 922b may then activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 922n may activate one or more nodes of the output layer 921, where an output is provided. In some cases, a node in neural network 900 (e.g., node 926) is shown as having multiple output lines; however, the node has a single output, and all lines shown as outputting from the node represent the same output value.
[0130] In some cases, each node or interconnections between nodes can have weights, which are a set of parameters derived from training of the neural network 900. Once the neural network 900 is trained, it may be referred to as a trained neural network and may be used to classify one or more activities. For example, the interconnections between nodes may represent information learned about the interconnected nodes. The interconnections may have adjustable numerical weights that may be adjusted (e.g., based on a training data set), allowing the neural network 900 to be adaptive to the input and to learn as more and more data is processed.
[0131] The neural network 900 is pre-trained to process features from data in the input layer 920 using different hidden layers 922a, 922b-922n to provide output through the output layer 921. In an example where the neural network 900 is used to identify features in images, the neural network 900 may be trained using training data that includes both images and labels. For example, training images may be input to the network, with each training image having a label indicating the feature in the image (for a feature segmentation machine learning system) or the class of activity in each image. In an example using object classification for illustration purposes, the training images may include images of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].
[0132] In some cases, the neural network 900 may use a training process called backpropagation to adjust the node weights. As described above, the backpropagation process may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed during one training iteration. The process may be repeated for a number of iterations for each set of training images until the neural network 900 is trained well enough that the layer weights are accurately adjusted.
[0133] For the example of identifying objects in an image, the forward pass may include passing a training image through the neural network 900. Before the neural network 900 is trained, the weights are first randomized. As an illustrative example, an image may include an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 representing the pixel intensity at that location in the array. In one example, the array may include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (such as a red component, a green component, and a blue component, or a luma component and two chroma components).
[0134] As mentioned above, in the first training iteration for the neural network 900, due to the weights being randomly selected at initialization, the output may contain values that do not give a preference to any particular class. For example, if the output is a vector with the probabilities that the object contains different classes, the probability values for each of the different classes may be equal or at least very similar (e.g., for 10 possible classes, each class may have a probability value of 0.1). With the initial weights, the neural network 900 cannot make an accurate determination of what the classification of the object is likely to be, since it is unable to determine low-level features. A loss function may be used to analyze the error in the output. Any suitable loss function definition may be used, such as cross-entropy loss. Another example of a loss function is:
number
[0135] The loss (or error) is large for the first training image because the actual values are very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 900 can perform a backward pass by determining which inputs (weights) contributed most to the network's loss, and the weights can be adjusted so that the loss is small and eventually minimized. To determine which weights contributed most to the network's loss, the derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight in a particular layer) can be calculated. After the derivatives are calculated, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated such that the weights change in the opposite direction of the gradient. The weight update can be
number
[0136] Neural network 900 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of a CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. Neural network 900 may include any other deep network other than a CNN, such as autoencoders, deep belief nets (DBNs), and Recurrent Neural Networks (RNNs), among others.
[0137] FIG. 10 is an illustrative example of a convolutional neural network (CNN) 1000. The input layer 1020 of the CNN 1000 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of the image, with each number in the array including a value from 0 to 255 representing the pixel intensity at that location in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (e.g., a red component, a green component, and a blue component, or a luma component and two chroma components, etc.). The image is passed through a convolutional hidden layer 1022a, an optional nonlinear activation layer, a pooling hidden layer 1022b, and a fully connected hidden layer 1022c to obtain an output at the output layer 1024. Although only one of each hidden layer is shown in Figure 10, one skilled in the art will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1000. As previously explained, the output may indicate a single class of object, or may include a probability of the class that best represents the object in the image.
[0138] The first layer of the CNN 1000 is the convolutional hidden layer 1022a. The convolutional hidden layer 1022a analyzes the image data of the input layer 1020. Each node of the convolutional hidden layer 1022a is connected to a region of nodes (pixels) of the input image called the receptive field. The convolutional hidden layer 1022a can be thought of as one or more filters (each filter corresponds to a different activation or feature map), and each convolutional iteration of a filter is a node or neuron of the convolutional hidden layer 1022a. For example, the region of the input image that the filter covers in each convolutional iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 1022a. Each connection between a node and the receptive field for that node learns a weight, and possibly an overall bias, so that each node learns to analyze its particular local receptive field in the input image. Each node in the hidden layer 1022a has the same weights and biases (called shared weights and shared biases). For example, a filter has an array of weights (numbers) and the same depth as the input. The filter has a depth of 3 in the example image frame (according to the three color components of the input image). The size of an illustrative example filter array is 5x5x3, which corresponds to the size of the receptive fields of the nodes.
[0139] The convolutional nature of the convolutional hidden layer 1022a results from each node of the convolutional layer being applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1022a may start at the upper left corner of the input image array and convolve around the input image. As mentioned above, each convolutional iteration of the filter may be considered a node or neuron of the convolutional hidden layer 1022a. At each convolutional iteration, the value of the filter is multiplied with a corresponding number of original pixel values of the image (e.g., a 5×5 filter array is multiplied with a 5×5 array of input pixel values at the upper left corner of the input image array). The multiplications from each convolutional iteration may be added together to obtain a total for that iteration or node. The process then continues at the next location in the input image according to the receptive field of the next node in the convolutional hidden layer 1022a. For example, the filter may be moved to the next receptive field by a certain step amount (called a stride). The stride may be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter is moved one pixel to the right in each convolution iteration. Processing the filter at each unique location of the input volume produces a number representing the filter result for that location, resulting in an aggregate value being determined for each node in the convolutional hidden layer 1022a.
[0140] The mapping from the input layer to the convolutional hidden layer 1022a is called an activation map (or feature map). The activation map contains the value of each node that represents the filter result at each location of the input volume. The activation map can include an array containing various aggregate values resulting from each iteration of the filter on the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (a stride of 1), the activation map includes a 24×24 array. The convolutional hidden layer 1022a can include several activation maps to identify multiple features in an image. The example shown in FIG. 10 includes three activation maps. Using the three activation maps, the convolutional hidden layer 1022a can detect three different types of features, each detectable throughout the image.
[0141] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 1022a. The nonlinear layer may be used to introduce nonlinearity into a system that previously computed linear operations. One illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply a function f(x)=max(0,x) to all of the values in the input volume, which changes all negative activations to 0. Thus, the ReLU may enhance the nonlinear nature of the CNN 1000 without affecting the receptive field of the convolutional hidden layer 1022a.
[0142] After the convolutional hidden layer 1022a (and after the non-linear hidden layer, when used), a pooling hidden layer 1022b may be applied. The pooling hidden layer 1022b is used to simplify the information in the output from the convolutional hidden layer 1022a. For example, the pooling hidden layer 1022b may take each activation map output from the convolutional hidden layer 1022a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is one example of a function implemented by the pooling hidden layer. Other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions, may be used by the pooling hidden layer 1022a. A pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1022a. In the example shown in FIG. 10, three pooling filters are used for the three activation maps in the convolutional hidden layer 1022a.
[0143] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 1022a. The output from the max pooling filter contains the maximum number of any subregion around which the filter convolves. Using a 2×2 filter as an example, each unit in the pooling layer may summarize a region of 2×2 nodes (each node is a value in the activation map) in the previous layer. For example, four values (nodes) in the activation map are analyzed by a 2×2 max pooling filter at each iteration of the filter, and the maximum value from the four values is output as the “max” value. If such a max pooling filter is applied to an activation filter from the convolutional hidden layer 1022a with dimensions of 24×24 nodes, the output from the pooling hidden layer 1022b is an array of 12×12 nodes.
[0144] In some examples, an L2 norm pooling filter may also be used, which involves computing the square root of the sum of the squares of the values in a 2×2 region (or other suitable region) of the activation map (rather than computing the maximum value as is done in max pooling) and using the computed value as the output.
[0145] Intuitively, a pooling function (e.g., max pooling function, L2 norm pooling function, or other pooling function) determines whether a given feature is found anywhere in a region of the image and discards the exact location information. This can be done without affecting the outcome of feature detection because once the feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (as well as other pooling methods) offers the advantage that there are many fewer features to pool, thus reducing the number of parameters required in later layers of the CNN 1000.
[0146] The final layer of connections in the network is a fully connected layer that connects every node from the pooling hidden layer 1022b to every one of the output nodes in the output layer 1024. Using the above example, the input layer includes 28×28 nodes that encode pixel intensities of the input image, the convolutional hidden layer 1022a includes 3×24×24 hidden feature nodes based on application of 5×5 local receptive fields (for the filters) to three activation maps, and the pooling hidden layer 1022b includes a layer of 3×12×12 hidden feature nodes based on application of max pooling filters to 2×2 regions across each of the three feature maps. Extending this example, the output layer 1024 can include 10 output nodes. In such an example, every node of the 3×12×12 pooling hidden layer 1022b is connected to every node of the output layer 1024.
[0147] The fully connected layer 1022c can take the output of the previous pooling hidden layer 1022b (which should represent the activation map of high-level features) and determine the features that are most correlated to a particular class. For example, the fully connected layer 1022c can determine the high-level features that are most strongly correlated to a particular class and can include weights (nodes) for the high-level features. A product can be calculated between the weights of the fully connected layer 1022c and the pooling hidden layer 1022b to obtain probabilities for different classes. For example, if the CNN 1000 is used to predict that an object in an image is a person, there will be high values in the activation map that represent the high-level features of a person (e.g., there are two legs, there is a face on the top of the object, there are two eyes on the top left and top right of the face, the nose is in the middle of the face, the mouth is on the bottom of the face, and / or other features that are common to people).
[0148] In some examples, the output from the output layer 1024 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of classes the CNN 1000 must choose from when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent a probability that the object is of a certain class. In one illustrative example, if a 10-dimensional output vector represents 10 different classes of objects as [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a 5% probability that the image is of the third class of object (e.g., a dog), an 80% probability that the image is of the fourth class of object (e.g., a human), and a 15% probability that the image is of the sixth class of object (e.g., a kangaroo). The probability of a class may be considered a confidence level that the object is part of that class.
[0149] 11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, FIG. 11 illustrates an example of a computing system 1100, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device constituting any of the components thereof, in which the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection to a processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a network connection, or a logical connection.
[0150] In some embodiments, computing system 1100 is a distributed system in which the functionality described in this disclosure may be distributed across one data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many components, each performing some or all of the functionality that is the subject of the component description. In some embodiments, the components may be physical or virtual devices.
[0151] The exemplary computing system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 coupling various system components to the processor 1110, including system memory 1115, such as read-only memory (ROM) 1120 and random access memory (RAM) 1125. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to the processor 1110, in close proximity to the processor 1110, or integrated as part of the processor 1110.
[0152] The processor 1110 may include any general-purpose processor, as well as hardware or software services, such as services 1132, 1134, and 1136, stored in a storage device 1130 and configured to control the processor 1110, as well as special-purpose processors whose software instructions are built into the actual processor design. The processor 1110 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.
[0153] To enable user interaction, computing system 1100 includes input device(s) 1145, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1100 may also include output device(s) 1135, which can be one or more of several output mechanisms. In some cases, a multi-modal system may enable a user to provide multiple types of input / output to communicate with computing system 1100. Computing system 1100 may generally include a communication interface 1140, which can govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) wireless signal transmission, Public Switched Telephone Network (PSTN), and the like. The communication device may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing PSTN (Provisional Telephone Network) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.The communication interface 1140 may include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a position of the computing system 1100 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite system (BDS), and the European Galileo GNSS. There is no constraint to operate with any particular hardware arrangement, and therefore the basic features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.
[0154] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (ICC), a circuit, IC chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASH EPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM),The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a memory, RRAM / ReRAM, phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.
[0155] The storage devices 1130 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by the processor 1110. In some embodiments, hardware services that perform particular functions may include software components stored in a computer-readable medium in conjunction with the necessary hardware components, such as the processor 1110, connections 1105, output devices 1135, etc., to perform the functions.
[0156] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may include non-transitory media on which data is stored and does not include carrier waves and / or transitory electronic signals propagating wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.
[0157] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.
[0158] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For ease of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components may be used other than the components shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.
[0159] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to a main function.
[0160] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions, or in some cases configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the examples described include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, etc.
[0161] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing in a single device, as further examples.
[0162] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.
[0163] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while illustrative embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative rather than restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.
[0164] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.
[0165] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor, or other suitable electronic circuitry) to perform the operation, or any combination thereof.
[0166] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).
[0167] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. Claim language reciting "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.
[0168] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
[0169] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, which may carry or communicate program code in the form of instructions or data structures and which may be accessed, read, and / or executed by a computer.
[0170] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, or alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.
[0171] Illustrative aspects of the present disclosure include the following.
[0172] Aspect 1: A method for processing one or more images, the method including: acquiring a two-dimensional (2D) input image from a camera; acquiring a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image; adjusting a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object; and generating a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object to be output on a display.
[0173] Aspect 2: The method of aspect 1, wherein at least a subportion of the modified version of the target object has an enlarged size in the 2D output image compared to a corresponding subportion of the target object in the 2D input image.
[0174] Aspect 3: The method of aspect 1 or 2, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image includes one or more of adjusting depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.
[0175] Aspect 4: A method as described in any of aspects 1 to 3, wherein adjusting the pose of the first portion of the 3D representation of the corresponding 2D input image includes reducing depth values of a first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object.
[0176] Aspect 5: The method of aspect 4, wherein adjusting the pose of a first portion of the 3D representation of the 2D input image corresponding to the target object further includes increasing depth values of a second one or more vertices of the first portion of the 3D representation of the 2D input image.
[0177] Aspect 6: A method described in any of aspects 1 to 5, further comprising obtaining user input identifying a target object, wherein a pose of a first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.
[0178] Aspect 7: The method of aspect 6, further comprising obtaining an indication of an amount of adjustment of the target object, and adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the amount of adjustment of the target object.
[0179] Aspect 8: The method described in aspect 6 or 7, further comprising determining an adjustment amount of the target object based on one or more of the adjustment range, the apparent size change, and the maximum amount of distortion, and adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.
[0180] Aspect 9: A method described in any of aspects 6 to 8, further comprising: obtaining an object mask associated with a target object in the 2D input image; determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask; and combining the first portion of the 3D representation with the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.
[0181] Example 10: The method of example 9, wherein generating the 2D output image includes converting the combined 3D mesh into the 2D output image.
[0182] Aspect 11: A method described in any of aspects 1 to 10, further comprising: obtaining an object mask associated with one or more additional objects in the 2D input image that are different from the target object; determining a first portion of a 3D representation of the 2D input image that corresponds to the target object based on the object mask; determining a second portion of the 3D representation that corresponds to at least one of the one or more additional objects based on the object mask; and adjusting the pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.
[0183] Aspect 12: The method of aspect 11, wherein adjusting the pose of a first portion of the 3D representation of the 2D input image corresponding to the target object includes reducing depth values of vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object such that the vertices have a predetermined numerical correspondence compared to one or more fixed vertices of the second portion of the 3D representation.
[0184] Example 13: A method according to any of examples 1 to 12, wherein generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.
[0185] Example 14: A method as described in any of examples 1 to 13, wherein determining a first portion of the 3D representation of the 2D input image includes at least one of feature detection and object detection.
[0186] Aspect 15: A method according to any of aspects 1 to 14, wherein determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.
[0187] Example 16: A method described in any of examples 1 to 15, further comprising adjusting the pose of a first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.
[0188] Example 17: A method according to any of examples 1 to 16, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on user input.
[0189] Example 18: A method according to any of examples 1 to 17, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration.
[0190] Example 19: A method according to any of examples 1 to 18, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior.
[0191] Aspect 20: A method described in any of aspects 1 to 19, comprising determining a second portion of the 3D representation of the 2D input image that corresponds to a second target object in the 2D input image, adjusting a pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object, and generating a 2D output image further based on the adjusted pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object.
[0192] Aspect 21: The method of aspect 20, wherein the adjustments to the pose of the second part are different from the adjustments to the pose of the first part.
[0193] Example 22: A method described in any of examples 1 to 21, wherein the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.
[0194] Aspect 23: A method according to any of aspects 1 to 22, further comprising acquiring a second 2D input image from a camera, the second 2D input image including at least a portion of a scene contained in the 2D input image, and acquiring a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.
[0195] Aspect 24: An apparatus for processing one or more images, comprising a memory and one or more processors coupled to the memory, wherein the one or more processors are configured to: acquire a two-dimensional (2D) input image from a camera; acquire a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image; adjust a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object; and generate a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object to be output on a display.
[0196] Example 25: The apparatus of example 24, wherein at least a subportion of the modified version of the target object has an enlarged size in the 2D output image compared to a corresponding subportion of the target object in the 2D input image.
[0197] Example 26: The apparatus of example 24 or 25, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image includes one or more of adjusting depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.
[0198] Aspect 27: An apparatus described in any of aspects 24 to 26, wherein the one or more processors are configured to reduce depth values of a first one or more vertices of a first portion of the 3D representation of the 2D input image corresponding to the target object.
[0199] Example 28: The apparatus of example 27, wherein adjusting the pose of a first portion of the 3D representation of the 2D input image corresponding to the target object further includes increasing a depth value of a second one or more vertices of the first portion of the 3D representation of the 2D input image.
[0200] Aspect 29: An apparatus described in any of aspects 24 to 28, wherein the one or more processors are further configured to obtain user input identifying a target object, and a pose of a first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.
[0201] Aspect 30: The apparatus of aspect 29, wherein one or more processors are configured to obtain an indication of an amount of adjustment of a target object, and adjust the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the amount of adjustment of the target object.
[0202] Aspect 31: An apparatus as described in aspect 29 or 30, wherein one or more processors are configured to determine an adjustment amount of the target object based on one or more of the adjustment range, the apparent size change, and the maximum amount of distortion, and adjust the pose of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.
[0203] Aspect 32: An apparatus described in any of aspects 29 to 31, wherein one or more processors are configured to obtain an object mask associated with a target object in a 2D input image, determine a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask, determine a second portion of the 3D representation corresponding to one or more additional objects based on the object mask, and combine the first portion of the 3D representation having the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.
[0204] Example 33: An apparatus described in any of examples 24 to 32, wherein generating the 2D output image includes converting the combined 3D mesh into the 2D output image.
[0205] Aspect 34: An apparatus described in any of aspects 24 to 33, wherein one or more processors are configured to obtain an object mask associated with one or more additional objects in the 2D input image, different from the target object, determine a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask, determine a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask, adjust the pose of the first portion of the 3D representation and fix one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.
[0206] Aspect 35: The apparatus described in aspect 34, wherein to adjust the pose of a first portion of the 3D representation of the 2D input image corresponding to the target object, one or more processors are configured to reduce depth values of vertices of the first portion of the 3D representation of the 2D input image corresponding to the 2D input image such that the vertices have a predetermined numerical correspondence compared to one or more fixed vertices of a second portion of the 3D representation.
[0207] Example 36: An apparatus described in example 34 or 35, wherein generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.
[0208] Example 37: An apparatus described in any of examples 24 to 36, wherein determining a first portion of the 3D representation of the 2D input image includes at least one of feature detection and object detection.
[0209] Example 38: An apparatus described in any of examples 24 to 37, wherein determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.
[0210] Aspect 39: An apparatus described in any of aspects 24 to 38, wherein one or more processors are configured to adjust the pose of a first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.
[0211] Example 40: An apparatus described in any of examples 24 to 39, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on user input.
[0212] Example 41: An apparatus described in any of examples 24 to 40, wherein the pose of a first portion of the 3D representation of the 2D input image is adjusted by an amount determined based on a default configuration.
[0213] Example 42: An apparatus described in any of examples 24 to 41, wherein the pose of the first portion of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior.
[0214] Aspect 43: An apparatus described in any of aspects 24 to 42, wherein one or more processors are configured to determine a second portion of the 3D representation of the 2D input image that corresponds to a second target object in the 2D input image, adjust a pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object, and generate a 2D output image further based on the adjusted pose of the second portion of the 3D representation of the 2D input image that corresponds to the second target object.
[0215] Aspect 44: The apparatus of aspect 43, wherein the adjustments to the pose of the second portion are different from the adjustments to the pose of the first portion.
[0216] Example 45: An apparatus described in any of examples 24 to 44, wherein the modification of at least a portion of the modified version of the target object includes one or more of a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, or a roll rotation relative to a corresponding sub-portion of the target object in the 2D input image.
[0217] Aspect 46: An apparatus described in any of aspects 24 to 45, wherein one or more processors are configured to acquire a second 2D input image from a camera, the second 2D input image including at least a portion of a scene contained in the 2D input image, and acquire a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.
[0218] Aspect 47: A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform any of the operations of aspects 1-46.
[0219] Embodiment 48: An apparatus comprising means for performing any of the operations of embodiments 1 to 46.
Claims
1. 1. A method for processing one or more images, comprising the steps of: acquiring a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image; adjusting a pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object; generating a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object to be output on a display; A method comprising:
2. The method of claim 1 , wherein at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image compared to a corresponding sub-portion of the target object in the 2D input image.
3. 2. The method of claim 1 , wherein adjusting the pose of the first portion of the 3D representation of the 2D input image includes one or more of adjusting a depth, a vertical shift, a horizontal shift, a pitch rotation, a yaw rotation, and a roll rotation.
4. adjusting the pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object, The method of claim 1 , comprising decreasing depth values of a first one or more vertices of the first portion of the 3D representation of the 2D input image that corresponds to the target object.
5. 5. The method of claim 4, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further comprises increasing a depth value of a second one or more vertices of the first portion of the 3D representation of the 2D input image.
6. 2. The method of claim 1, further comprising obtaining a user input identifying the target object, and wherein the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.
7. obtaining an indication of an amount of adjustment of the target object; adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the adjustment amount of the target object; The method of claim 6 further comprising:
8. determining an adjustment amount for the target object based on one or more of an adjustment range, an apparent size change, and a maximum amount of distortion; adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object; The method of claim 6 further comprising:
9. obtaining an object mask associated with the target object in the 2D input image; determining the first portion of the 3D representation of the 2D input image that corresponds to the target object based on the object mask; determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask; combining the first portion of the 3D representation with the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh; The method of claim 1 further comprising:
10. The method of claim 9 , wherein generating the 2D output image comprises converting the combined 3D mesh into the 2D output image.
11. obtaining an object mask associated with one or more additional objects in the 2D input image that are different from the target object; determining the first portion of the 3D representation of the 2D input image that corresponds to the target object based on the object mask; determining a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask; adjusting the pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh; The method of claim 1 further comprising:
12. adjusting the pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object, 12. The method of claim 11, comprising reducing depth values of vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object such that the vertices have a predetermined numerical correspondence compared to the fixed one or more vertices of the second portion of the 3D representation.
13. The method of claim 11 , wherein generating the 2D output image comprises converting the adjusted 3D mesh into the 2D output image.
14. The method of claim 1 , wherein determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.
15. The method of claim 1 , wherein determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.
16. 1. An apparatus for processing one or more images, comprising: Memory, one or more processors coupled to the memory; wherein the one or more processors: acquiring a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image that corresponds to a target object in the 2D input image; adjusting the pose of the first portion of the 3D representation of the 2D input image that corresponds to the target object; generating a 2D output image having a modified version of the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object to be output on a display; The apparatus is configured so as to
17. The apparatus of claim 16 , wherein at least a sub-portion of the modified version of the target object has an enlarged size in the 2D output image compared to a corresponding sub-portion of the target object in the 2D input image.
18. 17. The apparatus of claim 16, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image comprises one or more of adjusting depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation.
19. the one or more processors: decreasing depth values of a first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object; The apparatus of claim 16 configured to:
20. 20. The apparatus of claim 19, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object further comprises increasing a depth value of a second one or more vertices of the first portion of the 3D representation of the 2D input image.
21. 17. The apparatus of claim 16, wherein the one or more processors are configured to obtain user input identifying the target object, and wherein the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.
22. the one or more processors: Obtaining an indication of an amount of adjustment of the target object; adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the indication of the adjustment amount of the target object.
22. The apparatus of claim 21 configured to:
23. the one or more processors: determining an adjustment amount for the target object based on one or more of an adjustment range, an apparent size change, and a maximum amount of distortion; adjusting the pose of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.
22. The apparatus of claim 21 configured to:
24. the one or more processors: obtaining an object mask associated with the target object in the 2D input image; determining the first portion of the 3D representation of the 2D input image that corresponds to the target object based on the object mask; determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask; combining the first portion of the 3D representation having the adjusted pose with the second portion of the 3D representation to generate a combined 3D mesh. The apparatus of claim 16 configured to:
25. 25. The apparatus of claim 24, wherein generating the 2D output image comprises converting the combined 3D mesh into the 2D output image.
26. the one or more processors: obtaining an object mask associated with one or more additional objects in the 2D input image that are distinct from the target object; determining the first portion of the 3D representation of the 2D input image that corresponds to the target object based on the object mask; determining a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask; adjusting the pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh. The apparatus of claim 16 configured to:
27. to adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object, the one or more processors: reducing depth values of vertices of the first portion of the 3D representation of the 2D input image corresponding to the 2D input image such that the vertices have a predetermined numerical correspondence compared to the fixed one or more vertices of the second portion of the 3D representation; 27. The apparatus of claim 26 configured to:
28. 27. The apparatus of claim 26, wherein generating the 2D output image comprises converting the adjusted 3D mesh into the 2D output image.
29. The apparatus of claim 16 , wherein determining the first portion of the 3D representation of the 2D input image comprises at least one of feature detection and object detection.
30. The apparatus of claim 16 , wherein determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.
Citation Information
Patent Citations
Methods and Apparatus for Manipulating Images and Objects Within Images
US20130120457A1
Perspective distortion correction on faces
WO2019216879A1
