Method and apparatus utilizing image modification techniques

Abstract

This document describes systems and techniques for modifying the scale and / or position of objects in an image. For example, the system can acquire a two-dimensional (2D) input image from a camera and a three-dimensional (3D) representation of that 2D input image. The system can also determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image. The system can adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object. The system can also generate a modified 2D output image with the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image to be output on a display.

Description

[Technical Field]

[0001] This case generally relates to image adjustment or modification. For example, the nature of this case relates to systems and techniques for modifying the scale and / or position of objects in an image. [Previous Technology]

[0002] Many devices and systems allow a scene to be captured by generating images (or frames) and / or video data (including multiple frames). For example, a camera or a device that includes a camera can capture a sequence of frames of a scene (e.g., video of the scene). In some cases, the sequence of frames can be processed to perform one or more functions, can be output for display, can be output for processing and / or consumption by other devices, and for other purposes.

[0003] When using equipment to capture images, users may have to choose between different available fields of view offered by different image capture devices. In some cases, users may have to choose between using a tele-angle lens or telephoto lens that results in larger objects being shown in an image with a narrow field of view, or using a wide-angle lens with a wider field of view. For example, an image captured using a wide-angle lens can capture a larger portion of the scene (due to the wide field of view), but objects in the image may be relatively small.

[0004] Users can provide input (e.g., zoom input, such as pinch-to-zoom input) to increase the size of a portion of the image (e.g., zoom in on the two people in the lower left of the image above). However, the entire image will also be zoomed in, causing a large amount of background to be outside the field of view of the zoomed image. [Summary of the Invention]

[0005] In some instances, systems and techniques for adjusting one or more objects or features in an image are described. According to at least one instance, a method for processing one or more images is provided. The method includes: acquiring a two-dimensional (2D) input image from a camera; acquiring a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generating a modified 2D output image having the target object based on the adjusted pose corresponding to the first portion of the 3D representation of the 2D input image to be output on a display.

[0006] In another embodiment, an apparatus for processing one or more images is provided, comprising a memory (e.g., configured to store data such as virtual content data, one or more images, etc.) and one or more processors (e.g., implemented in circuitry) coupled to the memory. The one or more processors are configured to: acquire a two-dimensional (2D) input image from a camera; acquire a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generate a modified 2D output image having the target object based on the adjusted pose corresponding to the first portion of the 3D representation of the 2D input image to be output on a display.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: acquire a two-dimensional (2D) input image from a camera; acquire a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generate a modified 2D output image having the target object based on the adjusted pose corresponding to the first portion of the 3D representation of the 2D input image to be output on a display.

[0008] In another embodiment, an apparatus for processing one or more images is provided. The apparatus includes: components for acquiring a two-dimensional (2D) input image from a camera; components for acquiring a three-dimensional (3D) representation of the 2D input image; components for determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; components for adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and components for generating a modified 2D output image having the target object based on the adjusted pose corresponding to the first portion of the 3D representation of the 2D input image to be output on a display.

[0009] In some cases, at least a sub-part of the modified version of the target object has an enlarged size in the 2D output image relative to the corresponding sub-part of the target object in the 2D input image.

[0010] In some cases, adjusting the attitude of the first part of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

[0011] In some cases, the depth value of the first one or more vertices of the first part of the 3D representation of the 2D input image of the target object is reduced.

[0012] In some cases, adjusting the pose of the first part of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of the second one or more vertices of the first part of the 3D representation of the 2D input image.

[0013] In some cases, the methods, apparatus and computer-readable media described above also include: obtaining user input that identifies a target object; adjusting the pose of a first portion of a 3D representation of a 2D input image corresponding to the target object based on the user input.

[0014] In some cases, the methods, apparatus and computer-readable media described above also include: obtaining an indication of the adjustment amount of the target object; and adjusting the posture of the 3D representation of the 2D input image to correspond to the indication of the adjustment amount of the target object.

[0015] In some cases, the methods, apparatus and computer-readable media described above also include: determining the adjustment amount of the target object based on one or more of the adjustment range, apparent size change and maximum distortion; and adjusting the pose of the 3D representation of the 2D input image to correspond to the adjustment amount of the target object.

[0016] In some forms, the methods, apparatus, and computer-readable media described above also include: obtaining an object mask associated with a target object in a 2D input image; determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of a 3D representation corresponding to one or more additional objects based on the object mask; and combining the first portion of the 3D representation with an adjusted pose and the second portion of the 3D representation to produce a combined 3D mesh.

[0017] In some cases, generating a 2D output image includes converting a combined 3D mesh into a 2D output image.

[0018] In some forms, the methods, apparatus, and computer-readable media described above also include: obtaining an object mask associated with one or more additional objects in a 2D input image that are different from the target object; determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of a 3D representation corresponding to at least one of the one or more additional objects based on the object mask; adjusting the pose of the first portion of the 3D representation; and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.

[0019] In some cases, adjusting the pose of the first part of the 3D representation of the 2D input image of the target object includes reducing the depth values ​​of the vertices of the first part of the 3D representation of the 2D input image of the target object, such that these vertices have a predetermined numerical correspondence with one or more fixed vertices of the second part of the 3D representation.

[0020] In some cases, generating a 2D output image includes converting an adjusted 3D mesh into a 2D output image.

[0021] In some cases, the first part that determines the 3D representation of the 2D input image includes at least one of feature detection and object detection.

[0022] In some cases, the first part of determining the 3D representation of the 2D input image is based on the depth information corresponding to the first part of the 3D representation of the 2D input image.

[0023] In some cases, the methods, apparatus and computer-readable media described above also include: adjusting the pose of a first portion of a 3D representation of a 2D input image without adjusting a second portion of a 3D representation of a 2D input image that does not correspond to a target object.

[0024] In some cases, the pose of the first part of the 3D representation of the 2D input image is adjusted by an amount determined based on the user input.

[0025] In some cases, the pose of the first part of the 3D representation of the 2D input image is adjusted by an amount determined based on a preset configuration.

[0026] In some cases, the pose of the first part of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior.

[0027] In some cases, the methods, apparatus and computer-readable media described above also include: determining a second portion of a 3D representation of a 2D input image corresponding to a second target object in a 2D input image; adjusting the pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object; and further generating a 2D output image based on the adjusted pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object.

[0028] In some cases, the adjustment of the second part of the attitude is different from the adjustment of the first part of the attitude.

[0029] In some cases, modifications to at least a portion of the modified version of the target object include one or more of vertical displacement, horizontal displacement, pitch rotation, yaw rotation, or roll rotation relative to the corresponding sub-part of the target object in the 2D input image.

[0030] In some forms, the methods, apparatus and computer-readable media described above also include: acquiring a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image; and acquiring a second 3D representation of the second 2D input image, wherein at least a portion of a 2D output image is generated based on the second 3D representation of the second 2D input image.

[0031] In some embodiments, the one or more devices described above are, partially are, or include: mobile devices (e.g., mobile phones or so-called "smartphones" or other mobile devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), personal computers, laptops, server computers, vehicles (e.g., computing devices of vehicles), or other devices. In some embodiments, a device includes a camera or multiple cameras for capturing one or more images. In some embodiments, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the device may include one or more sensors. In some cases, one or more sensors may be used to determine the position and / or orientation of the device, the state of the device, and / or for other purposes.

[0032] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0033] The foregoing and other features and embodiments will become clearer with reference to the following description, requests and drawings.

Implementation Method

[0046] Certain aspects and embodiments of this invention are provided below. Some of these aspects and embodiments can be applied independently, and some can be applied in combination, as will be clear to those skilled in the art to which this invention pertains. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this invention. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting.

[0047] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this invention. Rather, the subsequent description of exemplary embodiments will provide enabling descriptions for implementing the exemplary embodiments to those skilled in the art to which this invention pertains. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of this invention as set forth in the appended claims.

[0048] When capturing images of a scene, photographers often need to choose between a wide field of view (FOV) that captures background objects and provides context for the photograph, and a narrow field of view (e.g., with a telephoto lens) to capture foreground objects more prominently and / or in more detail. Figures 1A to 1C illustrate images of different scenes based on different FOV choices. Figure 1A is a photograph of two people 102 posing in front of a large building 104, captured using a wide FOV. In Figure 1A, image 100 illustrates that the building 104 occupies almost the entire height of image 100, while the people 102 occupy less than a quarter of the height of image 100. Figures 1B and 1C illustrate two images of an event captured using different FOVs. In Figure 1B, image 110 is captured with a narrow FOV, and the people 112 near the camera (e.g., in the foreground) and performers 114 are prominently included in image 110, while only a limited portion of the background 116 is included in image 110. In Figure 1C, image 120 is captured with a wide field of view (FOV), and person 112 continues to be highlighted due to their proximity to the camera. In image 120, performer 114 appears smaller relative to person 112, and the larger portion of the background 116 provides further context for the event illustrated in image 120, which is taking place in a stadium.

[0049] In addition, in some cases, the device’s front camera may not have a zoom function that allows the user to zoom in on a portion of the image (e.g., objects in the background of the image). For example, when using the front camera of a mobile phone to capture a “selfie” image, the user may not be able to provide input to make the mobile phone zoom in on objects in the background of the selfie image.

[0050] Image adjustment techniques can be used to adjust the region of interest (ROI) or target object in the captured image to combine the benefits of different fields of view (FOV). For example, image adjustment techniques can be used to make performer 114 stand out more in image 120 (e.g., by scaling and / or increasing the size of the singer) while maintaining the proportion of background 116.

[0051] Figures 2A to 2C illustrate exemplary image adjustment techniques. Figure 2A illustrates an exemplary retouching technique 200 that can be used to adjust one or more ROIs or target objects in a 2D image. The input image 202 shows the initial state of a 2D image captured by a camera. In the input image 202, a person 203 is in the foreground of the image, and many other objects (such as trees, buildings, etc.) are in the background 205 of the input image 202. In the exemplary retouching technique 200 shown in Figure 2A, the target object selected for adjustment is the person 203, and the selected adjustment is to enlarge (or enlarge) the person 203.

[0052] Segmentation image 204 provides an example of segmentation that can be performed on input image 202 to separate an ROI or target object from other objects in input image 202. In segmentation image 204, the region of input image 202 associated with person 203 (represented by a white region shaped like person 203) is shown as segmented from the remainder of input image 202 (represented by a black background surrounding person 203). In some cases, the segmentation shown in segmentation image 204 can be performed wholly or partially by a machine learning model. For example, a machine learning model can be trained to detect features in input image 202. In some cases, based on the features detected in input image 202, the machine learning model can classify objects in input image 202. Other examples of segmentation are described in more detail below with respect to the feature segmentation machine learning model 404 shown in Figure 4. Exemplary classifications of person 203 may include "person", "foreground object", etc. In some cases, users can select which objects (or categories) in the input image 202 to target for adjustment (e.g., magnification) using the repair technique 200.

[0053] Based on the segmentation between the person 203 and the background 205 in the input image 202 shown in the segmentation image 204, the inpainting technique 200 can generate a target object image 206 containing primitives associated with the target object (e.g., person 203) in the input image 202, and an unadjusted image 208 containing primitives associated with the background 205 in the input image 202. In the target object image 206, primitives associated with the background 205 have been removed, as shown in blank area 207, and conversely, in the unadjusted image 208, primitives associated with the target object (e.g., person 203) have been removed, as shown in blank area 209. The adjusted image 210 shows an enlargement of the person 203. As shown, the enlargement includes enlarging the size of the person 203 to an enlarged person 211. In some cases, techniques for magnifying a target object (e.g., person 203) may include interpolation of primitives associated with the target object on a larger portion than the portion occupied by the target object in the input image 202. In some cases, after magnification, the primitives of the enlarged person 211 may not completely overlap with the blank area 209 corresponding to the original size of the person 203. Similarly, the blank area 209 may occupy primitives not covered by the enlarged person 211 after magnification. The repaired image 212 illustrates a repair procedure that attempts to fill the blank area 209 in the unadjusted image 208 with a color that approximates (or estimates) the captured scene portion occluded by the person 203 in the input image 202. In one illustrative example, repair may be performed by interpolating and / or blending the colors of primitives adjacent to the blank area 209 in the input image 202 and applying the interpolated colors to the blank area 209. As shown in patched image 212, the outline of scaled objects (e.g., person 203) in patched portion 213 can be preserved after patching is completed. Repair technique 200 can combine adjusted image 210 and patched image 212. In some cases, repair technique 200 can perform blending and smoothing around the edges of the enlarged person 211 and patched portion 213. In another illustrative example, a second input image (not shown) of the same scene as or a portion of the same scene as input image 202 can include scenes in input image 202 occluded by person 203. In some instances, repair technique 200 can combine primitives from the second input image with adjusted image 210, patched image 212, one or more additional images, or any combination thereof. Image 216 shows the final result of repair technique 200. The repair technique described with respect to Figure 2A can be computationally intensive and can lead to artifacts, where the repair must infer primitive color information for scenes not visible in the input image.

[0054] Figures 2B and 2C illustrate another exemplary image adjustment technique 220 for adjusting the appearance of a ROI and / or target object in an image. Figure 2B illustrates an example of a 2D image 240 and a corresponding 3D mesh 250. The 2D image 240 includes a 2D representation of a target object 243 (e.g., a vehicle), and the 3D mesh 250 includes a 3D representation of the target object 253. As shown in Figure 2B, the 2D image 240 can be considered as a projection of the 3D mesh 250 onto the image plane 248. In some cases, the 3D mesh 250 can also be considered as an unprojection of the 2D image on the image plane 248 into the 3D space of the 3D mesh 250. The position of the image plane 248 relative to the camera 244 can be determined based on camera parameters such as focal length, field of view, skew (e.g., misalignment between the camera and the image plane 248), etc. In the examples shown in Figures 2B and 2C, the field of view 270 of camera 244 is shown as a pyramid-shaped region in the 3D space of a 3D mesh. An exemplary primitive 242 in the 2D image 240 of an object is traversed by a ray originating from the projection center 245 of camera 244. The position corresponding to the projection center 245 of camera 244 in the physical world can be represented as a point in the 3D mesh 250. In some cases, if the distance to the target object 243 is a known distance d from the projection center 245, then the point 246 (e.g., a 3D vertex) corresponding to the exemplary primitive 242 can be located at a distance d from the camera projection center 245 in the ray direction.

[0055] In some cases, when distance information is included in the 2D image 240, the distance information corresponding to each pixel in the 2D image can be known. In an illustrative example, the 2D image 240 may include an RGB-D image, which includes color information and depth information for the location of each pixel in the 2D image 240. In some cases, the distance information corresponding to each pixel in the 2D image can be estimated from a 2D image that does not include explicit distance information.

[0056] In an illustrative example, depth information can be inferred by a machine learning system trained to determine depth (or the relative depth of objects in a 2D image). In some cases, the training dataset may include 2D images having depth information for each primitive in a 2D image (e.g., RGB-D). In some cases, the training dataset may include images with labeled features, objects, and / or classifications (e.g., cars, people, trees, buildings, foreground objects, background objects) and depth information corresponding to the labeled features, objects, and / or classifications. During inference (e.g., after the machine learning model has been trained), the machine learning model may estimate distances corresponding to primitives, features, objects, labels, and / or classifications associated with the 2D image.

[0057] FIG. 2C illustrates an exemplary technique for generating an adjusted 2D image 260 corresponding to 2D image 240, wherein the 2D representation of target object 263 is increased in size and / or has an enlarged appearance compared to the 2D representation of target object 243 shown in FIG. 2B. The enlargement effect can be achieved by moving the projection center 245 of camera 244 to a new position in 3D grid 250 (e.g., with different vertex coordinates). For example, the projection center 245 of the camera can be moved closer to the 3D representation of target object 253. As shown, the intersection point between the rays passing through point 246 of the 3D representation of target object 253 and the image plane 268 is at primitive position 262. In some cases, the adjusted 2D image 260 can be constructed by projecting rays from camera projection center 245 through each vertex of 3D grid 250 (including vertices of the 3D representation of target object 253) and determining the corresponding primitive position where the projection rays intersect. However, such technologies can be computationally intensive.

[0058] Another exemplary implementation for generating the adjusted 2D image 260 is to use perspective projection techniques. In an illustrative example, the projection from the 3D mesh 250 to the adjusted 2D image 260 may include perspective projection as shown in equation (1). (1)

[0059] where x and y are primitive coordinates of the projection in the 2D image, X is the X component of a vertex in the 3D mesh (e.g., horizontal axis position), Y is the Y component of a vertex in the 3D mesh (e.g., vertical axis position), and Z is the Z component of a vertex in the 3D mesh (e.g., depth). In an example of equation (1), the origin of the coordinate system of the 3D mesh may be located at the camera projection center 245, also known as the camera coordinate system. In some cases, if the coordinates of the vertices of the 3D mesh 250 are relative to a coordinate system different from the camera coordinate system, the image adjustment technique 220 may perform a coordinate transformation to the camera coordinate system before projecting the 3D mesh 250 onto the adjusted 2D image 260. As a supplement to or alternative to perspective projection techniques, other projection techniques from 3D to 2D may be used without departing from the scope of this case, including but not limited to orthographic projection and oblique projection. In some implementations, projection techniques corresponding to the projection techniques of a physical camera capturing 2D images may be used. In some cases, using projection techniques corresponding to physical camera projection techniques can reduce distortion in the adjusted 2D image, which may be introduced by a mismatch between physical camera projection techniques and projection techniques used to project the 3D grid 250 onto the adjusted 2D image 260.

[0060] The image adjustment technique 220 shown in Figures 2B and 2C can be superior to the patching technique 200 shown in Figure 2A because the 3D representation of the target object 253 and the remainder of the 3D mesh remain fixed. Therefore, no blank areas such as the blank area 209 shown in Figure 2A are created, and the image adjustment technique 220 may not require patching. One drawback of the image adjustment technique 220 shown is that moving the position of the camera 244 (and thus moving the projection center 245) simultaneously adjusts (e.g., enlarges) all vertices in the 3D mesh, including the 3D representation of the target object 253, as well as vertices in the 3D mesh that are not associated with the target object (not shown), such as other objects (e.g., cars, people, buildings, trees, etc.). As a result, while the image adjustment technique 220 achieves the desired effect of adjusting (e.g., enlarging) the target object, it does not provide the ability to adjust only (one or more) ROIs or (one or more) target objects without adjusting other parts of the original 2D image 240.

[0061] Systems and techniques are needed to accurately and effectively adjust one or more ROIs or target objects in an image without affecting other parts of the image. For example, adjusting the appearance of one or more ROIs or target objects in an image without adjusting other objects and / or parts of the image can combine the advantages of wide FOV and narrow FOV. For example, an ROI that appears small in the input image due to wide FOV image capture can be resized (e.g., enlarged) to appear larger and / or closer to the camera in the adjusted image.

[0062] This document describes systems, apparatuses, procedures (also referred to as methods), and computer-readable media (collectively, the “Systems and Techniques”) for selectively adjusting one or more regions of interest (ROIs) of an image without affecting other parts of the image. These systems and techniques allow users to provide user input to adjust the appearance (e.g., increase size and / or change position) of an ROI or target object in an image (e.g., selected by a user, a machine learning model, etc.) without affecting the appearance (e.g., size, position, etc.) of other parts of the image (e.g., foreground objects, background objects, etc.). In an illustrative example, by allowing users to provide input to selectively magnify one or more target objects or ROIs in an image while maintaining the field of view (FOV) (e.g., a wide FOV provided by a wide-angle lens), such that objects and the background other than the ROI / target object of the image are not scaled, or are scaled to a lesser extent compared to the target object, the systems and techniques can enhance the user experience.

[0063] In some instances, the image adjustment system can process one or more depth images and one or more color images (e.g., an image with red (R), green (G), and blue (B) components per pixel, referred to as an RGB image). The depth image can indicate the depth of each pixel in a color image from a camera used to capture the color image. In some cases, a wide-angle lens is used to capture one or more color images. In some cases, instead of one or more color images, or in addition to one or more color images, the image adjustment system can process one or more monochrome images. In some instances, a single image can include color information (e.g., RGB) and depth information per pixel, referred to as an RGB depth (RGB-D) image.

[0064] The image adjustment system can use a color image and a corresponding depth image to generate a three-dimensional (3D) representation (e.g., a 3D mesh) of a two-dimensional (2D) color image. For example, based on depth information of each primitive indicated by the depth image and the known field of view (FOV) of the camera (e.g., based on camera parameters such as intrinsic and / or extrinsic parameters), the system can determine the 3D position of a given primitive in the color image in 3D space (e.g., a 3D mesh). In an illustrative example of a primitive distance (from the camera) of 10 meters indicated by the depth image, the primitive can represent the intersection point between rays originating from the camera projection center pointing in a direction corresponding to the 3D position of the primitive. The 3D position corresponding to a primitive in the 2D color image can be located 10 meters from the camera center in the ray direction. Once each primitive of the color image is back-projected, the system can connect the corresponding 3D positions (e.g., vertices) into a mesh. The system can use color information from the color image to texture the mesh (with color). This type of technique is called perspective back-projection (from 2D RGB-D to 3D). In some cases, a 3D mesh can be a triangular or polygonal mesh representing the structure of items in an image (e.g., objects such as people and other physical objects in the image). For example, a 3D mesh can include a set of vertices, edges, and faces that define the shape of objects in the 3D space of a scene.

[0065] The image adjustment system can also perform object detection (e.g., face detection) on the color image to detect one or more objects (e.g., faces) in the image. The user can provide user input (e.g., via a touch screen interface, using voice input, using gesture input, gaze input, etc.) to instruct or select one or more ROIs in the color image that the user wants to adjust (e.g., enlarge, change rotation, etc.), which can be associated with one or more target objects from the detected one or more objects.

[0066] In some cases, the most maximally corrected object (e.g., the one most aligned with the vertical or horizontal axis of the 2D image) can be used to indicate or select one or more ROIs. In one illustrative example, the orientation of the camera and / or the device incorporating the camera can be used to determine the most maximally corrected object. In some cases, a scene or a portion of a scene captured in a 2D color image by the first camera can also be captured by the second camera. In some cases, objects in a scene or a portion of a scene captured by both the first and second cameras can be used to determine one or more ROIs. In some cases, the first and second cameras can be included in a single device (e.g., a mobile device with multiple cameras). In some cases, the first and second cameras can be included in a separate device.

[0067] In some instances, the image adjustment system can perform segmentation (e.g., binary segmentation, semantic segmentation, instance segmentation, etc.) on a color image to generate an object mask that identifies one or more objects in the color image. In one instance, such as when capturing a selfie image, the object mask can indicate primitives belonging to the object closest to the camera in the image and primitives not belonging to the object closest to the camera in the image. The image adjustment system can use the object mask to prevent one or more objects identified in the object mask from being adjusted (e.g., by preventing adjustment or fixing vertices associated with the masked object).

[0068] Based on the selection of 3D mesh, object mask, and ROI (corresponding to the target object), the image adjustment system can determine the adjusted portion of the 3D mesh corresponding to the ROI (or target object) and the unadjusted portion of the 3D mesh (e.g., the portion of the 3D mesh to be prevented from adjustment). The image adjustment system can deform the geometry (e.g., vertices, edges, and faces) of the adjusted portion of the 3D mesh corresponding to the ROI to effectively increase the size, change the rotation, or otherwise adjust the corresponding scaling of the deformed 3D mesh to which it will be reprojected in a two-dimensional (2D) output image. The deformation of the adjusted portion of the 3D mesh corresponding to the ROI can keep the unadjusted portion of the 3D mesh unchanged. In some cases, the deformation of the adjusted portion of the 3D mesh corresponding to the ROI can also keep the sub-parts of the adjusted portion unaffected by the geometry deformation of the ROI unchanged.

[0069] In an illustrative example, to deform the geometry of an adjusted portion of a 3D mesh to result in magnification of the ROI in a 2D image, the system and technique can reduce the depth values ​​(e.g., z-values) of vertices (and therefore edges and faces) within the adjusted portion of the 3D mesh, thereby bringing the vertices closer to the camera. Alternatively or supplementarily, to deform the geometry of the adjusted portion of the 3D mesh, the system can translate the geometry of the adjusted portion along a translation vector that maintains the position of a sub-part of the ROI in the 2D output image. For example, the translation vector can be chosen to maintain a fixed 2D primitive position of a sub-part of the ROI / object (e.g., the center of a face). In some cases, the direction of the translation vector can correspond to the direction of a ray passing from the camera center through the fixed 2D primitive position and the corresponding 3D vertex. As used herein, deforming the geometry of the adjusted portion can also be referred to as adjusting the pose of the geometry of a first portion of the 3D mesh. In some implementations, after deforming the adjusted portion, the image adjustment system can fuse or combine the deformed (or pose-adjusted) adjusted portion of the 3D mesh with the unadjusted portion of the 3D mesh. In some implementations, the adjusted and unadjusted portions can be maintained in a single 3D mesh, where the 3D vertices corresponding to the unadjusted portion are fixed and excluded from any deformation of the adjusted portion, which may occur close to the unadjusted portion.

[0070] The image adjustment system can then transform or translate the 3D mesh (including the deformed portion corresponding to the target object) into the 2D image domain, thereby producing an output 2D image with a modified version of the ROI / target object. In an illustrative example, after partial deformation of the 3D mesh corresponding to the ROI / target object, the ROI / target object in 2D can be enlarged (compared to the original image). For example, perspective projection techniques can be performed to transform the 3D mesh from the 3D domain to the 2D image domain. Therefore, the output image includes a wide field of view of a wide-angle lens and an enlarged version of the ROI / target object.

[0071] Although this document describes an example of adjusting one or more ROIs in an image obtained from a handheld camera, the image adjustment systems and techniques described herein can be used to perform image adjustment on images obtained from other devices, such as vehicles in a vehicle, robotic devices in a factory environment, and / or any other system that includes a camera.

[0072] The various forms of the technology described herein will be discussed below with reference to the accompanying drawings. Figure 3 is a block diagram showing the architecture of the image capture and processing system 300. The image capture and processing system 300 includes various components for capturing and processing scene images (e.g., images of scene 310). The image capture and processing system 300 can capture individual images (or photographs) and / or can capture video including multiple images (or video frames) comprising a specific sequence. The lens 315 of the image capture and processing system 300 faces the scene 310 and receives light from the scene 310. The lens 315 directs the light to the image sensor 330. The light received by the lens 315 passes through an aperture controlled by one or more control mechanisms 320 and is received by the image sensor 330.

[0073] One or more control mechanisms 320 may control exposure, focus, and / or zoom based on information from the image sensor 330 and / or information from the image processor 350. One or more control mechanisms 320 may include multiple mechanisms and components; for example, control mechanisms 320 may include one or more exposure control mechanisms 325A, one or more focus control mechanisms 325B, and / or one or more zoom control mechanisms 325C. One or more control mechanisms 320 may also include additional control mechanisms besides those shown, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0074] The focus control mechanism 325B of the control mechanism 320 can obtain the focus setting. In some instances, the focus control mechanism 325B stores the focus setting in a storage register. Based on the focus setting, the focus control mechanism 325B can adjust the position of the lens 315 relative to the position of the image sensor 330. For example, based on the focus setting, the focus control mechanism 325B can move the lens 315 closer to or further away from the image sensor 330 via an actuation motor or servo mechanism (or other lens mechanism) to adjust the focus. In some cases, the image capture and processing system 300 may include additional lenses, such as one or more microlenses on each photodiode of the image sensor 330, each microlens redirecting light received from the lens 315 to the corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast-detection autofocus (CDAF), phase-detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus settings can be determined using the control mechanism 320, the image sensor 330, and / or the image processor 350. The focus settings may be referred to as image capture settings and / or image processing settings.

[0075] The exposure control mechanism 325A of the control mechanism 320 can obtain the exposure settings. In some cases, the exposure control mechanism 325A stores the exposure settings in a storage register. Based on the exposure settings, the exposure control mechanism 325A can control the aperture size (e.g., aperture size or f / stop), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 330 (e.g., ISO speed or film speed), the analog gain applied to the image sensor 330, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0076] The zoom control mechanism 325C of the control mechanism 320 can obtain zoom settings. In some instances, the zoom control mechanism 325C stores the zoom settings in a storage register. Based on the zoom settings, the zoom control mechanism 325C can control the focal length of a lens element (lens assembly) including lens 315 and one or more additional lenses. For example, the zoom control mechanism 325C can control the focal length of the lens assembly by actuating one or more motors or servo mechanisms (or other lens mechanisms) to move one or more lenses relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some instances, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some instances, the lens assembly may include a focusing lens (in some cases, lens 315) that first receives light from scene 310, and then, before reaching image sensor 330, the light passes through the focusless zoom system between the focusing lens (e.g., lens 315) and image sensor 330. In some cases, the focusless zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference between them), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 325C moves one or more lenses in the focusless zoom system, such as a negative lens and one or two positive lenses.

[0077] The image sensor 330 includes one or more photodiode arrays or other photosensitive elements. Each photodiode measurement ultimately corresponds to the amount of light in a specific pixel in the image generated by the image sensor 330. In some cases, different photodiodes can be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, wherein each pixel of the image is generated based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald green") color filters to replace or supplement red, blue, and / or green color filters. Some image sensors (e.g., image sensor 330) may not have color filters at all, instead using different photodiodes (in some cases stacked vertically) throughout the primitive array. Different photodiodes throughout the primitive array can have different spectral sensitivity profiles, thus responding to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0078] In some cases, the image sensor 330 may alternatively or additionally include an opaque and / or reflective shield that blocks light from reaching a specific photodiode or a portion of a specific photodiode at a specific time and / or from a specific angle, which can be used for phase detection autofocus (PDAF). The image sensor 330 may also include an analog gain amplifier to amplify the analog signal output from the photodiode, and / or include an analog-to-digital converter (ADC) to convert the analog signal output from the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, specific components or functions discussed with respect to one or more control mechanisms 320 may alternatively or additionally be included in the image sensor 330. The image sensor 330 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0079] The image processor 350 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 354), one or more host processors (including host processor 352), and / or one or more processors 1110 of any other type discussed with respect to the computing system 1100. The host processor 352 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 350 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including the host processor 352 and the ISP 354. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 356), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, interconnect components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. I / O port 356 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer port or interface, Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports). In an illustrative example, host processor 352 may communicate with image sensor 330 using the I2C port, and ISP 354 may communicate with image sensor 330 using the MIPI port.

[0080] The image processor 350 can perform multiple tasks, such as demosaicing, color space conversion, image frame undersampling, primitive interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, image merging to form HDR images, image recognition, object recognition, feature recognition, input reception, output management, memory management, or combinations thereof. The image processor 350 can store image frames and / or processed images in random access memory (RAM) 140 / 3225, read-only memory (ROM) 145 / 1120, cache memory, memory units, another storage device, or combinations thereof.

[0081] Various input / output (I / O) devices 360 can be connected to the image processor 350. I / O devices 360 may include a display screen, keyboard, keypad, touchscreen, trackpad, touch-sensitive surface, printer, any other output device 1135, any other input device 1145, or combinations thereof. In some cases, subtitles can be input into the image processing device 305B via the physical keyboard or keypad of the I / O device 360 ​​or via the virtual keyboard or keypad of the touchscreen of the I / O device 360. I / O 360 may include one or more ports, jacks, or other connectors that enable wired connections between the image capture and processing system 300 and one or more peripheral devices, through which the image capture and processing system 300 can receive data from and / or transmit data to one or more peripheral devices. I / O 360 may include one or more wireless transceivers that enable wireless connectivity between the image capture and processing system 300 and one or more peripheral devices, through which the image capture and processing system 300 can receive data from and / or transmit data to one or more peripheral devices. Peripheral devices may include any of the aforementioned types of I / O devices 360, and once they are coupled to ports, jacks, wireless transceivers, or other wired and / or wireless connectors, they themselves may be considered I / O devices 360.

[0082] In some cases, the image capture and processing system 300 may be a single device. In some cases, the image capture and processing system 300 may be two or more separate devices, including an image capture device 305A (e.g., a camera) and an image processing device 305B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 305A and the image processing device 305B may be coupled together, for example, via one or more wires, cables or other electrical connectors and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 305A and the image processing device 305B may be disconnected from each other.

[0083] As shown in Figure 3, the vertical dashed line divides the image capture and processing system 300 of Figure 3 into two parts, representing the image capture device 305A and the image processing device 305B, respectively. The image capture device 305A includes a lens 315, a control mechanism 320, and an image sensor 330. The image processing device 305B includes an image processor 350 (including an ISP 354 and a host processor 352), RAM 340, ROM 345, and I / O 360. In some cases, certain components shown in the image capture device 305A (such as the ISP 354 and / or the host processor 352) may be included in the image capture device 305A.

[0084] The image capture and processing system 300 may include electronic devices such as mobile or static telephones (e.g., smartphones, cellular phones, etc.), desktop computers, laptops or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some instances, the image capture and processing system 300 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, or combinations thereof). In some implementations, the image capture device 305A and the image processing device 305B may be different devices. For example, the image capture device 305A may include a camera device, and the image processing device 305B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0085] Although the image capture and processing system 300 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 300 may include more components than those shown in FIG. 3. The components of the image capture and processing system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 300 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image capture and processing system 300. In some instances, the image adjustment system 400 may include an image capture and processing system 300, an image capture device 305A, an image processing device 305B, or a combination thereof.

[0086] Figure 4 is a block diagram illustrating an example of an image adjustment system 400. The image adjustment system 400 includes various components for processing one or more images, such as changing the size and / or apparent position of features or objects in one or more images. The image adjustment system 400 can generate a 3D mesh based on an input image, adjust (one or more) ROIs or objects in the 3D mesh to generate an adjusted 3D mesh, and project the adjusted 3D mesh back onto a 2D image to generate an adjusted 2D image. As shown, the components of the image adjustment system 400 include one or more image capture devices 402, a feature segmentation machine learning model 404, a 3D mesh engine 406, and a projection engine 408.

[0087] In the following description of FIG4 and the image adjustment system 400, reference is made to the exemplary images shown in FIGS. 5A to 5K, which illustrate different states of image adjustment performed by the image adjustment system 400. In the examples shown in FIGS. 5A to 5K, a second person 504 is selected for adjustment (e.g., the second person 504 is a ROI or a selected object). In the illustrated example, the adjustment applied to the second person 504 is selected to bring the second person 504 closer to the first person 502. In an illustrative example, the vertices associated with the second person 504 can be adjusted by bringing the vertices closer to the camera position in the 3D coordinate system.

[0088] Referring back to Figure 4, the image processing system 400 may include or be part of the following: mobile or stationary telephones (e.g., smartphones, cellular phones, etc.), server computers (e.g., communicating with vehicle computing systems), desktop computers, laptops or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video streaming devices, or any other suitable electronic devices. In some instances, the image capture and processing system 400 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications (such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, Bluetooth or other short-range communications, any combination thereof, and / or other communications). In some implementations, components of the image adjustment system 400 (e.g., one or more image capture devices 402, feature segmentation machine learning model 404, 3D mesh engine 406, and projection engine 408) may be part of the same computing device. In some implementations, components of the image adjustment system 400 may be part of two or more separate computing devices. In some cases, the image adjustment system 400 may be implemented as part of the computing system 1100 shown in FIG. 11.

[0089] Although the image adjustment system 400 is shown to include certain components, those skilled in the art will understand that the image adjustment system 400 may include more or fewer components than those shown in FIG. 3. In some cases, additional components of the image adjustment system 400 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, the image adjustment system 400 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. 3. In some implementations, additional components of the image adjustment system 400 may include and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image adjustment system 400.

[0090] One or more image capture devices 402 may capture one or more images. One or more image capture devices 402 (e.g., cameras or other image sensors) may be included in a mobile device and may be directed toward the user of the device (e.g., having one or more front cameras) or away from the user of the device (e.g., having one or more rear cameras).

[0091] Each of the one or more image capture devices 402 may include a camera or other type of image sensor. In some instances, the one or more image capture devices 402 may include an infrared (IR) camera configured to capture IR images and / or near-infrared (NIR) images. For example, an IR camera or sensor may capture IR signals. IR signals have wavelengths and frequencies falling within the IR electromagnetic spectrum. The IR electromagnetic spectrum includes wavelengths in the range of 2500 nanometers (nm) to 1 millimeter (mm), corresponding to frequencies in the range of 430 terahertz (THz) to 400 gigahertz (GHz). The infrared spectrum includes the NIR spectrum, which includes wavelengths in the range of 780 nm to 2500 nm. In some cases, the image adjustment system 400 may include an IR sensor configured to capture both IR and NIR signals. In some cases, separate IR and NIR sensors may be included in the image adjustment system 400. In some instances, the one or more image capture devices 402 may include a camera configured to capture color images and / or monochrome images. Color images may include: red, green, and blue (RGB) images; luminance, chromatic blue, and chromatic red (YCbCr or Y'CbCr) images; and / or any other suitable type of image. In one illustrative example, image adjustment system 400 may include one or more RGB cameras. In some cases, one or more image capture devices 402 may include one or more IR cameras and one or more RGB cameras.

[0092] In some embodiments, one or more image capture devices 402 may include one or more depth sensors. The one or more depth sensors can obtain distance measurements corresponding to objects in the captured scene. In one illustrative example, the depth sensor may take the form of a light source that can project a structured or textured light pattern onto one or more objects in the scene, the light pattern including one or more narrowband lights. Depth information can then be obtained by utilizing the geometric distortion of the projected pattern caused by the surface shape of the objects. In some cases, the one or more depth sensors can generate a depth image including depth values ​​corresponding to primitive locations in one or more images captured by the one or more image capture devices 402. In some cases, the depth sensor may be located at the same approximate location as other sensors of the one or more image capture devices 402. In some cases, the depth sensor may capture depth images simultaneously with images captured by one or more other sensors included in the one or more image capture devices 402. In some implementations, the systems and techniques described herein may also be used when inferring depth information from one or more images.

[0093] In one illustrative example, one or more image capture devices 402 and, in some cases, one or more depth sensors can capture one or more RGB-D images. In some cases, one or more image capture devices 402 can capture other image types that include depth information, such as monochrome depth, NIR depth, etc. For illustrative purposes, the examples described herein discuss image adjustments performed on RGB-D images, but without departing from the scope of this invention, the systems and techniques described herein can also be used for other image types that include depth information.

[0094] FIG5A illustrates an exemplary 2D input image 500 (e.g., an RGB image), and FIG5B illustrates an exemplary depth image 510. In some cases, the 2D input image 500 and the depth image 510 may jointly comprise an RGB-D image. Both the 2D input image 500 and the depth image 510 include representations of a first person 502 located closer to the camera and a second person 504 located at a greater distance from the camera. In the depth image 510, the brightness of each pixel may represent the distance of an object or feature from the camera. In the exemplary depth image 510, closer objects appear brighter (e.g., the first person 502), while farther objects have lower brightness (e.g., the second person 504 or the background 506).

[0095] Referring back to Figure 4, one or more images captured by one or more image capture devices 402 can be provided as input to a feature segmentation machine learning model 404. The feature segmentation machine learning model 404 can be trained to detect spatial information (e.g., features) associated with one or more images. In some cases, the feature segmentation machine learning model 404 can be further trained to provide one or more classifications of objects in one or more images based on the detected features. The feature segmentation machine learning model 404 can then use the classifications to segment the images into distinct parts associated with one or more classifications. For example, the feature segmentation machine learning model 404 can segment one or more images into distinct parts associated with people, buildings, cars, furniture, plants, etc.

[0096] The feature segmentation machine learning model 404 can also be trained to classify objects corresponding to features extracted from one or more images using one or more categories. A training dataset including exemplary images and classification labels can be used to train the feature segmentation machine learning model 404, such as using the techniques described below with reference to Figures 9 and 10. In an illustrative example, during inference (e.g., after the feature segmentation machine learning model 404 has been trained), the feature segmentation machine learning model 404 can use feature vectors to classify one or more input images. Exemplary classifications may include "people," "faces," "buildings," "trees," and any other categories that the feature segmentation machine learning model 404 is trained to classify.

[0097] In an illustrative example, the feature segmentation machine learning model 404 can perform semantic segmentation of an image. In semantic segmentation, the feature segmentation machine learning model 404 can associate each object in the scene with one or more categories (also referred to herein as labels). In some cases, if more than one object has the same label, semantic segmentation will not distinguish between the two objects. In such instances, depth information in one or more captured images (e.g., RGB-D images) can be used to distinguish objects and further segment between them. For example, a portion of an input image containing two people can be classified using a single classifier for "person". Figure 5C illustrates an exemplary image 520 illustrating a first classifier 508 (represented as white) associated with a first person 502 and a second classifier 512 associated with a second person 504. In some instances, the first classifier 508 and / or the second classifier 512 can represent categories such as "person" or "face". In some cases, using semantic segmentation, the feature segmentation machine learning model 404 may not be able to identify the first person 502 and the second person 504 as different objects based on the first classifier 508 and the second classifier 512.

[0098] In another illustrative example, the feature segmentation machine learning model 404 can perform instance segmentation. In instance segmentation, the feature segmentation machine learning model 404 can distinguish multiple instances of an object differently using the same classifier. For example, image 530 shown in FIG5D provides an illustrative example of instance segmentation, where a first instance 514 associated with a second person 504 is shown in gray, and a second instance 516 associated with a person 502 is shown in white. In some cases, using instance segmentation, the feature segmentation machine learning model 404 may be able to distinguish the first person 502 and the second person 504 as different objects based on the first instance 514 and the second instance 516.

[0099] In some cases, the 3D mesh engine 406 can utilize depth information and camera parameters (e.g., focal length, field of view, etc.) to generate a 3D mesh corresponding to an RGB-D image. In some cases, for each primitive in the input image, the 3D mesh engine 406 can generate a corresponding 3D vertex in a 3D model of the scene illustrated in the input image. In an illustrative example, the 3D mesh engine can generate a 3D mesh using depth values ​​and the field of view of the camera capturing the 2D input image (e.g., input image 500 of FIG. 5A). For example, the 3D mesh engine 406 can utilize back-projection techniques to generate a 3D mesh. As discussed above with reference to FIG. 2B and FIG. 2C, exemplary techniques for determining the position of the 3D vertex of the 3D mesh for a primitive at a coordinate may include projecting a ray from the center of the camera through that coordinate and determining the corresponding 3D vertex based on the 3D coordinate on the ray located at a distance from the camera, where the distance value in the depth image corresponds to that coordinate. The 3D mesh engine 406 can perform backprojection techniques on the coordinates of each primitive in the 2D input image. Any suitable backprojection technique may be used without departing from the scope of this case. Figure 5E illustrates an exemplary image 540 of a 3D mesh 542 generated from a 2D input image. As shown, the 3D mesh 542 may include 3D representations of a first person 502 and a second person 504.

[0100] In some cases, the features, objects, and / or classifications determined by the feature segmentation machine learning model 404 can be provided as input to the 3D mesh engine 406. In some cases, based on classification, the 3D mesh engine 406 can associate vertices in the 3D mesh with corresponding features, objects, and / or classifications associated with the corresponding primitive locations in the 2D input image. In some cases, such as when the feature segmentation machine learning model 404 provides instance segmentation, the 3D mesh engine can distinguish different features or objects associated with the same class based on the output from the feature segmentation machine learning model. In some cases, when the feature segmentation machine learning model 404 can provide only semantic segmentation, the 3D mesh engine 406 can distinguish different objects or features in the 3D mesh based on the depth value associated with each object. For example, the 3D mesh engine 406 can determine that two objects sharing the same classifier are different objects based on different parts of the 3D mesh associated with classifiers with different depth values. In an illustrative example, the feature segmentation machine learning model 404 can provide a classification of the "face" applicable to both first person 502 and second person 504. In such instances, the 3D mesh engine 406 can determine the depth value at a location in the 3D mesh associated with the "face" classifier (e.g., the center of the bounding box, the tip of the nose, between the eyes, etc.) and determine that other vertices in the 3D mesh within a threshold distance of the determined depth are part of the same object.

[0101] In some cases, the 3D mesh engine 406 and / or the feature segmentation machine learning model 404 may also apply one or more masks to regions (e.g., primitives) of the 2D input image and corresponding portions (e.g., vertices) of the 3D mesh. Figure 5F shows an image 550 including an exemplary mask, where the primitive associated with person 502 is shown as the masking region 552. In some instances, the 3D mesh engine 406 and / or the feature segmentation machine learning model 404 may select the foreground object closest to the camera to be included in the masking region 552. In the example shown, the first person 502 in the foreground of the 2D input image 500 is selected for masking because any incorrect adjustment of the vertices associated with the first person 502 after image adjustment is completed will create noticeable visual artifacts in the adjusted 2D image. In some cases, other masks and / or multiple masks may be used without departing from the scope of this case.

[0102] In some cases, the 3D mesh engine 406 can determine the unadjusted portion of the 3D mesh (e.g., 3D mesh 542) based on the occlusion region 552 in the 2D image. In some cases, any part of the 3D mesh outside the unadjusted portion can be referred to as an adjusted region. Including vertices in the adjusted portion does not necessarily mean that the included vertices will be adjusted by the 3D mesh engine 406. In some implementations, based on the determination of the adjusted and unadjusted portions in the 3D mesh, the 3D mesh engine 406 can divide the 3D mesh into an adjusted 3D mesh and separate unadjusted 3D meshes.

[0103] Figure 5G illustrates image 560, which includes a 3D representation of a person 502 and may be generated based on an exemplary unadjusted 3D mesh 562 shown in Figure 5F. Figure 5H illustrates an exemplary image 570 with an adjusted 3D mesh 572, which includes a second person 504 and other objects not included in the occlusion area 552, such as a sofa, a wall, etc.

[0104] Returning to Figure 4, in some implementations, the 3D mesh engine 406 may not divide the 3D mesh into separate non-adjustable and adjustable parts of the 3D mesh, but may instead fix the coordinates of the vertices of the 3D mesh within the non-adjustable part or otherwise prevent adjustment of it.

[0105] Once the 3D mesh engine 406 has generated a 3D mesh (or multiple 3D meshes) based on the 2D input image, determined one or more masks, and / or performed further segmentation of objects in the 3D mesh based on depth values, the 3D mesh engine 406 can adjust one or more portions of the 3D mesh. In some cases, the 3D mesh engine 406 can adjust one or more portions of the 3D mesh associated with one or more features detected and / or classified by the feature segmentation machine learning model 404 as being associated with 3D vertices in the 3D mesh. For example, the 3D mesh engine 406 can obtain indications of one or more selected objects (or (one or more) ROIs) to be adjusted in the input image. In some cases, indications of one or more selected objects to be adjusted in the input image (e.g., the 2D input image 500 of FIG. 5A) can be based on user input. In an illustrative example, a user may be able to indicate one or more selected objects using physical contact (e.g., tapping the screen, swiping, etc.), gestures (e.g., detected by a camera or other sensor), using an input device, or by any other means that allows the user to interact with the image adjustment system 400.

[0106] Based on indications of one or more selected objects to be adjusted in the input image, the 3D mesh engine 406 can adjust the 3D mesh (e.g., mesh 542 shown in Figure 5E). In some cases, the 3D mesh engine 406 can adjust vertices in the 3D mesh associated with one or more ROIs or objects. Adjustment of (one or more) ROIs or objects is also referred to as a change of (one or more) ROIs or objects. A change in the pose of (one or more) ROIs or objects can include depth changes, vertical shifts, horizontal shifts, pitch rotations, yaw rotations, roll rotations, and any combination thereof. In an illustrative example, mesh adjustment can include bringing one or more selected objects closer to a camera in the 3D model. Figure 5I shows image 580, which displays vertices in the 3D mesh associated with a second person 504 that have been shifted in the direction of arrow 582. In the example shown, the direction of arrow 582 is selected to maintain a fixed position of a sub-part near the center of the face of the second person 504 (e.g., between the nose and eyes) in the adjusted 2D image.

[0107] In one illustrative example, specific anchor points of an object can be selected to maintain a fixed position in an adjusted 2D image. In such examples, attitude changes may include adjusting the initial positions of one or more vertices of the selected object and the 3D vertices corresponding to the anchor points along a ray passing through the center of the camera. Another illustrative example of attitude adjustment may include reducing the depth values ​​of some or all vertices associated with the selected object. In some cases, reducing the depth values ​​of the vertices of the selected object results in enlarging the size of the selected object in the resulting 2D image. Another illustrative example of attitude adjustment may include (simultaneously) increasing and decreasing the depth values ​​of some or all vertices associated with the selected object. In some cases, increasing and decreasing the depth values ​​of the vertices of the selected object results in pitch rotation, yaw rotation, and / or roll rotation of the selected object in the resulting 2D image.

[0108] In some cases, the image adjustment system 400 can acquire multiple images of a scene or a portion of a scene captured in a 2D input image. For example, the electronic device may include a first image capture device and a second image capture device (e.g., included in image capture device 402), which simultaneously capture at least a common portion of the scene. In some cases, a single image sensor may capture multiple images of the scene to provide, for example, zero shutter lag capture, preview images, etc. In some instances, the image adjustment system may generate multiple 3D meshes based on multiple images of a scene or a portion of the scene. In some cases, the 3D mesh engine 406 may incorporate (e.g., merge, synthesize, patch, etc.) portions from multiple 3D meshes to generate a combined 3D mesh. In one illustrative example, the 3D mesh engine 406 may adjust 3D vertices from a first 3D mesh associated with selected objects in a first 2D image, and also includes 3D vertices from a second 3D mesh associated with a second 2D image. In an illustrative example, 3D vertices associated with the selected object can be horizontally shifted, and portions of the 3D mesh previously occupied by the selected object can be incorporated into portions from a second 2D mesh.

[0109] After the 3D mesh is adjusted by the 3D mesh engine 406, the projection engine 408 can project the 3D mesh onto the adjusted 2D image. The projection engine 408 can utilize any projection technique described above with respect to Figures 2B and 2C. In an illustrative example, the projection engine 408 can utilize perspective projection of equation (1). In some cases, without departing from the scope of this case, the projection engine 408 can use any other projection technique, including but not limited to orthographic or oblique projection, to produce the adjusted 2D image. Figure 5K shows an adjusted 2D image 595 that can be produced by the image adjustment system 400. In some implementations, the projection engine 408 can utilize a projection technique corresponding to the projection technique of a physical camera capturing the 2D image. In some cases, using a projection technique corresponding to the projection technique of a physical camera can reduce distortion in the adjusted 2D image that may be introduced by a mismatch between the physical camera projection technique and the projection technique utilized by the projection engine 408.

[0110] Figure 6 illustrates an exemplary program 600 for performing image adjustments according to an example of this case. In block 602, program 600 may acquire a 2D input image (e.g., from the image capture device 402 shown in Figure 4). In block 604, program 600 may perform segmentation of features or objects included in the input image. For example, program 600 may perform one or more of feature detection, semantic segmentation, instance segmentation, object classification, etc., as described with respect to the feature segmentation machine learning model 404 of Figure 4.

[0111] In block 606, program 600 may include generating a 3D mesh based on depth information and / or color information included in the RGB-D image obtained in block 602. In some cases, for each primitive in the 2D image, the corresponding 3D vertex in the 3D model of the scene may be generated using the back-projection technique described above with respect to the 3D mesh engine 406 of Figures 2B, 2C and 4, or using any other suitable technique for converting 2D images into 3D meshes.

[0112] In block 608, program 600 can perform occlusion of a 3D mesh. In some cases, this occlusion may include creating unadjusted and adjusted portions in the 3D mesh. In some cases, the unadjusted and adjusted portions may be divided into two separate 3D meshes to prevent adjustment of the unadjusted portion. In some cases, a single mesh may include both unadjusted and adjusted portions, and the vertices of the unadjusted portion may be fixed or otherwise prevented from being adjusted. In some cases, the occlusion may include vertices in the 3D mesh associated with one or more ROIs or objects.

[0113] In block 610, program 600 can perform mesh adjustments on the 3D mesh generated in block 608. In some cases, program 600 can obtain input indicating that one or more selected objects (or ROIs) in the 2D image have been selected for adjustment. For example, program 600 can adjust the pose of one or more selected objects by changing the coordinates of the vertices corresponding to the selected objects. As described above with respect to the 3D mesh engine 406 of FIG4, each 2D primitive position in the input image obtained at block 602 can have a corresponding 3D vertex in the 3D mesh generated at block 608. In some cases, program 600 can associate the 3D vertices generated in block 608 with one or more objects (or features). For example, features, objects, and / or categories determined in block 604 and associated with each 2D primitive position of the input image can also be associated with corresponding 3D vertices. Once the 3D vertices corresponding to the selected object are identified, the program 600 can adjust the vertices to provide the desired effect (e.g., magnification, translation, rotation, etc.) in the adjusted 2D image.

[0114] In block 612, program 600 can perform fusion between two or more parts of a 3D mesh. For example, if program 600 divides a 3D mesh into a non-adjusted 3D mesh and an adjusted 3D mesh in block 608, program 600 can combine the non-adjusted 3D mesh and the adjusted 3D mesh into a single combined 3D mesh (e.g., as shown in image 590 of FIG5J). In some cases, where program 600 does not generate separate 3D meshes but instead locks the vertices in the non-adjusted portion of the 3D mesh, program 600 may not need to perform any fusion of the 3D meshes.

[0115] At block 614, program 600 may use the projection techniques described herein with respect to Figures 2B and 2C and the projection engine 408 of Figure 4 to generate an adjusted 2D image. An illustrative example of a projection that may be performed by program 600 is the perspective projection shown in equation (1). However, without departing from the scope of this invention, any projection or rendering technique that can convert a 3D mesh into a 2D image may be used by program 600. In some implementations, program 600 may use a projection technique corresponding to the physical projection of a camera capturing the 2D image. In some cases, using a projection technique corresponding to the projection in a physical camera may reduce distortion in the adjusted 2D image that may be introduced by a mismatch between the physical camera projection technique and the projection technique utilized by program 600 at block 614.

[0116] The procedure 600 shown in Figure 6 provides an illustrative example of the image adjustment procedure according to this case. However, the order of operations of procedure 600 may be changed without departing from the scope of this case. In addition, one or more steps of procedure 600 may be omitted or additional steps may be performed without departing from the scope of this case.

[0117] In some cases, program 600 may also receive input corresponding to the adjustment amount to be applied to one or more objects or features. In some cases, program 600 may dynamically determine the adjustment amount to be applied to one or more selected objects. In one illustrative example, program 600 may determine the approximate size of a foreground object (e.g., using a bounding box, calculating the number of image primitives occupied by the foreground object, etc.) and the approximate size of each of the one or more objects. In such examples, program 600 may determine the adjustment amount based on the relative size of the foreground object and the one or more objects. In one illustrative example, one or more objects may be adjusted to enlarge the size of one or more objects in a 2D image, such that it has a size corresponding to a specific numerical correspondence (e.g., a specific percentage value, ratio, scale, function, relationship, etc.) with the size of the foreground object. In one illustrative example, one or more selected objects (or ROIs) can be adjusted to have a size corresponding to a predetermined ratio between 7 / 10 (e.g., seventy percent) and 9 / 10 (e.g., ninety percent) of the size of the foreground object. In some cases, the numerical correspondence (e.g., percentage values) and / or the range of numerical correspondence (e.g., percentage value range) can be limited based on the distance between the foreground object and the background object in the 3D mesh. In another illustrative example, if the distance (e.g., average distance) between the foreground object and the first object among the one or more selected objects is small (e.g., less than half a meter (m), less than 1 meter, less than 3 meters, etc.), the first object among the one or more selected objects can be adjusted to a ratio of 1 times (e.g., one hundred percent) of the size of the foreground object. In another illustrative example, if the distance (e.g., average distance) between a second object and a foreground object among one or more selected objects is large (e.g., greater than 5 m, greater than 10 m, or any other selected distance), the second object among the one or more selected objects can be adjusted to a ratio of 4 / 5 (e.g., 80 percent) of the size of the foreground object. Without departing from the scope of this case, other techniques may also be used to determine the amount of adjustment for each of the one or more selected objects. For example, the total change in object pose can be limited to an adjustment range. In some cases, the adjustment range for changing object pose can be determined based on the maximum amount of distortion in the 2D image resulting from the pose change. In another example, the maximum adjustment amount for each of the one or more selected objects can be based on the ratio of the apparent size of the object in the adjusted image to that of the original image. For example, the maximum adjustment amount can be limited to allowing the apparent size of each object to increase by 50 percent, 100 percent, 200 percent, or any other value.

[0118] As described above, the image adjustment system 400 and related techniques described herein allow the system to adjust one or more ROIs or one or more objects in an image without adjusting other objects in the image or scene. For example, the image adjustment system 400 can adjust the appearance of an image of two people sitting apart (e.g., as shown in Figures 5A and 5B) to make the two people appear closer together. In some cases, this adjustment may include reducing the depth of the target object, vertical shifting, pitch rotation, roll rotation, yaw rotation, etc. By performing adjustments on the 3D mesh, the image adjustment system can maintain continuity (e.g., the continuity of the 3D mesh between the adjusted target object and the unadjusted remainder of the image).

[0119] Figures 7A to 7E provide additional illustrative examples of image adjustments that can be performed using the systems and techniques of this case. Figure 7A shows a 2D input image 700, which includes a first person 702 in the foreground and a second person 704, a third person 706, a fourth person 708, and a fifth person 710 at different distances behind the first person 702. Figure 7B shows an exemplary instance segmentation 720 performed on the 2D input image 700, in which each of the first person 702, the second person 704, the third person 706, the fourth person 708, and the fifth person 710 in the image is individually identified. In one illustrative example, each person 702 to 710 is classified as a separate instance of the category "person," as indicated by different shades in Figure 7B. In the first adjusted image 740 shown in Figure 7C, each of the second, third, fourth, and fifth persons 704 to 710 has been adjusted using the image adjustment techniques described herein. For example, depth information included in the RGB-D input image is used to generate a 3D mesh corresponding to the 2D input image 700. In the example shown in FIG7C, the positions (e.g., vertices) corresponding to the second, third, fourth, and fifth people 704 to 710 in the 3D mesh are adjusted to be closer to the camera, and the adjusted 3D mesh is reprojected into 2D to generate the first adjusted image 740. Similarly, as shown in FIG7D, the second adjusted image 760 illustrates an instance where only the position of the third person 706 is adjusted to be closer to the camera, and the third adjusted image 780 shown in FIG7E illustrates an instance where only the position of the fifth person is adjusted to be closer to the camera. In some cases, the positions of the second, third, fourth, and fifth people 704 to 710 can be dynamically moved back and forth in the adjusted image based on user input. In all the examples shown in FIG7A and FIG7C to FIG7E, the first person 702 in the foreground remains fixed. In an exemplary 3D mesh generated based on 2D input image 700, the vertices associated with person 702 may have been included in the unadjusted portion and / or unadjusted mesh as described above with respect to the 3D mesh engine 406 of Figure 4. Figures 7A to 7E illustrate the flexibility of the systems and techniques for image adjustment described herein. For example, these systems and techniques allow adjustment of one or more ROIs or objects in a captured image while keeping other one or more ROIs or objects in the image fixed.

[0120] Figure 8 is a flowchart illustrating an example of a program 800 that processes one or more frames. In block 802, program 800 includes acquiring a 2D input image from a camera. For example, the 2D input image may be acquired from an image capture and processing system 300, an image capture device 305A, and / or an image processing device 305B.

[0121] In block 804, program 800 includes obtaining a 3D representation of a 2D input image. For example, program 800 can obtain a 3D representation from the 3D mesh engine 406 shown in FIG4.

[0122] In block 806, program 800 includes determining a first portion of a 3D representation of the 2D input image corresponding to a first target object in the 2D input image. For example, program 800 may obtain user input indicating the target object and / or ROI. In some cases, based on the user input, program 800 may determine vertices in a 3D mesh associated with the target object and / or ROI. In some cases, determining the first portion of the 3D representation of the 2D input image includes at least one of feature detection and object detection. In some instances, determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

[0123] In block 808, program 800 includes adjusting the pose of a first portion of the 3D representation of the 2D input image corresponding to the first target object. In some instances, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object includes decreasing the depth value of one or more first vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object. In some cases, adjusting the pose of the first portion of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation, and roll rotation. For example, the target object may be a person, and modifications to the 3D representation of a person may include changing the orientation of the person's head (e.g., rotation about the pitch axis, yaw axis, and / or roll axis). In some instances, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of one or more second vertices of the first portion of the 3D representation of the 2D input image.

[0124] In block 810, program 800 includes generating a modified 2D output image of the target object based on an adjusted pose of a first portion corresponding to a 3D representation of a 2D input image of the target object to be output on the display. At least a sub-part of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-part of the target object in the 2D input image.

[0125] In some instances, program 800 includes obtaining user input for identifying a target object, wherein the pose of a first portion of a 3D representation of a 2D input image corresponding to the target object is adjusted based on the user input.

[0126] In some instances, program 800 includes obtaining an indication of the adjustment amount of the target object and corresponding the attitude adjustment of the 3D representation of the 2D input image to the indication of the adjustment amount of the target object.

[0127] In some instances, procedure 800 includes obtaining an object mask associated with a target object in a 2D input image. In some cases, procedure 800 includes determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask. In some instances, procedure 800 includes determining a second portion of a 3D representation corresponding to one or more additional objects based on the object mask. In some cases, procedure 800 includes combining the first portion of the 3D representation with an adjusted pose and the second portion of the 3D representation to produce a combined 3D mesh. In some cases, producing a 2D output image includes converting the combined 3D mesh into a 2D output image.

[0128] In some instances, procedure 800 includes obtaining an object mask associated with one or more additional objects in the 2D input image that are different from the target object. In some instances, procedure 800 includes determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask. In some instances, procedure 800 includes determining a second portion of a 3D representation corresponding to at least one of the one or more additional objects based on the object mask. In some instances, procedure 800 includes adjusting the pose of the first portion of the 3D representation and fixing one or more vertices of the second portion of the 3D representation to produce an adjusted 3D mesh. In some instances, adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object includes reducing the depth values ​​of the vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object, such that these vertices have a predetermined numerical correspondence with the fixed one or more vertices of the second portion of the 3D representation. In some instances, producing a 2D output image includes converting the adjusted 3D mesh into a 2D output image.

[0129] In some cases, the pose of the first part of the 3D representation of the 2D input image is adjusted by an amount determined based on user input. In some cases, the pose of the first part of the 3D representation of the 2D input image is adjusted by an amount determined based on a preset configuration. In some instances, the pose of the first part of the 3D representation of the 2D input image is adjusted based on machine learning of user preferences and past behavior. In some implementations, modifications to at least a portion of the modified version of the target object include one or more of vertical displacement, horizontal displacement, pitch rotation, yaw rotation, or roll rotation relative to the corresponding sub-part of the target object in the 2D input image.

[0130] In some instances, program 800 includes adjusting the pose of a first portion of the 3D representation of the 2D input image without adjusting a second portion of the 3D representation of the 2D input image that does not correspond to the target object.

[0131] In some instances, program 800 includes determining a second portion of the 3D representation of the 2D input image corresponding to a second target object in the 2D input image. In some cases, program 800 includes adjusting the pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object. In some cases, program 800 includes further generating a 2D output image based on the adjusted pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object. In some implementations, the adjustment of the pose of the second portion differs from the adjustment of the pose of the first portion.

[0132] In some instances, program 800 includes acquiring a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image. In some other instances, program 800 includes acquiring a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.

[0133] In some instances, the programs described herein (e.g., program 800 and / or other programs described herein) can be executed by a computing device or apparatus. In one instance, one or more programs can be executed by the image adjustment system 400 of FIG. 4. In another instance, one or more programs can be executed by the computing system 1100 shown in FIG. 11. For example, a computing device having the computing system 1100 shown in FIG. 11 may include components of the image adjustment system 400 and may implement the operation of program 600 of FIG. 6, program 800 of FIG. 8, and / or other programs described herein.

[0134] A computing device may include any suitable device, such as a vehicle or a vehicle's computing device (e.g., a driver monitoring system (DMS) for a vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, AR headset, AR glasses, a connected watch or smartwatch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device with the resources to perform the programs described herein (including program 600, program 800, and / or other programs described herein). In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or one or more other components configured to perform the program steps described herein. In some instances, a computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or one or more other components. A network interface can be configured to transmit and / or receive Internet Protocol (IP) data based on data or other types of data.

[0135] Components of a computing device may be implemented using circuitry. For example, a component may include and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0136] Programs 600 and 800 are shown as logic flowcharts, whose actions represent sequences of operations that can be implemented using hardware, computer instructions, or combinations thereof. In the context of computer instructions, actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement these programs.

[0137] Furthermore, the programs 600, 800, and / or other programs described herein can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented via hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors. As mentioned above, the code can be stored, for example, in the form of a computer program containing a plurality of instructions executable by one or more processors on a computer-readable or machine-readable storage medium. The computer-readable or machine-readable storage medium can be non-transitory.

[0138] As mentioned above, various embodiments of this invention may utilize machine learning models or systems. Figure 9 is an illustrative example of a deep learning neural network 900 that can be used to implement the machine learning-based feature segmentation, instance segmentation, depth estimation, and / or classification described above. Input layer 920 includes input data. In one illustrative example, input layer 920 may include data representing primitives of an input image. Neural network 900 includes multiple hidden layers 922a, 922b to 922n. Hidden layers 922a, 922b to 922n include "n" hidden layers, where "n" is an integer greater than or equal to 1. For any given application, the number of hidden layers may include the number of layers required. Neural network 900 also includes an output layer 921, which provides the output generated by the processing performed by hidden layers 922a, 922b to 922n. In one illustrative example, output layer 921 may provide classification for objects in a 2D input image. Classification can include categories that identify the type of activity (e.g., looking up, looking down, closing eyes, yawning, etc.).

[0139] Neural network 900 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information while processing it. In some cases, neural network 900 may include a feedforward network, in which case no network output is fed back to its own feedback connection. In some cases, neural network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes when reading input.

[0140] Information can be exchanged between nodes via node-to-node interconnections between different layers. A node in input layer 920 can activate a set of nodes in the first hidden layer 922a. For example, as shown, each input node in input layer 920 is connected to each node in the first hidden layer 922a. Nodes in the first hidden layer 922a can transform the information of each input node by applying a activation function to the input node information. The information exported from the transformation can then be passed to and can activate nodes in the next hidden layer 922b, which can perform their own specified functions. Exemplary functions include spin, upsampling, data transformation, and / or any other suitable functions. The output of hidden layer 922b can then activate nodes in the next hidden layer, and so on. The output of the last hidden layer 922n can activate one or more nodes in output layer 921, at which output is provided. In some cases, although a node in neural network 900 (e.g., node 926) is shown to have multiple output lines, the node has a single output, and all lines shown as outputs from the node represent the same output value.

[0141] In some cases, each node or the interconnection between nodes can have weights, which are a set of parameters derived from the training of the neural network 900. Once the neural network 900 is trained, it can be referred to as a trained neural network, which can be used to classify one or more activities. For example, an interconnection between nodes can represent a piece of information about the interconnected nodes. The interconnections can have adjustable numerical weights, which can be adjusted (e.g., based on the training dataset), thereby allowing the neural network 900 to adapt to the input and learn as more and more data is processed.

[0142] The neural network 900 is pre-trained to process features from the data in the input layer 920 using different hidden layers 922a, 922b to 922n, so as to provide an output via the output layer 921. In an example of using the neural network 900 to identify features in an image, the neural network 900 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, wherein each training image has a label indicating features in the image (for a feature segmentation machine learning system) or a label indicating the activity category in each image. In an example of using object classification for illustrative purposes, the training images can include images with a digit of 2, in which case the image label can be [0 0 1 0 0 0 0 0 0 0].

[0143] In some cases, the neural network 900 can use a training procedure called backpropagation to adjust the weights of the nodes. As mentioned above, the backpropagation procedure may include forward pass, loss function, backpropagation, and weight update. For each training iteration, forward pass, loss function, backpropagation, and parameter update are performed. For each set of training images, the procedure may be repeated a certain number of times until the neural network 900 is trained well enough that the layer weights are precisely adjusted.

[0144] For instances of identifying objects in an image, forward propagation may include passing training images via neural network 900. Before neural network 900 is trained, the weights are initially randomized. As an illustrative example, the image may include a digital array representing image primitives. Each bit in the array may include a value from 0 to 255 to describe the intensity of the primitive at that location in the array. In one instance, the array may include a 28×28×3 digital array with 28 rows and 28 columns of primitives and 3 color components (such as red, green, and blue, or luminance and two chromaticity components, etc.).

[0145] As mentioned earlier, during the first training iteration of Neural Network 900, the output may include values ​​that do not provide preference for any particular category due to the random selection of weights during initialization. For example, if the output is a vector with probabilities that an object includes different categories, the probability values ​​for each category may be equal or at least very similar (e.g., for ten possible categories, each category may have a probability value of 0.1). Using the initial weights, Neural Network 900 cannot determine low-level features and therefore cannot accurately determine the possible classifications of objects. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), defined as [definition missing]. This loss can be set to a value equal to [definition missing].

[0146] For the first training image, the loss (or error) will be high because the actual value will differ significantly from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training label. The neural network 900 can perform backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight of a specific layer) can be calculated to determine the weights that contribute most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated to change in the opposite direction of the gradient. The weight update can be expressed as , where w represents the weight, wi represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a higher learning rate involves a larger weight update, and a lower value indicates a smaller weight update.

[0147] Neural Network 900 can include any suitable deep network. One example includes a spiral neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between them. The hidden layers of a CNN include a series of spiral, non-linear, pooling (for downsampling), and fully connected layers. Neural Network 900 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0148] Figure 10 is an illustrative example of a spiral neural network (CNN) 1000. The input layer 1020 of the CNN 1000 includes data representing an image or frame. For example, the data may include a digital array representing image primitives, where each number in the array includes a value from 0 to 255 that describes the intensity of the primitive at that location in the array. Using the previous example above, the array may include a 28×28×3 digital array with 28 rows and 28 columns of primitives and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a spiral hidden layer 1022a, an optional nonlinear activation layer, a pooling hidden layer 1022b, and a fully connected hidden layer 1022c to obtain an output at the output layer 1024. Although only one of each hidden layer is shown in Figure 10, those skilled in the art will understand that the CNN 1000 may include multiple spiral hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers. As previously described, the output can indicate a single category of an object, or it can include a probability that best describes the category of an object in the image.

[0149] The first layer of CNN 1000 is a cyclotron hidden layer 1022a. Cyclotron hidden layer 1022a analyzes the image data from input layer 1020. Each node in cyclotron hidden layer 1022a is connected to a node (primitive) region in the input image called a receptive field. Cyclotron hidden layer 1022a can be considered as one or more filters (each filter corresponding to a different activation or feature map), where each cyclotron iteration of the filter is a node or neuron in cyclotron hidden layer 1022a. For example, the input image region covered by each cyclotron iteration filter will be the receptive field of that filter. In an illustrative example, if the input image comprises a 28×28 array, and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in cyclotron hidden layer 1022a. Each connection between a node and its receptive field learns weights, and in some cases, learns an overall bias, allowing each node to learn to analyze its specific local receptive field in the input image. Each node in hidden layer 1022a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For an image frame instance, the filter depth will be 3 (based on the three color components of the input image). The illustrative instance size of the filter array is 5×5×3, corresponding to the size of the receptive field of the node.

[0150] The cyclotron nature of the cyclotron hidden layer 1022a is due to the fact that each node of the cyclotron layer is applied to its corresponding receptive field. For example, the filter of the cyclotron hidden layer 1022a can start from the upper left corner of the input image array and can cyclotron around the input image. As mentioned above, each cyclotron iteration of the filter can be considered as a node or neuron of the cyclotron hidden layer 1022a. In each cyclotron iteration, the value of the filter is multiplied by the corresponding number of original primitive values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input primitive values ​​at the upper left corner of the input image array). The multiplications from each cyclotron iteration can be summed to obtain the sum of the iteration or node. Next, the procedure continues at the next position in the input image according to the receptive field of the next node in the cyclotron hidden layer 1022a. For example, the filter can move a step size (called stride) to the next receptive field. The stride can be set to 1 or other suitable amounts. For example, if the stride is set to 1, the filter will move 1 primitive to the right during each cyclotron iteration. Processing the filter at each unique location in the input volume produces a number of bits representing the filter result at that location, resulting in a sum value being determined for each node of the cyclotron hidden layer 1022a.

[0151] The mapping from the input layer to the cyclotron hidden layer 1022a is called a start-up map (or feature map). The start-up map includes the value of each node representing the filtering result at each location in the input volume. The start-up map may include an array containing various sums of values ​​generated by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each primitive (stride 1) of a 28×28 input image, the start-up map would include a 24×24 array. The cyclotron hidden layer 1022a may include several start-up maps to identify multiple features in the image. The example shown in Figure 10 includes three start-up maps. Using three start-up maps, the cyclotron hidden layer 1022a can detect three different types of features, each of which is detectable throughout the image.

[0152] In some instances, a nonlinear hidden layer can be applied after the cyclotron hidden layer 1022a. Nonlinear layers can be used to introduce nonlinearity into a system that is constantly performing linear operations. An illustrative example of a nonlinear layer is the Rectified Linear Unit (ReLU) layer. A ReLU layer applies the function f(x) = max(0, x) to all values ​​in the input volume, thereby changing all negative initiations to 0. ReLU can thus increase the nonlinear properties of the CNN 1000 without affecting the receptive field of the cyclotron hidden layer 1022a.

[0153] Pooling hidden layer 1022b can be applied after spiral hidden layer 1022a (and when used, after nonlinear hidden layer). Pooling hidden layer 1022b is used to simplify information in the output of spiral hidden layer 1022a. For example, pooling hidden layer 1022b can take each startup graph output from spiral hidden layer 1022a and use a pooling function to produce a compressed startup graph (or feature map). Max pooling is an instance of a function performed by the pooling hidden layer. Pooling hidden layer 1022a can use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filter, L2 norm filter, or other suitable pooling filter) are applied to each startup graph included in spiral hidden layer 1022a. In the example shown in Figure 10, three pooling filters are used for three startup graphs in spiral hidden layer 1022a.

[0154] In some instances, max pooling can be used by applying a max pooling filter (e.g., of 2×2 size) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the startup graph output from the spiral hidden layer 1022a. The output of the max pooling filter includes the maximum number of nodes in each sub-region that the filter spirals through. Taking a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (each node is a value in the startup graph). For example, four values ​​(nodes) in the startup graph will be analyzed by the 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​output as the "maximum" value. If such a max pooling filter is applied to the startup filter from the spiral hidden layer 1022a having a dimension of 24×24 nodes, the output from the pooling hidden layer 1022b will be an array of 12×12 nodes.

[0155] In some instances, an L2 norm pooling filter can also be used. An L2 norm pooling filter involves calculating the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the starting graph (instead of calculating the maximum value as in max pooling), and using the calculated value as the output.

[0156] Intuitively, pooling functions (e.g., max pooling, L2 norm pooling, or other pooling functions) determine whether a given feature is found anywhere in an image region and discard the exact location information. This can be done without affecting the feature detection results, because once a feature is found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of pooling far fewer features, thereby reducing the number of parameters required in subsequent layers of the CNN 1000.

[0157] The final layer of the network is a fully connected layer, which connects each node from the pooling hidden layer 1022b to each output node of the output layer 1024. Using the example above, the input layer comprises 28×28 nodes encoding primitive intensity of the input image, the cyclotron hidden layer 1022a comprises 3×24×24 hidden feature nodes based on applying 5×5 local receptive fields (for filters) to three launch maps, and the pooling hidden layer 1022b comprises a layer of 3×12×12 hidden feature nodes based on applying a max-pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 1024 may include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 1022b is connected to each node of the output layer 1024.

[0158] The fully connected layer 1022c can take the output of the previous pooling hidden layer 1022b (which should represent the start-up graph of high-level features) and determine the features most relevant to a particular class. For example, the fully connected layer 1022c can determine the high-level features most relevant to a particular class and can include the weights (nodes) of the high-level features. The product between the weights of the fully connected layer 1022c and the pooling hidden layer 1022b can be calculated to obtain the probabilities of different classes. For example, if CNN 1000 is being used to predict that an object in an image is a person, high values ​​will appear in the start-up graph representing the high-level features of a person (e.g., two legs appear, the face appears at the top of the object, two eyes appear at the top left and top right of the face, the nose appears in the middle of the face, the mouth appears at the bottom of the face, and / or other features common to people).

[0159] In some instances, the output from output layer 1024 may include an M-dimensional vector (M=10 in the previous instance). M indicates the number of categories from which CNN 1000 must select when classifying objects in an image. Other exemplary outputs may also be provided. Each digit in the M-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents 10 different categories of objects as [0 0 0.05 0.8 0 0 0.15 0 0 0 0], then the vector indicates that the probability that the image is a third category object (e.g., a dog) is 5%, the probability that the image is a fourth category object (e.g., a person) is 80%, and the probability that the image is a sixth category object (e.g., a kangaroo) is 15%. The probability of a category can be considered as a confidence level that the object is part of that category.

[0160] FIG11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, FIG11 illustrates an example of a computing system 1100, which can be, for example, any computing device constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1105. Connection 1105 can be a physical connection using a bus, or a direct connection to processor 1110, such as in a chipset architecture. Connection 1105 can also be a virtual connection, a networking connection, or a logical connection.

[0161] In some embodiments, the computing system 1100 is a distributed system, wherein the functions described herein may be distributed across data centers, multiple data centers, peer-to-peer networks, etc. In some embodiments, one or more described system components represent a number of such components, each performing some or all of the functions described therein. In some embodiments, a component may be a physical or virtual device.

[0162] An exemplary computing system 1100 includes at least one processing unit (CPU or processor) 1110 and a connection 1105 that couples various system components, including system memory 1115 (such as read-only memory (ROM) 1120 and random access memory (RAM) 1125), to the processor 1110. The computing system 1100 may include a high-speed cache memory 1112, which is directly connected to, closely proximate to, or integrated into the processor 1110.

[0163] Processor 1110 may include any general-purpose processor and hardware or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, configured to control processor 1110 and dedicated processors, wherein software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a completely independent computing system, containing multiple cores or processors, buses, memory controllers, cache memory, etc. Multi-core processors may be symmetric or asymmetric.

[0164] To enable user interaction, the computing system 1100 includes an input device 1145, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The computing system 1100 may also include an output device 1135, which can be one or more of a plurality of output mechanisms. In some cases, a multi-mode system allows the user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which typically controls and manages user input and system output. The communication interface can use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including the use of audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, and 802.11. Wi-Fi wireless signal transmission, Wireless Local Area Network (WLAN) signal transmission, Visible Light Communication (VLC), Global Interoperability Microwave Access (WiMAX), Infrared (IR) wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE Cellular Data Network wireless signal transmission, Ad-hoc Network signal transmission, Radio Wave Signal Transmission, Microwave Signal Transmission, Infrared Signal Transmission, Visible Light Signal Transmission, Ultraviolet Light Signal Transmission, Wireless Signal Transmission along the Electromagnetic Spectrum, or combinations thereof. The communication interface 1140 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 1100 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There are no restrictions on operation on any particular hardware configuration, and therefore the fundamental features described here can be readily replaced by improved hardware or firmware configurations developed.

[0165] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable media that can store computer-accessible data, such as magnetic tape cassettes, flash memory cards, solid-state memory devices, digital multifunction discs, magnetic tape cartridges, floppy disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state memory, CD-ROM discs, rewritable compact discs (CDs), digital video discs (DVDs), Blu-ray discs (BDDs), holographic discs, another optical media, secure digital (SD) cards, micro secure digital (microSD) cards, memory Stick® cards, smart card chips, EMV chips, SIM cards, mini / micro / nano / pico SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another storage chip or tape cartridge and / or combinations thereof.

[0166] The storage device 1130 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by the processor 1110. In some embodiments, hardware services that perform a specific function may include software components stored in a computer-readable medium and associated with necessary hardware components, such as the processor 1110, connection 1105, output device 1135, etc., thereby performing the function.

[0167] As used herein, the term "computer-readable media" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (one or more) instructions and / or data. Computer-readable media may include non-transitory media on which data can be stored, and it does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Code and / or machine-executable instructions may be stored on a computer-readable medium, which may represent any combination of programs, functions, subroutines, programs, norms, subnorms, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuitry by transmitting and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be transmitted, forwarded, or transferred using any suitable means, including memory sharing, message passing, symbol passing, network transmission, etc.

[0168] In some embodiments, computer-readable storage devices, media, and memory may include cable or wireless signals containing bit streams, etc. However, when referred to as non-transitory computer-readable storage media, media such as energy, carrier signals, electromagnetic waves, and the signals themselves are explicitly excluded.

[0169] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art to which this invention pertains will understand that embodiments may be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks that contain devices, device components, steps, or conventions in a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, programs, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, programs, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0170] Individual embodiments may be described above as programs or methods illustrated as flowcharts, data flowcharts, structural diagrams, or block diagrams. Although a flowchart may describe operations as a sequential procedure, many operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. When an operation of a program completes, it terminates, but it may have additional steps not included in the diagram. A program may correspond to a method, function, routine, subroutine, subroutine, etc. When a program corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0171] The programs and methods according to the above examples can be implemented using computer-executable instructions that are stored or otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform certain functions or groups of functions. Some computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions, such as assembly languages, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, used information, and / or information created during the methods according to the examples include magnetic disks or optical disks, flash memory, USB devices equipped with non-volatile memory, networked storage devices, etc.

[0172] Devices implementing programs and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented with software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) performing the necessary tasks may be stored in a computer-readable or machine-readable medium. One or more processors may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablets or other small personal computers, personal digital assistants, rack-mount devices, standalone devices, and so on. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of other examples, such functionality may also be implemented on circuit boards of different chips or in different programs executing in a single device.

[0173] Instructions, media for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary components for providing the functionality described in this case.

[0174] In the foregoing description, the various embodiments of this invention have been described with reference to specific examples thereof; however, those skilled in the art will recognize that this invention is not limited thereto. Therefore, although illustrative embodiments of this invention have been described in detail herein, it should be understood that the concepts of the invention may be embodied and used differently in other ways, and the appended claims are intended to be construed as including such variations, unless limited by the prior art. Various features and variations of the above applications may be used alone or in combination. Furthermore, embodiments may be used in any number of environments and applications other than those described herein without departing from the scope of this specification. Therefore, the specification and drawings are to be considered illustrative and not restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, these methods may be performed in a different order than described.

[0175] Those skilled in the art will understand that, without departing from the scope of the specification, the symbols or terms less than (“<”) and greater than (“>”) used herein may be replaced by the symbols less than or equal to (“≦”) and greater than or equal to (“≧”), respectively.

[0176] When a component is described as being "configured" to perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0177] The phrase “coupled to” means any component that is physically connected to another component either directly or indirectly, and / or any component that communicates directly or with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0178] A request language or other language stating "at least one of" and / or "one or more" of a set indicates that one or more members of the set (in any combination) satisfy the request. For example, a request language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another instance, a request language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language stating "at least one of" and / or "one or more" of a set does not limit the set to items listed in the set. For example, a request language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0179] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art to which this invention pertains can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this invention.

[0180] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, mobile phones, or multi-purpose integrated circuit devices, including applications in wireless communication devices, mobile phones, and other devices. Any feature described as a module or component can be collectively implemented in an integrated logic device or implemented separately as a discrete but interoperable logic device. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium containing program code that includes instructions, when executed, to perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include encapsulation material. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (e.g., synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electronically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Alternatively or supplementarily, these technologies can be implemented at least in part by computer-readable communication media that carries or conveys program code in the form of instructions or data structures, and can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0181] The code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any known processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and DSP cores, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0182] The explanatory features of this case include:

[0183] State 1: A method for processing one or more images, comprising: obtaining a two-dimensional (2D) input image from a camera; obtaining a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generating a modified 2D output image having the target object based on the adjusted pose corresponding to the first portion of the 3D representation of the 2D input image to be output on a display.

[0184] State 2: The method of State 1, wherein at least a sub-part of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-part of the target object in the 2D input image.

[0185] Style 3: A method of any one of styles 1 to 2, wherein adjusting the attitude of the first part of the 3D representation of the 2D input image includes one or more of adjusting depth, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

[0186] Style 4: A method of any one of styles 1 to 3, wherein adjusting the pose of the first part of the 3D representation of the corresponding 2D input image comprises: reducing the depth value of the first one or more vertices of the first part of the 3D representation of the 2D input image of the target object.

[0187] State 5: The method of state 4, wherein adjusting the pose of the first part of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of the second one or more vertices of the first part of the 3D representation of the 2D input image.

[0188] State 6: The method of any one of states 1 to 5 also includes: obtaining user input that identifies the target object; wherein the pose of a first part of a 3D representation of a 2D input image corresponding to the target object is adjusted based on the user input.

[0189] State 7: The method of State 6 also includes: obtaining an indication of the adjustment amount of the target object; and the amount corresponding to the attitude adjustment of the 3D representation of the 2D input image and the indication of the adjustment amount of the target object.

[0190] Style 8: The method of any one of styles 6 to 7 also includes: determining the adjustment amount of the target object based on one or more of the adjustment range, apparent size change and maximum distortion amount; and adjusting the pose of the 3D representation of the 2D input image to correspond to the adjustment amount of the target object.

[0191] State 9: The method of any one of states 6 to 8 also includes: obtaining an object mask associated with a target object in a 2D input image; determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of a 3D representation corresponding to one or more additional objects based on the object mask; and combining the first portion of the 3D representation with an adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.

[0192] State 10: The method of State 9, wherein generating a 2D output image includes converting a combined 3D mesh into a 2D output image.

[0193] State 11: The method of any one of states 1 to 10 also includes: obtaining an object mask associated with one or more additional objects in a 2D input image that are different from the target object; determining a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of a 3D representation corresponding to at least one of the one or more additional objects based on the object mask; and adjusting the pose of the first portion of the 3D representation, and fixing one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.

[0194] Style 12: The method of style 11, wherein adjusting the pose of the first part of the 3D representation of the 2D input image of the target object includes: reducing the depth value of the vertices of the first part of the 3D representation of the 2D input image of the target object, such that these vertices have a predetermined numerical correspondence with one or more fixed vertices of the second part of the 3D representation.

[0195] State 13: A method of any one of states 1 to 12, wherein generating a 2D output image includes converting an adjusted 3D mesh into a 2D output image.

[0196] Sample 14: A method of any one of samples 1 to 13, wherein the first part of determining the 3D representation of the 2D input image includes at least one of feature detection and object detection.

[0197] Pattern 15: A method of any one of patterns 1 to 14, wherein determining a first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

[0198] Style 16: The method of any one of styles 1 to 15 also includes: adjusting the pose of a first part of the 3D representation of the 2D input image without adjusting a second part of the 3D representation of the 2D input image that does not correspond to the target object.

[0199] Style 17: A method of any one of styles 1 to 16, wherein the pose of a first portion of a 3D representation of a 2D input image is adjusted by an amount determined based on user input.

[0200] Style 18: A method of any one of styles 1 to 17, wherein the pose of the first part of the 3D representation of the 2D input image is adjusted by an amount determined based on a preset configuration.

[0201] Style 19: A method of any one of styles 1 to 18, wherein the pose of a first part of a 3D representation of a 2D input image is adjusted based on machine learning of user preferences and past behavior.

[0202] State 20: The method of any one of states 1 to 19 also includes: determining a second portion of the 3D representation of the 2D input image corresponding to the second target object in the 2D input image; adjusting the pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object; and further generating a 2D output image based on the adjusted pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object.

[0203] State 21: A method of any of the states 20, wherein the adjustment of the second part of the attitude is different from the adjustment of the first part of the attitude.

[0204] Style 22: A method of any one of styles 1 to 21, wherein the modification of at least a portion of the modified version of the target object includes one or more of vertical displacement, horizontal displacement, pitch rotation, yaw rotation or roll rotation relative to a corresponding sub-part of the target object in the 2D input image.

[0205] Style 23: The method of any one of styles 1 to 22 also includes: obtaining a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image; and obtaining a second 3D representation of the second 2D input image, wherein at least a portion of the 2D output image is generated based on the second 3D representation of the second 2D input image.

[0206] Style 24: An apparatus for processing one or more images, comprising: a memory; and one or more processors coupled to the memory and configured to: acquire a two-dimensional (2D) input image from a camera; acquire a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object; and generate a modified 2D output image having the target object based on the adjusted pose corresponding to the first portion of the 3D representation of the 2D input image to be output on a display.

[0207] Style 25: The apparatus of style 24, wherein at least a sub-part of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-part of the target object in the 2D input image.

[0208] Style 26: A device for adjusting the attitude of a first portion of the 3D representation of a 2D input image, wherein adjusting the attitude of the first portion of the 3D representation of the 2D input image includes one or more of depth adjustment, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

[0209] State 27: A device for any of states 24 to 26, wherein one or more processors are configured to: reduce the depth value of the first one or more vertices of the first portion of the 3D representation of the 2D input image of the target object.

[0210] Style 28: The apparatus of style 27, wherein adjusting the pose of the first part of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of the second one or more vertices of the first part of the 3D representation of the 2D input image.

[0211] State 29: A device for any of states 24 to 28, wherein one or more processors are configured to: obtain user input that identifies the target object; and adjust the pose of a first portion of a 3D representation of a 2D input image corresponding to the target object based on the user input.

[0212] State 30: The apparatus of state 29, wherein one or more processors are configured to: obtain an indication of the adjustment amount of the target object; and to adjust the posture of the 3D representation of the 2D input image to correspond to the indication of the adjustment amount of the target object.

[0213] State 31: A device for any of states 29 to 30, wherein one or more processors are configured to: determine the adjustment amount of the target object based on one or more of the adjustment range, apparent size change and maximum distortion amount; and adjust the pose of the 3D representation of the 2D input image to correspond to the adjustment amount of the target object.

[0214] State 32: An apparatus for any of states 29 to 31, wherein one or more processors are configured to: obtain an object mask associated with a target object in a 2D input image; determine a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determine a second portion of a 3D representation corresponding to one or more additional objects based on the object mask; and combine the first portion of the 3D representation with an adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.

[0215] State 33: A device for any of states 24 to 32, wherein generating a 2D output image includes converting a combined 3D mesh into a 2D output image.

[0216] State 34: An apparatus for any of states 24 to 33, wherein one or more processors are configured to: obtain an object mask associated with one or more additional objects in a 2D input image that are different from the target object; determine a first portion of a 3D representation of the 2D input image corresponding to the target object based on the object mask; determine a second portion of a 3D representation corresponding to at least one of the one or more additional objects based on the object mask; and adjust the pose of the first portion of the 3D representation, and fix one or more vertices of the second portion of the 3D representation to generate an adjusted 3D mesh.

[0217] Style 35: A device for style 34, wherein, in order to adjust the pose of a first portion of a 3D representation of a 2D input image corresponding to a target object, the one or more processors are configured to: reduce the depth value of the vertices of the first portion of the 3D representation of the 2D input image corresponding to the 2D input image, such that the vertices have a predetermined numerical correspondence with a fixed one or more vertices of a second portion of the 3D representation.

[0218] State 36: A device for any of states 34 to 35, wherein generating a 2D output image includes converting an adjusted 3D mesh into a 2D output image.

[0219] Specimen 37: A device for any of Specimens 24 to 36, wherein the first part determining the 3D representation of the 2D input image includes at least one of feature detection and object detection.

[0220] Style 38: A device for any of styles 24 to 37, wherein a first portion of the 3D representation of the 2D input image is determined based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

[0221] Style 39: A device for any of styles 24 to 38, wherein one or more processors are configured to: adjust the pose of a first portion of a 3D representation of a 2D input image without adjusting a second portion of a 3D representation of a 2D input image that does not correspond to a target object.

[0222] Style 40: A device for any of styles 24 to 39, wherein the pose of a first portion of a 3D representation of a 2D input image is adjusted by an amount determined based on user input.

[0223] Style 41: A device for any of styles 24 to 40, wherein the pose of the first part of the 3D representation of the 2D input image is adjusted by an amount determined based on a preset configuration.

[0224] Pattern 42: A device for any of patterns 24 to 41, wherein the pose of a first portion of a 3D representation of a 2D input image is adjusted based on machine learning of user preferences and past behavior.

[0225] State 43: An apparatus for any of states 24 to 42, wherein one or more processors are configured to: determine a second portion of a 3D representation of a 2D input image corresponding to a second target object in a 2D input image; adjust the pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object; and further generate a 2D output image based on the adjusted pose of the second portion of the 3D representation of the 2D input image corresponding to the second target object.

[0226] State 44: The apparatus of state 43, wherein the adjustment of the second part of the attitude is different from the adjustment of the first part of the attitude.

[0227] State 45: A device for any of states 24 to 44, wherein the modification of at least a portion of the modified version of the target object includes one or more of vertical displacement, horizontal displacement, pitch rotation, yaw rotation or roll rotation relative to a corresponding sub-part of the target object in the 2D input image.

[0228] State 46: An apparatus of any one of states 24 to 45, wherein one or more processors are configured to: acquire a second 2D input image from a camera, wherein the second 2D input image includes at least a portion of a scene included in the 2D input image; and acquire a second 3D representation of the second 2D input image, wherein at least a portion of a 2D output image is generated based on the second 3D representation of the second 2D input image.

[0229] Session 47: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations of modes 1 to 46.

[0230] State 48: An apparatus comprising components for performing any of the operations of states 1 to 46. [Simplified Explanation of the Diagram]

[0034] The illustrative embodiments of this case are described in detail below with reference to the accompanying drawings:

[0035] Figures 1A to 1C are illustrations of different fields of view in exemplary images based on some examples of this case;

[0036] Figures 2A to 2C illustrate exemplary image adjustment techniques according to some examples;

[0037] Figure 3 is a block diagram illustrating the architecture of an image capture and processing device according to some examples of this case;

[0038] Figure 4 is a block diagram illustrating an exemplary image adjustment system according to some examples of this case;

[0039] Figures 5A to 5K are images illustrating exemplary stages of image adjustment techniques according to some examples of this case;

[0040] Figure 6 is a flowchart illustrating an exemplary image adjustment procedure according to some examples of this case;

[0041] Figures 7A to 7E are images illustrating exemplary image adjustments according to some examples of this case;

[0042] Figure 8 is a flowchart illustrating an example of a program for processing one or more images according to some examples;

[0043] Figure 9 is a block diagram showing an example of a deep learning network based on some instances;

[0044] Figure 10 is a block diagram illustrating an example of a spiral neural network based on some examples;

[0045] Figure 11 is a diagram illustrating an example of a computing system used to implement some of the states described herein. [Biomaterial Storage]

[0232] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A method for processing one or more images, comprising the steps of: acquiring a two-dimensional (2D) input image from a camera; acquiring a three-dimensional (3D) representation of the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; acquiring an object mask associated with one or more additional objects in the 2D input image that are different from the target object, wherein the step of determining the first portion of the 3D representation is based on the object mask; determining a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask; adjusting a pose of the first portion of the 3D representation to produce an adjusted 3D representation; and generating a modified version of a 2D output image having the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object to be output on a display.

2. The method of claim 1, wherein at least one sub-part of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-part of the target object in the 2D input image.

3. The method according to claim 1, wherein adjusting the attitude of the first part of the 3D representation of the 2D input image includes adjusting one or more of depth, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

4. The method according to claim 1, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object comprises the following steps: reducing the depth value of the first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object.

5. The method according to claim 4, wherein adjusting the pose of the first part of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of the second one or more vertices of the first part of the 3D representation of the 2D input image.

6. The method according to claim 1 also includes the following steps: obtaining a user input that identifies the target object, wherein the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

7. The method according to claim 6 also includes the following steps: obtaining an indication of the adjustment amount of the target object; and adjusting the pose of the 3D representation of the 2D input image to correspond to the indication of the adjustment amount of the target object.

8. The method according to claim 6 also includes the following steps: determining an adjustment amount for the target object based on one or more of an adjustment range, an apparent size change, and a maximum distortion amount; and adjusting the pose of the 3D representation of the 2D input image to an amount corresponding to the adjustment amount of the target object.

9. The method according to claim 1 also includes the following steps: obtaining an object mask associated with the target object in the 2D input image; determining a first portion of the 3D representation of the 2D input image corresponding to the target object based on the object mask; determining a second portion of the 3D representation corresponding to one or more additional objects based on the object mask; and combining the first portion of the 3D representation having the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.

10. The method of claim 9, wherein generating the 2D output image includes converting the combined 3D mesh into the 2D output image.

11. The method of claim 1, wherein adjusting the pose of the first part of the 3D representation of the 2D input image corresponding to the target object comprises the following steps: reducing the depth value of a vertex of the first part of the 3D representation of the 2D input image corresponding to the target object, such that the vertex has a predetermined numerical correspondence with a fixed one or more vertices of the second part of the 3D representation.

12. The method according to claim 1, wherein generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.

13. The method of claim 1, wherein the first portion of determining the 3D representation of the 2D input image includes at least one of feature detection and object detection.

14. The method of claim 1, wherein determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

15. An apparatus for processing one or more images, comprising: a memory; and one or more processors coupled to the memory and configured to: acquire a two-dimensional (2D) input image from a camera; acquire a three-dimensional (3D) representation of the 2D input image; determine a first portion of the 3D representation of the 2D input image corresponding to a target object in the 2D input image; acquire an object mask associated with one or more additional objects in the 2D input image that are different from the target object, wherein the first portion of the 3D representation is determined based on the object mask; determine a second portion of the 3D representation corresponding to at least one of the one or more additional objects based on the object mask; adjust a pose of the first portion of the 3D representation to produce an adjusted 3D representation; and generate a modified version of a 2D output image having the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image corresponding to the target object to be output on a display.

16. The apparatus according to claim 15, wherein at least one sub-part of the modified version of the target object has an enlarged size in the 2D output image relative to a corresponding sub-part of the target object in the 2D input image.

17. The apparatus according to claim 15, wherein adjusting the attitude of the first portion of the 3D representation of the 2D input image includes one or more of adjusting depth, vertical shift, horizontal shift, pitch rotation, yaw rotation and roll rotation.

18. The apparatus according to claim 15, wherein the one or more processors are configured to: reduce the depth value of the first one or more vertices of the first portion of the 3D representation of the 2D input image corresponding to the target object.

19. The apparatus according to claim 18, wherein adjusting the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object also includes increasing the depth value of a second one or more vertices of the first portion of the 3D representation of the 2D input image.

20. The apparatus according to claim 15, wherein the one or more processors are configured to receive a user input that identifies the target object, wherein the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object is adjusted based on the user input.

21. The apparatus according to claim 20, wherein the one or more processors are configured to: obtain an indication of the adjustment amount of the target object; and adjust the posture of the 3D representation of the 2D input image to correspond to the indication of the adjustment amount of the target object.

22. The apparatus according to claim 20, wherein the one or more processors are configured to: determine an adjustment amount of the target object based on one or more of an adjustment range, an apparent size change, and a maximum distortion amount; and adjust the pose of the 3D representation of the 2D input image by an amount corresponding to the adjustment amount of the target object.

23. The apparatus of claim 15, wherein the one or more processors are configured to: obtain an object mask associated with the target object in the 2D input image; determine, based on the object mask, a first portion of the 3D representation of the 2D input image corresponding to the target object; determine, based on the object mask, a second portion of the 3D representation corresponding to one or more additional objects; and combine the first portion of the 3D representation having the adjusted pose and the second portion of the 3D representation to generate a combined 3D mesh.

24. The apparatus according to claim 23, generating the 2D output image includes converting the combined 3D mesh into the 2D output image.

25. The apparatus according to claim 15, wherein, in order to adjust the pose of the first portion of the 3D representation of the 2D input image corresponding to the target object, the one or more processors are configured to: reduce the depth value of a vertex of the first portion of the 3D representation of the 2D input image corresponding to the 2D input image, such that the vertex has a predetermined numerical correspondence with a fixed one or more vertices of the second portion of the 3D representation.

26. The apparatus according to claim 15, wherein generating the 2D output image includes converting the adjusted 3D mesh into the 2D output image.

27. The apparatus according to claim 15, wherein the first portion determining the 3D representation of the 2D input image includes at least one of feature detection and object detection.

28. The apparatus according to claim 15, wherein determining the first portion of the 3D representation of the 2D input image is based on depth information corresponding to the first portion of the 3D representation of the 2D input image.

29. A method of processing one or more images, comprising the steps of: acquiring a two-dimensional (2D) input image from an image capture device; acquiring a three-dimensional (3D) representation of the 2D input image, the 3D representation of the 2D input image including a first portion and a second portion, the first portion corresponding to a target object in the 2D input image, the second portion corresponding to at least one or more additional objects in the 2D input image; acquiring an object mask associated with the one or more additional objects, and determining the first portion and the second portion of the 3D representation based on the object mask; adjusting a pose of the first portion of the 3D representation to generate an adjusted 3D representation; and generating a modified version of a 2D output image having the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image to be output on a display.

30. A method of processing one or more images, comprising the steps of: generating a three-dimensional (3D) representation of a 2D input image, the 3D representation of the 2D input image including a first portion and a second portion, the first portion corresponding to a target object in the 2D input image, the second portion corresponding to at least one or more additional objects in the 2D input image; determining the first portion and the second portion of the 3D representation based on an object mask associated with the one or more additional objects; adjusting a pose of the first portion of the 3D representation to generate an adjusted 3D representation; and generating a modified version of a 2D output image having the target object based on the adjusted pose of the first portion of the 3D representation of the 2D input image to be output on a display.

Images