Smart identification of replacement areas for blending and replacing people in group portraits

Abstract

Embodiments of the present disclosure relate to intelligent identification of replacement regions for blending and replacement of people in group portraits. The present disclosure relates to intelligently blending and matching faces and / or people to generate enhanced images that reduce or minimize artifacts and other defects. For example, the disclosed system can selectively apply different alignment models to determine the relative alignment between a reference image and a target image having an improved instance of a person. After aligning the digital images, the disclosed system can intelligently identify a replacement region based on a boundary that includes a target instance and a reference instance of a person without intersecting other objects or people in the image. Using the size and shape of the replacement region around the target instance and the reference instance, the system replaces the instance of the person in the reference image with the target instance. Alignment of the images and intelligent selection of replacement regions minimize inconsistencies and / or artifacts in the final image.

Description

Background Art

[0001] In recent years, there has been a significant increase in the technology used for digital photography and its use. Indeed, the advent of smartphones with integrated high-quality cameras has led to a large percentage of people almost always carrying a digital camera with them. This has led to a large increase in the number of digital photos taken. Not only has the number of digital photos increased recently, the ease and speed of sharing digital photos has also increased. For example, smartphones with social networking applications allow users to share digital photos almost immediately after capture. Users now seek to capture and share the "perfect picture" as quickly as possible.

[0002] A recent development in helping photographers capture high-quality photos is burst mode or continuous shooting. A digital camera with burst mode captures multiple frames or images per second. Indeed, a digital camera with burst mode can capture dozens of images per second. Burst mode photography helps photographers capture high-quality images at a variety of different types of moments. Using burst mode to capture group portraits allows the photographer to capture many images and select instances where desired artifacts are minimized. For example, group portraits often have multiple artifacts, such as people blinking, looking in the wrong direction, or motion blur. Burst mode allows the photographer to increase the chances of capturing images with the fewest artifacts (i.e., images with all the people in the group looking forward with their eyes open).

[0003] While using burst mode photography for group portraits offers a number of advantages, burst mode photography also has a number of disadvantages. In particular, the chances of capturing a perfect photo of each person's face are generally low. For example, one person may look their best in one photo, while another person may look their best in another photo.

[0004] In addition, combing through multiple group photos using conventional burst mode photography systems is often inefficient. Typically, conventional burst mode photography systems require the photographer to examine and compare each person's face in potentially dozens of images to identify a desired or high-quality group portrait. Because it is difficult to compare such similar images, reviewing burst mode images can be cumbersome and time consuming. Furthermore, because the faces within the images may appear relatively small, the use of handheld devices with relatively small displays exacerbates these problems. The time and effort associated with selecting a high-quality group portrait may prevent users from adopting burst mode when trying to quickly capture and share images.

[0005] Conventional photo management systems provide the ability to mix and match portions of burst images. However, the mix and match functionality provided by conventional systems is generally inflexible. For example, in order to produce acceptable results, conventional systems require that burst photos be taken with a camera mounted on a tripod. Thus, the photographer is generally not provided with the flexibility to mix and match portions of images captured using a handheld camera or mobile phone. If a tripod is not used to take the photos, existing systems often produce images that contain inconsistencies or artifacts.

[0006] Additionally, even when a set of photos is taken using a tripod, mixing and matching images using conventional photo management systems often requires intensive manual editing and despite this often results in inconsistencies and / or visual artifacts. For example, replacing a face or face bounding box with another face or face bounding box often results in discontinuities and artifacts in the final result. For example, if a person moves between two photos in a sequence of photos, the face bounding box may include different parts of the background structure. As a result, the enhanced images produced by conventional photo management systems will often include inconsistencies.

[0007] These and other issues exist regarding capturing and editing group portraits. Summary of the invention

[0008] One or more embodiments described below provide benefits and / or solve one or more of the aforementioned or other problems in the art using systems, methods, and computer-readable media that intelligently mix and match faces and / or people between burst modes or real-time photos to generate enhanced images that reduce or minimize artifacts and other defects. For example, a photographer may capture a series of images, including an imperfect reference image. In particular, the person in the reference image may be blinking or looking away. In one or more embodiments, the disclosed system can selectively apply different alignment models to determine the relative alignment between the reference image and the target image with an improved instance of a person. After aligning the digital image, the disclosed system can intelligently identify a replacement area based on a boundary that includes a target instance and a reference instance of a person without intersecting other objects or people in the image. Using the size and shape of the replacement area around the target instance and the reference instance, the system, method, and computer-readable medium replace the instance of a person in the reference image with the target instance. The alignment of the image and the intelligent selection of the replacement area minimize inconsistencies and / or artifacts in the final image without the use of a tripod or intensive manual editing.

[0009] Additional features and advantages of one or more embodiments of the present disclosure will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of such example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1A-Figure 1C shows a series of example images including a reference image, a target image, and an enhanced image according to one or more embodiments;

[0012] Figure 2 A graphical user interface of a handheld device displaying multiple images according to one or more embodiments is shown;

[0013] Figure 3A-3B An example neural network for generating an aesthetic score in accordance with one or more embodiments is shown;

[0014] Figure 4 A flow chart illustrating a sequence for selecting and applying an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model to align a target image with a reference image in accordance with one or more embodiments;

[0015] Figure 5A An overview of a gyroscope alignment model applying pixel adjustment according to one or more embodiments is shown;

[0016] Figure 5B An overview of applying a feature-based alignment model according to one or more embodiments is shown;

[0017] Figure 6A-6B shows a schematic representation of a process of identifying replacement regions in a reference image and a target image in accordance with one or more embodiments;

[0018] Figure 7A-7E A series of example images are shown illustrating a process of automatically replacing a reference instance of a person with a target instance of a person in accordance with one or more embodiments;

[0019] Figure 8A-8E A graphical user interface is shown illustrating a process of replacing a reference instance of a person with a target instance of a person using manual feedback in accordance with one or more embodiments;

[0020] Fig. 9 A schematic diagram of an image region replacement system according to one or more embodiments is shown;

[0021] Fig.10 A schematic diagram showing a network environment in which an image region replacement system may be implemented according to one or more embodiments;

[0022] Fig.11 A flow chart showing a series of actions for switching replacement regions in a reference image and a target image according to one or more embodiments; and

[0023] Fig.12 A block diagram of an exemplary computing device is shown in accordance with one or more embodiments. DETAILED DESCRIPTION

[0024] One or more embodiments of the present disclosure include an image region replacement system that can generate seamless blended and matched images by intelligently identifying replacement regions around reference instances of people and target instances of people in multiple images. In particular, in one or more embodiments, the image region replacement system aligns multiple images, identifies one or more faces in a reference image that have a better appearance in other images, and replaces the intelligently determined replacement region around each identified face with a similar region in the image that has a better appearance of these faces. The selection of the replacement region is intended to minimize artifacts in the final image by ensuring that the entire face in the reference image is replaced with the entire face in the target image, and ensuring that the replacement region boundary passes through uniform and less structured pixels of the image.

[0025] More specifically, the image region replacement system aligns a reference image including a reference instance of a person with a target image including a more aesthetically pleasing instance of a person. Once the target image and the reference image are aligned, the image region replacement system intelligently determines a replacement region including the reference instance and the target instance of the person. For example, the image region replacement system selects a replacement region whose boundaries in the target image and the reference image pass through consistent pixels. Therefore, the boundaries of the replacement region should avoid structures in the reference image and the target image. In this way, the image region replacement system allows for artifact-free replacement of instances of people in photographs without the use of a tripod or intensive manual editing.

[0026] As described above, the image region replacement system can efficiently and accurately align a reference image and a target image. For example, in one or more embodiments, the image region replacement system determines a feature point deficiency metric and selects an alignment model based on the feature point deficiency metric. In particular, the image region replacement system can dynamically select an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model based on the feature point deficiency metric. The image region replacement system can align images taken with handheld cameras and mobile devices. Therefore, the image replacement region can align images for region replacement without using a tripod or other camera stabilization method. Once aligned, the image region replacement system can select a replacement region as described above.

[0027] As described, the image region replacement system can select a target instance of a person to replace a reference instance of the person as part of generating an enhanced image. In one or more embodiments, a user can manually select the target instance. In alternative embodiments, the image region replacement system can automatically identify the target instance without user input. For example, the image region replacement system can generate multiple aesthetic ratings for instances of a person in multiple images. The image region replacement system can identify the target instance of the person with the highest aesthetic rating.

[0028] As described above, the image region replacement system can generate multiple aesthetic grades for multiple instances of a person. For example, in one or more embodiments, the image region replacement system automatically generates aesthetic grades for multiple instances of a person across multiple images based on a weighted sum of different attributes. More specifically, the image region replacement system can extract attributes such as open eyes, frontal facial posture, facial expression, facial sharpness, etc. from each instance of a person. The image region replacement system can also weight each of these attributes. Based on the extracted attributes and their associated weights, the image region replacement system generates an aesthetic grade for each instance of a person.

[0029] In any case, once a target instance is selected and the target image and the reference image are aligned, the image region replacement system intelligently determines replacement regions in the target image and the reference image whose boundaries pass through consistent pixels. In particular, the image replacement region determines a reasonable boundary region for the replacement region boundary around the target instance of the person and the reference instance of the person, for example by receiving manual user input or automatically selecting a region around the instance of the person. Thereafter, the image region replacement system uses a graph cut algorithm that is constrained to find a boundary with maximum pixel consistency across the reference image and the target image within a reasonable boundary region. The image region replacement system determines a neighborhood around the reference instance of the person and the target instance of the person, and uses union and subtraction operations to identify the boundary of the replacement region. Therefore, the boundary of the replacement region attempts to avoid structures in the reference image and the target image.

[0030] In one or more embodiments, the image region replacement system generates an enhanced image by replacing a reference region corresponding to the replacement region with a target region corresponding to the replacement region. In particular, the reference region includes a boundary of the replacement region around a reference instance of a person. Similarly, the target region includes a boundary of the replacement region around a target instance of a person. The image region replacement system generates a new image by replacing the reference region with the target region, in which the reference image includes the target instance of a person.

[0031] As previously described, the image region replacement system has many advantages and benefits over conventional systems and methods. As an example, the image region replacement system can increase the speed at which a user can capture and share a desired group portrait. In particular, the image region replacement system can use a burst mode or real-time image capture technology to capture multiple images, each image including one or more people. The image region replacement system can then score each instance of a person across multiple images to identify a target instance of the person with the best appearance. The image region replacement system can replace the instance of the person in the reference image with the target instance of the person with the best appearance. Therefore, the image region replacement system can generate an image in which each person has the best appearance without the photographer having to attempt to compare and contrast potentially dozens of images.

[0032] In addition, the image region replacement system can enable a user to mix and match images captured using a handheld camera or mobile phone. In particular, the image region replacement system automatically aligns each image relative to a reference image in a collection. In particular, the image region replacement system can align multiple images using pixel-adjusted gyroscopic alignment or feature-based alignment. Once the images have been aligned, the image region replacement system can identify faces in the reference image and intelligently replace the identified faces with faces from multiple images. Thus, the image region replacement system can provide the user with the flexibility to mix and match portions of the captured images without the use of a tripod.

[0033] In addition, the image region replacement system reduces inconsistencies and visual artifacts that are typically a portion of an image produced by a conventional photo management system. In particular, the image region replacement system intelligently determines a replacement region based on the content of the image, rather than replacing a face or facial bounding box with another face or facial bounding box. For example, the image region replacement system utilizes a graph cut algorithm that is tasked with finding a region boundary that intersects the most consistent pixels around a reference instance of a person and a target instance of the person. In addition, the graph cut algorithm can be constrained to avoid structures in the reference image and the target image, while also selecting small boundaries around the reference instance of the person and the target instance of the person. Therefore, the image region replacement system can reduce the number of visual inconsistencies that occur when mixing and matching portions of the image.

[0034] As shown in the previous discussion, the present disclosure utilizes various terms to describe the features and advantages of the image region replacement system. Additional details about the meaning of these terms are now provided. For example, as used herein, the term "image" or "frame" refers to any digital item that can produce a visual representation. For example, the term "image" includes frames of digital images and digital videos. As used herein, the term "digital image" refers to any digital symbol, picture, icon, or illustration. For example, the term "digital image" includes digital files with the following file extensions: JPG, TIFF, BMP, PNG, RAW, or PDF.

[0035] As used herein, the term "replacement region" refers to the region around a reference instance of a person and a target instance of a person. In particular, the replacement region is defined by a replacement region boundary that avoids structures (e.g., buildings, objects, other people, etc.) in the reference image and the target image. The image region replacement system uses the region defined by the replacement region to identify a reference region on the reference image and a target region on the target image. The selection of the replacement region is intended to minimize artifacts in the enhanced image by ensuring that the entire instance of the reference person is replaced with the entire instance of the target person without cutting through structures in the image.

[0036] As used herein, the term "instance of a person" refers to the appearance of a person in an image. In particular, where a photographer captures multiple images (e.g., burst mode, video, or Live Photo), an "instance of a person" refers to the appearance of a person in one of the multiple images. Other images in the multiple burst mode images may also contain instances of the same person. An instance of a person may include only the face of the person, or may include both the face and body of the person, as captured in a digital image.

[0037] As used herein, a "burst mode image" refers to an image captured using a burst mode setting. In other words, the term "burst mode image" refers to one image in a group of images captured in rapid succession (e.g., at four frames per second or more). In particular, the burst mode setting of a handheld mobile device or camera can be used to capture a group portrait including two or more people. As used herein, the term "Live Photo" refers to a short video (e.g., up to 4 or 5 seconds) from which images (e.g., frames) can be selected.

[0038] As used herein, the term "attribute" refers to a characteristic of an object. In particular, the term "attribute" refers to an attribute of an instance of a person within a digital image. For example, an image region replacement system may extract attributes such as eye opening, face frontal pose, facial expression, face sharpness, etc. from an instance of a person. Facial landmark detection may be used to extract attributes from the appearance of eye landmarks, face contours, mouth landmarks, etc. Based on the extracted attributes, the image region replacement system may generate an aesthetic rating for an instance of a person across multiple images.

[0039] As used herein, the term "aesthetic grade" refers to a numerical grade that indicates the quality of appearance. For example, the aesthetic grade may include a value between 0 and 1, or between 0 and 100%, indicating the quality of appearance of an instance of an object or person in an image. Additionally, the aesthetic grade may include a weighted sum of attributes. For example, each attribute may be associated with a different weight. Certain attributes, such as facial sharpness, may have a greater weight than attributes, such as eye openness.

[0040] As used herein, the term "reference image" refers to an image from the plurality of images. In particular, the image region replacement system will identify a reference image from the plurality of images. The image region replacement system uses the reference image as a reference when removing and replacing instances of people. In other words, the image region replacement system replaces pixels in the reference image with pixels in the target image. The reference image can be the image with the least blur, the image with the most desired faces, or simply the first photo in a set of photos.

[0041] As used herein, the term "target image" refers to an image from the plurality of images. In particular, the image region replacement system identifies a target image from the plurality of images. The target image includes a target instance of a person, typically an instance of a person with a highest aesthetic rating. The image region replacement system identifies a replacement region in the target image to move to the reference image.

[0042] As used herein, the term "reasonable region" or "reasonable boundary region" refers to the region around a reference instance of a person and a target instance of a person that includes a replacement region. In particular, a reasonable region refers to an allowed region through which an image region replacement system can find the boundary of a replacement region. In particular, a reasonable region includes the union of the neighborhood around the reference instance and the target instance (i.e., an extension of the instance outline) minus other people and objects in the image. Reasonable regions will be discussed in further detail below.

[0043] As used herein, the term "machine learning" refers to the process of constructing and implementing algorithms that can learn from data and make predictions about it. In general, machine learning can operate by building models based on example inputs (e.g., training) (such as a training font set) to make data-driven predictions or decisions. In some example embodiments, machine learning is used for data mining and statistical pattern recognition, such as collaborative feature learning or learning features from a training image set, which can be supervised or unsupervised.

[0044] As used herein, the term "neural network" refers to a machine learning model that can be adjusted (e.g., trained) based on inputs to approximate unknown functions. In particular, the term "neural network" may include a model of interconnected neurons that communicate and learn to approximate complex functions and generate outputs based on multiple inputs provided to the model. For example, the term "neural network" includes one or more machine learning algorithms. In particular, the term "neural network" may include a deep convolutional neural network (i.e., "CNN"). In addition, a neural network is an algorithm (or a set of algorithms) that implements a deep learning technique that uses a set of algorithms to model high-level abstractions in data.

[0045] As used herein, the term "loss function" or "loss model" refers to a function indicating the error loss between feature vectors and / or probability vectors in a multidimensional vector space. A machine learning algorithm (e.g., a neural network) can be repeatedly trained to minimize and / or maximize classification error loss (e.g., the accuracy or inaccuracy of the classification for ground truth). In some embodiments, the image region replacement system employs a loss function at different training stages. An example of a loss function includes a classification loss model that uses a SoftMax classifier with a cross entropy loss, as described below.

[0046] As used in the present disclosure, the term "feature-based alignment model" refers to an image alignment model that aligns an input image with a reference image based on feature points within the input image and the reference image. In some embodiments, the feature-based alignment model includes an algorithm that identifies and matches feature points between the input image and the reference image to align the input image with the reference image. For example, in some embodiments, the image region replacement system applies the feature-based alignment model by executing a random sample consensus ("RANSAC") algorithm, a least median of squares ("LMedS") regression algorithm, or other suitable robust model estimation algorithm to generate feature alignment parameters, and aligns the input image with the reference image based on the feature alignment parameters. The present disclosure is referred to below in reference to Figure 4-5B The feature-based alignment model is further described.

[0047] The term "feature alignment parameters" refers to one or more attributes that associate feature points in one digital image with corresponding feature points in another digital image. For example, in some embodiments, the feature alignment parameters include attributes that, when applied to the first digital image, align the first digital image with the second digital image based on matching feature points between the first digital image and the second digital image. The feature alignment parameters can be homography transformation parameters or affine transformation parameters. Therefore, in some embodiments, the image region replacement system uses feature alignment parameters represented as a matrix associated with matching feature points.

[0048] As further used in the present disclosure, the term "pixel-adjusted gyroscope alignment model" refers to an image alignment model that aligns a digital image with another digital image based on a pixel-by-pixel comparison between a gyroscope data set corresponding to the digital image and the digital image. In some embodiments, the pixel-adjusted gyroscope alignment model includes an image alignment algorithm that aligns an input image with a reference image based on pixel-based alignment parameters and gyroscope alignment parameters. In some such cases, the image region replacement system uses the pixel-based alignment parameters to adjust or modify the gyroscope alignment parameters. The present disclosure further refers below to Figure 4-5B Gyroscope alignment model describing pixel adjustment.

[0049] Relatedly, the term "pixel-adjusted gyroscope alignment parameters" refers to one or more indicators that associate an input image with a reference image based on gyroscope data and individual pixel comparisons between the input image and the reference image. For example, in some embodiments, the pixel-adjusted gyroscope alignment parameters include attributes that align a first digital image with a second digital image based on both pixel-based alignment parameters and gyroscope alignment parameters when applied to a first digital image. In addition, the term "gyroscope alignment parameters" refers to one or more indicators that associate an input image with a reference image based on a gyroscope data set. For example, the gyroscope alignment parameters include transformation parameters that associate a gyroscope data set corresponding to one digital image with a gyroscope data set corresponding to another digital image. For example, in some embodiments, the gyroscope alignment parameters include a matrix that aligns a first digital image with a second digital image based on the focal length of the camera and the relative rotation between the input image and the reference image when applied to the first digital image, the relative rotation being indicated by the gyroscope data sets corresponding to the first digital image and the second digital image. Both the pixel-adjusted gyroscope alignment parameters and the gyroscope alignment parameters can be homography transformation parameters or (in some cases) affine transformation parameters. Thus, in some embodiments, the image region replacement system uses pixel adjusted gyro alignment parameters or gyro alignment parameters represented as a matrix relating the input image and the reference image.

[0050] As used in this disclosure, the term "gyroscope data set" refers to a gyroscope reading corresponding to a device for capturing a digital image. In some embodiments, the gyroscope data set refers to the gyroscope readings (e.g., angular momentum around each axis) captured by a gyroscope sensor or an inertial measurement unit ("IMU") when the camera captures the digital image. The image region replacement system can reference, store, or use the gyroscope data set in a specific format. For example, the image region replacement system can use a gyroscope data set that is stored as Euler angles (e.g., appropriate Euler angles or Tait-Bryan angles) and rotation data (e.g., rotation rates) or stored as quaternions. In either case, the image region replacement system can present such a gyroscope data set in a gyroscope data matrix (such as a rotation matrix for the gyroscope readings).

[0051] As described above, the image region replacement system intelligently generates an enhanced image by replacing a reference region of a reference image with a target region of a target image. Figure 1A-Figure 1C A series of example reference images, target images, and enhanced images are shown. Figure 1A-Figure 1C An example of a series of images 100 is shown in which a reference instance 106a of a person is replaced with a target instance 106b of a person. In particular, Figure 1A An example reference image 108 is shown, Figure 1B An example target image 110 is shown, and Figure 1C An example enhanced image 112 is shown. As shown, the reference image 108 and the target image 110 are two image portions of a plurality of images captured using burst mode, real-time photos, or video. Figure 1A and Figure 1B Only two images are shown, but the image region replacement system can process multiple images (eg, 5, 10, 15, etc.) The image region replacement system generates an enhanced image 112 using the reference image 108 and the target image 110 .

[0052] Figure 1A An example reference image 108 is shown that includes a reference instance 106a of a person. Figure 1A Also included are a second person's reference instance 102a and a third person's reference instance 104a. Figure 1A As shown, the second person's reference instance 102a and the third person's reference instance 104a have the expected expressions. However, the person's reference instance 106a is blinking.

[0053] The image region replacement system may use a variety of methods to identify the reference image 108. In at least one embodiment, the image region replacement system designates the image with the least blur as the reference image 108. The image region replacement system may also generate an aesthetic score for each instance of each person across multiple images. Based on the generated score, the image region replacement system may identify the image with the largest number of desired faces and designate that image as the reference image 108. In at least one embodiment, the image region replacement system selects the reference image 108 based on user input. In particular, the image region replacement system presents the group of images to the user and allows the user to select the reference image 108. Additionally, in at least one embodiment, the image region replacement system may designate an image with an average angle as the reference image 108. For example, the image region replacement system may determine the angles of multiple images and identify the image with the average angle. Therefore, since the multiple images will be aligned relative to the selected reference image, the reference image with the average angle reduces the alignment required for the largest number of images. In at least one embodiment, the image region replacement system simply uses the first image in a group of burst images as the reference image.

[0054] Once the reference image is selected, the image region replacement system identifies reference instances of the person for replacement. For example, the image region replacement system may receive user input or a selection of reference instances of the person to be replaced. Alternatively, the image region replacement system may automatically identify reference instances of the person to be replaced without user input. For example, the image region replacement system detects all instances of the person in reference image 108 and assigns an aesthetic grade to them. For example, Figure 1A As shown, the image region replacement system identifies a reference instance 106a of a person, a reference instance 102a of a second person, and a reference instance 104a of a third person and assigns an aesthetic grade to each of them. Figure 1A As shown, the image region replacement system assigns a lower aesthetic rating to the instance of person 106a because the instance of person 106a has its eyes closed. Figure 2 The method by which the image region replacement system assigns an aesthetic rating to each of the instances of persons is described in more detail.

[0055] In at least one embodiment, the image region replacement system identifies instances of a person in reference image 108 that have a low aesthetic rating. For example, the image region replacement system determines that reference image 106a of a person has a low rating, while reference image 102a of a second person and reference instance 104a of a third person are not low. In at least one embodiment, the image region replacement system quantifies the aesthetic score as low based on an aesthetic score threshold. When the image region replacement system determines that the aesthetic score of an instance of a person is below a predetermined threshold, the image region replacement system identifies it as a reference instance that should be replaced. In at least one embodiment, a user may determine the aesthetic score threshold.

[0056] Once the reference instance is selected, the image region replacement system identifies the target instance of the person to replace the reference instance. For example, the image region replacement system may receive user input or selection of the target instance of the person. Alternatively, the image region replacement system may automatically identify the target instance of the person without user input. For example, Figure 1A As shown, once the image region replacement system identifies the instance of the person as having a low aesthetic score, the image region replacement system identifies a boundary region 106a around the reference instance of the person. For example, the boundary region may be created by extending the outline of the reference instance 106a of the person. In other embodiments, the image region replacement system creates a boundary region around the reference instance 106a of the person by creating an inclusive geometric shape (such as a rectangle or a circle) around the reference instance 106a of the person. In other embodiments, the image region replacement system receives a user selection of the boundary region.

[0057] The image region replacement system detects instances of person 106 across multiple images using the boundary region. In particular, the image region replacement system identifies the boundary region in each of the multiple images. The image region replacement system uses facial detection technology to detect instances of person 106 within the boundary region in each of the multiple images. Therefore, the image region replacement system can specifically identify instances of person 106 across multiple images, rather than identifying instances of second other person 102 and third other person 104.

[0058] After identifying instances of person 106 across the plurality of images, the image region replacement system generates an aesthetic rating for each of the instances of person 106 across the plurality of images. Figure 1B As shown, based on the generated aesthetic rating for the instance of person 106 , the image region replacement system selects a target instance 106 b of the person for the target image 110 . Figure 1B An example target image 110 is shown that includes a target instance 106b of a person. Figure 1B Also included is an instance 102b of a second person and an instance 104b of a third person. Figure 1B As shown, the target instance of a person 106b has a desired expression (eg, her eyes are open).

[0059] The image region replacement system determines a replacement region around a reference instance 106a of a person and a target instance 106b of a person. In particular, the image region replacement system identifies a boundary of the replacement region that does not intersect an object or person in the image. In at least one embodiment, the image region replacement system employs a graph cut algorithm to identify the replacement region.

[0060] Figure 1C An example enhanced image 112 is shown in which the image region replacement system has replaced the reference instance 106a of the person with the target instance 106b of the person. As shown, the reference instance 102a of the second person, the reference instance 104a of the third person, and the target instance 106b of the person all have the desired expression. As further shown, the image region replacement system only replaces the reference instance of the person with the target instance 106b of the person. The instances of the other persons in the image are the same as their instances in the original reference image. Even though the example enhanced image 112 includes extraneous elements, the enhanced image 112 does not contain any inconsistencies or image artifacts.

[0061] Now turn to Figure 2 , Figure 2 It shows how an image region replacement system selects a target instance of a person based on generating multiple aesthetic ratings for multiple instances of the person. In particular, Figure 2 A client computing device 200 is shown having a display 202 showing a plurality of images 206 including multiple instances of a person. More specifically, a user interface 204 of an image region replacement system displays the plurality of images 206 (three different sets of burst mode images are shown).

[0062] like Figure 2 As shown, each of the plurality of images 206 has a small display size due to the size of the display of the client computing device 200. Due to the small size of the images 206, it may be difficult to compare and contrast the images 206, especially instances of people. This is especially the case when the images 206 all point to the same moment in time or include substantially the same content. For example, it may be relatively difficult to determine that a person's eyes are closed in one image and open in another image. As described above, the image reference region system can alleviate the time and effort of having to compare small image thumbnails or previews to identify high-quality instances of a person or image.

[0063] Figure 2A zoom-in of multiple images 210 is shown. The image region replacement system detects all instances of a person in multiple images 210. As shown, the image region replacement system may use facial detection techniques to identify faces in multiple images 210. For example, the image region replacement system may determine that each image in the multiple images includes an instance of a person. Thus, the image region replacement system will identify instances of each person across multiple group portraits.

[0064] The image region replacement system may generate an aesthetic rating for each instance of a person in the plurality of images 210. In at least one embodiment, the aesthetic rating comprises a weighted sum of different attributes for each instance of a person. Attributes considered by the image region replacement system include eye opening, frontal facial pose, facial expression, facial sharpness, etc. The image region replacement system may use facial landmark detection to extract attributes from the appearance of eye landmarks, facial contours, mouth landmarks, and other landmarks.

[0065] Figure 2 Multiple images 210 in may be provided to illustrate some attributes that the image region replacement system may consider when generating an aesthetic grade for an instance of a person. The image region replacement system may determine that the instance of the person in image 214 is not fully captured in the canvas of the image. Because image 220 includes the entire face of the person, the replacement region selection system may determine that image 220 has a higher quality than image 214. The image region replacement system may determine that the instance of the person in image 212 is centered in the canvas of the image, but the person's eyes are closed. The image region replacement system may determine that image 216 is small or blurry due to motion of a camera used to capture image 216. Figure 2 It is shown that the image region replacement system has identified the instance of the person in image 220 as the target instance of the person because it has the highest aesthetic rating.

[0066] The image region replacement system can generate, train and use one or more neural networks to perform one or more of the above functions. In particular, the image region replacement system can train a neural network to generate an aesthetic score for an instance of a person. Figure 3A and Figure 3B Some examples of neural networks that the image region replacement system can use to generate aesthetic scores are provided. Based on the disclosure herein, it should be appreciated that the image region replacement system can use other neural networks or scoring paradigms.

[0067] Figure 3AAn example of a neural network 300 trained to generate attribute ratings is shown. The attribute rating neural network 300, as well as other neural networks described herein, may be a convolutional neural network or other type of classification neural network. The image region replacement system may train and use the attribute rating neural network 300 to generate aesthetic ratings. Alternatively, the image region replacement system may train and use multiple attribute classification neural networks 300 to generate multiple attribute ratings. For example, the image region replacement system may train separate attribute rating neural networks 300 to generate ratings for attributes such as eye openness, facial pose, facial orientation, facial expression, facial sharpness, noise, darkness, contrast, etc. The image region replacement system may then use the various trained attribute classification neural networks 300 as references as follows: Figure 3B A portion of the multi-attribute rating neural network is described to generate an aesthetic rating.

[0068] like Figure 3A As shown, the attribute level neural network 300 includes a lower neural network layer 302, a higher neural network layer 304 and a loss layer 306. The loss layer 306 includes an attribute level loss model 308. In one or more embodiments, the lower neural network layer 302 is a convolutional layer that encodes an image (or an image patch) into a feature vector. The higher neural network layer 304 may include fully connected layers that classify the feature vectors and output attribute quality probability vectors. Due to their functions, the lower neural network layers 302 are collectively referred to as encoders and the higher neural network layers 304 are collectively referred to as classifiers. In some embodiments, the higher layers may include regressors instead of classifiers. In some embodiments, the attribute level neural network 300 is a deep learning convolutional neural network.

[0069] As shown, the attribute level neural network 300 in this example includes five convolutional layers and two fully connected layers. In other embodiments, the image region replacement system can employ a network with more layers. The image region replacement system configures the neural network layers 302, 304 to learn features or image characteristics (such as a certain amount of darkness, light, contrast, etc.), which together can be used to define higher-level image aesthetics and facial aesthetics by using attribute levels. For example, in order to understand whether an image is aesthetically "good", features of image characteristics (such as lighting, noise, whether the image is "upright", etc.) can be learned by the attribute level neural network 300 to define the attribute level.

[0070] In particular, the attribute-level neural network 300 may use the training image 310 or a patch thereof to train the attribute-level neural network 300. In order to efficiently process the training image 310, the attribute-level neural network 300 may adopt a patch-based neural network technique. For example, the attribute-level neural network 300 may process image patches of a fixed size (e.g., 128, 196, 256, 320 pixels in size, etc.) randomly sampled from the image, rather than processing the entire image. In addition, the attribute-level neural network 300 may process patches without downsampling the image, so that no information is lost. In one or more embodiments, the training image may include, for example, Figure 3A Portrait of the person shown.

[0071] The lower neural network layer 602 can encode the input image into a feature vector. Features can be expressed in a variety of ways. For example, for each patch, a vector with multiple dimensions is generated, where each dimension has a corresponding feature. Therefore, the vector represents the amount of each corresponding image feature. In one or more embodiments, the features include latent features.

[0072] The image region replacement system may initially train the attribute rating neural network 300 using the attribute classification loss model 308. As an overview, the image region replacement system initially trains the attribute rating neural network 300 by adjusting parameters based on the training images 310 (or training patches from the training images 310) used to generate feature vectors and attribute ratings.

[0073] In addition, the image region replacement system employs an attribute level loss model 308 to provide feedback based on the accuracy of the attribute level, which enables the image region replacement system to update adjustable parameters. More specifically, the image region replacement system back-propagates the error loss to adjust the parameters in the lower neural network layer 302 and the higher neural network layer 304 to classify the attribute level according to the input training image.

[0074] To illustrate, the image region replacement system may provide training images 310 as input to initially train the attribute level neural network 300. For example, in one or more embodiments, the image region replacement system trains the lower neural network layer 302 by adjusting feature extraction parameters that are used to generate feature vectors for each input training image 310. The image region replacement system then provides the generated feature vectors to the higher neural network layer 304, which compares the generated feature vectors from the lower neural network layer 302 with feature vectors of images with known high-quality attributes based on the attribute level parameters. Based on the feature vector comparison, the higher neural network layer 304 uses the attribute level parameters to generate an attribute quality probability vector that indicates a correspondence between the feature vectors of the input training image and the image with the known high-quality attributes.

[0075] Next, the image region replacement system provides the attribute quality probability vector to the attribute level loss model 308. The attribute level loss model 308 compares the identified attribute level indicated by the attribute quality probability vector with a known feature vector of an image having high-quality attributes to determine the amount of attribute level error loss (or simply "error loss"). In practice, the image region replacement system can provide the attribute quality probability vector to the attribute level loss model to determine the accuracy and / or error loss of the attribute level. In some embodiments, the attribute level loss model 308 uses a SoftMax cross entropy loss attribute quality classifier and / or a mean square error calculation to determine the amount of attribute level loss. For example, the attribute level loss model 308 identifies when the attribute quality probability vector exceeds a threshold distance, and / or how far the attribute quality probability vector exceeds a threshold distance to a feature vector of a known high-quality image (e.g., error loss).

[0076] Then, using the error loss to train and optimize the neural network layers of the attribute level neural network 300, the image region replacement system can employ back propagation and end-to-end learning to adjust the feature extraction parameters within the layers of the attribute level neural network 300. For example, in one or more embodiments, the image region replacement system obtains the error loss output from the attribute level loss model 308 and provides it back to the lower neural network layer 302 and / or the higher neural network layer 304 until the error loss from the attribute level loss model 308 is minimized. In particular, the attribute level loss model 308 provides feedback to the lower neural network layer 302 to further adjust the attribute quality feature extraction parameters, and / or provides feedback to the higher neural network layer 304 to further adjust the attribute level parameters. In this way, the image region replacement system iteratively trains the attribute level neural network 300 to learn a set of best-fit parameters that extract features from the image and accurately classify the corresponding attribute quality.

[0077] As described above, in some embodiments, the image region replacement system employs the higher neural network layer 304 to determine an attribute quality probability vector of the input attribute quality of the input image. As an example of an attribute quality probability vector, the higher neural network layer 304 outputs a two-dimensional attribute quality probability vector whose entries range between 0 and 1 (i.e., [0-1]). The first entry can be the probability that the input image has a high-quality attribute therein. The second entry in the attribute quality probability vector can be the probability that the input image has a low-quality attribute therein. The image region replacement system can use the first entry as the attribute ranking.

[0078] As described above, the image region replacement system can train the attribute rating neural network 300 to generate an overall aesthetic rating of the attributes of facial quality. When generating and training the overall facial quality rating neural network, the image region replacement system can use training images including faces with low quality or high quality labels. In one or more embodiments, a low quality face is a face that has motion blur, is out of focus, has poor lighting or exposure, is occluded (i.e., covered or not completely in the image), has one or more eyes closed, has low contrast or is hazy, has severe image compression, and has an unsatisfactory or ugly expression. In one or more embodiments, the image region replacement system warps the face into a fixed size for feature learning and classifier training. In order to apply the trained facial quality rating neural network, the facial quality rating neural network can use a face detection algorithm to locate the face in each image. The facial quality rating neural network can then perform a weighted average of all facial quality ratings based on a spatial face map that encodes multiple faces, face sizes, face positions, and facial quality ratings.

[0079] Alternatively, as described above, the image region replacement system may train multiple attribute rating neural networks 300 to rate multiple attributes, rather than training the attribute rating neural network 300 to generate an overall aesthetic rating. In such an embodiment, the image region replacement system may then employ a method such as Figure 3B The multi-attribute rating neural network 320 is shown. As described below, the multi-attribute rating neural network 320 may provide the following advantages: when determining the aesthetic rating, it is possible to control which attributes are weighted higher. In particular, the multi-attribute rating neural network 320 may include an attribute weighting model 322, which may allow a user to specify which attributes are ranked higher (open eyes are more important than smiling, etc.).

[0080] More specifically, when training multiple attribute-level neural networks, the image region replacement system generates Figure 3BThe multi-attribute hierarchical neural network 320 is shown. In particular, the image region replacement system can generate the multi-attribute hierarchical neural network 320 by combining higher neural network layers (eg, attribute classifiers 314a-314c) and the attribute weighting model 322. Figure 3B The multi-attribute hierarchical neural network 320 is shown sharing the lower neural network layer 302 (ie, the feature encoder). In an alternative embodiment, each classifier may have a separate feature encoder.

[0081] The multi-attribute ranking neural network 320 provides an improved image ranking. For example, upon receiving an input image, the multi-attribute ranking neural network 320 provides the image to a feature encoder (i.e., the lower neural network layer 302), which generates a feature vector for the image. The multi-attribute ranking neural network 320 then provides the feature vector to each of the attribute classifiers 314a-314c and the attribute weighting model 322. The attribute classifiers 314a-314c each output an attribute ranking for the input image, as described above with respect to Figure 3A described.

[0082] In addition, the attribute weighting model 322 outputs a multidimensional weighted vector that includes an attribute weight for each attribute having a corresponding attribute classifier 314a-314c. The respective attribute weights indicate how to combine the attribute grades output from the attribute classifiers 314a-314c to optimally generate an aesthetic grade for the image. In particular, to generate the aesthetic grade 318, the multi-attribute grade neural network 320 weights the attribute grades output from the attribute classifiers 314a-314c with the corresponding weights output from the attribute weighting model 322, and then sums the weight-adjusted attribute grade scores to generate the aesthetic grade 318.

[0083] Thus, to generate an aesthetic rating for each instance of a person in multiple images, the image region replacement system may feed the images to an attribute rating neural network 300 or a multi-attribute rating neural network 320, which is trained to rate overall facial quality, which returns an aesthetic score. The image region replacement system may then use the generated aesthetic ratings to select one or more images from a reference image or a target image. Once the target image and reference images for a given instance of a person are selected, the image region replacement system may align the target image and the reference image. In particular, the image region replacement system may select (and apply) an image alignment model selected from a feature-based alignment model or a pixel-adjusted gyroscope alignment model. For example, Figure 4 An image region replacement system for selecting between image alignment models is shown. In particular, Figure 4A sequence flow chart 400 of actions 402-426 is depicted, by which an image region replacement system selects and applies an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model to align a target image with a reference image (e.g., an input image) according to one or more embodiments. As shown in the sequence flow chart 400, when a lack of feature points corresponding to the feature-based alignment model is detected, the image region replacement system applies the pixel-adjusted gyroscope alignment model to align the input image with the reference image. Conversely, when it is determined that there is no lack of feature points, the image region replacement system applies the pixel-based alignment model to align the input image with the reference image.

[0084] like Figure 4 As shown, the image region replacement system performs an action 402 of receiving a target image and a corresponding gyroscope data set and an action 404 of receiving a reference image and a corresponding gyroscope data set. In some embodiments, the image region replacement system receives the target image and the reference image captured by a camera of a computing device (e.g., a mobile device) when such target image and reference image are received. In some cases, the computing device records or registers a gyroscope data set (e.g., an indication of the orientation or pose of the camera based on angular momentum around each axis) when the camera captures the target image, and records an additional gyroscope data set at the time the camera captures the reference image.

[0085] like Figure 4 As further shown, the image region replacement system performs an act of determining a relative rotation based on the gyroscope data set 406. For example, in some cases, the image region replacement system determines the relative rotation between the target image and the reference image based on (i) the gyroscope data set corresponding to the input image and (ii) the gyroscope data set corresponding to the reference image. In some such embodiments, the image region replacement system may determine the relative rotation from the reference image to the target image by comparing the gyroscope data set corresponding to the target image and the gyroscope data set corresponding to the reference image (and determining the difference therebetween).

[0086] In addition to determining the relative rotation, the image region replacement system also performs an action of identifying feature points within the target image and the reference image 408. For example, in some embodiments, the image region replacement system utilizes a SIFT, SURF, or ORB algorithm to detect a set of feature points within the target image and a set of feature points within the reference image. Upon detection, the image region replacement system can extract corresponding sets of feature points within the target image and the reference image.

[0087] As shown in the remaining actions 410-426, the image region replacement system uses the identified feature points and the corresponding gyroscope data set to generate and apply pixel-adjusted gyroscope alignment parameters, or to generate and apply feature alignment parameters. Figure 4 As shown, the image region replacement system performs actions 414, 418, and 422 to determine (at various decision points) whether feature points or feature alignment parameters corresponding to the target image and the reference image exhibit feature point deficiency. If the feature points or feature alignment parameters indicate feature point deficiency, the image region replacement system generates and applies pixel-adjusted gyroscope alignment parameters (as part of a pixel-adjusted gyroscope alignment model). If the feature points or feature alignment parameters indicate that there is no feature point deficiency, the image region replacement system continues to perform actions that result in generating and applying feature alignment parameters (as part of a feature-based alignment model).

[0088] like Figure 4 As shown, for example, the image region replacement system performs an action 410 of determining the focal length of the camera. For example, in some embodiments, the image region replacement system estimates the focal length of the camera that captured the target image and / or the reference image. When determining the focal length of the camera that captured a set of images, the image region replacement system can update the estimate of the focal length based on additional images captured by the camera. Alternatively, in some implementations, the image region replacement system queries the computing device that captured the target image for the focal length.

[0089] In addition to determining the focal length, the image region replacement system also performs an act 412 of generating gyroscope alignment parameters. Figure 4 As shown, in some embodiments, the image region replacement system generates gyroscope alignment parameters based on gyroscope data sets corresponding to the target image and the reference image, respectively, and the focal length of the camera. In some such embodiments, the image region replacement system generates gyroscope alignment parameters based on the relative rotation between the target image and the reference image determined by action 406.

[0090] like Figure 4 As further shown, the image region replacement system performs an action 414 of determining whether feature points within one or both of the target image and the reference image include a lack of feature points. For example, in some embodiments, the image region replacement system determines whether a set of feature points within the target image and / or a set of feature points within the reference image satisfies a threshold amount of feature points. Figure 4 As shown, if the feature points in one or both of the target image and the reference image do not satisfy the threshold amount of feature points and indicate that there is a lack of feature points at the decision point, the image region replacement system abandons applying the feature-based alignment model and generates pixel-adjusted gyroscope alignment parameters in action 424. However, if the feature points in both the target image and the reference image satisfy the threshold amount of feature points and indicate that there is no lack of feature points at the decision point, the image region replacement system continues with the action of the feature-based alignment model.

[0091] like Figure 4As shown, when it is determined in action 414 that there is no lack of feature points, the image region replacement system performs action 416 of matching feature points between the target image and the reference image. For example, in some implementations, the image region replacement system matches feature points from a set of feature points within the target image with feature points from a set of feature points within the reference image. In some such embodiments, the image region replacement system matches feature points between the target image and the reference image based on a brute force matcher algorithm, a fast library of approximate nearest neighbors ("FLANN") algorithm, or any other suitable feature point matching algorithm.

[0092] After matching the feature points, in some cases, the image region replacement system further removes (or filters out) a subset of the matched feature points that conflict with the gyroscope alignment parameters in action 406. By removing such conflicting matched feature points, the image region replacement system creates a set of filtered matched feature points between the target image and the reference image.

[0093] like Figure 4 As further shown, the sequence flow chart 400 includes another decision point, namely, action 418. In some embodiments, the image region replacement system performs action 418 of determining whether the matching feature points between the target image and the reference image include a lack of feature points. For example, in some implementations, the image region replacement system determines whether the matching feature points between the target image and the reference image meet a threshold amount of matching feature points. Figure 4 As shown, if the matched feature points do not satisfy the threshold amount of feature points and indicate that there is a lack of feature points at the decision point, the image region replacement system abandons applying the feature-based alignment model and generates pixel-adjusted gyroscope alignment parameters in action 424. However, if the matched feature points satisfy the threshold amount of matched feature points and indicate that there is no lack of feature points at the decision point, the image region replacement system continues to perform the actions of the feature-based alignment model.

[0094] When it is determined in action 418 that there is no lack of feature points, the image region replacement system performs action 420 of estimating feature alignment parameters. For example, in some embodiments, the image region replacement system estimates feature alignment parameters using matching feature points between the target image and the reference image. As described above, the image region replacement system can apply a RANSAC algorithm, a LMedS regression algorithm, or other suitable robust algorithms to parameter model estimation to generate feature alignment parameters.

[0095] like Figure 4As further shown, the sequence flow chart 400 includes a final decision point, namely, action 422. In some embodiments, the image region replacement system performs action 422 to determine whether the feature alignment parameters include a feature point absence. For example, in some implementations, the image region replacement system determines whether the feature alignment parameters reasonably align the target image with the reference image based on (or with reference to) a gyroscope data set or gyroscope alignment parameters captured by the computing device. For example, the image region replacement system can determine the difference between the feature alignment parameters corresponding to the feature-based alignment model and a range of expected feature alignment parameters for the input image determined from the relative rotation (e.g., a range that includes maximum and minimum expected feature alignment parameters for the input image based on the gyroscope data set or gyroscope alignment parameters).

[0096] Consistent with the decision point described above, if the feature alignment parameters do not reasonably align the target image with the reference image and indicate a lack of feature points, the image region replacement system forgoes applying the feature-based alignment model and generates pixel-adjusted gyroscope alignment parameters in act 424. However, if the feature alignment parameters reasonably align the target image with the reference image and indicate that a lack of feature points does not exist at the decision point, the image region replacement system proceeds to act 426 to align the target image with the reference image (based on the feature alignment parameters estimated in act 420).

[0097] As described above, if the image region replacement system detects a lack of feature points when performing any of actions 414, 418, or 422, the image region replacement system selects the pixel-adjusted gyroscope alignment model instead of the feature-based alignment model to align the target image with the reference image. Specifically, once a lack of feature points is detected in actions 414, 418, or 422, the image region replacement system abandons applying the feature-based alignment model and performs action 424 of generating pixel-adjusted gyroscope alignment parameters.

[0098] As a precursor to generating pixel-adjusted gyroscopic alignment parameters, in some embodiments, the image region replacement system warps the target image according to the gyroscopic alignment parameters generated in act 412, and estimates pixel-based alignment parameters that can align the warped image with the reference image. In some embodiments, the image region replacement system generates pixel-adjusted gyroscopic alignment parameters based on both the pixel-based alignment parameters and the gyroscopic alignment parameters.

[0099] like Figure 4As further shown, the image region replacement system performs an action 426 of aligning the target image with the reference image. As described above, if the feature points or feature alignment parameters indicate a feature point deficiency, the image region replacement system generates pixel-adjusted gyroscope alignment parameters and applies them to the target image to align the input with the reference image. If the feature points and feature alignment parameters indicate that there is no feature point deficiency, the image region replacement system generates feature alignment parameters and applies them to the target image to align the target with the reference image.

[0100] As mentioned above, Figure 4 A flexible and selective image alignment algorithm is shown. In contrast to some conventional image alignment systems, Figure 4 An image region replacement system disclosed in the disclosure uses a flexible approach to select between a pixel-adjusted gyroscope alignment model and a feature-based alignment model to align digital images. Alternatively, in some embodiments, the image region replacement system employs different selective image alignment algorithms by using a feature point lack metric to select between a pixel-based alignment model and a feature-based alignment model to align digital images. Thus, in some cases, based on identifying feature point lack, the image region replacement system optionally applies a pixel-based alignment model (rather than a feature-based alignment model) to align a target image with a reference image. In such an embodiment, the image region replacement system applies a pixel-based alignment model by comparing pixels across a target image and a reference image, and determining pixel-based alignment parameters for aligning an input image with a reference image based on overlapping pixels between the images.

[0101] As described above, in some embodiments, the image region replacement system may apply a pixel-adjusted gyroscope alignment model to align digital images. Figure 5A An embodiment of a pixel-adjusted gyroscope alignment model 512 is depicted. Figure 5A As an overview of the present invention, the image region replacement system identifies a first gyroscope data set 506a corresponding to a reference image 502 and a second gyroscope data set 506b corresponding to a target image 504. Based on the first gyroscope data set 506a and the second gyroscope data set 506b, the image region replacement system generates gyroscope alignment parameters 516 for aligning the target image 504 with the reference image 502. The image region replacement system also generates pixel-adjusted gyroscope alignment parameters 522 based on the gyroscope alignment parameters 516 and the pixel-based alignment parameters 520. Having generated the pixel-adjusted gyroscope alignment parameters 522, the image region replacement system applies the pixel-adjusted gyroscope alignment parameters 522 to the target image 504 to align the target image 504 with the reference image 502.

[0102] like Figure 5AAs shown, a camera of computing device 500 captures reference image 502 and target image 504. Computing device 500 then provides (and image region replacement system receives) reference image 502 and target image 504 for analysis. For example, in some embodiments, computing device 500 detects a user selection of a selectable option for aligning digital images (e.g., a selection of an option for aligning target image 504 with reference image 502). In some embodiments, based on detecting the user selection, image region replacement system analyzes reference image 502 and target image 504 against pixel-adjusted gyroscope alignment model 512.

[0103] After image capture, the image region replacement system identifies a first gyroscope data set 506a corresponding to the reference image 502 and a second gyroscope data set 506b corresponding to the target image 504. In some embodiments, the computing device 500 uses a gyroscope sensor or an IMU to record (i) gyroscope readings when the camera captured the reference image 502 and (ii) gyroscope readings when the camera captured the target image 504. Because the first gyroscope data set 506a and the second gyroscope data set 506b can be different from each other, the image region replacement system can use the difference to determine the relative rotation between the target image 504 and the reference image 502.

[0104] Additionally, in some embodiments, the computing device 500 stores the gyroscope readings as a first gyroscope data set 506a and a second gyroscope data set 506b for the image region replacement system. Consistent with the above disclosure, in some cases, the computing device 500 stores the first gyroscope data set 506a and the second gyroscope data set 506b as Euler angles and rotation data, or (to avoid gimbal lock that sometimes exists with Euler angles) as quaternions.

[0105] like Figure 5A As further shown, the image region replacement system determines a focal length 508 of a camera in the computing device 500. In some implementations, the image region replacement system queries the computing device 500 for the focal length 508. For example, when the camera captures the reference image 502 and / or the target image 504, the image region replacement system can query the computing device 500 for the focal length 508 based on an intrinsic calibration of the camera lens.

[0106] Alternatively, in some embodiments, the image region replacement system estimates the focal length 508 based on matching feature points between the reference image 502 and the target image 504. For example, the image region replacement system can estimate the focal length 508 when: (i) the matching feature points between the reference image 502 and the target image 504 meet a threshold amount of matching feature points, and (ii) the camera is panning between capturing the reference image 502 and the target image 504. In some embodiments, the image region replacement system estimates the focal length 508 using a calibration method, and represents the focal length 508 in pixels. In some such cases, the image region replacement system estimates the focal length 508 using a method for determining focal length described in "Image Alignment and Stitching: A Tutorial" by Richard Szeliski, in "Foundations and Trends in Computer Graphics and Vision", Vol. 2, No. 1 (2006), hereinafter referred to as "Szeliski", the entire contents of which are incorporated herein by reference. Because pixels are sometimes not quadratic, in some implementations, the image region replacement system determines a first focal length (fx) for the horizontal dimension and a second focal length (fy) for the vertical dimension when determining the focal length 508 .

[0107] like Figure 5A As shown, in addition to determining the focal length 508, the image region replacement system also identifies the image size 510. For example, in some embodiments, the image region replacement system also identifies the height and width of the reference image 502 and / or the height and width of the target image 504. Additionally or alternatively, in some implementations, the image region replacement system identifies the camera's principal point offset and / or axis skew when identifying the image size 510.

[0108] After determining the focal length 508 and the image size 510, the image region replacement system applies a pixel adjusted gyroscope alignment model 512 by using the first gyroscope data set 506a, the second gyroscope data set 506b, the focal length 508, and the image size 510. Figure 5AAs shown, for example, based on the focal length 508 and the image size 510, the image region replacement system determines the camera intrinsic matrix 514. In some cases, the image region replacement system determines the camera intrinsic matrix 514 by creating a matrix of intrinsic camera parameters including the focal length 508 and the principal axes. Further, in some such embodiments, the image region replacement system determines the camera intrinsic matrix 514 including the focal length, the principal point offset, and the axis skew. In some cases, the image region replacement system determines the camera intrinsic matrix 514 by determining the intrinsic calibration matrix described by Szeliski or by determining the calibration matrix described in Chapter 8 of "Multiple View Geometry in Computer Vision" by Richard Hartley and Andrew Zisserman (2nd edition, 2004), the entire contents of which are incorporated herein by reference.

[0109] like Figure 5A As further shown, after determining the camera intrinsic matrix 514, the image region replacement system generates gyroscope alignment parameters 516 based on the first gyroscope data set 506a and the second gyroscope data set 506b and the camera intrinsic matrix 514. For example, in some embodiments, the image region replacement system generates the gyroscope alignment parameters 516 based on the relative rotation between the reference image 502 and the target image 504 indicated by the first gyroscope data set 506a and the second gyroscope data set 506b. In particular, in some such embodiments, the image region replacement system generates the gyroscope alignment parameters 516 based on the camera intrinsic matrix 514, the first gyroscope data set 506a (represented as a rotation matrix), and the second gyroscope data set 506b (also represented as a rotation matrix).

[0110] like Figure 5A As shown, in some embodiments, the image region replacement system generates gyroscope alignment parameters 516 such that when the gyroscope alignment parameters 516 are applied to the target image 504, the gyroscope alignment parameters 516 approximately align the target image 504 with the reference image 502. Here, the gyroscope alignment parameters 516 approximately align the target image 504 with the reference image 502 because, in the pixel-adjusted gyroscope alignment model 512, the image region replacement system also adjusts the gyroscope alignment parameters 516 to generate more accurate alignment parameters (e.g., by adjusting the gyroscope alignment parameters 516 or combining them with pixel-based alignment parameters).

[0111] In some embodiments, after generating the gyroscope alignment parameters 516, the image region replacement system warps the target image 504 based on the gyroscope alignment parameters 516 to generate a warped input image 518. For example, in some implementations, the image region replacement system applies the gyroscope alignment parameters 516 to the target image 504 to perform a homography transformation and generate the warped input image 518. Thus, the warped input image 518 is a warped version of the target image 504.

[0112] like Figure 5A As further shown, after generating the warped input image 518, the image region replacement system estimates pixel-based alignment parameters 520. In particular, in some embodiments, the image region replacement system estimates translation parameters that align the warped input image 518 with the reference image 502 based on comparing pixels between the two images (e.g., reducing their alignment errors). For example, in some implementations, the image region replacement system downsamples the reference image 502 and the warped input image 518. In some such implementations, the image region replacement system then applies an error metric (e.g., sum of squared differences or sum of absolute differences, root mean square intensity pixel error, normalized cross-correlation), hierarchical motion estimation, or Fourier-based alignment to the downsampled reference image 502 and the downsampled warped input image 518. Based on such error metric, hierarchical motion estimation, or Fourier-based alignment, the image region replacement system estimates alignment parameters that translate pixels from the downsampled warped input image 518 to the reference image 502.

[0113] By employing an error metric, hierarchical motion estimation, or Fourier-based alignment to perform a simple pixel-based translation on the downsampled image (i.e., applying a pixel-based translation to a warped image that has already been coarsely aligned based on gyroscope data), the image region replacement system can avoid the processing load consumed by conventional pixel-based alignment methods. Even compared to the hierarchical sampling of pixels employed by some conventional pixel-based alignment methods, the image region replacement system also speeds up the estimation of pixel-based alignment parameters 520 by utilizing both gyroscope data and the downsampled pixel-based translation. In an alternative, in some embodiments, the image region replacement system estimates the pixel-based alignment parameters 520 using a hierarchical coarse-to-fine or Fourier transform technique described by Szeliski.

[0114] In fact, in one or more embodiments, the gyroscope data reflects the rotation data as well as the pan (but not necessarily the translation data) of the camera used to capture the digital image. By using a fast pixel-based translation estimate of the downsampled input image, the image region replacement system can determine the translation information with little computational expense. The image region replacement system can then combine the translation transform (e.g., pixel-based alignment parameters) with the rotation transform (e.g., gyroscope alignment parameters) to form the final alignment parameters (e.g., pixel-adjusted gyroscope alignment parameters).

[0115] like Figure 5A As shown, in some embodiments, the image region replacement system estimates pixel-based alignment parameters 520 such that when applied to the target image 504, the pixel-based alignment parameters 520 approximately align the target image 504 with the reference image 502. Indeed, with respect to Figure 5A , the image region replacement system uses the pixel-based alignment parameters 520 to refine or generate more accurate alignment parameters (e.g., by adjusting the gyroscope alignment parameters 516 or combining them with the pixel-based alignment parameters). Moreover, as described above, the image region replacement system can estimate the pixel-based alignment parameters 520 using less computational processing and avoids searching every constituent pixel of the image (as required by some conventional pixel-based methods).

[0116] like Figure 5A As further shown, after estimating the pixel-based alignment parameters 520, the image region replacement system generates pixel-adjusted gyroscope alignment parameters 522. In particular, the image region replacement system generates the pixel-adjusted gyroscope alignment parameters 522 based on the gyroscope alignment parameters 516 and the estimated pixel-based alignment parameters 520. For example, in some embodiments, the image region replacement system adjusts the gyroscope alignment parameters 516 based on the estimated pixel-based alignment parameters 520 to generate the pixel-adjusted gyroscope alignment parameters 522. In certain implementations, the image region replacement system combines or multiplies the gyroscope alignment parameters 516 and the estimated pixel-based alignment parameters 520 to generate the pixel-adjusted gyroscope alignment parameters 522.

[0117] like Figure 5AAs shown, after generating the pixel-adjusted gyroscope alignment parameters 522, the image region replacement system applies the pixel-adjusted gyroscope alignment parameters 522 to the target image 504 to generate an aligned input image 524. By applying the pixel-adjusted gyroscope alignment parameters 522 to the target image 504, the image region replacement system shifts or warps the target image 504 to align with the reference image 502. For example, in some embodiments, the image region replacement system warps the target image 504 according to a homography encoded by the pixel-adjusted gyroscope alignment parameters 522 to generate the aligned input image 524.

[0118] like Figure 5A As shown, the image region replacement system can apply the pixel-adjusted gyroscope alignment model 512 independently of detecting the lack of feature points and independently of the feature-based alignment model. In contrast, in some implementations, the pixel-adjusted gyroscope alignment model 512 can be part of a selective image alignment algorithm that selects between the pixel-adjusted gyroscope alignment model 512 and the feature-based alignment model. Figure 5B An example of such a feature-based alignment model is shown.

[0119] Figure 5B An image region replacement system is shown that applies a feature-based alignment model 536 to align a target image 534 with a reference image 532 in accordance with one or more embodiments. As an overview, the image region replacement system identifies a first set of feature points 538a within the reference image 532 and a second set of feature points 538b within the target image 534. The image region replacement system also matches feature points between the reference image 532 and the target image 534 to create matched feature points 542. Based on a filtered set of matched feature points 546, the image region replacement system estimates feature alignment parameters 550 corresponding to the feature-based alignment model 536, and applies the feature alignment parameters 550 to the target image 534. In some embodiments, to guide implementation of the feature-based alignment model 536, the image region replacement system determines Figure 5B The various feature points indicated in are lacking metrics, and the metrics are compared with a lack threshold.

[0120] like Figure 5B As shown, a camera of computing device 500 captures reference image 532 and target image 534. Computing device 500 then provides (and image region replacement system receives) reference image 532 and target image 534 for analysis. For example, in some embodiments, computing device 500 detects a user selection of a selectable option for aligning digital images. Based on detecting the user selection, the image region replacement system analyzes reference image 532 and target image 534 against feature-based alignment model 536.

[0121] After image capture, the image region replacement system identifies a first set of feature points 538a within the reference image 532 and a second set of feature points 538b within the target image 534. In some embodiments, the image region replacement system uses BREIF, FAST, ORB, or other suitable detection algorithms as part of identifying the first set of feature points 538a within the reference image 532 and the second set of feature points 538b within the target image 534. In certain implementations, upon detection, the image region replacement system also extracts the first set of feature points 538a and the second set of feature points 538b and formats them as feature descriptors or feature vectors, respectively.

[0122] like Figure 5B As further shown, in addition to identifying the first set of feature points 538a and the second set of feature points 538b, the image region replacement system also determines a feature point deficiency metric for each set of feature points in the first set of feature points 538a and the second set of feature points 538b, and then compares the feature point deficiency metric to a deficiency threshold. Figure 5B As shown, for example, the image region replacement system: (i) determines a first feature point lack metric by determining a plurality of feature points in a first set of feature points 538a, and (ii) determines a second feature point lack metric by determining a plurality of feature points in a second set of feature points 538b. The image region replacement system also compares each feature point in the plurality of feature points to a threshold amount 540 of the feature point.

[0123] In some implementations, the threshold amount of feature points 540 can be a predetermined amount or a relative amount of feature points. For example, in some embodiments, the image region replacement system determines whether each of the first set of feature points 538a and the second set of feature points 538b includes a predetermined threshold number of feature points. Conversely, in some embodiments, the image region replacement system determines whether the first set of feature points 538a includes a threshold number of feature points relative to the number of feature points in the second set of feature points 538b. Figure 5B For example, the image region replacement system determines that each of the first set of feature points 538a and the second set of feature points 538b satisfies a threshold amount 540 of feature points.

[0124] like Figure 5B As further shown, the image region replacement system creates matched feature points 542 by matching feature points in a first set of feature points 538a within the reference image 532 and a second set of feature points 538b within the target image 534. For example, in some embodiments, the image region replacement system matches feature points between the reference image 532 and the target image 534 based on a brute force matcher algorithm, a FLANN algorithm, or other suitable feature point matching algorithm.

[0125] After matching the feature points, in some embodiments, the image region replacement system filters the feature points based on the gyroscope data to create a set of filtered matched feature points 546. For example, in some cases, the image region replacement system generates gyroscope alignment parameters that align the reference image 532 with the target image 534 using the method described above.

[0126] The image region replacement system also compares the matching feature points 542 to the gyroscopic alignment parameters and determines whether any matching feature points 542 are inconsistent with the gyroscopic alignment parameters. For example, the image region replacement system can apply the gyroscopic alignment parameters and determine that two "matched" feature points are on opposite sides of a scene depicted in the digital image. Specifically, the image region replacement system can compare the gyroscopic alignment parameters to the alignment indicated by the matching feature points. If the two alignments differ by a threshold amount, the image region replacement system can determine that the matching feature points are incorrect. Therefore, the image region replacement system can use the gyroscopic alignment parameters to determine that the matching feature points are not actually matched.

[0127] After determining the inconsistency between the matching feature points 542 and the gyroscope alignment parameters, the image region replacement system then removes any conflicting matching feature points from the matching feature points 542. After removing or filtering out the conflicting matching feature points, the image region replacement system identifies the remaining matching feature points as a set of filtered matching feature points 546.

[0128] In addition to creating a set of filtered matching feature points 546, in some embodiments, the image region replacement system also determines a feature point deficiency metric based on the set of filtered matching feature points 546. Figure 5B As shown, the image region replacement system determines the feature point lack metric by determining a plurality of matching feature points within the set of filtered matching feature points 546. In some embodiments, the image region replacement system may also determine the feature point lack metric by determining a plurality of matching feature points within the matching feature points 542.

[0129] like Figure 5B As shown, the image region replacement system also compares the number of matching feature points within the set of filtered matching feature points 546 (or alternatively, the number of matching feature points within the matching feature points 542) to a threshold amount 548 of matching feature points. For example, in some embodiments, the image region replacement system determines whether the set of filtered matching feature points 546 includes a predetermined threshold number of matching feature points. Additionally or alternatively, in some embodiments, the image region replacement system determines whether the set of filtered matching feature points 546 includes a threshold number of matching feature points given the number of pixels or size of the reference image 532 and the target image 534. Figure 5B, the image region replacement system determines that the filtered set of matching feature points 546 (or alternatively, matching feature points 542 ) satisfies a threshold amount 548 of matching feature points.

[0130] like Figure 5B As further shown, the image region replacement system estimates feature alignment parameters 550. As suggested by the feature-based alignment model 536, the image region replacement system estimates the feature alignment parameters 550 based on the set of filtered matching feature points 546 or alternatively based on the matching feature points 542. As described above, the image region replacement system can apply a RANSAC algorithm, a LMedS regression algorithm, or other suitable feature-based image alignment algorithm to the set of filtered matching feature points 546 (or matching feature points 542) to generate the feature alignment parameters 550.

[0131] As described above, the feature alignment parameters 550 can take the form of homography transformation parameters or affine transformation parameters. In some embodiments, the image region replacement system determines whether the set of filtered matching feature points 546 (or matching feature points 542) includes a threshold amount of feature points to create homography transformation parameters. If the set of filtered matching feature points 546 (or matching feature points 542) includes the threshold amount, the image region replacement system estimates the homography transformation parameters as the feature alignment parameters 550. Conversely, if the set of filtered matching feature points 546 (or matching feature points 542) does not include the threshold amount, the image region replacement system estimates the affine transformation parameters as the feature alignment parameters 550.

[0132] In addition to estimating the feature alignment parameters 550, in some embodiments, the image region replacement system determines a feature point lack metric based on the feature alignment parameters 550. Figure 5B As shown, in some implementations, the image region replacement system determines the feature point lack metric by determining a difference between the feature alignment parameters 550 and a range of expected feature alignment parameters for the target image 534. In some such embodiments, the image region replacement system determines a difference between the feature alignment parameters 550 and a range that includes a maximum and a minimum expected feature alignment parameters for the target image 534 relative to the reference image 532 based on a gyroscope data set or gyroscope alignment parameters.

[0133] As described above, in some cases, the image region replacement system compares the feature alignment parameters 550 (or the difference in the feature alignment parameters 550) to the gyroscope data set or gyroscope alignment parameters corresponding to the reference image 532 and the target image 534. Therefore, the image region replacement system can determine whether the feature alignment parameters 550 reasonably align the target image 534 with the reference image 532 based on the comparison (or reference) to the gyroscope data set or gyroscope alignment parameters. For example, in some embodiments, the image region replacement system determines whether the difference in the feature alignment parameters 550 is equal to, exceeds, or is below an alignment threshold that indicates a percentage or number of values ​​within the feature alignment parameters 550 that may differ from the expected feature alignment parameters indicated by the gyroscope data set or gyroscope alignment parameters. Figure 5B As shown, the image region replacement system determines that the feature alignment parameters 550 reasonably align the target image 534 with the reference image 532 based on a comparison (or reference) to a gyroscope data set or gyroscope alignment parameters (e.g., by determining that the feature alignment parameters 550 or the difference in the feature alignment parameters 550 satisfies an alignment threshold).

[0134] like Figure 5B As further shown, after generating the feature alignment parameters 550, the image region replacement system applies the feature alignment parameters 550 to the target image 534 to generate an aligned input image 552. By applying the feature alignment parameters 550 to the target image 534, the image region replacement system shifts or warps the target image 534 to align with the reference image 532. For example, in some embodiments, the image region replacement system warps the target image 534 according to a homography (or affine transformation) encoded by the feature alignment parameters 550 to generate the aligned input image 552.

[0135] Although this disclosure primarily relates to feature-based alignment models, Figure 5B The feature-based alignment model 536 in (and embodiments of feature-based alignment models described elsewhere in this disclosure) may likewise be referred to as a “guided feature-based alignment model.” Unlike conventional keypoint-based alignment, in some embodiments, the image region replacement system determines one or more feature point lack metrics and compares such feature point lack metrics to one or more lack thresholds to guide implementation of the feature-based alignment model 536. Furthermore, unlike conventional keypoint-based alignment, in some implementations, the image region replacement system removes conflicting matching feature points to create a set of filtered matching feature points to further guide implementation of the feature-based alignment model 536.

[0136] As described above, in some embodiments, the image region replacement system selects between a pixel-adjusted gyroscope alignment model and a feature-based alignment model to align one or more target images in a set with a reference image. More specifically, the image region replacement system may select between a set of target images I = {I i ; i∈{1,...,N}} and the reference image I r Align, where r∈{1,...,N}. The set of target images I is aligned with the reference images I by using a pixel-adjusted gyroscope alignment model or a feature-based alignment model. r Alignment, image region replacement system creates a set of aligned images

[0137] In some embodiments, the image region replacement system may implement the high-level algorithm shown in Table 1 below to select and apply an image alignment model from a pixel-adjusted gyroscope alignment model or a feature-based comparison model to align the set of target images I with the reference images I r As described above, the image region replacement system can read and store the gyroscope data corresponding to each input image at the capture time. In the following algorithm, each target image from the set of input images I corresponds to the gyroscope data set matrix G i , where G = {G i ; i∈{1,...,N}}. In addition, the image region replacement system extracts a set of feature points F from each target image of the set of input targets I i , where F = {F i ; i∈{1,...,N}}.

[0138] Table 1

[0139]

[0140]

[0141]

[0142] As shown in the aforementioned algorithm, the image region replacement system starts with feature-based alignment, extracts feature points using a fast algorithm (e.g., ORB keypoint detection), and continues to match feature points to form candidate matches using robust alignment parameter estimation (e.g., RANSAC or LMedS). In the presence of foreground motion, many feature points may correspond to foreground objects and therefore mislead the background alignment. Therefore, the image region replacement system uses information from the gyroscope to predict the direction of camera motion and correct the matching feature points. At each step along the path, if any failure is detected, the image region replacement system falls back to alignment based on gyroscope data (and pixel alignment) rather than feature points. As described above, fault detection may include, but is not limited to: not enough feature points are detected; not enough feature point matches after gyroscope-based correction; or unreasonable alignment parameters based on the scene.

[0143] For gyroscope (and pixel) based alignment, the image region replacement system coarsely aligns the input image with the reference image by computing the relative rotation between the input image and the reference image using the gyroscope dataset G. This step utilizes the focal length of the camera. The image region replacement system can estimate the focal length from matching feature points (if there are enough feature points and there is some camera pan between the digital images), or the focal length can be programmatically queried from a mobile device. In addition, since the rotation data may only capture the camera pan (not translation), the image region replacement system performs a fast pixel-based translation estimation on the downsampled input image. The image region replacement system then combines the translation and rotation transformations to form the final alignment parameters.

[0144] As described above, once the target image and the reference image are aligned, the image region replacement system can determine the replacement region. Figure 6A-6B A schematic representation of a process for identifying replacement regions in a reference image and a target image is shown. In particular, Fig. 6A An example replacement region boundary 602 is shown around a reference instance 606 of a person and a target instance 604 of a person. Figure 6B Elements of an image are shown that are considered by the image region replacement system when determining the boundaries 602 of the replacement region.

[0145] Fig. 6AAn example replacement region boundary 602 is shown around a reference instance 606 of a person and a target instance 604 of the person. In particular, the target instance 604 of the person includes a smiling face of a person with eyes open from a target image. The source instance 606 of the person to be replaced includes an image of a person blinking from a reference image. In addition, as shown, the replacement region boundary 602 includes an irregular shape. As will be discussed in further detail below, the image region replacement system generates the replacement region boundary 602 by using a graph cut algorithm that is constrained to pass through consistent pixels while avoiding structures in the reference image and the target image. Therefore, the irregularity of the replacement region boundary 602 allows the image region replacement system to create an enhanced image with a minimal number of artifacts and inconsistencies.

[0146] like Fig. 6A As shown, the image region replacement system identifies a replacement region boundary 602 that includes a reference instance 606 of a person that is offset and a target instance 604 of a person. Even after the image replacement system has aligned the target image relative to the reference image, the instance of the person may be offset. This may be due to the movement of the person when capturing the burst of images. Although the target instance 604 of the person and the reference instance 606 of the person are offset as shown, in at least another embodiment, the target instance 604 of the person and the reference instance 606 of the person are evenly matched or more closely matched.

[0147] Figure 6B Elements in an image that the image region replacement system considers to create a replacement region boundary 602 are shown. Figure 6B Including the above Fig. 6A The replacement region boundary 602, the target instance of a person 604 and the reference instance of a person 606 are discussed. In addition, Figure 6B Included are a reference neighborhood 608 around a reference instance 606 of a person and a target neighborhood 610 around a target instance 604 of a person. Figure 6B Also included are a first other face 612a and a second other face 612b (collectively referred to as "other faces 612") and image structures 614a-614c. Figure 6B Two other faces 612 and three image structures 614 are shown, but the region replacement system may generate the replacement region boundary 602 around any number of other faces and image structures.

[0148] The image region replacement system determines a reasonable area around the reference instance of the person and the target instance of the person in which the graph cut algorithm will generate a replacement region boundary. For the reference instance of the person 606 (S) and the target instance of the person 604 (T), the goal is to find a replacement region boundary (R) such that the replacement region boundary does not cross the reference instance of the person 606 or the target instance of the person 604. In addition, the image region replacement system can try to place the replacement region boundary 602 so that it does not cross S or T and does not even cross other faces (denoted as F s，k and F t，m ). As shown in the figure, Figure 6B The first other face portion 612a (F s，k ) and the second other face 612b (F t，m ). Because the reasonable area of ​​the replacement area boundary 602 avoids instances of faces and people, the facial appearance is not destroyed in any way after the pixel replacement. Figure 6B Only two other faces are shown, but the image region replacement system can identify any number of other faces and human instances.

[0149] To begin defining a reasonable area for the replacement region boundary 602, the image region replacement system defines a reference neighborhood 608 (N) around a reference instance 606 (S) of a person. S ), and define a target neighborhood 610 (N) around a target instance 604 (T) of a person T ). In at least one embodiment, reference neighborhood 608 and reference neighborhood 610 are simple extensions of the facial contour, or any arbitrarily defined neighborhood covering the corresponding face. Figure 8A-8E To further describe the method in detail, a user may define a reference neighborhood 608, a target neighborhood 610, or both.

[0150] like Figure 6B As shown, the image region replacement system uses the identified reference neighborhood 608 (N S ), target neighborhood 610 (N T ), a reference instance of a person 606 (S), a target instance of a person 604 (T), a first other face 612a (F s，k ) and the second other face 612b (F t，m ) to define the reasonable area. The reasonable area P is defined as:

[0151]

[0152] Once the image region replacement system determines a plausible region, the image region replacement system uses a graph cut algorithm to find a replacement region boundary within the plausible region 602. As used herein, the term "graph cut algorithm" is used to describe an algorithm for determining an optimal replacement region boundary in an image. In particular, the graph cut algorithm is constrained to identify a replacement region boundary that passes through consistent pixels while avoiding structures in the reference image and the target image.

[0153] The image region replacement system uses dynamic programming that selects the minimum cost path from one of the overlapping regions to another overlapping region (e.g., Figure 6B Overlap of the aligned target image and reference image shown). To do this, the image region replacement system places the patches so that they partially overlap each other, typically 4 to 8 pixels. The image region replacement system determines which patch (e.g., a patch from the reference image or the target image) contributes to the pixels at different positions in the overlapping region. The image region replacement system selects a path between pixels that minimizes the quality cost by selecting a path that passes through pixels where the reference and target patch characteristics (e.g., color) are similar. In particular, s and t can represent two adjacent pixel positions in the overlapping region, and A(s) and B(s) represent the pixel colors at position s in the old patch and the new patch, respectively. The image region replacement system defines the quality cost M between two adjacent pixels s and t copied from patches A and B, respectively, as:

[0154] M(s,t,A,B)=||A(s)-B(s)||+||A(t)-B(t)||

[0155] where ||·|| represents an appropriate norm. The image region replacement system calculates the quality cost M between all adjacent pixels in the overlapping region and cuts between pixels with the lowest quality cost M.

[0156] The image region replacement system uses an intelligent graph cut algorithm that uses a method similar to the method described above for intelligently generating a replacement region boundary 602 within a plausible region. The image region replacement system uses the graph cut algorithm to avoid image artifacts caused by cutting image structures 614a, 614b, and 614c by cutting the most consistent pixels. In particular, the image region replacement system can calculate a quality cost based on pixel frequency and consistency between adjacent pixels located within the plausible region, and use the graph cut algorithm to identify the optimal replacement region boundary 602. The graph cut algorithm can determine a quality cost M between two adjacent pixels s and t copied from a plausible region of a reference image A and a plausible region of a target image B. In particular, the image region replacement system utilizes the following cost function:

[0157]

[0158] Here, d represents the direction of the gradient, and is the same as the direction of the edge between s and t. and is the gradient along direction d in the reasonable area of ​​the reference image A and the reasonable area of ​​the target image B. M' penalizes the boundary passing through the high-frequency area more than the boundary passing through the low-frequency area.

[0159] "Frequency" in an image refers to the rate of change of intensity per pixel. A high frequency region in an image has a high rate of change of intensity per pixel. On the other hand, if changes occur over many pixels within a region, the region has a low frequency. For example, a video with instances of a person (including a target instance of a person 604, a reference instance of a person 606, first other faces 612a and 612b) in a clear sky background and image structures 614a, 614b, and 614c in the foreground

[0160] Figure 6B The image in may include high and low frequency regions. The clear sky background is quantified as a low frequency region because the changes (if any) span many pixels. On the other hand, the edges of the image structures 614a, 614b, and 614c represent high frequency regions because the pixels change rapidly over the short span of the edges of the image structures 614a, 614b, and 614c.

[0161] The image region replacement system uses a graph cut algorithm to identify the lowest quality cost M′ within a reasonable region to intelligently create a replacement region boundary 602 that will produce an enhanced image with the least possible image artifacts. In particular, Figure 6B As shown, the image region replacement system identifies the replacement region boundary 602 with the lowest quality cost M′. Therefore, the replacement region boundary 602 cuts through the most consistent pixels and does not intersect with the image structures 614a, 614b and 614c. In addition, the pixels within the replacement region boundary 608 are constrained to be from the target image containing the target instance 604 of the person. The pixels around the replacement region boundary 608 are constrained to be from the reference image containing the reference instance 606 of the person. Therefore, the image replacement system generates a new image by replacing the reference region with the target region. More specifically, the reference region includes the reference instance 606 of the person and the surrounding region defined by the replacement region boundary 602. The target region includes the target instance 610 of the person and the surrounding region defined by the replacement region boundary 602.

[0162] As described above, the image region replacement system replaces a reference instance of a person with a target instance of a person. Figure 7A-7E A series of example images are shown that illustrate the automatic replacement of a reference instance of a person with a target instance of a person in a fully automatic paradigm. In particular, Fig. 7AAn example reference image 702 with a reference instance 706 of a person and an example target image 704 with a target instance 708 of a person are shown. Figure 7B-7D Steps automatically taken by an image region replacement system in a fully automatic paradigm to identify replacement region boundaries 714 around a reference instance 706 of a person and a target instance 708 of a person are shown. Figure 7B An identified reference neighborhood 710 is shown. Figure 7C A target neighborhood 712 is shown around a target instance 708 identifying a person. Fig.7D A replacement region boundary 714 generated by a graph cut algorithm is shown. Fig. 7E An example enhanced image is shown where the reference region is replaced with the target region using a Gaussian mixture.

[0163] Fig. 7A A reference image 702 including a reference instance 706 of a person and a target image 704 including a target instance 708 of a person are shown. Fig. 7A , the image replacement system has rated each score in the reference image 702 and determined that the reference instance of the person 706 has a low score (e.g., the reference instance of the person 706 has eyes closed). In particular, the image region replacement system detects that the reference image I r All faces in , and denote the face as F r，k ; k∈{1,...,K}. For each face F r，k ; k∈{1,...,K}, image region replacement system assigned score S r，k ; k∈{1,...,K}. For r，k <δ) for each face F r，k , the image region replacement system is represented as T i，k Detect the image around (D r，k ), that is, the face of the person in the target image i corresponding to the face k in the reference image. The image region replacement system calculates T i，k ; i∈{1,...,N}; i≠r, and denote it as S i，k .

[0164] The image region replacement system selects a target image 704 from the plurality of images based on rating each of the instances of the person represented by the reference instance of the person 706 across all images. In particular, the image region replacement system uses the target face t=argmax i S i，k A target instance of a person (ie, a target face) is found. In at least one embodiment, the image region replacement system selects 708 the target instance of the person having the highest aesthetic rating.

[0165] The image region replacement system determines a reference neighborhood on the reference image 702 . Figure 7B The image region replacement system is shown automatically identifying a reference neighborhood 710 around a reference instance 706 of a person on a reference image 702 .

[0166] Figure 7C A target neighborhood 712 is shown automatically determining a person around a target instance 708 on the target image 704. Because the image region replacement system independently determines the reference neighborhood 710 and the target neighborhood 712, the reference neighborhood 710 and the target neighborhood 712 may occupy different regions of the reference image 702 and the target image 704, respectively.

[0167] Image region replacement system uses the above reference Figure 6A-6B The graph cut algorithm described above is used to identify the replacement region boundary 714. In particular, for face F r，k and the target face F in frame t t，k , the image region replacement system selects the region R k The replacement region R with maximum consistency across images r and t on the boundary of k .For example, Fig.7D The replacement region boundary 714 is shown to cut through reasonable regions and avoid intersections with structures within the image. The image region replacement system uses the replacement region on the reference image 702 to define a reference region. In particular, the reference region includes a replacement region boundary 714 around a reference instance 712 of a person. The image region replacement system also uses the replacement region on the target image 704 to define a target region. In particular, the target region includes a replacement region boundary 714 around a target instance 708 of a person. Therefore, Fig.7D The replacement region boundary 714 shown also defines a target region of the target image 704 .

[0168] Fig. 7E An enhanced image 716 is shown in which the image region replacement system has replaced the reference instance 706 of the person with the target instance 708 of the person. In particular, the image region replacement system replaces the reference region with the target region using a Gaussian mixture. In particular, in the reference image I r In the image region replacement system, the region R k (specified by a mask with values ​​0 and 1) replaced by I t The image region replacement system applies Gaussian smoothing with kernel size d and sigma σ to R k To obtain Next, the image replacement system will I r The alpha channel is set to Drawing I t , and then draw I on top of it r, to execute I t and I r alpha blending, and I r Thus, the enhanced image 716 will have the final intelligent mixing and matching result.

[0169] Figure 8A-8E An example of an interactive example for replacing a reference area with a target area presented to a user via a client computing device is shown. Specifically, Fig. 8A A client computing device 800 is shown having a display 802 (a touch screen in one or more embodiments) including a user interface 804. Fig. 8A As shown, user interface 804 is associated with an application. The application may be an embodiment of an image management system. User interface 804 includes selectable elements 806 (for interacting with a reference image) and 808 (for interacting with a target image).

[0170] like Fig. 8A As shown, the image region replacement system automatically selects a reference image having a reference instance 810 of a person for the user. In particular, the image region replacement system displays the reference image I to the user. r In at least one embodiment, the user selects a face to be replaced. In particular, the user can select a target image by tapping a face in reference image 806. In response, the image region replacement system can present multiple instances of the selected face to the user. The user can then select a target face.

[0171] In at least one other embodiment, the image replacement area automatically selects a target image having a target instance 812 of a person. In this way, the user does not need to search through multiple images captured by a camera or client device. In addition, the user continues to gain the benefit of taking multiple images to capture a common moment. In this way, the user is provided with a selected reference image having a reference instance 810 of a person and a selected target image having a target instance 812 of a person.

[0172] Figure 8B The selected reference image 806 is shown. The user can interact with the reference image 806 to identify the person to be replaced. In particular, Figure 8B As shown, the user clicks and selects a user-selected plausible region 814. The user can use the brush tool to interactively select plausible regions. For each face that the user wants to replace, the image region replacement system sets a default target frame t=0, sets the plausible region 814 selected by the user to blank, and updates t for each user interaction with the target frame selection slider. For each click of the user on the screen (the touched region is collectively referred to as R) or update in the target frame selection slider, the reference instance I of the person is updated.r In the image region replacement system, the region R (specified by a mask with values ​​0 and 1) is replaced by a human target instance I via a Gaussian mixture. t Pixels in .

[0173] In at least one embodiment, the image replacement system uses the user-selected plausible region 814 as a plausible region in which the graph cut algorithm automatically identifies the replacement region boundary. In at least one embodiment, the image region replacement system detects faces within the user-selected plausible region 814 and constrains the graph cut algorithm to avoid crossing any detected faces. The image region replacement system automatically identifies a reference region and a target region and replaces the reference region with the target region. In at least one embodiment, the graph cut algorithm is constrained to cross the boundary of the user-selected plausible region 814 as closely as possible while still crossing consistent pixels and avoiding image structure.

[0174] Based on the reasonable area 814 selected by the user, the image area replacement system returns a draft image 816 to the user, such as Figure 8C As shown. The draft image 816 is created using a Gaussian mixture. In particular, the image region replacement system applies Gaussian smoothing with a kernel size d and sigma σ to R to obtain R (s) The image area replacement system will r The alpha channel is set to R (s) , draw the target instance of person I t , and then draw the reference instance of the person I on top of it r , to I t and I r Perform alpha blending.

[0175] As shown, the image region replacement system has replaced the target portion 818 of the person corresponding to the user-selected plausible region 814. However, the draft image 816 includes an image artifact 822 resulting from the misalignment of the target portion 818 of the person with the reference portion 820 of the person. In some embodiments, the image region replacement system can automatically detect the image artifact 822 and notify the user that the draft image 816 contains the image artifact 822. In at least one embodiment, the image region replacement system presents only the draft image 816 to the user and provides the user with the option of adjusting the user-selected plausible region. In either case, the image region replacement system presents the user with the reference image 806 showing the user-selected plausible region 814.

[0176] The user may correct a user-selected reasonable area on the reference image 806 to eliminate inconsistencies or artifacts in the draft image. Fig.8DAs shown, the user selects a corrected user-selected plausible region 824 by extending the user-selected plausible region 814. As shown, the corrected user-selected plausible region 824 includes the region where the image artifact 822 is located. Based on the corrected user-selected plausible region 824, the image region replacement system again uses the graph cut algorithm to traverse the boundary of the user-selected plausible region 824 to generate a replacement region boundary.

[0177] like Fig. 8E As shown, the enhanced image 826 does not have image artifacts 822 like the draft image 816. This is because the user expanded the reasonable area selected by the user. In at least another embodiment, the image area replacement system can continue to present the user with options for adjusting the reasonable area selected by the user to eliminate the image artifacts that appear.

[0178] Now turn to Fig. 9 , additional details regarding the components and functionality of the image region replacement system 904 according to one or more embodiments will be provided. In particular, Fig. 9 A computing device 900 is shown implementing an image management system 902. Image management system 902 may include an image region replacement system 904. Image region replacement system 904 may include an image aligner 906, a face identifier 908, an aesthetic grade generator 910, a reference image identifier 912, a target image identifier 914, a replacement region manager 916, a new image generator 918, a neural network manager 920, and a storage manager 922. Storage manager 922 may include a trained neural network 924, an image buffer 926, and a memory 928.

[0179] As described above, the computing device 900 may include an image management system 902. The image management system 902 may access, identify, modify, revise, and / or provide digital visual media (such as images and videos). For example, the image management system 902 may include applications that allow for the capture, editing, and / or sharing of digital images or videos. For example, the image management system 902 may help a user capture, edit (add filters, masks, frames, etc.), and share digital images (e.g., via a social network or upload to a cloud-based repository). In one or more embodiments, the image management system 902 may include digital image editing applications or digital image capture applications, including but not limited to or “ADOBE”, “PHOTOSHOP”, “LIGHTROOM” and “PREMIERE” are registered trademarks or trademarks of Adobe Systems Incorporated in the United States and / or other countries. In any case, the image management system 902 can implement the image area replacement system 904.

[0180] The image region replacement system 904 may include multiple components as described above. As described in detail above, using multiple components, the image region replacement system 904 can mix and match regions of an image. By way of example, the image region replacement system 904 can generate multiple aesthetic ratings for multiple instances of a person in multiple images. The image region replacement system 904 identifies a reference image and a target image. Using a graph cut algorithm, the image region replacement system 904 identifies a replacement region having a boundary that passes through consistent pixels around a reference instance of a person and a target instance of a person in the reference image and the target image, respectively. The image region replacement system 904 generates a new image by replacing a reference region in the reference image corresponding to the replacement region with a target region in the target image corresponding to the replacement region.

[0181] The image region replacement system 904 includes an image aligner 906. The image aligner 906 aligns the multiple images relative to the reference image. For example, the image aligner 906 can use the multiple images and the corresponding gyroscope data sets to align the multiple images in the image burst.

[0182] The image region replacement system 904 also includes a face identifier 908. In particular, the face identifier 908 generally identifies instances of faces and persons. The face identifier identifies faces within a reference image. The face identifier 908 may utilize facial detection techniques to identify instances of faces or persons in the reference image. The face identifier 908 may also identify instances of persons associated with faces. For example, while the face identifier 908 may identify unique faces in the reference image and the target image, the face identifier 908 may also identify the body of a person corresponding to a face.

[0183] Image region replacement system 904 also includes an aesthetic rating generator 910. Aesthetic rating generator 910 can generate an aesthetic rating for an instance of a person, as described in detail above. For example, using one or more weight-adjusted attribute scores associated with the instance of the person, aesthetic rating generator 910 can generate an aesthetic score for the instance of the person.

[0184] The image region replacement system 904 includes a reference image identifier 912. The reference image identifier 912 designates an image as a reference image. For example, the reference image identifier 912 may select the image with the least blur, the image with the largest number of desired faces, or simply the first image in a set of photos.

[0185] The image region replacement system 904 also includes a target image identifier 914. The target image identifier 914 identifies the target image by identifying the image containing the instance of the person with the highest aesthetic rating. In particular, the target image identifier 914 can create a boundary region around the reference instance of the person and identify other instances of the person in the corresponding boundary region across multiple images. The target image identifier 914 can efficiently identify the instance of the person across multiple images instead of using facial detection technology to identify each face in each of the multiple images. The target image identifier 914 accesses the aesthetic rating of each instance of the person detected within the boundary region across multiple images. The target image identifier 914 designates the image of the instance of the person with the highest score as the target image.

[0186] The replacement region manager 916 is located on the image region replacement system 904 and identifies a replacement region that includes a reference instance of a person and a target instance. In addition, the replacement region manager 916 ensures that the boundary of the replacement region passes through consistent pixels and the target image by using a graph cut algorithm, which is constrained to pass through consistent pixels while avoiding structures in the reference image and the target image. More specifically, the replacement region manager determines a reasonable region based on the neighborhood around the reference instance of the person and the target instance of the person. The replacement region manager 916 also creates a replacement region boundary using a graph cut algorithm, which is constrained to pass through consistent pixels in the target image and the reference image. The replacement region manager 916 identifies a reference region that includes a replacement region boundary around the reference instance of the person. In addition, the replacement region manager 916 identifies a target region that includes a replacement region boundary around the target instance of the person.

[0187] The image region replacement system 904 further includes a new image generator 918. The new image generator 918 replaces pixels in the reference image including the reference region with pixels in the target image including the target region. More specifically, the new image generator 918 generates a new image using a Gaussian mixture.

[0188] In addition to the above components, the image region replacement system 904 may include a neural network manager 920. The neural network manager 920 may provide, manage, train and / or control (multiple) neural networks, as described above. For example, the neural network manager 920 applies the trained neural network to the digital image.

[0189] Image region replacement system 904 also includes a storage manager 922. Specifically, storage manager 922 may maintain data of any type, size, or kind required to perform the functions of image region replacement system 904. Although not shown, storage manager 922 may maintain other data including aesthetic ratings, attribute weight values, or other information discussed herein.

[0190] In addition, storage manager 922 may provide, manage, and / or control trained neural network 924, image buffer 926, and memory 928. For example, storage manager 922 determines when to store multiple images in an image burst in image buffer 926 and when to delete multiple images from image buffer 926. In addition, storage manager 922 maintains the trained neural network, including updating trained neural network 924 as directed by image region replacement system 904. In addition, storage manager 922 controls interaction with client device memory 928 (e.g., indicating where to store a selected image provided by image region replacement system 904 in client device memory).

[0191] Each of the components 904, 906, 908, 910, 912, 914, 916, 918, 920, and 922 ("components") of the image region replacement system 904 and their respective elements may communicate with each other using any suitable communication technology. It should be appreciated that although the components and their respective elements may be described in detail in detail, the image region replacement system 904 may be described in detail in detail. Fig. 9 Although shown as being separated, any of the multiple components and their respective elements may be combined into fewer components (such as into a single device or module), divided into more components, or configured into different components for use in a particular embodiment.

[0192] Multiple components and their corresponding elements may include software, hardware, or both. For example, multiple components and their corresponding elements may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices. Multiple components and their corresponding elements may include hardware, such as a dedicated processing device that performs a specific function or group of functions. Additionally or alternatively, multiple components and their corresponding elements may include a combination of computer-executable instructions and hardware.

[0193] In addition, the components of the image area replacement system 904 can be implemented, for example, as one or more independent applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that can be called by other applications, and / or as a cloud computing model. Therefore, the components of the image area replacement system 904 can be implemented as independent applications, such as desktop or mobile applications. In addition, the components of the image area replacement system 904 can be implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, the components of the image area replacement system 904 can be implemented in a set of mobile device applications or "apps".

[0194] The above-mentioned components for operating the image region replacement system can be implemented in the system environment. Specifically, Fig.10A schematic diagram of one embodiment of an exemplary system environment ("environment") 1000 in which the image region replacement system 904 may operate is shown. Fig.10 As shown, environment 1000 may include server(s) 1010, network 1012, and multiple client devices 1014a-1014n. Server(s) 1010, network 1012, multiple client devices 1014a-1014n, and other components of environment 1000 may be communicatively coupled to each other directly or indirectly (e.g., via network 1012).

[0195] like Fig.10 As shown, environment 1000 may include (multiple) servers 1010. (Multiple) servers 1010 may generate, store, receive and / or transmit any type of data. (Multiple) servers 1010 may include communication servers or web hosting servers. In one or more embodiments, (multiple) servers 1010 may include data servers.

[0196] In addition, if Fig.10 As shown, the environment 1000 may include multiple client devices 1014a-1014n. The multiple client devices 1014a-1014n may include a variety of different computing devices, such as personal computers, laptops, mobile devices, smart phones, tablet computers, dedicated computers, TVs, or other computing devices. Fig.10 As shown, multiple client devices 1014a - 1014n and / or server(s) 1010 may communicate via a network 1012 .

[0197] As shown, the image management system 902 and the image region replacement system 904 can be implemented by various components in the environment 1000. Specifically, the image management system 902 and the image region replacement system 904 can be implemented in whole or in part via (multiple) servers 1010 and / or multiple client devices 1014a-1014n. For example, the (multiple) servers 1010 can host a neural network for calculating an aesthetic grade, while the multiple client devices 1014a-1014n can implement an image replacement region selection process. When implemented partially on the (multiple) servers 1010 and partially on the multiple client devices 1014a-1014n, the image management system 902 and the image region replacement system 904 are communicatively coupled (i.e., via the network 1012).

[0198] The image management system 902 can manage the image region replacement system 904. For example, the image management system 902 can activate the image region replacement system 904, send and receive requests to and from the image region replacement system 904, and provide the selected image received from the image region replacement system 904 to the client device 1014.

[0199] In addition, the image region replacement system 904 may be as described above with respect to FIG. Fig. 9 Specifically, the image region replacement system 904 may generate a plurality of aesthetic ratings for a plurality of instances of a person in a plurality of images including a reference image having a reference instance of the person. The image region replacement system 904 may also identify a target instance of the person based on the plurality of aesthetic ratings. The image region replacement system 904 will determine a replacement region including the reference instance of the person and the target instance, the replacement region having a boundary passing through coincident pixels in the target image and the reference image, and the image region replacement system 904 will generate a new image by replacing the reference region in the reference image with the target region in the target image.

[0200] although Fig.10 A single server 1010 is shown, but it should be understood that server 1010 may represent any number of server computing devices. Fig.10 A particular arrangement of server(s) 1010, network 1012, and multiple client devices 1014a-1014n is shown, but various additional arrangements are possible.

[0201] In Figure 1- Fig.10 In the accompanying text and examples, many different systems and devices are provided that allow users to facilitate the generation of high-quality group portraits. In addition to the foregoing, embodiments may also be described in terms of a series of actions for achieving a particular result. For example, Fig.11 1100 is a flow chart showing a series of actions 1100 for image replacement region selection according to one or more embodiments. Fig.11 Actions according to one embodiment are shown, but alternative embodiments may omit, add to, reorder, and / or modify Fig.11 Any action shown. Fig.11 The actions of may be performed as part of a method. Alternatively, the non-transitory computer readable medium may include a program that, when executed by one or more processors, causes a computing device to perform Fig.11 In other embodiments, the system may execute Fig.11 action.

[0202] like Fig.11As shown, a series of actions 1100 may include an action 1110 of identifying a target instance. For example, action 1100 may include identifying a target instance of a person in a target image in a plurality of images. The plurality of images may include a plurality of instances of a person and a reference image having a reference instance of a person. In one or more embodiments, action 1110 includes receiving a user selection of a target instance of a person from a user of a computing device. For example, action 1110 may include receiving a user selection of a target instance of a person (e.g., receiving or detecting a click on touch screen 802 of computing device 800 in an area showing a face of a person).

[0203] Alternatively, action 1110 may include generating multiple aesthetic ratings for multiple instances of a person in multiple images. Then, action 1110 may include selecting the instance of the person with the highest aesthetic rating as the target instance. Action 1110 may include selecting the instance of the person automatically or without user input. Therefore, in one or more embodiments, action 1110 includes automatically identifying the highest quality instance of the face of the person from multiple images without displaying each image to the user or requiring the user to scan multiple images to identify the target instance. In one or more embodiments, action 1110 includes generating multiple aesthetic ratings using one or more neural networks 300, 230, the neural network 300, 230 having classifiers 304, 314a-314n, and the classifiers 304, 314a-314n are trained to generate attribute ratings for multiple instances of the person. In addition, action 1110 may include associating a weight with each attribute rating and combining the weighted attribute weights to generate the aesthetic rating.

[0204] The series of actions 1100 also includes an action 1120 of aligning the target image and the reference image. For example, action 1120 may include: identifying a first gyroscope data set corresponding to the reference image and a second gyroscope data set corresponding to the target image; detecting a first set of feature points within the reference image and a second set of feature points within the target image; and applying an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model based on the first gyroscope data set and the second gyroscope data set and one or both of the first and second sets of feature points to align the target image with the reference image. Alternatively, action 1120 may include: determining a feature point lack metric; selecting an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model based on the feature point lack; and applying the selected image alignment model to the target image and the reference image to determine alignment between the target image and the reference image. In addition, in one or more embodiments, action 1120 may also include superimposing the aligned target image on the reference image.

[0205] The series of actions 1100 includes an action 1130 of determining a replacement region. For example, action 1130 may include determining a replacement region that includes a reference instance of a person and a target instance, the replacement region having a boundary that passes through consistent pixels in the target image and the reference image. In addition, action 1130 may include using a graph cut algorithm that identifies a boundary from a reasonable boundary region by identifying pixels with maximum consistency and minimum passing through structured objects in the target image and the reference image. In particular, action 1130 may include identifying adjacent pixels within a reasonable region with a minimum quality cost. More particularly, action 1130 may include using a graph cut algorithm that is constrained to pass through consistent pixels while avoiding structures in the reference image and the target image. For example, action 1130 may include: more penalizing potential boundaries that pass through high-frequency regions compared to potential boundaries that pass through low-frequency regions.

[0206] In such an embodiment, action 1130 may include determining a reasonable boundary area by combining a reference neighborhood around a reference instance of the person and a target neighborhood around a target instance of the person, and wherein the reasonable area excludes instances of other persons. In particular, action 1130 may include: generating a union of the target neighborhood and the reference neighborhood, and subtracting the portions of the target neighborhood and the reference neighborhood that include the face. Subtracting the portions of the target neighborhood and the reference neighborhood that include the face may include: subtracting the face of the target instance of the person, the reference instance of the person, and the faces of other persons. Action 1130 may also include: generating a reference neighborhood by extending the outline of the reference instance of the person, and generating a target neighborhood by extending the outline of the target instance of the person. Therefore, in one or more embodiments, action 1130 includes determining a reasonable boundary area automatically or without user input.

[0207] In an alternative embodiment, action 1130 may include receiving a selection of a plausible area on the reference image from a user of the computing device. For example, action 1130 may include detecting that the user has clicked on and selected the user-selected plausible area 814. In another embodiment, action 1130 includes detecting the use of a brush tool activated by the user to interactively select the plausible area.

[0208] The series of actions 1100 includes an action 1140 of generating an enhanced image. For example, action 1140 may include: generating an enhanced image by replacing a reference region in the reference image corresponding to the replacement region with a target region in the target image corresponding to the replacement region. More specifically, action 1140 may include: replacing the reference region with pixels in the target region using Gaussian blending. For example, action 1140 may include: applying alpha blending on the target region and the reference region.

[0209] In addition to (or alternatively to) the above actions, in some embodiments, action 900 includes performing steps for aligning a plurality of images including a plurality of instances of a person, wherein the plurality of images includes a reference image having a reference instance of a person. Figure 4-5B The described algorithms and actions may include corresponding actions for performing steps for aligning multiple images including multiple instances of a person, wherein the multiple images include a reference image having a reference instance of a person. Figure 6A-6B The described algorithms and acts may also include corresponding acts for performing the step of determining a replacement region, the replacement region including a reference instance in the reference image and a target instance of a person, the boundary of the replacement region passing through coincident pixels in the target image and the reference image. Figure 3A-3B The described algorithms and acts may also include corresponding acts for the step of generating multiple aesthetic ratings for multiple instances of a person.

[0210] As used herein, the term "digital environment" generally refers to an environment implemented, for example, as a standalone application (e.g., a personal computer or mobile application running on a computing device), an element of an application, a plug-in for an application, (multiple) library functions, a computing device, and / or a cloud computing system.

[0211] Embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer including computer hardware (such as, for example, one or more processors and system memory), as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions, thereby performing one or more processes, including one or more processes described herein.

[0212] Computer-readable media can be any available media that can be accessed by a general or special purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct computer-readable media: a non-transitory computer-readable storage medium (device) and a transmission medium.

[0213] Non-transitory computer-readable storage media (devices) include: RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code devices in the form of computer-executable instructions or data structures and can be accessed by a general or special purpose computer.

[0214] "Network" is defined as one or more data links that enable electronic data to be transmitted between computer systems and / or modules and / or other electronic devices. When information is delivered or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer appropriately regards the connection as a transmission medium. The transmission medium may include a network and / or a data link, which may be used to carry desired program code means in the form of computer executable instructions or data structures and may be accessed by a general or special purpose computer. The above combination should also be included in the scope of computer readable media.

[0215] In addition, upon reaching various computer system components, program code means in the form of computer executable instructions or data structures may be automatically delivered from a transmission medium to a non-transitory computer readable storage medium (device) (or vice versa). For example, a computer executable instruction or data structure received over a network or data link may be buffered in RAM within a network interface module (e.g., "NIC") and then ultimately transferred to the computer system RAM and / or a less volatile computer storage medium (device) at the computer system. Thus, it should be understood that a non-transitory computer readable storage medium (device) may be included in a computer system component that also (or even primarily) utilizes a transmission medium.

[0216] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions when executed at a processor. In some embodiments, computer executable instructions are executed on a general-purpose computer to turn a general-purpose computer into a special-purpose computer that implements the elements of the present disclosure. Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly language, or even source code. Although the subject matter is described in a language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the above-mentioned features or actions. Instead, the described features and actions are disclosed as an example form of implementing the claims.

[0217] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be implemented in a distributed system environment, in which local and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or by a combination of hardwired and wireless data links) all perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0218] Embodiments of the present disclosure may also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be used in a marketplace to provide universal and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be rapidly provisioned via virtualization and published with low management effort or service provider interaction, and then scaled accordingly.

[0219] The cloud computing model may consist of various features, such as, for example, on-demand self-service, broad network access, resource pools, rapid elasticity, measured services, etc. The cloud computing model may also expose various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In this specification and claims, a "cloud computing environment" is an environment in which cloud computing is employed.

[0220] Fig.12 1 shows a block diagram of an exemplary computing device 1200 that can be configured to perform one or more of the above processes. It is understood that one or more computing devices (such as computing device 1200) can implement one or more components of the image region replacement system. Fig.12 As shown, computing device 1200 may include a processor 1202, memory 1204, storage device 1206, I / O interface 1208, and communication interface 1210, which may be communicatively coupled via a communication infrastructure 1212. Fig.12 An exemplary computing device 1200 is shown in FIG. Fig.12 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in some embodiments, the computing device 1200 may include more than Fig.12 The components shown are few and far between. We will now describe in more detail Fig.12 Components of computing device 1200 are shown.

[0221] In one or more embodiments, the processor 1202 includes hardware for executing instructions, such as instructions that constitute a computer program. By way of example and not limitation, to execute instructions, the processor 1202 may retrieve (or obtain) instructions from an internal register, an internal cache, memory 1204, or storage device 1206, and decode and execute the instructions. In one or more embodiments, the processor 1202 may include one or more internal caches for data, instructions, or addresses. By way of example and not limitation, the processor 1202 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 1204 or storage device 1206.

[0222] Memory 1204 may be used to store data, metadata, and programs for execution by processor(s). Memory 1204 may include one or more of volatile and non-volatile memory, such as random access memory ("RAM"), read-only memory ("ROM"), solid-state disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage devices. Memory 1204 may be internal or distributed memory.

[0223] Storage device 1206 includes a storage device for storing data or instructions. As an example, but not as a limitation, storage device 1206 may include the above-mentioned non-transitory storage medium. Storage device 1206 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, storage device 1206 may include a removable or non-removable (or fixed) medium. Storage device 1206 may be inside or outside computing device 1200. In one or more embodiments, storage device 1206 is a non-volatile solid-state memory. In other embodiments, storage device 1206 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0224] The I / O interface 1208 allows a user to provide input to the computing device 1200, receive output from the computing device 1200, and otherwise transmit data to and receive data from the computing device 1200. The I / O interface 1208 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of such I / O interfaces. The I / O interface 1208 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O interface 1208 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be used for a particular implementation.

[0225] The communication interface 1210 may include hardware, software, or both. In any case, the communication interface 1210 may provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device 1200 and one or more other computing devices or networks. By way of example and not limitation, the communication interface 1210 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network such as WI-FI.

[0226] Additionally or alternatively, the communication interface 1210 can facilitate communication with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more portions of one or more of these networks can be wired or wireless. As an example, the communication interface 1210 can facilitate communication with a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination thereof.

[0227] In addition, the communication interface 1210 can facilitate communication across various communication protocols. Examples of communication protocols that can be used include, but are not limited to: data transmission media, communication devices, transmission control protocol ("TCP"), Internet protocol ("IP"), file transfer protocol ("FTP"), Telnet, hypertext transfer protocol ("HTTP"), secure hypertext transfer protocol ("HTTPS"), session initiation protocol ("SIP"), simple object access protocol ("SOAP"), extensible markup language ("XML") and its variants, simple mail transfer protocol ("SMTP"), real-time transport protocol ("RTP"), user datagram protocol ("UDP"), global system for mobile communications ("GSM") technology, code division multiple access ("CDMA") technology, time division multiple access ("TDMA") technology, short message service ("SMS"), multimedia messaging service ("MMS"), radio frequency ("RF") signaling technology, long term evolution ("LTE") technology, wireless communication technology, in-band and out-of-band signaling technology, and other suitable communication networks and technologies.

[0228] The communication infrastructure 1212 may include hardware, software, or both that couples components of the computing device 1200 to each other. By way of example and not limitation, the communication infrastructure 1212 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a HyperTransport (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or other suitable busses or combinations thereof.

[0229] The foregoing description has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the present disclosure are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and the accompanying drawings are illustrative of the present disclosure and should not be construed as limiting. Many specific details are described to provide a thorough understanding of the various embodiments.

[0230] Additional or alternative embodiments may be implemented in other specific forms without departing from its spirit or essential features. The described embodiments should be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the preceding description. All changes falling within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause a computing device to: identifying a target instance of a person in a target image among a plurality of images, wherein the plurality of images include a plurality of instances of the person and a reference image having a reference instance of the person; aligning the target image and the reference image; determining a replacement region, the replacement region comprising the reference instance and the target instance of the person, the replacement region having a boundary passing through coincident pixels in the target image and the reference image; and An enhanced image is generated by replacing a reference region in the reference image corresponding to the replacement region with a target region in the target image corresponding to the replacement region.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: The replacement region is determined using a graph cut algorithm that identifies the boundary from a region of plausible boundaries by identifying pixels in the target image and the reference image that have maximum consistency and minimum passage across structured objects.

3. The non-transitory computer-readable medium of claim 2, further comprising instructions that, when executed by the at least one processor, cause the computing device to: The reasonable bounding region is determined by combining a reference neighborhood around the reference instance of the person and a target neighborhood around the target instance of the person, and wherein the reasonable bounding region does not include instances of other persons.

4. The non-transitory computer-readable medium of claim 3, further comprising instructions that, when executed by the at least one processor, cause the computing device to: generating the reference neighborhood by extending a contour of the reference instance of the person; and The target neighborhood is generated by extending a contour of the target instance of the person. 5 . The non-transitory computer-readable medium of claim 2 , wherein determining the reasonable bounding region comprises receiving a selection of the reasonable bounding region on the reference image from a user of the computing device. 6 . The non-transitory computer-readable medium of claim 1 , wherein identifying the target instance of the person comprises receiving a user selection of the target instance of the person from a user of the computing device.

7. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: generating a plurality of aesthetic ratings for the plurality of instances of the person in the plurality of images; and The instance of the person having the highest aesthetic rating is selected as the target instance.

8. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to align the target image and the reference image by: identifying a first gyroscope data set corresponding to the reference image and a second gyroscope data set corresponding to the target image; Detecting a first set of feature points in the reference image and a second set of feature points in the target image; as well as Based on the first gyroscope data set and the second gyroscope data set and one or two groups of feature points in the first group of feature points and the second group of feature points, an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model is applied to align the target image with the reference image.

9. A system operable to mix and match faces in a group digital portrait, the system comprising: Memory, including: a plurality of images including a plurality of instances of a person, wherein the plurality of images includes a reference image having a reference instance of the person; at least one processor; and A computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to: generating a plurality of aesthetic ratings for the plurality of instances of the person in the plurality of images; identifying the target instance from a target image in the plurality of images by determining that the target instance of the person has a highest aesthetic grade based on the plurality of aesthetic grades; aligning the target image and the reference image; determining a replacement region by using a graph cut algorithm, the replacement region comprising the reference instance and the target instance of the person, the graph cut algorithm being constrained to pass through consistent pixels while avoiding structure in the reference image and the target image; and An enhanced image is generated by replacing a reference region in the reference image corresponding to the replacement region with a target region in the target image corresponding to the replacement region.

10. The system of claim 9, wherein the instructions, when executed by the at least one processor, further cause the system to identify reasonable boundary regions to which the graph cut algorithm is to be applied by: defining a reference neighborhood around the reference instance of the person; defining a target neighborhood around the target instance of the person; generating a union of the target neighborhood and the reference neighborhood; as well as The portion of the target neighborhood and the reference neighborhood that includes the face is subtracted.

11. The system of claim 10, wherein subtracting the target neighborhood and the portion of the reference neighborhood that includes the face comprises: The face of the target instance of the person, the reference instance of the person, and the faces of other persons are subtracted.

12. The system of claim 9, wherein the instructions, when executed by the at least one processor, cause the system to generate the plurality of aesthetic ratings using one or more neural networks having classifiers trained to generate attribute ratings for the plurality of instances of the person.

13. The system of claim 12, wherein the instructions, when executed by the at least one processor, cause the system to associate a weight with each attribute level.

14. The system of claim 9, wherein the instructions, when executed by the at least one processor, cause the system to align the target image and the reference image by: There is a lack of metrics to determine feature points; selecting an image alignment model from a pixel-adjusted gyroscope alignment model and a feature-based alignment model based on the feature point deficiency metric; as well as The selected image alignment model is applied to the target image and the reference image to determine an alignment between the target image and the reference image.

15. The system of claim 10, wherein the instructions, when executed by the at least one processor, cause the system to: replace the reference region with pixels from the target region by using Gaussian mixture to generate a new image.

16. The system of claim 15, wherein using a Gaussian mixture comprises: Alpha blending is applied to the target area and the reference area.

17. The system of claim 10, wherein using the graph cut algorithm comprises: Potential boundaries passing through high frequency regions are penalized more than potential boundaries passing through low frequency regions, wherein the graph cut algorithm is constrained to pass through consistent pixels while avoiding structures in the reference image and the target image.

18. A method for capturing and manipulating digital images in a digital media environment, mixing and matching faces of group digital portraits, comprising: performing steps for aligning a plurality of images including a plurality of instances of a person, wherein the plurality of images includes a reference image having a reference instance of the person; identifying a target instance of the person on a target image among the plurality of images; performing a step for determining a replacement region having a boundary passing through coincident pixels in the target image and the reference image; as well as A new image is generated by replacing a reference area in the reference image corresponding to the replacement area with a target area in the target image corresponding to the replacement area.

19. The method of claim 18, further performing a step for generating a plurality of aesthetic ratings for the plurality of instances of the person.

20. The method of claim 19, further comprising using the aesthetic rating to select the target instance of the person.

Images