Modifying the appearance of hair

Abstract

According to various embodiments, a method of modifying the appearance of hair in an image of a subject's head is disclosed. The method includes providing an image of a subject's head having a hair region as input to a first trained neural network model; using the first trained neural network model and based on the image of the head, generating a hair layer and a face layer, the hair layer including an estimated representation of the portion of the image containing hair, the face layer including an estimated representation of the subject's head with the hair region removed; providing an indication of a defined hairstyle to be incorporated into the image and the generated hair layer as input to a second trained neural network model; using the second trained neural network model, based on the indication of the defined hairstyle and the generated hair layer, generating a modified hair layer; and using a processor, generating a modified image of the subject's head by applying the modified hair layer to the generated face layer.

Description

TECHNICAL FIELD

[0001] The present invention relates to modifying images, and in particular to modifying the appearance of hair in images of a subject's head. BACKGROUND

[0002] Personal care activities can form an important part of a person's day. For example, a human can spend time applying make-up or skin care products, and adjusting, restyling and cutting the hair on their head and / or on their face (e.g. facial hair). Typically, a person can look at their image in a mirror whilst performing a personal care activity to achieve a desired result from that activity.

[0003] When a person wishes to adjust, restyle or cut the hair on their head or face, it can be helpful to be able to visualise how a particular hairstyle can appear before implementing a physical change to their appearance. In existing systems, a user can superimpose an image of a particular facial hairstyle over their facial image so that they can envisage how they can look with that particular facial hairstyle. However, in such systems, the pixels in the image of the new hairstyle are simply placed on top of (e.g. on top of the pixels of) the user's image. Therefore, a user with existing hair on their head (i.e. facial hair or cranial hair) can not be able to view a realistic representation of their head with the modified hairstyle as it can not be clearly visible in view of their existing hair. It would therefore be desirable to be able to modify the appearance of hair in an image of a subject in a more realistic manner regardless of whether the subject has existing hair. There is therefore a need for an improved system for generating and viewing images of how they can appear with a particular hairstyle.

[0004] Yi Zhou et al in their paper "HairNet: Single-View Hair Reconstruction Using Convolutional Neural Networks" provide a method to generate complete 3D hair geometry from images. A convolutional neural network takes as input a 2D orientation field of a hair image and generates a strand feature uniformly distributed on a parameterized 2D scalp. SUMMARY

[0005] As indicated above, the limitations of the prior art for generating images of people with modified hairstyles is that the existing hair in the image can adversely affect the appearance of the new or modified hair. For example, for a user with a mustache, it can be difficult to use the prior art to generate a realistic image of the user with a goatee, as the goatee can not be clearly visible over the existing mustache. It has therefore been recognised that a system that is able to generate realistic images of users with modified hairstyles, regardless of the user’s existing hairstyle, would be of value. According to the embodiments disclosed herein, it is an object of the present invention to overcome the shortcomings of existing systems by creating a modified image of a user in which at least a portion of the user’s existing hair has been removed before a new hairstyle is added to the image.

[0006] According to a first aspect, there is provided a method of modifying the appearance of hair in an image of a head of a subject, the method comprising: providing an image of the head of the subject having a region of hair as an input to a first trained neural network model; generating, using the first trained neural network model and based on the image of the head, a hair layer and a face layer, the hair layer comprising an estimated representation of the portion of the image comprising hair, the face layer comprising an estimated representation of the head of the subject with the region of hair removed; providing an indication of a defined hairstyle to be incorporated into the image and the generated hair layer as an input to a second trained neural network model; generating, using the second trained neural network model, a modified hair layer based on the indication of the defined hairstyle and the generated hair layer; and generating, using a processor, a modified image of the head of the subject by applying the modified hair layer to the generated face layer.

[0007] By using trained neural network models to generate the face layer, a realistic estimated representation of the user’s head can be generated in which the existing hair has been removed. Similarly, by using trained neural network models to generate a hair layer to be applied to the generated face layer, the resulting image is realistic.

[0008] In some embodiments, the method can further comprise generating, using the first trained neural network model and based on the image of the head, a hair mask, the hair mask defining a region of the image containing hair.

[0009] In some embodiments, the method can further comprise receiving, at the processor, an annotation of the image of the head of the subject, the annotation comprising an indication of a region of the image containing hair; and calculating, using the processor, an accuracy of the generated hair mask relative to the received annotation.

[0010] In some embodiments, the method can further comprise providing the generated hair mask as an input to a second trained neural network model. Generating the modified hair layer can comprise generating the modified hair layer based on the generated hair mask such that the hairs in the modified hair layer are generated only within the region defined by the generated hair mask.

[0011] In some embodiments, the method can further comprise receiving, via a user interface, a user input to modify a parameter of at least one of: the generated hair layer, the indication defining the hairstyle, and the hair mask.

[0012] In some embodiments, at least one of the first trained neural network model and the second trained neural network model comprises, or forms part of, a generative adversarial network.

[0013] In some embodiments, the method can further comprise providing the modified image for presentation to a user.

[0014] In some embodiments, the method can further comprise using a discriminative network to assess a quality of the generated facial layer relative to an image of the subject’s head in which there are no visible hairs in the region.

[0015] In some embodiments, the method can further comprise using a discriminative network to assess a quality of the generated modified image of the subject’s head.

[0016] According to a second aspect, there is provided a method of training a neural network model to manipulate the appearance of hair in an image of a subject’s head, the method comprising: generating a training data set comprising a plurality of images of a subject’s head each having hair in a particular region, and a plurality of images of a subject’s head each having no hair in the particular region; and training the neural network model to generate, based on an input image of a subject’s head having hair in the particular region, an estimated representation of the subject’s head in which the hair in the particular region has been removed.

[0017] According to a third aspect, there is provided a method of training a neural network model to manipulate the appearance of hair in an image, the method comprising: generating a training data set comprising a plurality of hair layers and a plurality of indicators of a hairstyle to be incorporated into an image, each hair layer comprising an estimated representation of a hair-containing portion of an image; and training the neural network model to generate, based on an input hair layer and a particular indicator of a hairstyle, a modified hair layer.

[0018] In some embodiments, the method can further comprise providing, to the neural network model during said training, a hair mask and a noise element, the hair mask defining a range of the hair-containing portion of the input hair layer.

[0019] According to a fourth aspect, a computer program product is provided, comprising a computer-readable medium having computer-readable code embodied therein, the computer-readable code being configured to cause the computer or processor, when executed by a suitable computer or processor, to perform the steps of the methods disclosed herein.

[0020] According to a fifth aspect, an apparatus for modifying an image of a subject's head is provided, the apparatus comprising: a memory including instruction data representing a set of instructions; and a processor configured to communicate with the memory and execute the set of instructions, wherein, when executed by the processor, the processor causes the processor to: provide an image of a subject's head having hair regions as input to a first trained neural network model; generate a hair layer and a facial layer using the first trained neural network model and based on the image of the head, the hair layer including an estimated representation of the hair-containing portion of the image, and the facial layer including a representation of the subject's head with the hair regions removed; provide an indication of a defined hairstyle to be incorporated into the image and the generated hair layer as input to a second trained neural network model; generate a modified hair layer similar to the defined hairstyle using the second trained neural network model and based on the indication of the defined hairstyle and the generated hair layer; and generate a modified image of the subject's head by applying the modified hair layer to the generated facial layer.

[0021] In some embodiments, the device may further include a display for displaying a modified image of the subject's head.

[0022] These and other aspects of the invention will become apparent and will be illustrated with reference to the embodiments described below. Attached Figure Description

[0023] To better understand the invention and to more clearly illustrate how it can be implemented, reference will now be made to the accompanying drawings by way of example only, wherein:

[0024] Figure 1 This is a flowchart illustrating an example of a method for modifying the appearance of hair in an image according to various embodiments;

[0025] Figure 2 These are illustrations of multiple hairstyle examples;

[0026] Figure 3 This is a flowchart of another example of a method for modifying the appearance of hair in an image according to various embodiments;

[0027] Figure 4 This is a flowchart of yet another example of a method for modifying the appearance of hair in an image according to various embodiments;

[0028] Figure 5 is a flowchart of one example of a method of training a neural network to manipulate the appearance of hair in an image in accordance with various embodiments;

[0029] Figure 6 is a flowchart of another example of a method of training a neural network to manipulate the appearance of hair in an image in accordance with various embodiments;

[0030] Figure 7 is a schematic diagram of one example of a computer-readable medium in communication with a processor; and

[0031] Figure 8 is a schematic diagram of an apparatus for modifying an image of a subject’s head in accordance with various embodiments. DETAILED DESCRIPTION

[0032] As noted above, improved methods for modifying the appearance of hair in an image are provided, as are apparatus for performing these methods. More specifically, methods are provided that utilize a trained classifier, such as a neural network model, to generate an image of a subject in which some of the subject’s hair has been removed, and to generate another image of the subject with a new hairstyle or modified hairstyle.

[0033] As used herein, the term “hairstyle” is intended to include the arrangement or appearance of hair on the top, sides, or back of a subject’s head (i.e., scalp hair) or on a user’s face, including eyebrows, sideburns, upper lip hair, and a beard (i.e., facial hair). Thus, while examples described herein are in the context of facial hair, the present invention is equally applicable to hair located on other parts of a subject, such as scalp hair.

[0034] As used herein, the term“trained classifier” or“trained neural network” is intended to include any type of model or algorithm trained using machine learning techniques. One example of a type of trained classifier is a trained artificial neural network model. Artificial neural networks (or simply neural networks) will be familiar to those skilled in the art, but briefly, a neural network is a type of model that can be used to classify data (e.g. classify or identify the contents of image data) or to predict or estimate a result given some input data. The structure of a neural network is inspired by the human brain. A neural network is composed of layers, each layer comprising a plurality of neurons. Each neuron comprises a mathematical operation. In the process of classifying a portion of data, the mathematical operation of each neuron is performed on that portion of data to produce a numerical output, and the output of each layer in the neural network is fed in turn to the next layer. Typically, the mathematical operation associated with each neuron comprises one or more weights that are tuned during a training process (e.g. the weight values are updated during the training process to tune the model to produce more accurate classifications).

[0035] For example, in a neural network model used to generate a portion of an image (e.g. a portion of an image of a subject’s head in which a portion of hair has been removed), each neuron in the neural network can comprise a mathematical operation comprising a weighted linear sum of pixel (or, in the three-dimensional case, voxel) values in the image, followed by a non-linear transformation. Examples of non-linear transformations used for neural networks include sigmoid functions, hyperbolic tangent functions, and rectified linear functions. The neurons in each layer of a neural network typically comprise different weighted combinations of a single type of transformation (e.g. the same type of transformation, sigmoid, etc. but with different weights). As will be familiar to the skilled person, in some layers, each neuron can apply the same weight in the linear sum; this applies, for example, in the case of convolutional layers. The weights associated with each neuron can cause certain features to be more prominent (or, conversely, less prominent) in the classification process, and so, adjusting the weights of the neurons during a training process trains the neural network to attribute increased importance to particular features when generating images. Typically, a neural network can have weights associated with neurons and / or weights between neurons (e.g. that modify the data values passed between neurons).

[0036] As briefly noted above, in some neural networks, such as convolutional neural networks, lower layers in the neural network (i.e. layers towards the beginning of the series of layers in the neural network), such as input layers or hidden layers, are activated by small features or patterns in the data portion (i.e. their output depends on small features or patterns in the data portion), while higher layers (i.e. layers towards the end of the series of layers in the neural network) are activated by larger and larger features in the data portion.

[0037] Generally, the neural network model can comprise a feed-forward model (such as a convolutional neural network, an auto-encoder neural network model, a probabilistic neural network model, and a time-delay neural network model), a radial basis function network model, a recurrent neural network model (such as a fully recurrent model, a Hopfield model, or a Boltzmann machine model), or any other type of neural network model comprising weights.

[0038] According to a first aspect, the present invention provides a method of modifying the appearance of hair in an image. Figure 1 is a flowchart of one example of a method 100 for modifying the appearance of hair in an image of a subject's head. The method 100 comprises, at step 102, providing an image of a subject's head having a hair region as input to a first trained neural network model. The image can be a previously acquired image that has been stored in a memory or storage medium (e.g. in a database) and subsequently retrieved and provided to the neural network model. Alternatively, the image can be acquired in real-time by an image acquisition device such as a camera or any other type of image capturing device. For example, the image can be captured using a camera device associated with a smartphone, a tablet computer, a laptop computer, a wearable device (e.g. a smartwatch) or an interactive mirror (also known as a smart mirror). An interactive mirror is a unit that, in addition to functioning as a mirror showing a user their image, is also capable of displaying information to the user. Information such as text, images and videos can be displayed on a display portion of the interactive mirror, which can for example be located behind a mirror (or partial mirror) panel or mirror (or partial mirror) surface. In this way, the display screen or portion thereof can be visible through the mirror portion, enabling the user to view their image and the information presented on the display screen simultaneously.

[0039] The "hair region" of the user can comprise one or more hair regions on the user's head, including skull hair and / or facial hair. While the user can have both skull hair and facial hair, the "region" of hair can be considered to be the portion of the image in which the hair is to be digitally removed and replaced with a representation of the hair in the new hairstyle. Thus, in some embodiments, the user can define in the image the portion of their hair that is to be modified using the claimed method.

[0040] At step 104, method 100 includes: using a first trained neural network model and based on an image of the head, generating a hair layer (H) and a face layer (F), the hair layer (H) comprising an estimated representation of the hair-covered portion of the image, and the face layer (F) comprising an estimated representation of the subject's head with the hair-covered region removed. In some embodiments, the hair layer may be considered to comprise an estimated representation of the hair-covered region of the subject's head. Thus, the first neural network is trained to acquire an image of a head with a hair-covered region (e.g., a beard) and generate an image of a head without that hair-covered region as output. In other words, the trained neural network predicts what the subject's head will look like without the hair-covered region. Details of how the neural network is trained to perform this task are provided below. However, in general, the first neural network is trained using training data comprising multiple images of heads of people with and without skull and facial hair. During the training process, the neural network learns to recognize features and the associations between features that allow the neural network to predict what the skin of the subject's head might look like beneath the hair-covered region. The neural network model then generates a representation of that part of the subject's head without hair (e.g., replacing the pixels of the hairy areas in the image), which is referred to in this paper as the "facial layer" (F).

[0041] In addition to the facial layer (F), the first neural network also estimates the hair-containing portions of the image, referred to as the hair layer (H). In some embodiments, the first neural network may estimate all hair-containing regions of the image (e.g., both cranial hair and facial hair regions), while in other embodiments, the first neural network may estimate only the regions containing cranial hair, or only the regions containing facial hair. For example, a user may provide instructions on which hair-containing portions (e.g., cranial or facial hair) should be included in the hair layer (H) generated by the first neural network.

[0042] Method 100 includes: at step 106, providing an indication of a defined hairstyle to be incorporated into an image and a generated hair layer (H) as input to a second trained neural network model. The hair layer (H) is the hair layer generated by the first trained neural network in step 104. The indication of the defined hairstyle may include an indication selected by the user (e.g., a selection of a desired hairstyle made by the user via a user interface) or one of a plurality of predetermined hairstyles. For example, in an example where a user wants to see how they would look with different facial hair, the defined hairstyle may include Figure 2 One of the defined facial hairstyles shown. Figure 2In particular embodiments, the set of facial hair styles includes: full goatee 202, goatee and mustache 204, anchor mustache 206, extended goatee 208, goatee with chin strap 210, V-shaped mustache with goatee 212, V-shaped mustache with chin hair 214, and "Norwegian Captain" style 216. In other examples, other styles can be included in the plurality of styles from which the defined style can be selected.

[0043] The second neural network is trained to alter or transform the layer of hair generated by the first neural network so that it more closely resembles the defined hair style. Thus, at step 108, the method 100 includes generating a modified layer of hair (H2) using the second trained neural network model based on the indication of the defined hair style and the generated layer of hair. For example, if the image provided to the first neural network includes a chin curtain, the layer of hair generated by the first neural network will include a chin curtain. If the defined hair style includes an extended goatee 208, the modified layer of hair (H2) will include a smaller region of hair because an extended goatee includes fewer hairs than a chin curtain.

[0044] Details of how the second neural network is trained to generate the modified layer of hair (H2) are provided below. In general, however, the second neural network is trained using training data that includes a plurality of images of heads of people having different skull styles and / or facial hair styles. During the training process, the neural network learns to recognize features and associations between features that allow the neural network to generate a modified layer of hair (H2) based on a defined (e.g., desired) hair style and the subject's existing hair (i.e., as defined by the layer of hair). For example, the second neural network can be trained to generate a modified layer of hair (H2) by predicting how the subject's existing hair would look if it were shaved or trimmed to the defined hair style. The neural network model then generates a representation of new regions of hair (e.g., pixels added to the generated facial layer) based on the defined hair style, which is referred to herein as the "modified layer of hair" (H2).

[0045] At step 110, the method 100 includes generating a modified image of the subject's head using the processor by applying the modified layer of hair (H2) to the generated facial layer (F). Thus, the modified layer of hair (H2) is combined (e.g., by superimposition) with the facial layer (F) generated at step 104 to create an image of the subject in which the existing hair regions have been replaced with the new hair style.

[0046] In some embodiments, the method 100 can also include providing the modified image for presentation to the user. For example, the processor executing the method 100 can generate (e.g., render) the modified image in a form that can be displayed on a display screen (such as the screen of a computing device, smart phone, tablet computer, or desktop computer), interactive / smart mirror, etc. In this way, the user can see what they will look like with the modified hairstyle and the existing hair region removed.

[0047] Figure 3 is a flowchart of another example of a method 300 of modifying the appearance of hair in an image. The method 300 includes some of the steps of the method 100 discussed above. An image 302 of a head of a subject having a region of hair 304 (e.g., a beard) is provided (step 102) as input to a first trained neural network 306. In this example, the first neural network 306 includes an encoder-decoder neural network model; however, in other examples, other types of neural network models can be used. Figure 3 The encoder-decoder neural network model in the illustrated example includes an encoder network 308 and one or more decoder networks 310. The first neural network model 306 generates (step 104) a face layer (F) 312 and a hair layer (H) 314 as its output.

[0048] In some embodiments, the methods 100, 300 can also include, at step 316, generating a hair mask (M) 318 using the first trained neural network model 306 and based on the image 302 of the head, the hair mask (M) 318 defining a region of the image that contains hair. As discussed in more detail below, the second trained neural network can use the hair mask (M) 318 to ensure that hair (in the modified hair layer (M2)) is not added where there is no hair in the original image 302. In this way, for example, if the subject is generating the modified image during a shaving activity, only those hairstyles that can be achieved from the subject’s existing hair will be generated.

[0049] In some embodiments, an annotated image of a subject’s head having a hair region can be used by a processor performing the methods 100, 300. For example, the annotated image 320 can include an image with an indication of the regions of the image that contain hair. In some embodiments, this indication can be made by a human operator who manually indicates (e.g., by marking a contour) the regions of the image where hair is present. In other embodiments, this indication can be made by a computer that indicates, pixel by pixel, which regions of the image include hair. In some examples, a separate neural network can be used to generate the segmentation annotation, which is trained to estimate the hairstyle annotation (i.e., to estimate the contour of the hair region). Thus, according to some embodiments, the methods 100, 300 can further include, at step 322, receiving, at the processor, an annotation 320 of an image of the subject’s head, the annotation including an indication of the regions of the image 302 that contain hair. For example, the annotation 320 can be provided as an input to the first neural network 306, and the first neural network will attempt to generate a hair mask (M) 318 that matches or closely resembles the annotation. This can be done by comparing the annotation 320 to the hair mask 318 at each pixel point and determining the degree of overlap. The neural network learns to maximize the overlapping portion. In some embodiments, the annotation 320 can be used to calculate the accuracy of the generated hair mask (M) 318 in the form of a segmentation loss or a patch loss. Thus, the methods 100, 300 can include, at step 322, calculating, using the processor, an accuracy 324 of the generated hair mask (M) 318 relative to the received annotation 320. The accuracy can be referred to as a segmentation loss.

[0050] Figure 3 Block 328 represents how a more accurate representation of the subject’s head with the hair region removed can be generated according to some embodiments. While the first trained neural network generates an estimate of how the subject’s skin would look if the hair region were removed (i.e., the facial layer (F)), the regions of the head outside of the hair region can include artifacts introduced by the processing performed by the network. Thus, since a representation of those parts of the subject’s head that fall outside of the hair region can be obtained from the original image 302 provided to the first trained neural network model 306, block 328 illustrates how the original input image can be used for those parts of the image that are outside of the hair region. According to the expression F*M + (1-M)*FH in block 328, for any region of the head that falls within the hair mask (M) 318, the generated facial layer (F) is used in the estimated representation, while for any region of the head that falls outside of the hair mask (M) 318, the original image 302 is used in the estimated representation. Thus, the output of block 328 is a more accurate representation of the subject’s head with the hair region removed, as compared to the facial layer (F) generated by the first neural network.

[0051] According to some embodiments, during training of the first neural network model, the quality of the generated facial layer (F) is evaluated or assessed. Thus, in some embodiments, the method 100, 300 can comprise the step of assessing the quality of the generated facial layer using a discriminative network relative to an image of the subject’s head without the region of visible hair. In some embodiments, this can be done using a discriminative network that forms part of a generative adversarial network. A generative adversarial network (GAN) is a type of machine learning architecture in which neural networks compete with each other. A generative network generates a candidate, while a discriminative network assesses the candidate. According to the present disclosure, the first neural network 306 can be considered the generative network as it generates a candidate facial layer (F) to be assessed by the discriminative network 330. The discriminative network 330 can be a neural network trained to discriminate between the generated facial layer (F) and an image 332 of the subject’s face without the region of hair. If the discriminative network 330 is able to tell that the generated facial layer (F) is a computer generated image, rather than an actual image of the subject without the region of hair, then it can be determined that the quality of the generated facial layer (F) is below a desired threshold (i.e. does not reach an expected level of confidence) and the first neural network 306 (i.e. the generative network) can need to generate a revised facial layer (F) to be re-evaluated. However, if the discriminative network 330 is not able to determine which of the generated face (F) and the image 332 is computer generated, then it can be determined that the quality of the generated facial layer (F) is above a desired threshold, such that the generated facial layer F is accepted (i.e. reaches an expected level of confidence). In some embodiments, the measure by which the discriminative network 330 assesses the quality of the generated facial layer F can be referred to as a discriminator loss.

[0052] Figure 3block 334 represents how the facial layer (F) 312 and the hair layer (H) 314 can be combined to attempt to obtain the original input image 302, according to some embodiments. Thus, according to the expression H*M + (1-M)*F in block 334, for any region of the head that falls within the hair mask (M) 318, the generated hair layer (H) is applied, while for any region of the head that falls outside the hair mask, the facial layer (F) 312 is applied. During training of the first neural network model 306, the reconstruction of the original image obtained by combining the hair layer (H) 314 and the facial layer (F) 312 can be evaluated or assessed to establish the accuracy of the facial layer and the hair layer generated by the first neural network 306. Thus, in some embodiments, the method 100, 300 can include the step of evaluating the quality of the generated facial layer and the beard layer using a discriminative network. In some embodiments, this can be done using a discriminative network that forms part of a generative adversarial network. In this example, because the first trained neural network 306 generates a candidate facial layer and a hair layer, it can be considered a generative network, and the combination of the candidate facial layer and the hair layer can be evaluated by a discriminative network 336. The discriminative network 336 can discriminate between the original input image 302 and the reconstruction of the image of the subject’s head formed using the expression of block 334. If the discriminative network 336 is able to determine that one of the images is not the original input image 302, the first neural network 306 can need to be generated to generate a revised facial layer and hair layer. Otherwise, if the discriminative network 336 is not able to determine that one of the images is not the original input image 302, the generated facial layer and hair layer can be accepted. In some embodiments, the measure by which the discriminative network 336 evaluates the quality of the hair layer and the facial layer can be referred to as a reconstruction loss.

[0053] Figure 4 is a flowchart of yet another example of a method 400 of modifying the appearance of the hair of an image. The method 400 includes some of the steps of the method 100 discussed above. The hair layer (H) 314 generated by the first trained neural network model 306 is provided (step 106) as a first input to a second trained neural network model 402, in addition, a hairstyle indication 404 is provided (step 106) as a second input to the second trained neural network model 402. The hairstyle indication 404 includes an indication of a defined hairstyle to be incorporated into the image. As discussed above, in some embodiments, the defined hairstyle can include one of a plurality of standard hairstyles, such as Figure 2The defined hairstyle can include a hairstyle defined by the user or a hairstyle selected by the user. In one example, the indication 404 can be provided in the form of a hairstyle image that the user wants to incorporate into their head image. In other examples, the user can select the defined hairstyle from a list and provide the indication 404 via a user interface.

[0054] Based on the input (i.e. the layer of hair (H) 314 and the defined hairstyle indication 404), the second trained neural network 402 is configured to generate (step 108) a modified layer of hair (H2) 406. The modified layer of hair 406 includes a representation of the area of hair 304 from the original image 302 that has been manipulated or adjusted to more closely match or resemble the defined hairstyle 404.

[0055] In some embodiments, the method 100, 300, 400 can further comprise the step of providing 408 the generated hair mask (M) 318 as input to the second trained neural network model 402. In such embodiments, the step of generating a modified layer of hair 108 can comprise generating a modified layer of hair based on the generated hair mask (M) 318 such that the hair in the modified layer of hair is only generated within the area defined by the generated hair mask. In this way, the functionality of the second trained neural network model can be constrained to only generate those hairstyles that will“fit” within the area defined by the hair mask. That way, the user can choose to only see hairstyles that can be achieved by cutting, shaving or trimming their hair; in other words, those that can be achieved with their current hair.

[0056] In some embodiments, the method 100, 300, 400 can further comprise the step of providing 410 a noise element 412 as input to the second trained neural network model 402. The noise element 412 can for example comprise random noise and can be provided in the form of a vector. By including the noise element 412 in the second neural network 402, variations of the defined hairstyle 404 can be incorporated into the modified layer of hair 406. In this way, the defined hairstyle 404 can be altered, ranging from very small alterations to large alterations, which introduces elements of diversity into the modified layer of hair generated by the second neural network 402.

[0057] In some embodiments, the methods 100, 300, 400 can further comprise the step of receiving 414, via the user interface UI, a user input to modify a parameter of at least one of the generated hair layer (H) 314, the indication defining the hairstyle 404, and the hair mask 318. For example, the user input can be provided as a further input to the second trained neural network model 402. For example, the user can modify the extent of the hair (e.g. the boundary), the length of the hair, the color of the hair, and / or the coarseness of the hair, such that the hairstyle to be incorporated into the image is customized in a way that suits the user. In other embodiments, other parameters can be modified. By modifying the parameters defining the hairstyle 404, the user can customize the appearance of the hairstyle to be incorporated onto the image of their head. Because the hair layer has already been generated by the first neural network model, by modifying the generated hair layer (H) 314, the user can manually adjust or correct the appearance of the hair layer. By modifying the hair mask (M) 318, the user can enlarge or reduce the size of the region on their head within which the hair can be modified.

[0058] According to some embodiments, the methods 100, 300, 400 can comprise the step 416 of generating a modified hair mask (M2) 418. The modified hair mask 418 can define, from the modified hair layer 406, a new region of the subject’s head within which hair exists. For example, if the subject’s original facial hair defines a chin strap, the hair mask 318 will define the extent of the chin strap. If, based on the defining hairstyle 404, the subject’s modified hair layer 406 is in the form of a goatee, the modified hair mask 418 will be reduced to define the extent of the goatee, rather than the extent of the chin strap.

[0059] Block 420 represents how the output of the second trained neural network 402 can be used to generate a modified image of the subject’s head, according to some embodiments. At step 422, the facial layer (F) 312 is provided as an input, and in block 420, the modified hair layer (H2), the modified hair mask (M2), and the facial layer (F) can be combined to generate a modified image, according to the expression M2*H2 + (1-M2)*F in block 420, applying the modified hair layer (H2) 406 for any region of the head that falls within the modified hair mask (M2) 418, and applying the facial layer (F) 312 for any region of the head that falls outside the modified hair mask. In this way, the modified hair layer is applied to the generated facial layer, resulting in a modified image.

[0060] According to some embodiments, during training of the second neural network model 402, the quality of the generated modified images of the subject’s head can be evaluated or assessed. Thus, in some embodiments, the methods 100, 200, 400 can comprise a step 424 of assessing the quality of the generated modified images of the subject’s head using a discriminative network. As with the embodiments discussed above, the discriminative network can form part of a generative adversarial network, where the second neural network 402 can be considered a generative network generating modified images to be assessed by the discriminative network 426. In some embodiments, the measure of the quality of the generated modified images of the subject’s head assessed by the discriminative network 426 can be referred to as an adversarial loss.

[0061] According to a second aspect, the present application provides a method of training a neural network model to manipulate the appearance of hair in an image. Figure 5 is a flowchart of one example of a method 500 of training a neural network model to manipulate the appearance of hair in an image of a subject’s head. For example, the neural network model can comprise the first neural network model 306 discussed above. The method 500 comprises, at step 502, generating a training dataset comprising a plurality of images of a subject’s head each having hair in a particular region, and a plurality of images of a subject’s head each having no hair in the particular region. In other words, a plurality of images of subjects are provided for training the neural network model, some of the subjects having a region of hair (e.g. a beard), and some of the subjects having no hair in that region (e.g. shaved clean). At step 504, the method 500 further comprises training the neural network model to generate, based on an input image of a subject’s head having hair in the particular region, an estimated representation of the subject’s head with the hair in the particular region removed. The training dataset allows the neural network model to identify patterns and correlations between features in the images, such that given an image of a particular subject having hair in a particular region, the trained model is able to estimate or predict what that subject would look like with no hair in that particular region.

[0062] According to a third aspect, the present application provides a further method of training a neural network model to manipulate the appearance of hair in an image. Figure 6is a flowchart of another example of a method 600 of training a neural network model to manipulate the appearance of hair in an image. For example, the neural network model can comprise the second neural network model 402 discussed above. The method 600 comprises, at step 602, generating a training dataset comprising a plurality of hair layers and a plurality of indicators of defined hairstyles to be incorporated into an image, each hair layer comprising an estimated representation of a portion of the image comprising hair. In other words, a training dataset is created from a plurality of representations (e.g. images) of portions of images containing hair and various hairstyles such as those illustrated above. At step 604, the method 600 further comprises training a neural network model to generate a modified hair layer based on an input hair layer and a particular defined hairstyle. In other words, the training dataset allows the neural network model to modify or adjust the input hair layer to more closely resemble the particular defined hairstyle. Figure 2

[0063] In some embodiments, the method 600 can further comprise, during said training, providing the neural network model with a hair mask and noise elements, the hair mask defining the extent of the hair-containing portion of the input hair layer. Providing the hair mask enables the neural network model to generate a modified hair layer that is compatible with (e.g. fits within) the existing hair of the image. Providing the noise elements facilitates variation within the modified hair layer generated by the neural network model.

[0064] The methods disclosed herein can be implemented in the form of a computer application. In some examples, the methods can be used as part of a shaving simulation method whereby a user is able to view one or more possible hairstyles on a representation of their face before commencing the action of shaving or trimming their hair.

[0065] According to a fourth aspect, the present application provides a computer program product. Figure 7 is a schematic diagram of one example of a processor 702 in communication with a computer readable medium 704. According to various embodiments, the computer program product comprises a computer readable medium 704 having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor 702, the computer or processor is caused to perform the methods 100, 300, 400, 500, 600 as disclosed herein.

[0066] Any of the methods 100, 300, 400, 500, 600 discussed herein can be performed using a processor or multi-processing device, which can form part of one or more devices. Thus, according to a fifth aspect, the present application provides a device for modifying an image of a subject’s head. Figure 8 ​A block diagram illustrating an apparatus 800 that can be used to train a neural network model, in accordance with one embodiment, is shown. Referring to Figure 8 The apparatus 800 includes a processor 802 that controls operation of the apparatus 800 and can implement methods described herein. The apparatus 800 also includes memory 806 including instruction data representing a set of instructions. The memory 806 can be configured to store instruction data in the form of program code executable by the processor 802 to carry out methods described herein. In some implementations, the instruction data can include a plurality of software and / or hardware modules, each software and / or hardware module configured to perform or to be used to perform an individual step or a plurality of steps of the methods described herein. In some embodiments, the memory 806 can be part of a device that also includes one or more other components of the apparatus 800 (e.g., the processor 802 and / or one or more other components of the apparatus 800). In alternative embodiments, the memory 806 can be part of a device separate from the other components of the apparatus 800.

[0067] In some embodiments, the memory 806 can include a plurality of sub- memories, each sub-memory capable of storing a segment of instruction data. In some embodiments in which the memory 806 includes a plurality of sub-memories, the instruction data representing the set of instructions can be stored at a single sub-memory. In other embodiments in which the memory 806 includes a plurality of sub-memories, the instruction data representing the set of instructions can be stored at a plurality of sub-memories. For example, at least one sub-memory can store instruction data representing at least one instruction of the set of instructions, while at least one other sub-memory can store instruction data representing at least one other instruction of the set of instructions. Thus, according to some embodiments, instruction data representing different instructions can be stored at one or more different locations in the apparatus 800. In some embodiments, the memory 806 can be used to store information, data (e.g., images), signals, and measurements acquired or made by the processor 802 of the apparatus 800 or from any other component of the apparatus 800.

[0068] The processor 802 of the apparatus 800 can be configured to communicate with the memory 806 to execute a set of instructions. The set of instructions, when executed by the processor 802, can cause the processor 802 to perform the methods described herein. The processor 802 can comprise one or more processors, processing units, multi-core processors, and / or modules configured or programmed to control the apparatus 800 in the manner described herein. In some embodiments, for example, the processor 802 can comprise a plurality of (e.g., interoperating) processors, processing units, multi-core processors, and / or modules configured for distributed processing. Those skilled in the art will appreciate that such processors, processing units, multi-core processors, and / or modules can be located in different places and can perform different steps and / or different parts of individual steps of the methods described herein.

[0069] Again, returning to Figure 8 In some embodiments, the apparatus 800 can comprise at least one user interface 804. In some embodiments, the user interface 804 can be part of a device that also includes one or more other components of the apparatus 800 (e.g., the processor 802, the memory 806, and / or one or more other components of the apparatus 800). In alternative embodiments, the user interface 804 can be part of a device that is separate from the other components of the apparatus 800.

[0070] The user interface 804 can be used to provide information to a user of the apparatus 800 generated according to the methods of the embodiments herein. The set of instructions, when executed by the processor 802, can cause the processor 802 to control the one or more user interfaces 804 to provide information generated according to the methods of the embodiments herein. Alternatively or additionally, the user interface 804 can be configured to receive user input. In other words, the user interface 804 can allow a user of the apparatus 800 to manually enter instructions, data, or information. The set of instructions, when executed by the processor 802, can cause the processor 802 to acquire user input from the one or more user interfaces 804.

[0071] User interface 804 can be any user interface that enables the rendering (or outputting or displaying) of information, data, or signals to a user of device 800. For example, user interface 804 can display a modified image of a subject. Alternatively or additionally, user interface 804 can be any user interface that enables a user of device 800 to provide user input, interact with device 800, and / or control device 800. For example, user interface 804 may include one or more switches, one or more buttons, a keypad, a keyboard, a mouse, a mouse wheel, a touchscreen or application (e.g., on a tablet or smartphone), a display screen, a graphical user interface (GUI) or other visual rendering component, one or more speakers, one or more microphones or any other audio component, one or more lights, a component for providing haptic feedback (e.g., vibration function), or any other user interface, or a combination of user interfaces.

[0072] In some embodiments, such as Figure 8 As shown, device 800 may further include a communication interface (or circuitry) 808 for enabling device 800 to communicate with interfaces, memories, and / or devices that are part of device 800. The communication interface 808 may communicate wirelessly or via a wired connection with any interface, memory, and device.

[0073] General Figure 8 Only the components necessary to illustrate this aspect of the present disclosure are shown, and in a practical implementation, device 800 may include components other than those shown. For example, device 800 may include a battery or other power source for powering device 800, or means for connecting device 800 to a mains power source.

[0074] According to some embodiments, processor 802 may be configured to communicate with memory 806 and execute a set of instructions, which, when executed by the processor, cause the processor to: provide an image of a subject's head with hairy regions as input to a first trained neural network model; generate a hair layer and a face layer using the first trained neural network model and based on the image of the head, the hair layer including an estimated representation of the hairy portion of the image, and the face layer including a representation of the subject's head with the hairy regions removed; provide an indication of a defined hairstyle to be incorporated into the image and the generated hair layer as input to a second trained neural network model; generate a modified hair layer similar to the defined hairstyle using the second trained neural network model and based on the indication of the defined hairstyle and the generated hair layer; and generate a modified image of the subject's head by applying the modified hair layer to the generated face layer.

[0075] In some embodiments, the apparatus 800 can also include a display (e.g., the user interface 804) to display the modified image of the subject’s head. In some embodiments, the apparatus 800 can include or form part of a computing device, such as a smartphone, tablet computer, laptop or desktop computer, or an interactive mirror or smart mirror.

[0076] As noted above, the processors 702, 802 can include one or more processors, processing units, multi-core processors, or modules configured or programmed to control the apparatus 800 in the manner described herein. In particular implementations, the processors 702, 802 can include a plurality of software and / or hardware modules each configured to perform or for performing individual steps or a plurality of steps of the methods described herein.

[0077] According to the embodiments disclosed herein, the trained neural network model is used to manipulate or modify an image of a subject having a region of hair, such as facial hair or skull hair. First, the trained neural network model modifies the image to “remove” the region of hair by predicting how the region would look if the hair were removed. Second, the trained neural network model generates a representation of an alternative region of hair to add to the modified image. The resulting image is a realistic prediction of how the subject would look with the modified hairstyle. However, with existing techniques, it is difficult to generate a modified realistic image in a situation where the subject has existing hair, using the techniques disclosed herein, a modified hairstyle can be applied to an image of a subject regardless of whether the subject has existing hair.

[0078] The methods and apparatus disclosed herein can be implemented and used in a number of different ways. In one example, the present invention can be used to use existing defined hairstyles (such as Figure 2In another example, the application can be used to generate a simulation of a hairstyle from a picture of a different person (i.e. a source image). With this example, the hairstyle can not resemble one of the existing defined hairstyles. Thus, the method can involve segmenting the source image to extract the shape of the hair (e.g. beard) region from the source image so that it can be placed on the target image (e.g. the image of the subject) and modified to fit the subject’s head. In another example, as discussed above, constraints can be implemented to ensure that a hairstyle is only available for addition to the subject’s image if it can be achieved using the subject’s existing hair. This can be achieved by using a hair mask to see which defined hairstyles are available for use. In another example, a hairstyle can be modifiable or manipulable via user input. For example, a user can modify the length of the hair in a particular hairstyle, or the shape of the boundary of the hairstyle from the subject’s skin. This provides the user with the opportunity to design a unique hairstyle. In another example, a subject can be provided with recommendations based on other subjects. For example, the method can involve querying a database of other subjects with a variety of hairstyles. The method can find subjects with similar characteristics (e.g. face shape, chin shape or beard or hairstyle) to the subject and recommend a hairstyle to the subject based on those characteristics. Another example, the subject’s modified image can be shared with other users (e.g. via social media). In another example, the subject’s modified image can be used as a guide for trimming or shaving their face or head. Based on the selected hairstyle, the subject can be provided with a recommendation of a personal care device to use.

[0079] As used herein, the term“module” is intended to include a hardware component, such as a processor or a component of a processor, configured to perform a specific function, or a software component, such as a set of instructions data that has a specific function when executed by a processor.

[0080] It will be appreciated that embodiments of the application also apply to computer programs, particularly computer programs on or in a carrier, adapted to put the embodiments of the application into practice. The program can be in the form of a software program, or a computer program product. The program can be stored on a carrier, which can be a storage medium or a data stream. The carrier is typically tangible, although it can be a transient signal in one example. The carrier is typically non-transitory in one example. The carrier can be a record medium, computer memory, read-only memory, erasable programmable read-only memory, programmable read-only memory, electrically erasable programmable read-only memory, compact disc read-only memory, random access memory, or a hard magnetic or optical storage medium, or a magnetic tape or other magnetic medium, or a storage internal to or external to a computer or device. The carrier can be a computer readable medium having stored the computer program. The computer readable medium can be a non-transitory computer readable medium in one example. The carrier can be a data stream, for example, in telecommunications or network communications. The carrier can be a computer data signal, in which case the carrier can be a propagated signal in one example. The computer program can be embodied in the carrier.

[0081] The carrier of a computer program can be any entity or device capable of carrying the program. For example, the carrier can include a data storage, such as a ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example a hard disk. Further, the carrier can be a transmissible carrier such as an electrical or optical signal, which can be conveyed via electrical or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier can be constituted by such a cable or other device or means. Alternatively, the carrier can be an integrated circuit in which the program is embedded, the integrated circuit being suitable for performing, or used in the performance of, the relevant method.

[0082] Variations to the disclosed embodiments can become apparent to those of ordinary skill in the art from the foregoing description and disclosure of the application, and are intended to be included within the scope of the application. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state storage medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method of modifying the appearance of hair in an image of a subject's head, the method comprising: providing an image of a subject's head having a region of hair as input to a first trained neural network model; using the first trained neural network model and based on the image of the head, generating a hair layer and a face layer, the hair layer comprising an estimated representation of the portion of the image comprising hair, the face layer comprising an estimated representation of the subject's head with the region of hair removed; providing an indication of a defined hairstyle to be incorporated into the image and the generated hair layer as input to a second trained neural network model; using the second trained neural network model, based on the indication of the defined hairstyle and the generated hair layer, generating a modified hair layer; and using a processor, generating a modified image of the subject's head by applying the modified hair layer to the generated face layer.

2. The method of claim 1, further comprising: using the first trained neural network model and based on the image of the head, generating a hair mask, the hair mask defining a region of the image containing hair.

3. The method of claim 2, further comprising: receiving, at a processor, an annotation of the image of the subject's head, the annotation comprising an indication of the region of the image containing hair; and using a processor, computing an accuracy of the generated hair mask relative to the received annotation.

4. The method of claim 2 or claim 3, further comprising: providing the generated hair mask as input to the second trained neural network model; wherein generating the modified hair layer comprises generating a modified hair layer based on the generated hair mask such that hair in the modified hair layer is generated only within a region defined by the generated hair mask.

5. The method of claim 2 or claim 3, further comprising: receiving, via a user interface, user input to modify a parameter of at least one of the following: the generated hair layer, the indication of the defined hairstyle, and the hair mask.

6. The method of any one of claims 1-3, wherein at least one of the first trained neural network model and the second trained neural network model comprises or forms part of a generative adversarial network.

7. The method of any one of claims 1-3, further comprising: providing the modified image for presentation to a user.

8. The method of any one of claims 1-3, further comprising: using a discriminative network to evaluate a quality of the generated face layer relative to an image of a subject's head having no visible hair in the region.

9. The method of any one of claims 1-3, further comprising: using a discriminative network to evaluate a quality of the generated modified image of the subject's head. ​ 10. The method of any one of claims 1 to 3, further comprising training a neural network model to obtain the first trained neural network model by: generating a training data set comprising a plurality of images of heads of subjects each having hair in a particular region, and a plurality of images of heads of subjects each not having hair in the particular region; and training the neural network model to generate, based on an input image of a head of a subject having hair in the particular region, an estimated representation of the head of the subject in which the hair in the particular region has been removed.

11. The method of any one of claims 1 to 3, further comprising training a neural network model to obtain the second trained neural network model by: generating a training data set comprising a plurality of hair layers and a plurality of indicators of defined hairstyles to be incorporated into an image, each hair layer comprising an estimated representation of a hair-containing portion of the image; and training the neural network model to generate, based on an input hair layer and a particular defined hairstyle, a modified hair layer.

12. The method of claim 11, further comprising: providing, to the neural network model during the training, a hair mask and a noise element, the hair mask defining a range of a hair-containing portion of the input hair layer.

13. A computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of any one of the preceding claims.

14. An apparatus for modifying an image of a head of a subject, the apparatus comprising: a memory comprising instruction data representing a set of instructions; and a processor configured to communicate with the memory and to execute the set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to: provide an image of a head of a subject having a hair region as input to a first trained neural network model; generate, using the first trained neural network model and based on the image of the head, a hair layer comprising an estimated representation of a hair-containing portion of the image and a face layer comprising a representation of the head of the subject in which the hair region has been removed; provide an indication of a defined hairstyle to be incorporated into the image and the generated hair layer as input to a second trained neural network model; generate, using the second trained neural network model and based on the indication of the defined hairstyle and the generated hair layer, a modified hair layer similar to the defined hairstyle; and generate a modified image of the head of the subject by applying the modified hair layer to the generated face layer.

15. The apparatus of claim 14, further comprising: a display for displaying the modified image of the head of the subject.

Images