Method of image processing and apparatus therefor

Abstract

The application discloses a kind of image processing method and its device, belong to artificial intelligence technical field.The image processing method includes: obtaining image to be handled;Target image region is input to target residual network model, and the residual output of the target residual network model is obtained;Wherein, the target image region includes: the image region in the image to be handled that shows hair and forehead;The residual output is superimposed to the image to be handled, and the target image after the image to be handled increases hair is obtained.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to an image processing method and apparatus. Background Technology

[0002] As people age and face various social pressures, more and more people are experiencing hair loss and becoming distressed by it. Hair thinning has a significant impact on some individuals. Currently, the StyleGAN (Style Generative Adversarial Network) model can be used to generate images showing the effect of hair restoration. This allows even users with relatively little hair to generate images that appear to have more hair.

[0003] like Figure 1 The diagram illustrates the process of generating the post-hair augmentation image using the StyleGAN model. Encoder 11 encodes the input image into W+, a matrix of numbers representing user information. StyleGAN 12 decodes the W+ information and adjusts the hair information to achieve the hair augmentation.

[0004] However, the above-mentioned use of the StyleGAN model for augmentation has high hardware performance requirements and high power consumption. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method and apparatus that can solve the problems in related technologies where the hardware performance requirements are high and the power consumption is large when implementing the hair enhancement effect for a person.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0007] Obtain the image to be processed;

[0008] The target image region is input into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image region includes: the image region in the image to be processed that displays hair and forehead;

[0009] The residual output is superimposed on the image to be processed to obtain the target image after the image to be processed is augmented.

[0010] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0011] The first acquisition module is used to acquire the image to be processed;

[0012] The residual module is used to input the target image region into the target residual network model and obtain the residual output of the target residual network model; wherein, the target image region includes: the image region in the image to be processed that displays hair and forehead;

[0013] The processing module is used to superimpose the residual output onto the image to be processed to obtain the target image after the image to be processed is augmented.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0016] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0017] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first aspect.

[0018] In this embodiment, for the image to be processed, the target image region displaying hair and forehead is input into the target residual network model to obtain a residual output representing the difference between the image before and after hair augmentation. Then, by superimposing the residual output onto the image to be processed, the target image after hair augmentation can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption, this application can reduce hardware performance requirements and power consumption while achieving the hair augmentation effect. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the process of generating an image of the augmented hair using the StyleGAN model.

[0020] Figure 2 This is a flowchart of the steps of an image processing method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram illustrating the changes in the image to be processed before and after augmentation in the embodiments of this application;

[0022] Figure 4 This is a schematic diagram of the training style generative adversarial network model provided in the embodiments of this application;

[0023] Figure 5 This is a schematic diagram illustrating the obtaining of a pair of before-and-after comparison images provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram illustrating the practical application of the image processing method provided in the embodiments of this application;

[0025] Figure 7 This is a structural block diagram of an image processing apparatus provided in an embodiment of this application;

[0026] Figure 8 This is one of the hardware structure diagrams of the electronic device provided in the embodiments of this application;

[0027] Figure 9 This is the second schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0031] like Figure 2 The diagram shown illustrates the steps of an image processing method according to an embodiment of this application. The image processing method may include:

[0032] Step 201: Obtain the image to be processed.

[0033] In this step, the image to be processed is the image for which the augmentation effect needs to be added, i.e., the image before augmentation. It can be an image captured by an electronic device such as a mobile phone or camera. Here, the image to be processed can be obtained locally; for example, in this application, the method is applied to a mobile phone, and the image to be processed can be a locally stored image on the phone. Alternatively, it can receive images transmitted from other electronic devices. For example, in this application, the method is applied to a server, and the image to be processed can be an image uploaded to the server by a mobile phone user.

[0034] It is understood that the image content to be processed includes at least hair and a forehead. Specifically, the image to be processed includes images showing a hairline. It is worth noting that the size of the image to be processed is not limited here. That is, the size of the image to be processed can be any size. For example, the size of the image to be processed can be 1536*768, 4096*4096, but it is not limited to these.

[0035] Step 202: Input the target image region into the target residual network model to obtain the residual output of the target residual network model.

[0036] It should be noted that the target image region includes the image region in the image to be processed that displays hair and forehead. Specifically, the target image region can be the entire image region or a portion of the image to be processed. In the case where the target image region is a portion of the image to be processed, that portion is the image region that displays hair and forehead.

[0037] The target residual network model is used to calculate the difference in a person's hair volume before and after the hair increase. Specifically, the target residual network model is a model built based on residual networks. This target residual network model can calculate the difference in a person's hair volume before and after the hair increase and output this difference as the residual. Here, hair increase refers to the increase in the amount of hair or hair volume of a person. The amount of hair after the hair increase is significantly higher than the amount of hair before the hair increase.

[0038] Step 203: Superimpose the residual output onto the image to be processed to obtain the target image after the image to be processed is augmented.

[0039] It should be noted that the residual output represents the difference between the person before and after the hair extension. Here, the image to be processed is taken as the image before the hair extension, and by superimposing the residual output onto the image before the hair extension, we can obtain the image after the hair extension, which is the target image after the hair extension. It can be understood that the location of the extension can be the position of the hairline in the image to be processed. This makes the target image appear, compared to the image to be processed, as if the person's hairline has moved towards their forehead. Figure 3As shown, the target image 33 is obtained by superimposing the image to be processed 31 and the residual output 32. Obviously, compared with the image to be processed 31, the hairline in the target image 33 is closer to the face, making the exposed forehead area smaller. Of course, the hair augmentation can also be applied to the sparse hair areas in the image to be processed, so that the target image appears to have thick hair in the sparse areas of the person compared to the image to be processed.

[0040] In this embodiment, for the image to be processed, the target image region displaying hair and forehead is input into the target residual network model to obtain a residual output representing the difference between the image before and after hair augmentation. Then, by superimposing the residual output onto the image to be processed, the target image after hair augmentation can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption, this application can reduce hardware performance requirements and power consumption while achieving the hair augmentation effect.

[0041] Optionally, before inputting the target image region into the target residual network model and obtaining the residual output of the target residual network model, the method further includes:

[0042] Obtain multiple pairs of before-and-after comparison images of the share issuance; each pair of comparison images includes one before-and-after image.

[0043] Using the image before the injection as input and the image after the injection as the ground truth, the initial residual network model is trained to obtain the target residual network model.

[0044] It should be noted that the initial residual network model is an untrained model built upon the residual network. The residual network can be built using an encoder-decoder structure, such as UNet. During the training of the initial residual network model, the learning rate and loss function can be set empirically. For example, the learning rate can be set to 0.001, and the loss function can be a combination of LPIPS (Learned Perceptual Image Patch Similarity) loss function and L1 loss function, but is not limited to these. In multiple pairs of before-and-after hair augmentation comparison images, each pair consists of images of the same person, i.e., the image of that person before hair augmentation and the image of that person after hair augmentation, with the post-hair augmentation image being an image obtained by increasing the amount of hair on top of the pre-hair augmentation image. When obtaining multiple pairs of before-and-after comparison images of hair thickening, one can first obtain a certain number of person images, and then draw a person image after hair thickening for each person image, or use an application that can currently achieve the hair thickening effect to generate a person image after hair thickening, and use the person image and the person image after hair thickening as a pair of before-and-after comparison images of hair thickening, but it is not limited to this.

[0045] Multiple pairs of before-and-after comparison images are used to train the initial residual network model, resulting in the trained model, i.e., the target residual network model. There is no limit to the number of before-and-after comparison images. Specifically, each pair of before-and-after comparison images can serve as a training sample for one iteration. Supervised learning is employed, using multiple pairs of before-and-after comparison images to train the initial residual network model until the model converges or the number of training iterations reaches a threshold. It is worth noting that for supervised learning of the residual network, in each model training process, the image before hair augmentation from one pair of before-and-after comparison images is used as input (INPUT), and the image after hair augmentation is used as the ground truth (GT). Through continuous model training, the residual output of the model gradually becomes more accurate in representing the difference between the person before and after hair augmentation.

[0046] In this embodiment of the application, multiple pairs of before-and-after comparison images of hair extension are used as training samples. Then, the initial residual network model is trained using the training samples, thereby obtaining a target residual network model for calculating the difference between a person before and after hair extension.

[0047] Optionally, multiple pairs of before-and-after comparison images of the issuance can be obtained, including:

[0048] Obtain multiple three-dimensional matrices that conform to a Gaussian distribution.

[0049] For each 3D matrix, a pair of before-and-after comparison images are generated based on a pre-trained style generative adversarial network model.

[0050] It should be noted that the input to the trained style generative adversarial network (SGN) model consists of multiple 3D matrices conforming to a Gaussian distribution, and the output is an image containing hair and forehead. In other words, inputting any Gaussian-distributed 3D matrix into the trained SGN model will yield an image containing hair and forehead. Finally, the model output can be used to obtain a pair of images, each containing hair and forehead, as a before-and-after comparison of hair augmentation. Preferably, the 3D matrices obtained here are randomly generated.

[0051] In this embodiment of the application, based on a three-dimensional matrix and a pre-trained style generative adversarial network model, comparison images before and after hair augmentation can be generated quickly, while simplifying the process of creating comparison images before and after hair augmentation.

[0052] Optionally, before generating a pair of before-and-after comparison images based on a pre-trained style generative adversarial network model for each three-dimensional matrix, the method further includes:

[0053] Obtain a training sample set; wherein the training sample set includes multiple random three-dimensional matrices conforming to a Gaussian distribution and training images containing foreheads and hair;

[0054] The target model is trained based on the training sample set to obtain a pre-trained style generative adversarial network model;

[0055] The target model includes an initial style generative adversarial network model and a discriminator model, with the output of the initial style generative adversarial network model and the training images serving as the input to the discriminator model.

[0056] It should be noted that the training sample set is used to train the target model. Since a trained style generative adversarial network (GAN) model can generate images including hair and forehead, training images containing both are required. The size of the training images is not limited, but all training images must have the same size. Specifically, when acquiring training images, high-resolution portrait data taken with a DSLR camera can be obtained first; then, face detection and face rotation correction / alignment can be performed on the high-resolution portrait data. Finally, the hair region of the rotated and aligned face is cropped to a uniform width-to-height ratio of 2:1 (other sizes are also acceptable; 1536*768 is used as an example here). The cropped image is the training image.

[0057] Since the target model comprises an initial style generative adversarial network (SGAN) model and a discriminator model, with the output of the initial SGAN model and the training image serving as the input to the discriminator model, training the target model is essentially training the initial SGAN model within it. This trained initial SGAN model is the well-trained SGAN model. The initial SGAN model acts as a generator, generating an image based on a random 3D matrix conforming to a Gaussian distribution. Therefore, the initial SGAN model can also be understood as the synthesis network component of the SGAN model, which consists of a mapping network and a synthesis network. In each training iteration of the target model, a random 3D matrix is ​​input to the initial SGAN model, and then the model output of the initial SGAN model and a training image are input to the discriminator model to obtain its output. After calculating the model loss, the backpropagation algorithm is used to update the model parameters of both the initial SGAN model and the discriminator model. Training is stopped once the initial SGAN model has fully converged.

[0058] Specifically, such as Figure 4 As shown, a generator 41 can be initialized first, and a discriminator model 42 can be initialized.

[0059] First, step 401 is executed: a three-dimensional matrix W+ conforming to a Gaussian distribution is randomly generated, and W+ is input into generator 41 to obtain the RGB image 43 output by generator 41, where RGB represents the colors of the red, green, and blue channels. At this point, a training image 44 is used as a real hair image, and its corresponding label is set to True (correct). The RGB image 43 is used as a fake hair image, and its corresponding label is set to False (incorrect). The loss is then calculated using loss function formula 1, and the model parameters of the discriminator model are updated via backpropagation.

[0060] loss=log(exp(D(G(W+)))+1)+log(exp(-D(x))+1) (1)

[0061] Where G(W+) represents the RGB image 43, x represents the training image 44, and D represents the discriminator model 42.

[0062] Step 402: Regenerate a three-dimensional matrix W+' that conforms to a Gaussian distribution, and input W+' into generator 41 to obtain a new RGB image. Now, with the trained discriminator model 42 and a new RGB image, calculate the loss using loss function formula 2, and backpropagate to update the weights of generator 41;

[0063] loss=-log(exp(D(G(W+)))+1) (2)

[0064] By repeating steps 401 and 402 in that order, generator 41 can be trained until it fully converges. At this point, a mapping from W+ to a 1536*768 image will be obtained. Inputting any random 3D matrix conforming to a Gaussian distribution into generator 41 will produce a corresponding 1536*768 image containing hair and forehead.

[0065] In this embodiment of the application, a style generative adversarial network model can be trained using a random three-dimensional matrix and training images containing foreheads and hair, thereby obtaining a network model for generating images containing foreheads and hair using a three-dimensional matrix.

[0066] Optionally, for each 3D matrix, a pair of before-and-after comparison images are generated based on a pre-trained style generative adversarial network model, including:

[0067] Multiple 3D matrices are input into the style generative adversarial network model to obtain the un-enhanced images output by the style generative adversarial network model.

[0068] The input of the style generative adversarial network model is back-adjusted based on the output of the style generative adversarial network model and the guide image until the number of back-adjustments reaches the target number; where the guide image is a comparison image after the augmentation drawn based on the image before augmentation.

[0069] After the final inverse adjustment, the output of the style generative adversarial network model is determined as the augmented image.

[0070] It should be noted that a pair of before-and-after comparison images of hair augmentation can be generated for each three-dimensional matrix. This embodiment only illustrates the generation of a pair of before-and-after comparison images for a single three-dimensional matrix. Since any randomly selected three-dimensional matrix conforming to a Gaussian distribution will produce a corresponding image containing hair and forehead when input into the style generative adversarial network model, the first image obtained is used as the before-and-after image in the pair of before-and-after comparison images. Then, a comparison image after hair augmentation is drawn based on the before-and-after image, and this comparison image is used as the guide image to adjust the three-dimensional matrix in reverse. When the number of reverse adjustments reaches the target number, the image output by the style generative adversarial network model will be very close to the guide image, and thus the image output at this point is used as the after-and-after image in the pair of before-and-after comparison images.

[0071] like Figure 5 The flowchart shown illustrates the process of obtaining a pair of before-and-after images of a share issuance, specifically including:

[0072] Step 501: Input a three-dimensional matrix W+ that conforms to a Gaussian distribution into the style generative adversarial network model 53 to obtain the image 51 before the hair growth, and store the image 51 before the hair growth as the image before the hair growth in a pair of before and after comparison images.

[0073] Step 502: Use a brush to draw the hair in the hair-addition area in the image 51 before hair addition (you can just draw the hair color) to generate the guide image 54.

[0074] Step 503: Use the L1 loss function to calculate the difference between the image 51 before the augmentation and the guide image 54, and obtain the effect difference between the two images: loss = |image 51 before augmentation - guide image 54|.

[0075] Step 504: Set the learning rate to 0.01 during model training and set the optimizer to Adam (Adaptive Moment Estimation) optimizer. Use the backpropagation method to backpropagate the loss along the style generative adversarial network model 53 to the three-dimensional matrix W+. At this time, update the numbers in the three-dimensional matrix W+ through the gradient from backpropagation to obtain the new three-dimensional matrix W+*.

[0076] Step 505: Input the new 3D matrix W+* into the style generative adversarial network model 53 to obtain the new image before augmentation.

[0077] Step 506: Use the L1 loss function to calculate the difference between the new pre-expansion image and the guide image 54, and obtain the effect difference between the two images: loss = |new pre-expansion image - guide image 54|.

[0078] Steps 504 to 506 are repeated for the target number of times. During this repetition, the style generative adversarial network model 53 refers to the guiding image 54 to gradually grow hair and fill in hair texture in the pre-hair augmentation image 51. The target number of times can be set arbitrarily, for example, 100 times, but is not limited to this. The final output image of the style generative adversarial network model 53 is used as the post-hair augmentation image in a pair of before-and-after comparison images.

[0079] In this embodiment, after generating the image before the augmentation, a contrasting image after the augmentation is used as the guiding image. By continuously adjusting the input to the style generative adversarial network model, the image after the augmentation can be obtained, and the images before and after the augmentation are the same size.

[0080] Optionally, the input to the target residual network model is an image of a first size. When the size of the image to be processed is smaller than or larger than the first size, the target image region is input to the target residual network model to obtain the residual output of the target residual network model, including:

[0081] Perform an affine transformation on the target image region to generate a first intermediate image of the first size;

[0082] The first intermediate image is input into the target residual network model to obtain the residual output of the target residual network model;

[0083] The residual output is superimposed onto the image to be processed to obtain the target image after augmentation, including:

[0084] Perform an inverse affine transformation on the residual output to generate a second intermediate image of a second size, where the second size is the size of the image to be processed;

[0085] The second intermediate image is superimposed on the image to be processed to obtain the target image.

[0086] It should be noted that the target residual network model is a pre-trained network model, and its input image size will be fixed after training. Since the size of the image to be processed may not necessarily meet the target residual network model's input image size requirements, the size of the image to be processed can be detected first. If its size is the first size, meaning the size of the image to be processed meets the target residual network model's input image size requirements, then the image to be processed can be used directly. Otherwise, an affine transformation is required, and an inverse affine transformation is performed on the residual output to ensure that the size of the residual output matches the size of the image to be processed, avoiding various problems caused by size inconsistencies. For example, the target residual network model can be trained using a 1024 resolution image. If the original image (the image to be processed) is 4096 resolution, it is downsampled to 1024 resolution using an affine transformation, then fed into the target residual network model to generate a 1024 resolution residual output. This residual output is then enlarged to 4096 resolution and superimposed onto the original image.

[0087] In this embodiment, by performing affine transformation on the image to be processed and inverse affine transformation on the residual output, images of various sizes can be processed without affecting the clarity of the target image.

[0088] Optionally, an affine transformation is performed on the target image region to generate a first intermediate image of a first size, including:

[0089] The target image region is cropped from the image to be processed to obtain the cropped image.

[0090] Perform an affine transformation on the cropped image to generate a first intermediate image of the first size.

[0091] It should be noted that the method provided in this application achieves an augmentation effect on the image to be processed. Therefore, processing can be performed only on the target image region within the image to be processed. Here, the target image region is cropped out to obtain a cropped image.

[0092] In this embodiment, a cropped image is obtained by cropping the target image region from the image to be processed. The cropped image is then used to obtain the residual output, which avoids the influence of information from other image regions besides the target image region in the image to be processed.

[0093] like Figure 6 The diagram shown is a schematic representation of the actual application of the image processing method provided in this application embodiment. This application uses resolution as the image size, and for the image to be processed 61 with a size of 4096*4096, it generates its augmented target image 62. Assuming the input image size of the already trained target residual network model 63 is 1536*768, the process includes:

[0094] Step 601: Obtain the image to be processed with a size of 4096*4096.

[0095] Step 602: For the image to be processed 61, detect the face bounding box and crop out the image areas of the hair and forehead to obtain the cropped image 64.

[0096] Step 603: Scale the cropped image 64 to 1536*768 to obtain the original intermediate image 65.

[0097] Step 604: Input the original intermediate image 65 (1536*768) into the target residual network model 63 to obtain the residual output 66 (1536*768). The residual output 66 can be understood as the residual corresponding to the original intermediate image 65.

[0098] Step 605: Scale the 1536*768 residual output 66 to an intermediate residual output 67 of 4096*4096. The intermediate residual output 67 can be understood as corresponding to the cropped image 64.

[0099] Step 606: Based on the intermediate residual output 67, generate the residual corresponding to the image to be processed 61, i.e., the target residual 68. Superimpose the target residual 68 onto the cropped area of ​​the image to be processed 61 to obtain the final image, i.e., the target image 62.

[0100] In this embodiment, a residual network is designed to improve the clarity of the final image while controlling performance and power consumption, thus solving the shortcomings of traditional style generative adversarial networks, such as decreased clarity, excessive performance and power consumption, and lack of operator support.

[0101] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing apparatus or a control module within the image processing apparatus for executing the image processing method. This application embodiment uses an image processing apparatus executing the image processing method as an example to illustrate the image processing apparatus provided in this application embodiment.

[0102] like Figure 7 As shown, this application embodiment also provides an image processing apparatus, which includes:

[0103] The first acquisition module 71 is used to acquire the image to be processed;

[0104] The residual module 72 is used to input the target image region into the target residual network model and obtain the residual output of the target residual network model; wherein, the target image region includes: the image region in the image to be processed that displays hair and forehead;

[0105] The processing module 73 is used to superimpose the residual output onto the image to be processed to obtain the target image after the image to be processed is augmented.

[0106] Optionally, the device further includes:

[0107] The second acquisition module is used to acquire multiple pairs of before-and-after comparison images of the share issuance; wherein each pair of before-and-after comparison images includes one before-and-after image.

[0108] The first training module is used to train the initial residual network model by taking the image before the augmentation as input and the image after the augmentation as the ground truth, so as to obtain the target residual network model.

[0109] Optionally, the second acquisition module includes:

[0110] The acquisition unit is used to acquire multiple three-dimensional matrices that conform to a Gaussian distribution;

[0111] The generation unit is used to generate a pair of before-and-after comparison images for each 3D matrix, based on a pre-trained style generative adversarial network model.

[0112] Optionally, the device further includes:

[0113] The third acquisition module is used to acquire the training sample set; wherein, the training sample set includes multiple random three-dimensional matrices that conform to a Gaussian distribution and training images containing foreheads and hair;

[0114] The second training module is used to train the target model based on the training sample set to obtain a pre-trained style generative adversarial network model.

[0115] The target model includes an initial style generative adversarial network model and a discriminator model, with the output of the initial style generative adversarial network model and the training images serving as the input to the discriminator model.

[0116] Optionally, generating units are specifically used for:

[0117] Multiple 3D matrices are input into the style generative adversarial network model to obtain the un-enhanced images output by the style generative adversarial network model.

[0118] The input of the style generative adversarial network model is back-adjusted based on the output of the style generative adversarial network model and the guide image until the number of back-adjustments reaches the target number; where the guide image is a comparison image after the augmentation drawn based on the image before augmentation.

[0119] After the final inverse adjustment, the output of the style generative adversarial network model is determined as the augmented image.

[0120] Optionally, the input to the target residual network model is an image of a first size. When the size of the image to be processed is smaller or larger than the first size, the residual module 72 includes:

[0121] An affine transformation unit is used to perform an affine transformation on the target image region to generate a first intermediate image of a first size.

[0122] The residual unit is used to input the first intermediate image into the target residual network model to obtain the residual output of the target residual network model;

[0123] Processing module 73 includes:

[0124] The inverse affine transformation unit is used to perform an inverse affine transformation on the residual output to generate a second intermediate image of a second size, wherein the second size is the size of the image to be processed;

[0125] The processing unit is used to overlay the second intermediate image onto the image to be processed to obtain the target image.

[0126] Optionally, the affine transformation unit is specifically used for:

[0127] Cropping the target image region from the image to be processed yields the cropped image;

[0128] Perform an affine transformation on the cropped image to generate a first intermediate image of the first size.

[0129] In this embodiment, the target image region displaying hair and forehead is input into a target residual network model for the image to be processed. Since the target residual network model is used to calculate the difference in appearance before and after hair augmentation, a residual output representing this difference can be obtained. Furthermore, by superimposing the residual output onto the image to be processed, the target image after hair augmentation can be obtained. Because the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption, this application can achieve the hair augmentation effect while reducing hardware performance requirements and power consumption.

[0130] The image processing apparatus in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0131] The image processing apparatus in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0132] The image processing apparatus provided in this application embodiment can achieve... Figures 2-6 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0133] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0134] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0135] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0136] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0137] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0138] Processor 910 is used to acquire the image to be processed.

[0139] The processor 910 is also used to input the target image region into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image region includes: the image region in the image to be processed that displays hair and forehead.

[0140] The processor 910 is also used to superimpose the residual output onto the image to be processed to obtain the target image after the image to be processed is augmented.

[0141] In this embodiment, for the image to be processed, the target image region displaying hair and forehead is input into the target residual network model to obtain a residual output representing the difference between the image before and after hair augmentation. Then, by superimposing the residual output onto the image to be processed, the target image after hair augmentation can be obtained. Since the target residual network model utilizes a residual network with low hardware performance requirements and low power consumption, this application can reduce hardware performance requirements and power consumption while achieving the hair augmentation effect.

[0142] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0143] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0144] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0145] This application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, these will not be described again here.

[0146] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0147] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0148] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0149] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the image processing method embodiment described above, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0152] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed; Acquiring multiple pairs of before-and-after comparison images of hair augmentation includes: acquiring multiple three-dimensional matrices conforming to a Gaussian distribution; inputting each of the multiple three-dimensional matrices into a style generative adversarial network (SGN) model to obtain the before-hair augmentation image output by the SGN model; performing reverse adjustment on the input of the SGN model based on the output of the SGN model and a guide image until the number of reverse adjustments reaches a target number; wherein, the guide image is a comparison image of the after-hair augmentation drawn based on the before-hair augmentation image; determining the output of the SGN model after the last reverse adjustment as the after-hair augmentation image; wherein, each pair of before-hair augmentation comparison images includes one before-hair augmentation image and one after-hair augmentation image; using the before-hair augmentation image as input and the after-hair augmentation image as the ground truth, training an initial residual network model to obtain a target residual network model; the target residual network model is used to calculate the difference between the person before and after hair augmentation; The target image region is input into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image region includes: the image region in the image to be processed that displays hair and forehead; The residual output is superimposed on the image to be processed to obtain the target image after the image to be processed is augmented.

2. The method according to claim 1, characterized in that, The input to the target residual network model is an image of a first size. When the size of the image to be processed is smaller than or larger than the first size, the step of inputting the target image region into the target residual network model to obtain the residual output of the target residual network model includes: Perform an affine transformation on the target image region to generate a first intermediate image of a first size; The first intermediate image is input into the target residual network model to obtain the residual output of the target residual network model; The step of superimposing the residual output onto the image to be processed to obtain the target image after augmentation of the image to be processed includes: Perform an inverse affine transformation on the residual output to generate a second intermediate image of a second size, wherein the second size is the size of the image to be processed; The second intermediate image is superimposed on the image to be processed to obtain the target image.

3. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the image to be processed; The second acquisition module is used to acquire multiple pairs of before-and-after comparison images of the injection, including: acquiring multiple three-dimensional matrices conforming to a Gaussian distribution; inputting the multiple three-dimensional matrices into a style generative adversarial network (SGAN) model respectively to obtain the before-injection image output by the SGAN model; performing reverse adjustment on the input of the SGAN model based on the output of the SGAN model and a guide image until the number of reverse adjustments reaches a target number; wherein, the guide image is a comparison image of the after-injection drawn based on the before-injection image; and determining the output of the SGAN model after the last reverse adjustment as the after-injection image; wherein each pair of injection comparison images includes one before-injection image and one after-injection image; The first training module is used to train the initial residual network model by taking the image before hair extension as input and the image after hair extension as the ground truth, to obtain the target residual network model; the target residual network model is used to calculate the difference between the person before and after hair extension. The residual module is used to input the target image region into the target residual network model to obtain the residual output of the target residual network model; wherein, the target image region includes: the image region in the image to be processed that displays hair and forehead; The processing module is used to superimpose the residual output onto the image to be processed to obtain the target image after the image to be processed is augmented.

4. The apparatus according to claim 3, characterized in that, The input to the target residual network model is an image of a first size. When the size of the image to be processed is smaller than or larger than the first size, the residual module includes: An affine transformation unit is used to perform an affine transformation on the target image region to generate a first intermediate image of a first size; The residual unit is used to input the first intermediate image into the target residual network model to obtain the residual output of the target residual network model; The processing module includes: The inverse affine transformation unit is used to perform an inverse affine transformation on the residual output to generate a second intermediate image of a second size, wherein the second size is the size of the image to be processed; The processing unit is used to overlay the second intermediate image onto the image to be processed to obtain the target image.

Images