Aesthetics scoring model-based portrait beautifying method and system

Abstract

The invention discloses a portrait beautifying method and system based on an aesthetic scoring model, and belongs to the field of artificial intelligence and image processing, and the method comprises the steps: obtaining a to-be-processed portrait image; inputting the obtained to-be-processed portrait image into the trained convolutional neural network for attribute recognition, and obtaining an attribute tag of the to-be-processed portrait image; retrieving a plurality of candidate images with the same attribute tag from a database according to the attribute tag of the portrait image to be processed to obtain a candidate image set; scoring each candidate image in the candidate image set by using an aesthetic scoring model, and taking the candidate image with the highest aesthetic score as a recommended target image; and adjusting the portrait attitude in the to-be-processed portrait image into the portrait attitude in the recommended target image by using an attitude migration algorithm. By utilizing the method and the system, the human body posture in the portrait image to be processed can be conveniently and quickly adjusted into the target image meeting aesthetic requirements.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and image processing technology, and more particularly to a portrait beautification method and system based on an aesthetic scoring model. Background Art

[0002] At present, the image processing technology in existing image processing software is not simple and convenient enough. For example, the popular Meitu Xiuxiu can only achieve simple functions such as skin smoothing and whitening, face slimming and body slimming, and requires users to make manual adjustments bit by bit, which is very cumbersome. As for the more professional PhotoShop, although it has more functions for processing images, it can only be used on the computer, and its operation is relatively cumbersome, which is not very friendly to ordinary people and is not very versatile.

[0003] Therefore, how to provide a portrait beautification method that is powerful and easy to operate is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] In view of this, the present invention provides a portrait beautification method and system based on an aesthetic scoring model, which are used to at least solve some of the technical problems in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] On one hand, the present invention discloses a portrait beautification method based on an aesthetic scoring model, comprising the following steps:

[0007] Obtain a portrait image to be processed;

[0008] Input the obtained portrait image to be processed into the trained convolutional neural network for attribute recognition to obtain the attribute label of the portrait image to be processed;

[0009] According to the attribute label of the portrait image to be processed, multiple candidate images with the same attribute label are retrieved from the database to obtain a candidate image set;

[0010] Each candidate image in the candidate image set is scored using the aesthetic scoring model, and the candidate image with the highest aesthetic score is used as the recommended target image;

[0011] The pose transfer algorithm is used to adjust the portrait pose in the portrait image to be processed to the portrait pose in the recommended target image.

[0012] Furthermore, the attribute tags include the person's gender, age and shooting environment.

[0013] Furthermore, each candidate image in the candidate image set is scored using an aesthetic scoring model, specifically including the following steps:

[0014] Using the OpenPose algorithm to identify and detect key features of the human body in each candidate image, the key features of the human body include limb features and facial features;

[0015] An aesthetic score is given to each candidate image based on the key point features of the human body, and candidate images whose limb stretch angles meet the set threshold and whose facial features meet the scene requirements are selected as recommended target images.

[0016] Furthermore, the posture transfer algorithm is used to adjust the posture of the portrait in the portrait image to be processed to the posture of the portrait in the recommended target image, which specifically includes the following steps:

[0017] Input the original image to be processed into the coordinate completion model to generate the original image pose coordinates;

[0018] Input the recommended target image into the coordinates to complete the model to generate the target pose coordinates;

[0019] Encode the target pose coordinates using a pose encoder;

[0020] By adjusting the style vector of the generator, the target pose coordinates after encoding the recommended target image are mapped to the original image to be processed to complete the pose transfer.

[0021] Furthermore, the recommended target images also include designated recommended target images manually selected by a user.

[0022] A portrait beautification system based on an aesthetic scoring model, comprising:

[0023] An attribute recognition module is used to use a trained convolutional neural network to perform attribute recognition on the portrait image to be processed and obtain attribute labels of the portrait image to be processed;

[0024] The recommended target image acquisition module is used to retrieve multiple candidate images with the same attribute labels from the database according to the attribute labels of the portrait image to be processed, obtain a candidate image set, and score each candidate image in the candidate image set using the aesthetic scoring model, and select the candidate image with the highest aesthetic score as the recommended target image;

[0025] Posture transfer module: used to adjust the portrait posture in the portrait image to be processed to the portrait posture in the recommended target image by using the posture transfer algorithm.

[0026] Preferably, in the attribute recognition module, the attribute tags include the gender of the person, the age of the person and the shooting environment.

[0027] Preferably, in the recommended target image acquisition module, each candidate image in the candidate image set is scored using an aesthetic scoring model, specifically comprising the following steps:

[0028] Using the OpenPose algorithm to identify and detect key features of the human body in each candidate image, the key features of the human body include limb features and facial features;

[0029] An aesthetic score is given to each candidate image based on the key point features of the human body, and candidate images whose limb stretch angles meet the set threshold and whose facial features meet the scene requirements are selected as recommended target images.

[0030] Preferably, in the posture transfer module, the posture transfer algorithm is used to adjust the posture of the portrait in the portrait image to be processed to the posture of the portrait in the recommended target image, which specifically includes the following steps:

[0031] Input the original image to be processed into the coordinate completion model to generate the original image pose coordinates;

[0032] Input the recommended target image into the coordinates to complete the model to generate the target pose coordinates;

[0033] Encode the target pose coordinates using a pose encoder;

[0034] By adjusting the style vector of the generator, the target pose coordinates after encoding the recommended target image are mapped to the original image to be processed to complete the pose transfer.

[0035] Preferably, the recommended target image in the recommended target image acquisition module is also acquired by:

[0036] The user manually specifies the recommended target image.

[0037] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a portrait beautification method and system based on an aesthetic scoring model, which has the following beneficial effects:

[0038] The present invention combines artificial intelligence with image processing, takes the research on the connection between aesthetics and human posture attributes as theoretical basis, and uses OpenPose human posture recognition algorithm and human posture migration algorithm as technical support, thus achieving the purpose of modifying and beautifying the human posture in photos.

[0039] The present invention can choose to modify the human body posture in the portrait to be processed into a human body posture with higher aesthetic perception in the database, or can choose to modify it into a posture in a specified picture, which is both automated and personalized. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 The present invention provides a schematic diagram of the overall process of the method.

[0042] Figure 2 This is a schematic diagram of the attribute analysis network structure provided by the present invention.

[0043] Figure 3 A schematic diagram of obtaining the recommended target image provided by the present invention.

[0044] Figure 4 This is a schematic diagram of the posture migration process provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 The embodiment of the present invention discloses, on the one hand, a portrait beautification method based on an aesthetic scoring model, comprising the following steps:

[0047] Obtain a portrait image to be processed;

[0048] Input the obtained portrait image to be processed into the trained convolutional neural network for attribute recognition to obtain the attribute label of the portrait image to be processed;

[0049] According to the attribute label of the portrait image to be processed, multiple candidate images with the same attribute label are retrieved from the database to obtain a candidate image set;

[0050] Each candidate image in the candidate image set is scored using the aesthetic scoring model, and the candidate image with the highest aesthetic score is used as the recommended target image;

[0051] The pose transfer algorithm is used to adjust the portrait pose in the portrait image to be processed to the portrait pose in the recommended target image.

[0052] Furthermore, the attribute tags include the person's gender, age and shooting environment.

[0053] Furthermore, each candidate image in the candidate image set is scored using an aesthetic scoring model, specifically including the following steps:

[0054] Using the OpenPose algorithm to identify and detect key features of the human body in each candidate image, the key features of the human body include limb features and facial features;

[0055] An aesthetic score is given to each candidate image based on the key point features of the human body, and candidate images whose limb stretch angles meet the set threshold and whose facial features meet the scene requirements are selected as recommended target images.

[0056] Furthermore, the posture transfer algorithm is used to adjust the posture of the portrait in the portrait image to be processed to the posture of the portrait in the recommended target image, which specifically includes the following steps:

[0057] Input the original image to be processed into the coordinate completion model to generate the original image pose coordinates;

[0058] Input the recommended target image into the coordinates to complete the model to generate the target pose coordinates;

[0059] Encode the target pose coordinates using a pose encoder;

[0060] By adjusting the style vector of the generator, the target pose coordinates after encoding the recommended target image are mapped to the original image to be processed to complete the pose transfer.

[0061] Furthermore, the recommended target images also include designated recommended target images manually selected by a user.

[0062] Combine the following Figure 2-Figure 4 Each step of the present invention is described in detail.

[0063] Figure 2 The attribute analysis network diagram provided by the present invention can parse out the scene, age and gender of each picture in the database, and use the portrait aesthetic scoring model to score and annotate the posture attributes of the photos.

[0064] Figure 3 This is the flowchart of the pose recommendation algorithm. We select a batch of portrait images from the database that are consistent with the scene, age and gender of the characters in the pose to be changed. Then we use openpose to obtain the skeletal key points of each body, calculate the Euclidean distance to select the portrait that is closest to the pose to be changed, and then select the image with the highest pose aesthetic score from this batch of portrait images as our recommended pose (this process considers two aspects. The first priority is that the label attributes are the same, and then index the gallery with the same label attributes to find the portrait with the closest pose. One is to facilitate the change of pose, and the other is to make the result more aesthetic).

[0065] Figure 4 This is a posture transfer flow chart, which is used to use the StyleGAN network to achieve the migration of source posture to target posture. There are four main steps. The first step is to input the original image to be processed into the coordinate completion model to generate the original image posture coordinates; the second step is to input the recommended target image into the coordinate completion model to generate the target posture coordinates; the third step is to use the posture encoder to encode the target posture coordinates; the fourth step is to map the target posture coordinates encoded by the recommended target image to the original image to be processed by adjusting the style vector of the generator Generator to complete the posture migration. This step may require multiple iterations and optimizations to ensure that the final generated image not only matches the recommended target image in appearance, but also that the recommended posture is correctly migrated to the original image to be processed.

[0066] Furthermore, the transferred images need further adjustments to improve their quality or meet specific requirements, including adjusting color, light, details, etc., to ensure that the final result not only matches the visual style of the original image, but also retains the precise pose of the target object in the recommended target image.

[0067] A portrait beautification system based on an aesthetic scoring model, comprising:

[0068] An attribute recognition module is used to use a trained convolutional neural network to perform attribute recognition on the portrait image to be processed and obtain attribute labels of the portrait image to be processed;

[0069] The recommended target image acquisition module is used to retrieve multiple candidate images with the same attribute labels from the database according to the attribute labels of the portrait image to be processed, obtain a candidate image set, and score each candidate image in the candidate image set using the aesthetic scoring model, and select the candidate image with the highest aesthetic score as the recommended target image;

[0070] Posture transfer module: used to adjust the portrait posture in the portrait image to be processed to the portrait posture in the recommended target image by using the posture transfer algorithm.

[0071] Preferably, in the attribute recognition module, the attribute tags include the gender of the person, the age of the person and the shooting environment.

[0072] Preferably, in the recommended target image acquisition module, each candidate image in the candidate image set is scored using an aesthetic scoring model, specifically comprising the following steps:

[0073] Using the OpenPose algorithm to identify and detect key features of the human body in each candidate image, the key features of the human body include limb features and facial features;

[0074] Each candidate image is aesthetically scored based on the key features of the human body, and candidate images whose limb extension angles meet the set threshold and whose facial features meet the scene requirements are recommended as target images. For example, for different scenes and backgrounds, different opening angles of human limbs are recommended as target images with the highest aesthetic scores. For example, in a certain scene, the image with the largest opening angle is used as the recommended target image. At the same time, the corresponding facial features in different scenes are selected as recommended target images. For example, in a meeting, the recommended facial feature is serious, and in a scene with a landscape background, smile is used as the recommended target feature.

[0075] Preferably, in the posture transfer module, the posture transfer algorithm is used to adjust the posture of the portrait in the portrait image to be processed to the posture of the portrait in the recommended target image, which specifically includes the following steps:

[0076] Input the original image to be processed into the coordinate completion model to generate the original image pose coordinates;

[0077] Input the recommended target image into the coordinates to complete the model to generate the target pose coordinates;

[0078] Encode the target pose coordinates using a pose encoder;

[0079] By adjusting the style vector of the generator, the target pose coordinates after encoding the recommended target image are mapped to the original image to be processed to complete the pose transfer.

[0080] Preferably, the recommended target image in the recommended target image acquisition module is also acquired by:

[0081] The user manually specifies the recommended target image.

[0082] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0083] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A portrait beautification method based on an aesthetic scoring model, characterized in that: The following steps are involved: Obtain a portrait image to be processed; Input the obtained portrait image to be processed into the trained convolutional neural network for attribute recognition to obtain the attribute label of the portrait image to be processed; According to the attribute label of the portrait image to be processed, multiple candidate images with the same attribute label are retrieved from the database to obtain a candidate image set; Each candidate image in the candidate image set is scored using the aesthetic scoring model, and the candidate image with the highest aesthetic score is used as the recommended target image; The pose transfer algorithm is used to adjust the portrait pose in the portrait image to be processed to the portrait pose in the recommended target image.

2. The portrait beautification method based on the aesthetic scoring model according to claim 1, characterized in that: The attribute tags include the gender of the person, the age of the person and the shooting environment.

3. The portrait beautification method based on the aesthetic scoring model according to claim 1, characterized in that: Each candidate image in the candidate image set is scored using the aesthetic scoring model, which specifically includes the following steps: Using the OpenPose algorithm to identify and detect key features of the human body in each candidate image, the key features of the human body include limb features and facial features; An aesthetic score is given to each candidate image based on the key point features of the human body, and candidate images whose limb stretch angles meet the set threshold and whose facial features meet the scene requirements are selected as recommended target images.

4. The portrait beautification method based on the aesthetic scoring model according to claim 1, characterized in that: The posture transfer algorithm is used to adjust the posture of the portrait in the portrait image to be processed to the posture of the portrait in the recommended target image, which specifically includes the following steps: Input the original image to be processed into the coordinate completion model to generate the original image pose coordinates; Input the recommended target image into the coordinates to complete the model to generate the target pose coordinates; Encode the target pose coordinates using a pose encoder; By adjusting the style vector of the generator, the target pose coordinates after encoding the recommended target image are mapped to the original image to be processed to complete the pose transfer.

5. The portrait beautification method based on the aesthetic scoring model according to claim 1, characterized in that: The recommended target images also include designated recommended target images manually selected by a user.

6. A portrait beautification system based on an aesthetic scoring model, characterized in that: include: An attribute recognition module is used to use a trained convolutional neural network to perform attribute recognition on the portrait image to be processed and obtain attribute labels of the portrait image to be processed; The recommended target image acquisition module is used to retrieve multiple candidate images with the same attribute labels from the database according to the attribute labels of the portrait image to be processed, obtain a candidate image set, and score each candidate image in the candidate image set using the aesthetic scoring model, and select the candidate image with the highest aesthetic score as the recommended target image; Posture transfer module: used to adjust the portrait posture in the portrait image to be processed to the portrait posture in the recommended target image by using the posture transfer algorithm.

7. The portrait beautification system based on the aesthetic scoring model according to claim 6, characterized in that: In the attribute recognition module, the attribute tags include the gender of the person, the age of the person and the shooting environment.

8. The portrait beautification system based on the aesthetic scoring model according to claim 6, characterized in that: In the recommended target image acquisition module, each candidate image in the candidate image set is scored using an aesthetic scoring model, specifically including the following steps: Using the OpenPose algorithm to identify and detect key features of the human body in each candidate image, the key features of the human body include limb features and facial features; An aesthetic score is given to each candidate image based on the key point features of the human body, and candidate images whose limb stretch angles meet the set threshold and whose facial features meet the scene requirements are selected as recommended target images.

9. The portrait beautification system based on the aesthetic scoring model according to claim 6, characterized in that: In the posture transfer module, the posture transfer algorithm is used to adjust the posture of the portrait in the portrait image to be processed to the posture of the portrait in the recommended target image, which specifically includes the following steps: Input the original image to be processed into the coordinate completion model to generate the original image pose coordinates; Input the recommended target image into the coordinates to complete the model to generate the target pose coordinates; Encode the target pose coordinates using a pose encoder; By adjusting the style vector of the generator, the target pose coordinates after encoding the recommended target image are mapped to the original image to be processed to complete the pose transfer.

10. The portrait beautification system based on the aesthetic scoring model according to claim 6, characterized in that: The recommended target image in the recommended target image acquisition module also includes obtaining the recommended target image in the following manner: The user manually specifies the recommended target image.

Images