Image processing method, medium, device and computing equipment

Abstract

The embodiments of the present disclosure provide an image processing method, medium, device and computing equipment, which relate to the field of image processing technology. The image processing method includes: receiving a style conversion instruction for a target image to a target facial style; performing face detection on the target image according to the style conversion instruction, obtaining the position information of the facial parts in the target image, and performing conversion processing on the target image to obtain an intermediate image; performing segmentation processing on the intermediate image based on the position information of the facial parts, obtaining multiple facial part images contained in the intermediate image; for each facial part image, determining a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold; outputting the target material so that the user can select the material corresponding to the facial part to obtain an image of the target image corresponding to the target facial style. The present disclosure can accurately provide users with suitable materials to be selected.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and more specifically, the embodiments of the present disclosure relate to an image processing method, medium, apparatus and computing device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Face pinching is a function that allows users to use pre-defined templates of various facial parts to freely combine and form a personalized complete face. Its main feature is that various parts of the face can be freely edited. For example, game applications, short video applications, image processing applications, etc. can all provide users with the face pinching function.

[0004] In the related art, the face pinching function is usually implemented in the following way: for different facial styles (such as comic style), a variety of materials corresponding to various parts of the face (such as round face, square face or melon seed face, etc.) are manually drawn to obtain a material library; based on the material library, users can select the corresponding materials that are most similar to their own or other people's photos, combine the materials, and finally obtain a complete face. However, when the face pinching function is implemented in the above way, it is not possible to accurately provide suitable materials for different users. Summary of the invention

[0005] The present disclosure provides an image processing method, medium, apparatus and computing device to solve the problem that when a face pinching function is implemented by related technologies, appropriate materials to be selected cannot be accurately provided for different users.

[0006] In a first aspect of the embodiments of the present disclosure, there is provided an image processing method, comprising:

[0007] Receiving an instruction to perform style conversion of a target facial style on a target image;

[0008] According to the style conversion instruction, face detection is performed on the target image to obtain the position information of the facial parts in the target image, and the target image is converted into the target facial style to obtain an intermediate image;

[0009] Based on the position information of the facial parts, the intermediate image is segmented to obtain a plurality of facial part images contained in the intermediate image;

[0010] For each facial part image, determining a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample facial part image contained in sample images obtained based on a plurality of sample images containing faces of sample objects and different facial styles;

[0011] Output the target material so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

[0012] In a possible implementation, the material library is obtained in the following manner: a sample image is acquired, the sample image includes the face of the sample object; for each sample image, the following operations are performed: face detection is performed on the sample image to obtain position information of facial parts in the sample image; different facial style conversion processing is performed on the sample image to obtain sample intermediate images corresponding to different facial styles; based on the position information of facial parts in the sample image, a segmentation processing is performed on the sample intermediate image to obtain multiple sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images.

[0013] In a possible implementation, face detection is performed on a sample image to obtain position information of facial parts in the sample image, including: performing face detection on the sample image to obtain multiple facial parts in the sample image and confidence levels corresponding to the multiple facial parts; and obtaining, based on the confidence levels, position information of facial parts in the sample image whose confidence levels are greater than a confidence threshold.

[0014] In a possible implementation, for each facial part image, determining a target material whose feature similarity between the facial part image and the material in the material library is greater than a similarity threshold includes: for each facial part image, obtaining a target material whose feature similarity between the facial part image and the material in the material library is greater than a similarity threshold; if the number of target materials is greater than a preset number, sorting the target materials from high to low according to the feature similarity, and determining a preset number of target materials whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold; if the number of target materials is less than or equal to the preset number, determining a target material whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold.

[0015] In a possible implementation, performing face detection on a target image to obtain position information of facial parts in the target image includes: inputting the target image into a face detection model to perform face detection to obtain position information of facial parts in the target image.

[0016] In a possible implementation, performing a conversion process of a target facial style on a target image to obtain an intermediate image includes: inputting the target image into a facial stylization model to perform a conversion process of the target facial style to obtain an intermediate image.

[0017] In a possible implementation, after segmenting the intermediate image based on the position information of the facial parts to obtain a plurality of facial part images contained in the intermediate image, the image processing method further includes: inputting the facial part images into an image feature extraction model to extract image features to obtain features corresponding to the facial part images.

[0018] In a second aspect, an embodiment of the present disclosure provides an image processing device, including:

[0019] A receiving module, used for receiving an instruction to perform style conversion on a target image into a target facial style;

[0020] The first processing module is used to perform face detection on the target image according to the style conversion instruction, obtain the position information of the facial parts in the target image, and perform a conversion process of the target facial style on the target image to obtain an intermediate image;

[0021] A second processing module is used to segment the intermediate image based on the position information of the facial parts to obtain a plurality of facial part images contained in the intermediate image;

[0022] A determination module, for determining, for each facial part image, a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample facial part image contained in sample images obtained based on a plurality of sample images containing faces of sample objects and different facial styles;

[0023] The output module is used to output the target material so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

[0024] In a possible embodiment, the image processing device also includes an acquisition module, which is used to obtain a material library in the following manner: obtain a sample image, the sample image includes the face of the sample object; for each sample image, perform the following operations: perform face detection on the sample image to obtain position information of facial parts in the sample image; perform conversion processing of different facial styles on the sample image to obtain sample intermediate images corresponding to different facial styles; based on the position information of facial parts in the sample image, perform segmentation processing on the sample intermediate image to obtain multiple sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images.

[0025] In a possible implementation, when the acquisition module is used to perform face detection on a sample image and obtain position information of facial parts in the sample image, it is specifically used to: perform face detection on the sample image to obtain multiple facial parts in the sample image and confidence levels corresponding to the multiple facial parts; and obtain, based on the confidence levels, the position information of the facial parts in the sample image whose confidence levels are greater than a confidence threshold.

[0026] In a possible implementation, the determination module is specifically used to: for each facial part image, obtain target materials whose feature similarity between the facial part image and the materials in the material library is greater than a similarity threshold; if the number of target materials is greater than a preset number, sort the target materials from high to low according to the feature similarity, and determine a preset number of target materials whose feature similarity between the facial part image and the materials in the material library is greater than the similarity threshold; if the number of target materials is less than or equal to the preset number, determine the target materials whose feature similarity between the facial part image and the materials in the material library is greater than the similarity threshold.

[0027] In a possible implementation manner, the first processing module is specifically configured to: input the target image into a face detection model to perform face detection, and obtain position information of facial parts in the target image.

[0028] In a possible implementation manner, the first processing module is specifically configured to: input the target image into the facial stylization model to perform conversion processing of the target facial style, and obtain an intermediate image.

[0029] In a possible implementation, the second processing module is also used to: perform segmentation processing on the intermediate image based on the position information of the facial parts, and after obtaining a plurality of facial part images contained in the intermediate image, input the facial part images into an image feature extraction model for image feature extraction to obtain features corresponding to the facial part images.

[0030] In a third aspect, an embodiment of the present disclosure provides a computing device, including: a processor, and a memory communicatively connected to the processor;

[0031] Memory stores computer-executable instructions;

[0032] The processor executes the computer-executable instructions stored in the memory to implement the image processing method as described in the first aspect of the present disclosure.

[0033] In a fourth aspect, an embodiment of the present disclosure provides a storage medium, in which computer program instructions are stored. When the computer program instructions are executed, the image processing method as described in the first aspect of the present disclosure is implemented.

[0034] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the image processing method as described in the first aspect of the present disclosure when the computer program is executed by a processor.

[0035] The image processing method, medium, apparatus and computing device provided by the embodiments of the present disclosure receive an instruction to perform style conversion of a target image into a target facial style, perform face detection on the target image according to the style conversion instruction, obtain position information of facial parts in the target image, and perform conversion processing of the target facial style on the target image to obtain an intermediate image; segment the intermediate image based on the position information of the facial parts to obtain multiple facial part images contained in the intermediate image; for each facial part image, determine a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample facial part image contained in a sample image automatically obtained based on multiple sample images of faces of sample objects and different facial styles, thereby reducing the cost of manually drawing materials; through feature comparison, a suitable material range can be dynamically obtained according to the target image without manual intervention, thereby reducing time consumption; output the target material so that the user can select the material corresponding to the facial part to obtain an image of the target image corresponding to the target facial style, thereby accurately providing the user with suitable materials to be selected, greatly reducing the cost of selecting materials for the user, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0037] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0038] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0039] Figure 3 A flowchart of an image processing method provided by another embodiment of the present disclosure;

[0040] Figure 4 A flowchart of a method for obtaining a material library provided in an embodiment of the present disclosure;

[0041] Figure 5 A schematic diagram of obtaining and using a material library provided in an embodiment of the present disclosure;

[0042] Figure 6 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0043] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present disclosure;

[0044] Figure 8 A schematic diagram of the structure of a computing device provided in one embodiment of the present disclosure.

[0045] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0046] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, device, equipment, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. The data involved in the present disclosure can be data authorized by the user or fully authorized by all parties, and the embodiments / examples of the present disclosure can be combined with each other.

[0048] According to an embodiment of the present disclosure, an image processing method, medium, apparatus and computing device are proposed.

[0049] In this article, it is important to understand that the terms involved are:

[0050] Face pinching refers to the function of users using pre-defined templates of various facial parts to freely combine and form a personalized complete face. Its main feature is that various parts of the face can be freely edited.

[0051] Face stylization refers to processing face photos into a certain unified style, such as comic style, sketch style, oil painting style, etc., which can be done with the help of Convolutional Neural Networks (CNN) algorithm or traditional algorithms (such as machine learning algorithms) for image processing.

[0052] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0053] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0055] The inventors have found that in the related art, a pre-hand-drawn material library is usually provided, and users can select the corresponding material that is most similar to their own or other people's photos from the material library, combine the materials, and finally obtain a complete face. Among them, a function of adjusting each part within a certain range can be added on the basis of the material library. For example, the user selects a round face, but hopes that the face to be pinched has some ovals, and there is no pre-hand-drawn material that meets the requirements in the material library. At this time, at the technical level, a function of editing the arc within a certain range can be provided, so that a small range of adjustment can be achieved to achieve the final face pinching effect. However, when the face pinching function is realized by the above method, the following problems mainly exist: (1) It is necessary to manually draw facial part materials of various styles. If a very similar face pinching effect is to be achieved, a very rich material library is required; (2) If the face pinching style changes, all materials need to be redrawn; (3) If the material library is too small, the face pinching will not be similar to the actual face. If the material library is too large, it will not be possible to pinch out a suitable face in a short time (high selection cost), and it is not possible to accurately provide suitable materials for different users.

[0056] Based on the above problems, the present disclosure provides an image processing method, medium, device and computing equipment, which save manpower by automatically and quickly generating a large number of material libraries of facial parts with different facial styles; by similarly matching the facial part images in the target image in the material library, the user can be accurately provided with suitable materials to be selected, so that the user can choose to pinch out a face with a target facial style that is more similar to a real face, which can greatly reduce the cost of selecting materials for the user.

[0057] Application Scenario Overview

[0058] First reference Figure 1 The application scenarios of the solutions provided in the present disclosure are illustrated by examples. Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as Figure 1 As shown, in this application scenario, the user uploads a face photo to the server 102 via the mobile phone 101. In response to the user selecting a style conversion instruction to convert the face in the photo into a comic style, the mobile phone 101 sends a style conversion instruction to the server 102. After receiving the style conversion instruction, the server 102 determines multiple comic style candidate materials corresponding to each facial part in the photo based on the face photo, and sends the candidate materials to the mobile phone 101. The mobile phone 101 displays multiple candidate materials corresponding to each facial part for the user to select and combine. Among them, the specific implementation process of the server 102 determining the comic style candidate materials corresponding to each facial part in the photo based on the face photo after receiving the style conversion instruction can be referred to the solutions of the following embodiments.

[0059] It should be noted that Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices in the Figure 1 In the application scenario shown, a data storage device may also be included. The data storage device may be an external memory relative to the server 102 or an internal memory integrated in the server 102 .

[0060] Exemplary Methods

[0061] Combine the following Figure 1 For application scenarios, refer to Figure 2 To describe the image processing method according to the exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0062] First, the image processing method is introduced through a specific embodiment.

[0063] Figure 2 The flowchart of the image processing method provided by one embodiment of the present disclosure is shown in FIG. The method of the embodiment of the present disclosure can be applied to a computing device, which can be a server or a server cluster. Figure 2 As shown, the method of the embodiment of the present disclosure includes:

[0064] S201: Receive an instruction to perform style conversion on a target image into a target facial style.

[0065] In the disclosed embodiment, the target image is, for example, a face photo input by a user or uploaded to an electronic device executing the method embodiment, or an image containing a face sent by other devices to the electronic device executing the method embodiment. The instruction to perform style conversion of the target image to the target facial style may be input by a user to the electronic device executing the method embodiment, or may be sent by other devices to the electronic device executing the method embodiment. Exemplarily, if the target facial style is a comic style, for example, an instruction to perform style conversion of the target image to the comic style is received.

[0066] S202. According to the style conversion instruction, face detection is performed on the target image to obtain position information of facial parts in the target image, and target facial style conversion is performed on the target image to obtain an intermediate image.

[0067] In this step, after receiving the instruction to perform style conversion of the target facial style on the target image, according to the style conversion instruction, face detection can be performed on the target image by a preset face detection method to obtain the position information of each facial part in the target image. If the facial part is, for example, an eyebrow, the position information of the eyebrow can be obtained, specifically, the position information of the eyebrow is, for example, the coordinates, length and width of the upper left corner of the smallest rectangular frame including the eyebrow. And the target image can be converted into the target facial style by a preset face stylization method to obtain an intermediate image. If the target facial style is, for example, a comic style, the intermediate image obtained is an image in comic style. For how to perform face detection on the target image, obtain the position information of the facial parts in the target image, and how to perform target facial style conversion on the target image to obtain the intermediate image, please refer to the subsequent embodiments, which will not be repeated here.

[0068] S203 . Segment the intermediate image based on the position information of the facial parts to obtain a plurality of facial part images included in the intermediate image.

[0069] In this step, after obtaining the position information of each facial part in the target image and the intermediate image, the intermediate image can be segmented based on the position information of each facial part to obtain multiple facial part images contained in the intermediate image. For example, if the intermediate image is a comic-style image and the facial part is an eyebrow, the intermediate image can be segmented based on the position information of the eyebrow to obtain a comic-style eyebrow image contained in the intermediate image.

[0070] S204. For each facial part image, determine a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample facial part image contained in sample images obtained based on a plurality of sample images containing faces of sample objects and different facial styles.

[0071] In this step, the present disclosure does not specifically limit the value of the similarity threshold. After obtaining multiple facial part images contained in the intermediate image, multiple target materials whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold can be determined for each facial part image. Among them, the features of each facial part image can be obtained by a preset image feature extraction method, and the features of the material in the material library can also be obtained by a preset image feature extraction method. Exemplarily, for example, the cosine similarity between the features of the facial part image and the features of the material in the material library can be calculated to obtain the feature similarity, and then the feature similarity can be compared with the similarity threshold to determine the target material whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold. Among them, the features of the facial part image are the feature vectors of the facial part image, and the features of the material in the material library are the feature vectors of the material. In machine learning problems, the features are usually expressed in the form of feature vectors, so when analyzing the similarity between two feature vectors, cosine similarity is often used to express it. Cosine similarity is used to measure the similarity between two feature vectors by measuring the cosine value of the angle between the two feature vectors, and the value range is [-1,1]. The cosine value of the angle between two eigenvectors can be used to determine whether the two eigenvectors point to roughly the same direction. When the two eigenvectors have the same direction, the cosine similarity value is 1; when the angle between the two eigenvectors is 90°, the cosine similarity value is 0. It can be understood that different facial part images can obtain different multiple ranges of candidate materials.

[0072] For how to obtain the material library, please refer to the subsequent embodiments, which will not be described in detail here.

[0073] S205: Output the target material so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

[0074] In this step, after obtaining the target material corresponding to each facial part image, the target material can be output for the user to select the material corresponding to the facial part to obtain an image of the target facial style corresponding to the target image. Exemplarily, the facial part is, for example, eyebrows, and the target facial style is, for example, comic style. After obtaining, for example, 5 comic-style target materials (i.e., target eyebrow images) corresponding to the eyebrows, 5 comic-style target eyebrow images are output to the user, and the user can select a suitable target eyebrow from the 5 comic-style target eyebrow images.

[0075] The image processing method provided by the embodiment of the present disclosure receives an instruction to perform style conversion of a target facial style on a target image, performs face detection on the target image according to the style conversion instruction, obtains position information of facial parts in the target image, and performs conversion processing of the target facial style on the target image to obtain an intermediate image; based on the position information of the facial parts, performs segmentation processing on the intermediate image to obtain multiple facial part images contained in the intermediate image; for each facial part image, determines a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample facial part image contained in a sample image automatically obtained based on multiple sample images of faces of sample objects and different facial styles, thereby reducing the cost of manually drawing materials; through feature comparison, a suitable material range can be dynamically obtained according to the target image without manual intervention, thereby reducing time consumption; outputs the target material so that the user can select the material corresponding to the facial part to obtain an image of the target image corresponding to the target facial style, thereby accurately providing the user with suitable materials to be selected, greatly reducing the cost of selecting materials for the user, and improving the user experience.

[0076] Figure 3 FIG. 1 is a flowchart of an image processing method provided by another embodiment of the present disclosure. Based on the above embodiment, the present disclosure further describes the image processing method. Figure 3 As shown, the method of the embodiment of the present disclosure may include:

[0077] S301: Receive an instruction to perform style conversion on a target image into a target facial style.

[0078] The detailed description of this step can be found in Figure 2 The relevant description of S201 in the illustrated embodiment will not be repeated here.

[0079] In the disclosed embodiment, Figure 2 The step S202 may further include the following two steps S302 and S303:

[0080] S302: According to the style conversion instruction, the target image is input into a face detection model for face detection to obtain position information of facial parts in the target image.

[0081] In this step, the face detection model is pre-trained and is used to output the position information (i.e., coordinate information) of the facial parts in the image. Therefore, according to the style conversion instruction, the target image can be input into the face detection model for face detection to obtain the position information of each facial part of the face in the target image. Exemplarily, the main process of face detection by the face detection model may include: first defining the number of positions of key facial parts to be detected, etc.; then performing data annotation, allowing the face detection model to learn the annotation data; when the face detection model receives a new face image, the output of the face detection model is the content in the annotation format, that is, the position information of the key facial parts of the face. Specifically, the face detection model is, for example, a Practical Facial Landmark Detector (PFLD) model.

[0082] S303: Input the target image into the facial stylization model to perform conversion processing on the target facial style, and obtain an intermediate image.

[0083] In this step, different facial styles have different facial stylization models, and the facial stylization model corresponding to each facial style is pre-trained and used to output facial images of corresponding styles. Therefore, the target image is input into the facial stylization model for conversion processing of the target facial style, and an intermediate image can be obtained, which is an image of the target facial style. Exemplarily, the main process of the facial stylization model for facial style conversion processing may include: firstly obtaining a large number of real face images, then determining the target facial style to be converted, and drawing an image corresponding to the target facial style according to the face image; for example, the target facial style is a comic style, and for 100 face images, 100 corresponding comic style face images can be obtained; the face stylization model outputs a comic style face image according to the input face image, and calculates the loss value (loss) with the annotated comic style face image, that is, adjusting the parameters of the face stylization model according to the annotated data, and training the face stylization model; after the training is completed, the face image is input into the face stylization model, and the face stylization model outputs the corresponding comic style face image.

[0084] S304: Segment the intermediate image based on the position information of the facial parts to obtain a plurality of facial part images included in the intermediate image.

[0085] The detailed description of this step can be found in Figure 2 The relevant description of S203 in the illustrated embodiment will not be repeated here.

[0086] S305: Input the facial part image into an image feature extraction model to extract image features, and obtain features corresponding to the facial part image.

[0087] In this step, the image feature extraction model is pre-trained and is used to extract a feature representation, such as a 128-bit single-precision floating point (float) feature, based on the input image. Exemplarily, for example, by inputting 5 facial part images into the image feature extraction model for image feature extraction, 5 corresponding features can be obtained. Exemplarily, the image feature extraction model is equivalent to a classification network. For example, 100 faces are 100 categories. A classification network (i.e., image feature extraction model) can be trained to correctly classify faces. Then, the output of the image feature extraction model is used as a feature vector. An image of a facial part is input into the image feature extraction model. The image feature extraction model will output a feature vector of, for example, 128 bits (bit) or other lengths. The image feature extraction model is commonly used, for example, in the face recognition (arcface) structure.

[0088] In the disclosed embodiment, Figure 2 The step S204 may further include the following three steps S306 to S308:

[0089] S306: For each facial part image, obtain a target material whose feature similarity between the facial part image and the material in the material library is greater than a similarity threshold.

[0090] Referring to the example of step S204, a plurality of target materials whose feature similarities between the facial part images and the materials in the material library are greater than a similarity threshold can be obtained.

[0091] S307: If the number of target materials is greater than a preset number, the target materials are sorted from high to low according to feature similarity, and a preset number of target materials whose feature similarity between the facial part images and the materials in the material library is greater than a similarity threshold are determined.

[0092] The present disclosure does not specifically limit the preset number. For example, assuming that the preset number is represented by N, after obtaining, for example, M target materials corresponding to each facial part image, that is, the number of target materials is M, if M is greater than N, the target materials are sorted from high to low according to feature similarity, and N target materials whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold are determined.

[0093] S308: If the number of target materials is less than or equal to the preset number, determine the target material whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold.

[0094] Exemplarily, based on the example of step S307, if the number M of target materials is less than or equal to the preset number N, M target materials whose feature similarity between the facial part images and the materials in the material library is greater than the similarity threshold can be determined.

[0095] S309: Output the target material so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

[0096] The detailed description of this step can be found in Figure 2 The relevant description of S205 in the illustrated embodiment will not be repeated here.

[0097] The image processing method provided by the embodiment of the present disclosure performs face detection on a target image through a face detection model to obtain position information of facial parts in the target image; inputs the target image into a face stylization model to perform conversion processing of the target facial style to obtain an intermediate image; based on the position information of the facial parts, performs segmentation processing on the intermediate image to obtain multiple facial part images contained in the intermediate image; inputs the facial part images into an image feature extraction model to perform image feature extraction to obtain features corresponding to the facial part images; can quickly and accurately obtain facial part images in the target image and features corresponding to the facial part images; for each facial part image, obtain Take the target material whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold. If the number of target materials is greater than the preset number, sort the target materials from high to low according to the feature similarity, and determine the preset number of target materials whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold; if the number of target materials is less than or equal to the preset number, determine the target material whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold; through feature comparison, based on the similarity threshold and the preset number, it is possible to accurately provide users with suitable materials to be selected, greatly reducing the cost of users selecting materials and improving user experience.

[0098] Based on the above embodiments, Figure 4 Flow chart of a method for obtaining a material library provided in one embodiment of the present disclosure. Figure 4 As shown, the method of the embodiment of the present disclosure may include:

[0099] S401: Acquire a sample image, where the sample image includes a face of a sample object.

[0100] For example, a batch of images containing real human faces may be collected as sample images by manual collection or web crawling. For example, 10,000 sample images containing real human faces may be collected.

[0101] For each sample image, do the following:

[0102] S402: Perform face detection on the sample image to obtain position information of the face parts in the sample image.

[0103] Exemplarily, referring to the example of step S302, face detection may be performed on the sample image using a pre-trained face detection model to obtain position information of facial parts in the sample image.

[0104] Further, optionally, performing face detection on the sample image to obtain position information of facial parts in the sample image may include: performing face detection on the sample image to obtain multiple facial parts in the sample image and confidence levels corresponding to the multiple facial parts; and obtaining, based on the confidence levels, position information of facial parts in the sample image whose confidence levels are greater than a confidence threshold.

[0105] The present disclosure does not specifically limit the value of the confidence threshold. For example, by performing face detection on a sample image using a pre-trained face detection model, multiple facial parts in the sample image and the confidences corresponding to the multiple facial parts can be obtained, and the confidence corresponding to each facial part is compared with the confidence threshold to obtain the position information of the facial parts in the sample image whose confidence is greater than the confidence threshold, thereby removing the position information of the facial parts that are incorrectly detected. The position information of the facial parts that are incorrectly detected can also be removed manually.

[0106] S403: Perform conversion processing of different facial styles on the sample image to obtain sample intermediate images corresponding to different facial styles.

[0107] In this step, for example, different facial styles are comic style, sketch style, oil painting style, etc. Referring to the example of step S303, the sample image can be converted into different facial styles by using the pre-trained facial stylization model to obtain sample intermediate images corresponding to different facial styles. It can be understood that for a newly added facial style, a corresponding facial stylization model can be added.

[0108] S404: Based on the position information of the facial parts in the sample image, segment the sample intermediate image to obtain a plurality of sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images.

[0109] In this step, after obtaining the position information (i.e., coordinate information) of each facial part in the sample image, the sample intermediate image can be segmented based on the position information of the facial part in the sample image to obtain multiple sample facial part images contained in the sample intermediate image, and sample facial part images with incorrect segmentation processing can be manually eliminated. Referring to the example of step S305, image feature extraction can be performed on each sample facial part image using a pre-trained image feature extraction model to obtain features corresponding to the sample facial part image.

[0110] It can be understood that by repeatedly executing steps S402 to S404, multiple sample facial part images and features corresponding to the sample facial part images contained in each sample image can be obtained, that is, a material library is obtained. If it is necessary to dynamically increase the material, the material library can be quickly expanded by collecting face images and executing steps S402 to S404.

[0111] The method for obtaining a material library provided by the embodiment of the present disclosure performs face detection on a sample image to obtain position information of facial parts in the sample image; performs conversion processing of different facial styles on the sample image to obtain sample intermediate images corresponding to different facial styles; based on the position information of the facial parts in the sample image, performs segmentation processing on the sample intermediate image to obtain multiple sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images. The method can automatically and quickly generate a large number of material libraries of facial parts in different facial styles without manual intervention, thereby reducing the cost of manually drawing materials.

[0112] Based on the above embodiments, Figure 5 A schematic diagram of obtaining and using a material library provided in an embodiment of the present disclosure. Figure 5 As shown, (1) for obtaining a material library, it may include: collecting n face images, namely face image 1 to face image n, wherein each face image is represented by n i Represents; for each face image n i , perform the following operations: for face image n i Perform face detection to obtain face image n i The position information of the facial parts in the face image n i Perform different facial style conversion processing to obtain intermediate images corresponding to different facial styles; based on the face image n iThe position information of the facial parts in the image is used to segment the intermediate image, and multiple facial part images and features corresponding to the facial part images contained in the intermediate image are obtained. Taking the facial style as a comic style as an example, a comic style nose image and features corresponding to the nose image, a comic style eye image and features corresponding to the eye image, etc. can be obtained, wherein the feature representation is, for example, a 128-bit single-precision floating-point feature, and the feature value is represented by an id. Therefore, a material library can be obtained. (2) Using the material library may include: obtaining a real face image of the user, performing face detection on the real face image according to a style conversion instruction, obtaining the position information of the facial parts in the real face image, and converting the real face image into a target facial style to obtain an intermediate image; based on the position information of the facial parts, performing segmentation processing on the intermediate image to obtain a plurality of facial part images contained in the intermediate image, and obtaining features corresponding to each facial part image; taking the facial style as a comic style as an example, a comic style nose image and features corresponding to the nose image, a comic style eye image and features corresponding to the eye image, etc. may be obtained, wherein the feature representation is, for example, a 128-bit single-precision floating-point feature, and the feature value is represented by an id. For each facial part image, the features of the facial part image are compared with the features of the materials of the corresponding parts of the target facial style in the material library to determine the most similar m target materials whose feature similarity is greater than the similarity threshold; the most similar m target materials corresponding to each facial part are output, that is, the most similar m candidate materials corresponding to each facial part are obtained, so that the user can select the material corresponding to the facial part to obtain an image of the target facial style corresponding to the real face image.

[0113] Exemplary Devices

[0114] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 6 The image processing device according to the exemplary embodiment of the present disclosure is described. The device according to the exemplary embodiment of the present disclosure can implement each process in the aforementioned image processing method embodiment and achieve the same function and effect.

[0115] Figure 6 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the image processing device 600 of the embodiment of the present disclosure includes: a receiving module 601, a first processing module 602, a second processing module 603, a determination module 604 and an output module 605. Among them:

[0116] The receiving module 601 is used to receive an instruction for performing style conversion on a target image into a target facial style.

[0117] The first processing module 602 is used to perform face detection on the target image according to the style conversion instruction to obtain the position information of the facial parts in the target image, and perform a target facial style conversion process on the target image to obtain an intermediate image.

[0118] The second processing module 603 is used to segment the intermediate image based on the position information of the facial parts to obtain a plurality of facial part images included in the intermediate image.

[0119] The determination module 604 is used to determine, for each facial part image, a target material whose feature similarity between the facial part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample facial part image contained in sample images obtained based on a plurality of sample images of the face of the sample object and different facial styles.

[0120] The output module 605 is used to output the target material so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

[0121] In a possible implementation, the image processing device may further include an acquisition module 606, which is used to obtain a material library in the following manner: obtain a sample image, the sample image includes the face of the sample object; for each sample image, perform the following operations: perform face detection on the sample image to obtain position information of facial parts in the sample image; perform conversion processing of different facial styles on the sample image to obtain sample intermediate images corresponding to different facial styles; based on the position information of facial parts in the sample image, perform segmentation processing on the sample intermediate image to obtain multiple sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images.

[0122] In a possible implementation, when the acquisition module 606 is used to perform face detection on a sample image and obtain position information of facial parts in the sample image, it can be specifically used to: perform face detection on the sample image to obtain multiple facial parts in the sample image and confidence levels corresponding to the multiple facial parts; and obtain, based on the confidence levels, position information of facial parts in the sample image whose confidence levels are greater than a confidence threshold.

[0123] In a possible implementation, the determination module 604 may be specifically used to: for each facial part image, obtain a target material whose feature similarity between the facial part image and the material in the material library is greater than a similarity threshold; if the number of target materials is greater than a preset number, sort the target materials from high to low according to the feature similarity, and determine a preset number of target materials whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold; if the number of target materials is less than or equal to the preset number, determine the target material whose feature similarity between the facial part image and the material in the material library is greater than the similarity threshold.

[0124] In a possible implementation, the first processing module 602 may be specifically configured to: input the target image into a face detection model to perform face detection, and obtain position information of facial parts in the target image.

[0125] In a possible implementation, the first processing module 602 may be specifically configured to: input the target image into the facial stylization model to perform conversion processing of the target facial style, and obtain an intermediate image.

[0126] In a possible implementation, the second processing module 603 can also be used to: based on the position information of the facial parts, segment the intermediate image to obtain multiple facial part images contained in the intermediate image, and then input the facial part images into the image feature extraction model to extract image features to obtain features corresponding to the facial part images.

[0127] The device of the embodiment of the present disclosure can be used to execute the solution of the image processing method in any of the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0128] Exemplary Media

[0129] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Figure 7 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0130] Figure 7 A schematic diagram of a storage medium provided by an embodiment of the present disclosure. Figure 7 As shown, the storage medium 700 stores a program product for implementing the above method according to an embodiment of the present disclosure, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0131] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0132] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium.

[0133] Program code for performing the disclosed operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0134] Exemplary Computing Devices

[0135] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Figure 8 A computing device according to an exemplary embodiment of the present disclosure is described.

[0136] Figure 8 The computing device 800 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0137] Figure 8 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown in FIG. Figure 8As shown, the computing device 800 is in the form of a general computing device. The components of the computing device 800 may include, but are not limited to: the at least one processing unit 801, the at least one storage unit 802, and a bus 803 connecting different system components (including the processing unit 801 and the storage unit 802). Exemplarily, the processing unit 801 may be specifically a processor, the storage unit 802 stores computer-executable instructions, and the processing unit 801 executes the computer-executable instructions stored in the storage unit 802 to implement the above-mentioned image processing method.

[0138] The bus 803 includes a data bus, a control bus, and an address bus.

[0139] The storage unit 802 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 8021 and / or a cache memory 8022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 8023 .

[0140] The storage unit 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024, such program modules 8024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0141] The computing device 800 may also communicate with one or more external devices 804 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 805. Furthermore, the computing device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 806. Figure 8 As shown, the network adapter 806 communicates with other modules of the computing device 800 via the bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] It should be noted that, although several units / modules or sub-units / modules of the image processing device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided to be embodied by multiple units / modules.

[0143] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0144] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An image processing method, comprising: Receiving an instruction to perform style conversion of a target facial style on a target image; According to the style conversion instruction, face detection is performed on the target image to obtain position information of facial parts in the target image, and conversion processing of the target face style is performed on the target image to obtain an intermediate image; Based on the position information of the facial parts, segmenting the intermediate image to obtain a plurality of facial part images included in the intermediate image; For each face part image, determining a target material whose feature similarity between the face part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample face part image contained in the sample image obtained based on a plurality of sample images containing faces of sample objects and different facial styles; The target material is outputted so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

2. The image processing method according to claim 1, wherein the material library is obtained by: Acquire a sample image, wherein the sample image includes a face of a sample object; For each sample image, do the following: Performing face detection on the sample image to obtain position information of the face part in the sample image; Performing different facial style conversion processing on the sample image to obtain sample intermediate images corresponding to the different facial styles; Based on the position information of the facial parts in the sample image, the sample intermediate image is segmented to obtain a plurality of sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images.

3. The image processing method according to claim 2, wherein the step of performing face detection on the sample image to obtain position information of facial parts in the sample image comprises: Performing face detection on the sample image to obtain multiple facial parts in the sample image and confidence levels corresponding to the multiple facial parts; According to the confidence level, position information of the facial part in the sample image whose confidence level is greater than a confidence level threshold is obtained.

4. The image processing method according to claim 1, wherein for each facial part image, determining a target material whose feature similarity between the facial part image and a material in the material library is greater than a similarity threshold comprises: For each facial part image, obtaining a target material whose feature similarity between the facial part image and a material in the material library is greater than a similarity threshold; If the number of the target materials is greater than a preset number, the target materials are sorted from high to low according to the feature similarity, and the preset number of target materials whose feature similarity between the facial part image and the materials in the material library is greater than a similarity threshold are determined; If the number of the target materials is less than or equal to a preset number, a target material whose feature similarity between the facial part image and the materials in the material library is greater than a similarity threshold is determined.

5. The image processing method according to any one of claims 1 to 4, wherein the step of performing face detection on the target image to obtain position information of facial parts in the target image comprises: The target image is input into a face detection model for face detection to obtain position information of the face parts in the target image.

6. The image processing method according to any one of claims 1 to 4, wherein the step of performing the target facial style conversion process on the target image to obtain an intermediate image comprises: The target image is input into a facial stylization model to perform conversion processing on the target facial style, thereby obtaining the intermediate image.

7. The image processing method according to any one of claims 1 to 4, further comprising: The facial part image is input into an image feature extraction model to extract image features, and features corresponding to the facial part image are obtained.

8. An image processing device, comprising: A receiving module, used for receiving an instruction to perform style conversion on a target image into a target facial style; A first processing module, configured to perform face detection on the target image according to the style conversion instruction, obtain position information of facial parts in the target image, and perform conversion processing of the target face style on the target image to obtain an intermediate image; a second processing module, configured to segment the intermediate image based on the position information of the facial parts, to obtain a plurality of facial part images contained in the intermediate image; A determination module, configured to determine, for each face part image, a target material whose feature similarity between the face part image and a material in a material library is greater than a similarity threshold, wherein the material library is a sample face part image contained in the sample image obtained based on a plurality of sample images containing faces of sample objects and different facial styles; The output module is used to output the target material so that the user can select the material corresponding to the facial part to obtain the target image corresponding to the target facial style.

9. The image processing device according to claim 8, further comprising an acquisition module, configured to obtain the material library in the following manner: Acquire a sample image, wherein the sample image includes a face of a sample object; For each sample image, do the following: Performing face detection on the sample image to obtain position information of the face part in the sample image; Performing different facial style conversion processing on the sample image to obtain sample intermediate images corresponding to the different facial styles; Based on the position information of the facial parts in the sample image, the sample intermediate image is segmented to obtain a plurality of sample facial part images contained in the sample intermediate image and features corresponding to the sample facial part images.

10. The image processing device according to claim 9, when the acquisition module is used to perform face detection on the sample image to obtain the position information of the face part in the sample image, is specifically used to: Performing face detection on the sample image to obtain multiple facial parts in the sample image and confidence levels corresponding to the multiple facial parts; According to the confidence level, position information of the facial part in the sample image whose confidence level is greater than a confidence level threshold is obtained.

11. The image processing device according to claim 8, wherein the determining module is specifically configured to: For each facial part image, obtaining a target material whose feature similarity between the facial part image and a material in the material library is greater than a similarity threshold; If the number of the target materials is greater than a preset number, the target materials are sorted from high to low according to the feature similarity, and the preset number of target materials whose feature similarity between the facial part image and the materials in the material library is greater than a similarity threshold are determined; If the number of the target materials is less than or equal to a preset number, a target material whose feature similarity between the facial part image and the materials in the material library is greater than a similarity threshold is determined.

12. The image processing device according to any one of claims 8 to 11, wherein the first processing module is specifically configured to: The target image is input into a face detection model for face detection to obtain position information of the face parts in the target image.

13. The image processing device according to any one of claims 8 to 11, wherein the first processing module is specifically configured to: The target image is input into a facial stylization model to perform conversion processing on the target facial style, thereby obtaining the intermediate image.

14. The image processing device according to any one of claims 8 to 11, wherein the second processing module is further configured to: Based on the position information of the facial parts, the intermediate image is segmented to obtain a plurality of facial part images contained in the intermediate image, and then the facial part images are input into an image feature extraction model to extract image features to obtain features corresponding to the facial part images.

15. A computing device comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the image processing method according to any one of claims 1 to 7. 16 . A storage medium storing computer program instructions, wherein the computer program instructions, when executed, implement the image processing method according to claim 1 . 17 . A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image processing method according to claim 1 is implemented.

Images