Model training, image processing method, device, equipment and storage medium

By training the model and adjusting the parameters, the problem of time-consuming and labor-intensive beauty methods of traditional facial image beauty is solved, and efficient personalized makeup and makeup removal are achieved, improving the detailed effect of image processing.

CN113269719BActive Publication Date: 2024-11-05BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110409627.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-16
Publication Date
2024-11-05
Estimated Expiration
2041-04-16

AI Technical Summary

Technical Problem

The traditional facial image beauty method requires manual operation, which is time-consuming and labor-intensive. The existing image processing algorithm cannot achieve detailed personalized makeup effects.

Method used

By obtaining the set of face images, the model is trained to generate the intermediate face image, and the loss function value is determined through area segmentation and comparison, and the model parameters are adjusted to achieve refined processing of face images and personalized makeup.

Benefits of technology

It realizes efficient personalized makeup and makeup removal for facial images, reduces labor costs, improves processing efficiency, and makes the processed image details more obvious.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113269719B_ABST
    Figure CN113269719B_ABST
Patent Text Reader

Abstract

The present disclosure provides a model training, image processing method, device, equipment and storage medium, which relates to the field of image processing, specifically to the field of computer vision, augmented reality, and deep learning technology. The specific implementation scheme is: obtain a face image set, the face image set includes a subset of face images before makeup and a subset of face images after makeup, each face image before makeup corresponds to each face image after makeup; based on the face image set, perform the following training steps multiple times: input each face image before makeup or each face image after makeup into the model to obtain an intermediate face image output by the model; perform regional segmentation on the intermediate face image and the face image corresponding to the input face image; determine the loss function value based on the comparison results of each region in the intermediate face image and each region in the face image corresponding to the input face image; adjust the parameters of the model according to the loss function value. This implementation method can make up or remove makeup on face images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, specifically to the field of computer vision, augmented reality, and deep learning technology, and especially to model training, image processing methods, devices, equipment, and storage media. Background Art

[0002] Traditionally, makeup for human images is generated manually through image software, including common image processing such as skin smoothing, whitening, and beauty contact lenses. This method is time-consuming and labor-intensive. Later, some image processing algorithms were developed to replace this manual operation effect, such as Gaussian blur and language algorithms. These image processing algorithms generally perform overall makeup. Summary of the invention

[0003] Provided are a model training, image processing method, apparatus, device and storage medium.

[0004] According to a first aspect, a model training method is provided, comprising: obtaining a face image set, wherein the face image set comprises a subset of face images before makeup and a subset of face images after makeup, and each face image before makeup in the subset of face images before makeup corresponds one to one to each face image after makeup in the subset of face images after makeup; based on the face image set, performing the following training steps multiple times: inputting each face image before makeup in the subset of face images before makeup or each face image after makeup in the subset of face images after makeup into a model to obtain an intermediate face image output by the model; performing region segmentation on the intermediate face image and the face image corresponding to the input face image; determining a loss function value based on comparison results of each region in the intermediate face image and each region in the face image corresponding to the input face image; and adjusting the parameters of the model according to the loss function value.

[0005] According to a second aspect, an image processing method is provided, comprising: obtaining a target facial image; determining a matching model from at least one pre-trained model based on the target facial image, wherein at least one model is trained by the model training method described in the first aspect; determining a processed facial image corresponding to the target facial image based on the target facial image and the matching model.

[0006] According to a third aspect, a model training device is provided, comprising: a first acquisition unit, configured to acquire a face image set, wherein the face image set comprises a subset of face images before makeup and a subset of face images after makeup, and each face image before makeup in the subset of face images before makeup corresponds one to one to each face image after makeup in the subset of face images after makeup; a model training unit, configured to perform training steps multiple times based on the face image set through the following modules: an input module, configured to input each face image before makeup in the subset of face images before makeup or each face image after makeup in the subset of face images after makeup into a model to obtain an intermediate face image output by the model; a segmentation module, configured to perform regional segmentation on the intermediate face image and the face image corresponding to the input face image; a determination module, configured to determine a loss function value based on a comparison result of each region in the intermediate face image and each region in the face image corresponding to the input face image; and an adjustment module, configured to adjust the parameters of the model according to the loss function value.

[0007] According to a fourth aspect, an image processing device is provided, comprising: a second acquisition unit, configured to acquire a target facial image; a model determination unit, configured to determine a matching model from at least one pre-trained model based on the target facial image, wherein at least one model is trained by the model training method described in the first aspect; and an image processing unit, configured to determine a processed facial image corresponding to the target facial image based on the target facial image and the matching model.

[0008] According to the fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the method described in the first aspect or executes the method described in the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect or the method described in the second aspect.

[0010] According to a seventh aspect, a computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method described in the first aspect or executes the method described in the second aspect.

[0011] The technology disclosed herein provides an image processing model capable of applying or removing makeup on a facial image.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0015] Figure 2 is a flowchart of an embodiment of a model training method according to the present disclosure;

[0016] Figure 3 is a flowchart of another embodiment of the model training method according to the present disclosure;

[0017] Figure 4 is a flow chart of an embodiment of an image processing method according to the present disclosure;

[0018] Figure 5 It is a schematic diagram of an application scenario of the model training method and image processing method disclosed in the present invention;

[0019] Figure 6 is a structural schematic diagram of an embodiment of a model training device according to the present disclosure;

[0020] Figure 7 is a structural schematic diagram of an embodiment of an information output device according to the present disclosure;

[0021] Figure 8 It is a block diagram of an electronic device used to implement the model training method and image processing method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] Figure 1An exemplary system architecture 100 is shown to which an embodiment of the present disclosure may be applied for a model training method, an image processing method or a model training device, or an image processing device.

[0025] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, network 104 and server 105. Network 104 is used to provide a medium for communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0026] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104, for example, to receive a trained target model, etc. Various communication client applications, such as image processing applications, social platform applications, etc., can be installed on the terminal devices 101, 102, 103. The user can process the face image through the image processing application and the model obtained from the server 105.

[0027] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, e-book readers, car computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0028] The server 105 may be a server that provides various services, such as a background server that processes the face images provided by the terminal devices 101, 102, 103. The background server may train the initial model using the face image collection to obtain the target model, and feed the target model back to the terminal devices 101, 102, 103.

[0029] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0030] It should be noted that the model training method provided in the embodiment of the present disclosure is generally executed by the server 105, and the image processing method provided in the embodiment of the present disclosure can be executed by the terminal devices 101, 102, 103, or by the server 105. Accordingly, the model training device is generally provided in the server 105, and the image processing device can be provided in the terminal devices 101, 102, 103, or in the server 105.

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0032] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a model training method according to the present disclosure. The model training method of this embodiment includes the following steps:

[0033] Step 201, obtaining a set of face images.

[0034] In this embodiment, the execution subject of the model training method (for example Figure 1 The server 105 shown in the figure may first obtain a set of facial images. The set of facial images includes a subset of facial images before makeup and a subset of facial images after makeup. And each facial image before makeup in the subset of facial images before makeup corresponds one to one with each facial image after makeup in the subset of facial images after makeup. The facial image before makeup refers to an image taken of a bare face, or an image of a facial image processed with bare face. The facial image after makeup may be an image taken of a face after makeup, or an image of a facial image processed with makeup. The facial images in this embodiment may come from a public data set, or the acquisition of the facial images is authorized by the user corresponding to the facial images.

[0035] In some specific applications, each of the makeup-applied face images in the above-mentioned makeup-applied face image subset is processed by the same person, or each of the makeup-applied face images in the makeup-applied face image subset represents a makeup style (e.g., Tang Dynasty style, Qing Dynasty style, etc.). It is understandable that a makeup style has some unique characteristics of its own. For example, a makeup style has thin eyebrows, which can be thin curved eyebrows or thin raised eyebrows. If it is a raised eyebrow, it is generally raised at two-thirds of the eye and then elongated along the end of the eye. The makeup of the eye is relatively plain.

[0036] Step 202, based on the face image set, perform the following training steps multiple times:

[0037] Step 2021, input each face image before makeup in the face image subset before makeup or each face image after makeup in the face image subset after makeup into the model to obtain an intermediate face image output by the model.

[0038] In this embodiment, after acquiring the facial image set, the execution subject may input each pre-makeup facial image in the pre-makeup facial image subset or each post-makeup facial image in the post-makeup facial image subset into the model. And the output of the model is recorded as the intermediate facial image. Specifically, when training the makeup model, the execution subject may input each pre-makeup facial image in the pre-makeup facial image subset into the model. When training the makeup removal model, the execution subject may input each post-makeup facial image in the post-makeup facial image subset into the model. The above-mentioned model may be a neural network, such as a convolutional neural network, an adversarial neural network, etc.

[0039] Step 2022, performing region segmentation on the intermediate face image and the face image corresponding to the input face image.

[0040] The execution subject may perform regional segmentation on the intermediate face image and the face image corresponding to the input face image. Specifically, the execution subject may perform regional segmentation on the intermediate face image and the corresponding face image according to the facial features. The purpose of this is to refine the processing of the facial features when applying or removing makeup.

[0041] Here, if the face image of the input model is a face image before makeup, the face image corresponding to the input face image is the face image after makeup. If the face image of the input model is a face image after makeup, the face image corresponding to the input face image is the face image before makeup.

[0042] Step 2023, determining a loss function value based on the comparison results of each region in the intermediate face image and each region in the face image corresponding to the input face image.

[0043] The execution subject may compare each region of the intermediate face image with each region in the face image corresponding to the input face image to obtain a comparison result. And determine the loss function value based on the comparison result. Specifically, the execution subject may compare the facial features in the intermediate face image with the facial features in the face image corresponding to the input face image. The comparison result may include the similarity of the facial features in the two face images, and the loss function value may be determined based on the comparison result.

[0044] Step 2024, adjusting the parameters of the model according to the loss function value.

[0045] After obtaining the loss function value, the execution subject can adjust the parameters of the model in combination with the preset threshold. The parameters of the model can include the weights of the convolution layer, the parameters of the convolution kernel, etc.

[0046] The model training method provided by the above embodiment of the present disclosure can use a face image set to train a face makeup model and a face makeup removal model. Refinement processing can be achieved by comparing each area of ​​the intermediate face image output by the model with each area in the face image corresponding to the input face image.

[0047] Continue to see Figure 3 , which shows a process 300 of another embodiment of the model training method according to the present disclosure. Figure 3 As shown, the method of this embodiment may include the following steps:

[0048] Step 301, obtaining a set of face images.

[0049] Step 302, based on the face image set, perform the following training steps multiple times:

[0050] Step 3021, input each face image before makeup in the face image subset before makeup or each face image after makeup in the face image subset after makeup into the model to obtain an intermediate face image output by the model.

[0051] Step 3022, performing region segmentation on the intermediate face image and the face image corresponding to the input face image.

[0052] Step 3023, for each area in the intermediate face image, determine a first comparison result based on a comparison between the area and a corresponding area in the face image corresponding to the input face image; for each pixel in the intermediate face image, determine a second comparison result based on a comparison between the pixel and a corresponding pixel in the face image corresponding to the input face image; determine the loss function value based on the first comparison result and the second comparison result.

[0053] In this embodiment, when determining the first comparison result, the execution subject may compare each area in the intermediate face image with the corresponding area in the face image corresponding to the input face image. Specifically, if the input face image is a face image before makeup, each area in the intermediate face image is compared with each corresponding area in the face image after makeup. If the input face image is a face image after makeup, each area in the intermediate face image is compared with each corresponding area in the face image before makeup.

[0054] The execution subject can extract the features of each region of the intermediate face image respectively, and extract the features of each region in the corresponding face image respectively, and use the distance between the features of the corresponding regions as the similarity of each region, and use the above similarity as the first comparison result.

[0055] The execution subject may also compare the pixel value of each pixel in the intermediate face image with the pixel value of each pixel in the corresponding face image. Specifically, the execution subject may use the difference between each pixel of the two face images as the second comparison result.

[0056] The execution entity may weight the first comparison result and the second comparison result, and determine the loss function value according to the weighted result.

[0057] In some optional implementations of this embodiment, the first comparison result can be specifically obtained by Figure 3 The following steps not shown in the figure are used to achieve: determining a local color histogram of a corresponding area in the face image corresponding to the area and the input face image; and determining a first comparison result according to the local color histogram.

[0058] In this implementation, the execution subject can calculate the local color histogram of each area in the intermediate face image and the corresponding area in the corresponding face image. Color feature is a global feature that describes the surface properties of the scene corresponding to the image or image area. Generally, color features are based on pixel features, and all pixels belonging to the image or image area have their own contributions. Since color is insensitive to changes in the direction, size, etc. of the image or image area, color features cannot capture the local features of the object in the image well. In addition, when only color features are used for query, if the database is large, many unnecessary images are often retrieved. Color histogram is the most commonly used method to express color features. Its advantage is that it is not affected by image rotation and translation changes, and further with the help of normalization, it can be unaffected by changes in image scale. The execution subject can use the local color histogram to determine the first comparison result. In this way, local training can be achieved.

[0059] In some optional implementations of this embodiment, the second comparison result can be specifically obtained by Figure 3 The following steps not shown in the figure are used to achieve this: according to the difference between the pixel value of the pixel and the pixel value of the corresponding pixel in the face image corresponding to the input face image; according to the difference, determine the second comparison result.

[0060] In this implementation, the execution entity may calculate the difference between each pixel in the intermediate face image and the corresponding pixel in the corresponding face image, and use the difference as the second comparison result.

[0061] Step 3024, adjust the parameters of the model according to the loss function value.

[0062] The model training method provided by the above embodiment of the present disclosure can use the differences between regions and the differences between pixels to determine the loss function value, and adjust the parameters of the model according to the above loss function value, thereby achieving the learning of the details of the face image.

[0063] See also Figure 4 , which shows a process 400 of an embodiment of the image processing method according to the present disclosure. Figure 4 As shown, the method of this embodiment may include the following steps:

[0064] Step 401, obtaining a target face image.

[0065] In this embodiment, the execution subject of the image processing method (for example Figure 1 The terminal devices 101, 102, 103) shown in the figure can obtain the target face image in various ways. The target face image can be obtained by the user in various authorized ways.

[0066] Step 402: Determine a matching model from at least one pre-trained model based on the target face image.

[0067] The execution subject can determine a matching model from at least one pre-trained model through the target face image. The at least one model is Figure 2 or Figure 3 The at least one model may include a makeup model and a makeup removal model. The execution subject may analyze the target face image to determine whether the target face image is a makeup face image. If the target face image is a makeup face image, the makeup removal model may be used as a matching model. If the target face image is not a makeup face image, the makeup model may be used as a matching model.

[0068] Step 403: Determine a processed facial image corresponding to the target facial image according to the target facial image and the matching model.

[0069] After determining the matching model, the execution subject can directly input the target face image into the matching model to obtain a processed face image corresponding to the target face image, thereby realizing makeup application or makeup removal of the target face image.

[0070] The image processing method provided by the above-mentioned embodiments of the present disclosure can apply makeup or remove makeup on a target facial image, and because the model learns the details of the facial image, the details of the processed facial image are more obvious.

[0071] Continue to see Figure 5 , which shows a schematic diagram of an application scenario of the model training method and image processing method according to the present disclosure. Figure 5 In the application scenario, the server 501 trains a convolutional neural network through a face image set. Specifically, the server 501 can use the face image before makeup in the face image set as input, and the corresponding face image after makeup as the expected output, to train a makeup model. It can also use the face image after makeup in the face image set as input, and the corresponding face image before makeup as the expected output, to train a makeup removal model. The trained makeup model and makeup removal model are sent to the terminal device 502. When processing a face image, the user of the terminal device 502 first obtains the target face image, and then selects the makeup model to obtain the face image after makeup.

[0072] Further references Figure 6 , as a response to the above Figure 2-3 The present disclosure provides an embodiment of a model training device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0073] like Figure 6 As shown, the model training device 600 of this embodiment includes: a first acquisition unit 601 and a model training unit 602. The model training unit 602 may include: an input module 6021, a segmentation module 6022, a determination module 6023 and an adjustment module 6024.

[0074] The first acquisition unit 601 is configured to acquire a face image set, wherein the face image set includes a pre-makeup face image subset and a post-makeup face image subset, and each pre-makeup face image in the pre-makeup face image subset corresponds to each post-makeup face image in the post-makeup face image subset.

[0075] The model training unit 602 is configured to perform the training steps multiple times based on the face image set through the following modules:

[0076] The input module 6021 is configured to input each pre-makeup facial image in the pre-makeup facial image subset or each post-makeup facial image in the post-makeup facial image subset into the model to obtain an intermediate facial image output by the model.

[0077] The segmentation module 6022 is configured to perform region segmentation on the intermediate face image and the face image corresponding to the input face image.

[0078] The determination module 6023 is configured to determine the loss function value based on the comparison results of each area in the intermediate face image and each area in the face image corresponding to the input face image.

[0079] The adjustment module 6024 is configured to adjust the parameters of the model according to the loss function value.

[0080] In some optional implementations of this embodiment, the determination module 6023 can be further configured to: for each area in the intermediate face image, determine a first comparison result based on a comparison between the area and a corresponding area in the face image corresponding to the input face image; for each pixel in the intermediate face image, determine a second comparison result based on a comparison between the pixel and a corresponding pixel in the face image corresponding to the input face image; determine the loss function value based on the first comparison result and the second comparison result.

[0081] In some optional implementations of this embodiment, the determination module 6023 can be further configured to: determine a local color histogram of the area corresponding to the input face image and the area in the face image corresponding to the area; and determine a first comparison result based on the local color histogram.

[0082] In some optional implementations of this embodiment, the determination module 6023 can be further configured to: determine the second comparison result based on the difference between the pixel value of the pixel and the pixel value of the corresponding pixel in the facial image corresponding to the input facial image; and

[0083] It should be understood that the units 601 to 602 and modules 6021 to 6024 described in the model training device 600 are respectively the same as those described in the reference Figure 2 Therefore, the operations and features described above for the model training method are also applicable to the device 600 and the units contained therein, and will not be repeated here.

[0084] Further references Figure 7 , as a response to the above Figure 4 The present disclosure provides an embodiment of an image processing device, and the device embodiment is Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0085] like Figure 7 As shown, the image processing device 700 of this embodiment includes: a second acquisition unit 701 , a model determination unit 702 and an image processing unit 703 .

[0086] The second acquisition unit 701 is configured to acquire a target face image;

[0087] The model determination unit 702 is configured to determine a matching model from at least one pre-trained model according to the target face image. Figure 2 or Figure 3 The model is obtained by training using the model training method described in the embodiment.

[0088] The image processing unit 703 is configured to determine a processed facial image corresponding to the target facial image according to the target facial image and the matching model.

[0089] It should be understood that the units 701 to 703 described in the image processing device 700 are respectively the same as those described in the reference Figure 4 Therefore, the operations and features described above for the image processing method are also applicable to the device 700 and the units included therein, and will not be described in detail here.

[0090] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0091] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0092] Figure 8 A block diagram of an electronic device 800 for executing a model training method and an image processing method according to an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0093] like Figure 8 As shown, the electronic device 800 includes a processor 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a memory 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The processor 801, ROM 802 and RAM 803 are connected to each other via a bus 804. An I / O interface (input / output interface) 805 is also connected to the bus 804.

[0094] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a memory 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0095] The processor 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 801 performs the various methods and processes described above, such as model training methods and image processing methods. For example, in some embodiments, the model training method and the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable storage medium, such as a memory 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the processor 801, one or more steps of the model training method and the image processing method described above may be performed. Alternatively, in other embodiments, the processor 801 may be configured to perform the model training method and the image processing method in any other appropriate manner (e.g., by means of firmware).

[0096] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. The above program code can be packaged into a computer program product. These program codes or computer program products can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor 801, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0098] In the context of the present disclosure, a machine-readable storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable storage medium may be a machine-readable signal storage medium or a machine-readable storage medium. A machine-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0101] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0102] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document does not limit this.

[0103] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, comprising: Acquire a face image set, wherein the face image set includes a pre-makeup face image subset and a post-makeup face image subset, and each pre-makeup face image in the pre-makeup face image subset corresponds one-to-one to each post-makeup face image in the post-makeup face image subset; Based on the face image set, the following training steps are performed multiple times: Inputting each pre-makeup facial image in the pre-makeup facial image subset or each post-makeup facial image in the post-makeup facial image subset into a model to obtain an intermediate facial image output by the model; Performing region segmentation on the intermediate face image and the face image corresponding to the input face image; Determining the loss function value based on the comparison results of each area in the intermediate face image and each area in the face image corresponding to the input face image, including: comparing the facial features in the intermediate face image with the facial features in the face image corresponding to the input face image to obtain the similarity of the facial features in the two face images, and determining the loss function value according to the comparison result; or determining the loss function value by using the difference between the area of ​​the face image corresponding to the intermediate face image and the input face image and the difference between the pixels; The parameters of the model are adjusted according to the loss function value.

2. The method according to claim 1, wherein: The determining of the loss function value based on the comparison results of each region in the intermediate face image and each region in the face image corresponding to the input face image comprises: For each region in the intermediate facial image, determining a first comparison result based on a comparison between the region and a corresponding region in the facial image corresponding to the input facial image; For each pixel in the intermediate facial image, determining a second comparison result based on a comparison between the pixel and a corresponding pixel in the facial image corresponding to the input facial image; A loss function value is determined according to the first comparison result and the second comparison result.

3. The method according to claim 2, wherein: The step of determining a first comparison result based on a comparison between the region and a corresponding region in the face image corresponding to the input face image includes: Determine a local color histogram of a corresponding region in the face image corresponding to the region and the input face image; A first comparison result is determined according to the local color histogram.

4. The method according to claim 2, wherein: Determining a second comparison result based on the comparison between the pixel and a corresponding pixel in the face image corresponding to the input face image includes: According to the difference between the pixel value of the pixel and the pixel value of the corresponding pixel in the face image corresponding to the input face image; A second comparison result is determined according to the difference.

5. An image processing method, comprising: Get the target face image; Determine a matching model from at least one pre-trained model according to the target face image, wherein the at least one model is trained by the model training method according to any one of claims 1 to 4; A processed facial image corresponding to the target facial image is determined according to the target facial image and the matching model.

6. A model training device, comprising: a first acquisition unit configured to acquire a face image set, wherein the face image set includes a pre-makeup face image subset and a post-makeup face image subset, and each pre-makeup face image in the pre-makeup face image subset corresponds one-to-one to each post-makeup face image in the post-makeup face image subset; The model training unit is configured to perform the training steps multiple times through the following modules based on the face image set: An input module is configured to input each pre-makeup facial image in the pre-makeup facial image subset or each post-makeup facial image in the post-makeup facial image subset into a model to obtain an intermediate facial image output by the model; a segmentation module configured to perform region segmentation on the intermediate face image and the face image corresponding to the input face image; The determination module is configured to determine the loss function value based on the comparison results of each area in the intermediate face image and each area in the face image corresponding to the input face image, including: comparing the facial features in the intermediate face image with the facial features in the face image corresponding to the input face image to obtain the similarity of the facial features in the two face images, and determining the loss function value according to the comparison result; or determining the loss function value by using the difference between the area of ​​the face image corresponding to the intermediate face image and the input face image and the difference between the pixels; An adjustment module is configured to adjust the parameters of the model according to the loss function value.

7. The device according to claim 6, wherein: The determination module is further configured to: For each region in the intermediate facial image, determining a first comparison result based on a comparison between the region and a corresponding region in the facial image corresponding to the input facial image; For each pixel in the intermediate facial image, determining a second comparison result based on a comparison between the pixel and a corresponding pixel in the facial image corresponding to the input facial image; A loss function value is determined according to the first comparison result and the second comparison result.

8. The device according to claim 7, wherein: The determination module is further configured to: Determine a local color histogram of a corresponding region in the face image corresponding to the region and the input face image; A first comparison result is determined according to the local color histogram.

9. The device according to claim 7, wherein: The determination module is further configured to: According to the difference between the pixel value of the pixel and the pixel value of the corresponding pixel in the face image corresponding to the input face image; A second comparison result is determined according to the difference.

10. An image processing device, comprising: A second acquisition unit is configured to acquire a target face image; A model determination unit, configured to determine a matching model from at least one pre-trained model according to the target face image, wherein the at least one model is trained by the model training method according to any one of claims 1 to 4; The image processing unit is configured to determine a processed facial image corresponding to the target facial image according to the target facial image and the matching model.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 4 or the method of claim 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of any one of claims 1 to 4 or the method of claim 5.

Citation Information

Patent Citations

  • Cross-modal face recognition model training method, device and equipment and storage medium

    CN112149635A

  • Face makeup evaluation method and device

    WO2018214115A1