Face recognition model construction method, portrait face changing method and related device

Abstract

The embodiment of the invention provides a face recognition model construction method, a portrait changing method and a related device, and the method comprises the steps: carrying out the processing of the definition of each source face image, and obtaining a blurred face image and a high-definition face image corresponding to each source face image; storing all face blurred images into a blurred data set, and storing all face high-definition images into a high-definition data set; according to a set sampling period, determining a first sampling probability and a second sampling probability adopted by each time of sampling in the sampling period; acquiring sample images from the high-definition data set and the fuzzy data set in each sampling according to the corresponding first sampling probability and second sampling probability and a set sample number so as to obtain a sample set of each sampling; and training the initial network model by using the sample set to obtain a face recognition model. Therefore, the face recognition model meeting the face changing task requirement is obtained, and the AI face changing effect can be improved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a face recognition model construction method, a portrait face replacement method and related devices. Background Art

[0002] As AI technology gradually matures and live broadcasts and short videos become popular, in order to improve the playability, diversity or interactivity of live broadcasts or short videos, a variety of AI live functions have been added to the live broadcast field. For example, AI face-changing is a function that uses artificial intelligence technology to replace the face in the original face image with the face in the target face image, which involves the technology of extracting facial features using a face recognition model.

[0003] However, since the face recognition model used in AI face-changing is trained for face recognition tasks, face recognition tasks refer to face feature recognition tasks that use face features to identify people, and the focus is on extracting features that play a key role in identifying people, so as to identify the differences between different people. For example, if a face has obvious features of one big eye and one small eye, the face recognition model will focus all its attention on the eyes and ignore other facial features. It can be seen that the face recognition model trained based on face recognition tasks, which relies on face discrimination features, may not meet the needs of face feature extraction in face-changing tasks, and may result in the face in the image obtained by face-changing processing only having some of the features of the face in the target face image, resulting in poor face-changing effects. Summary of the invention

[0004] In view of this, in order to improve the face-changing effect of AI face-changing, the purpose of the present invention is to provide a face recognition model construction method, a portrait face-changing method and related devices.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] A first aspect of an embodiment of the present invention provides a method for constructing a face recognition model, comprising:

[0007] Processing the clarity of each source face image to obtain a blurred face image and a high-definition face image corresponding to each source face image;

[0008] All blurred face images are stored in the blurred dataset, and all high-definition face images are stored in the high-definition dataset;

[0009] According to the set sampling period, determine the first sampling probability and the second sampling probability used for each sampling in the sampling period; wherein the first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a blurred face image each time; in each sampling period, the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps;

[0010] In each sampling, according to the corresponding first sampling probability and second sampling probability and the set number of samples, a sample image is obtained from the high-definition data set and the fuzzy data set to obtain a sample set for each sampling;

[0011] The sample set is used to train the initial network model to obtain a face recognition model.

[0012] A second aspect of an embodiment of the present invention provides a method for face replacement of a portrait, comprising:

[0013] Processing the original face image to obtain a first feature image of a non-face area in the original face image;

[0014] Processing the target face image by using a pre-built face recognition model to obtain a second feature image of the face area in the target face image; the face recognition model is constructed by the face recognition model construction method provided by the first aspect above;

[0015] The first feature image and the second feature image are fused to obtain a face replacement image; the face replacement image is an image in which the face area in the original face image is replaced with the face area in the target face image.

[0016] A third aspect of an embodiment of the present invention provides a face recognition model construction device, comprising:

[0017] The clarity processing module is configured to: process the clarity of each source face image to obtain a blurred face image and a high-definition face image corresponding to each source face image;

[0018] A storage module is configured to: store all blurred face images into a blurred data set, and store all high-definition face images into a high-definition data set;

[0019] The probability determination module is configured to: determine, according to a set sampling period, a first sampling probability and a second sampling probability adopted for each sampling in the sampling period; wherein the first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a blurred face image each time; in each sampling period, the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps;

[0020] The sampling module is configured to: in each sampling, according to the corresponding first sampling probability and second sampling probability and the set number of samples, obtain a sample image from the high-definition data set and the fuzzy data set to obtain a sample set for each sampling;

[0021] The training module is configured to: use the sample set to train the initial network model to obtain a face recognition model.

[0022] A fourth aspect of an embodiment of the present invention provides a face-changing device for a human portrait, comprising:

[0023] The original image processing module is configured to: process the original face image to obtain a first feature image of a non-face area in the original face image;

[0024] The target image processing module is configured to: process the target face image by using a pre-trained face recognition model to obtain a second feature image of the face area in the target face image; the face recognition model is constructed by the face recognition model construction device provided by the third aspect;

[0025] The replacement processing module is configured to: fuse the first feature image and the second feature image to obtain a face replacement image; the face replacement image is an image in which the face area in the original face image is replaced with the face area in the target face image.

[0026] According to a fifth aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the face recognition model construction method provided in the first aspect above, and / or the portrait face changing method provided in the second aspect above.

[0027] According to a sixth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the face recognition model construction method provided in the first aspect and / or the portrait face-changing method provided in the second aspect is implemented.

[0028] The face recognition model construction method, portrait face-changing method and related devices provided by the embodiments of the present invention, on the one hand, by processing the clarity of the source face image, a blurred face image and a high-definition face image with lower clarity are obtained, and then the blurred face image and the high-definition face image are used as samples in the sample set, so as to realize the ability to distinguish the relative positions and proportions of facial features by using the blurred face image training model, and the ability to distinguish the particularity of the details of facial features or special features of the face by using the high-definition face image training model, such as moles or scars on the face, so that the face recognition model obtained based on this training has the ability to distinguish and extract global facial features and local facial features at the same time, so that more comprehensive and representative facial features can be identified, which is not only suitable for face recognition tasks, but also can meet the recognition requirements of facial features in face-changing tasks, so that the features of the target face to be replaced can be extracted more comprehensively and accurately to replace the original face, and have a good face-changing effect. On the other hand, by using the first sampling probability and the second sampling probability corresponding to each sampling, the blurred face image and the high-definition face image are sampled to obtain a sample set after each sampling, wherein the first sampling probability is configured to be positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps, so that the model can first learn the global features of the face based on the blurred face image, and then learn the local features of the face based on the high-definition face image, so as to gradually improve the model's fine-grained recognition ability of facial details, so that the model can have better recognition ability for fine-grained features of the face. Therefore, combining the above two aspects, the trained face recognition model can better meet the needs of the face-changing task, so that the subsequent face-changing has a better effect, thereby better improving the user experience.

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A structural block diagram of an electronic device provided by an embodiment of the present invention is shown;

[0032] Figure 2 A flow chart of a method for constructing a face recognition model provided by an embodiment of the present invention is shown;

[0033] Figure 3 A network structure block diagram of an initial network model provided by an embodiment of the present invention is shown;

[0034] Figure 4 A flow chart of a portrait face-changing method provided by an embodiment of the present invention is shown;

[0035] Figure 5 A functional module diagram of a face recognition model building device provided by an embodiment of the present invention is shown;

[0036] Figure 6 A functional module diagram of a portrait face-changing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0038] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0039] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0040] In order to solve the technical problems that the face recognition model trained for the face recognition task is applied to the AI ​​face-changing task in the traditional technology, the face feature extraction is not comprehensive enough and cannot meet the face feature extraction requirements in the face-changing task, resulting in poor face-changing effect, the present invention provides a face recognition model construction method. On the one hand, by processing the clarity of the source face image, a blurred face image and a high-definition face image with low clarity are obtained, and then the blurred face image and the high-definition face image are used as samples in the sample set to realize the analysis of the relative positions and proportions of the facial features by using the blurred face image training model. The face recognition model based on this training has the ability to distinguish and extract both global and local facial features, so that more comprehensive and representative facial features can be identified. It is not only suitable for face recognition tasks, but also can meet the recognition needs of facial features in face-changing tasks, so that the features of the target face to be replaced can be extracted more comprehensively and accurately to replace the original face, achieving a good face-changing effect. On the other hand, by using the first sampling probability and the second sampling probability corresponding to each sampling, the blurred face image and the high-definition face image are sampled to obtain a sample set after each sampling, wherein the first sampling probability is configured to be positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps, so that the model can first learn the global features of the face based on the blurred face image, and then learn the local features of the face based on the high-definition face image, so as to gradually improve the model's fine-grained recognition ability of facial details, so that the model can have better recognition ability for fine-grained features of the face. Therefore, combining the above two aspects, the trained face recognition model can better meet the needs of the face-changing task, so that the subsequent face-changing has a better effect, thereby better improving the user experience.

[0041] The face recognition model construction method provided by the present invention can be applied to electronic devices, please refer to Figure 1 , is a block diagram of the structure of an electronic device. The electronic device 100 includes a memory 110, a processor 120 and a communication module 130. The memory 110, the processor 120 and the communication module 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0042] The memory is used to store programs or data. The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0043] The processor is used to read / write data or programs stored in the memory and execute corresponding functions.

[0044] The communication module is used to establish a communication connection between the electronic device and other communication terminals through a network, and is used to send and receive data through the network.

[0045] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device, and the electronic device may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0046] The following combination Figure 2 The face recognition model construction method provided by the embodiment of the present invention is described. Figure 2 : is a flowchart of a method for constructing a face recognition model provided by an embodiment of the present invention, the method for constructing a face recognition model comprising:

[0047] In step S100, the clarity of each source face image is processed to obtain a blurred face image and a high-definition face image corresponding to each source face image;

[0048] In step S200, all blurred face images are stored in a blurred data set, and all high-definition face images are stored in a high-definition data set;

[0049] In step S300, according to the set sampling period, the first sampling probability and the second sampling probability adopted for each sampling in the sampling period are determined; wherein the first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a blurred face image each time; in each sampling period, the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps;

[0050] In step S400, in each sampling, according to the corresponding first sampling probability and second sampling probability and the set number of samples, a sample image is obtained from the high-definition data set and the fuzzy data set to obtain a sample set for each sampling;

[0051] In step S500, the sample set is used to train the initial network model to obtain a face recognition model.

[0052] In order to obtain a face recognition model that can take into account the extraction of global and local features of the face and has excellent face fine-grained feature recognition capabilities, a sample set suitable for the face swap task can be obtained through the above steps S100 to S500, and the sample set can be used to train the initial network model until the model converges to obtain a face recognition model. The following is a technical principle of constructing a face recognition model in combination with steps S100 to S500:

[0053] First, a sufficient number of source face images can be collected from a free resource library or a licensed resource library, and each source face image shows a person's face.

[0054] In some examples, all source face images may include a sufficient number of images at different face angles. It is understandable that there are sufficient images at different face angles to meet the training requirements for the recognition accuracy of the model, so that the model can be suitable for extracting features of faces at different angles. The different angles can be understood as the angle between the plane where the face is located and the plane where the lens is located. For example, the front of the face facing the lens is one angle, and the side of the face facing the lens is another angle. In addition, the specific number of sufficient numbers can be determined based on experience or experiments, and the embodiments of the present invention do not limit this.

[0055] In some other examples, if the trained face recognition model is expected to be applicable to Asian face recognition, all source face images may be Asian face images; if the trained face recognition model is expected to be applicable to Western face recognition, all source face images may be Western face images; if the trained face recognition model is expected to be applicable to both Asian and Western face recognition, all source face images may include both Asian and Western face images. The embodiments of the present invention are not limited to this.

[0056] After a sufficient number of source face images are collected through any of the above embodiments, all source face images can be stored in a database, so that when a face recognition model needs to be constructed, source face images can be obtained from the database.

[0057] After the source face image is obtained, the clarity of each source face image obtained can be processed through step S100. For example, each source face image is processed by a super-resolution algorithm that does not introduce artifacts to obtain a high-definition face image of each source face image; each source face image is processed by a downsampling algorithm to obtain a blurred face image of each source face image; thereby, a blurred face image and a high-definition face image corresponding to each source face image are obtained. Among them, a blurred face image refers to a face image with lower clarity, and a high-definition face image refers to a face image with higher clarity. The clarity of the blurred face image and the high-definition face image can be set by the developer according to the face-changing effect or actual needs, or according to the experimental setting, and is not limited here. In addition, the relevant principles of the super-resolution algorithm and the downsampling algorithm that do not introduce artifacts can be found in the relevant technology, which will not be explained in detail here.

[0058] It should be noted that blurred face images are used to train the model's ability to distinguish the proportions and relative positions of facial features, that is, the global features of the face; high-definition face images are used to train the model's ability to distinguish the detailed features of facial features or representative features of the face, that is, the local features of the face.

[0059] After obtaining the blurred face image and the high-definition face image corresponding to each source face image through step S100, step S200 can be executed to store all the blurred face images into the blurred data set and store all the high-definition face images into the high-definition data set.

[0060] After obtaining the fuzzy data set and the high-definition data set through step S200, step S300 can be performed to first determine the first sampling probability and the second sampling probability used in each sampling in each sampling cycle. The first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a fuzzy face image each time; and in each sampling cycle, the first sampling probability is positively correlated with the number of sampling steps in the sampling cycle, and the second sampling probability is negatively correlated with the number of sampling steps. Thus, the samples input to the model can be controlled to be fuzzy face images first, and then high-definition face images, so that the model first learns the global features of the face based on the fuzzy face images, and then learns the local features of the face based on the high-definition face images, so as to gradually improve the model's fine-grained recognition ability of face details, so that the model can have better recognition ability of fine-grained face features.

[0061] In some embodiments, the determination of the first sampling probability and the second sampling probability in step S300 can be sampled in the following manner: for each sampling period, the values ​​of the first sampling probability and the second sampling probability used for each sampling can be pre-set so that the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps; for example, assuming that n sampling steps are used as a sampling period, based on this, the total number of sampling steps is n, the number of sampling steps for the first sampling is 0, the number of sampling steps for the second sampling is 1, the number of sampling steps for the third sampling is 2... and the number of sampling steps for the nth sampling is n-1. Correspondingly, since the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps, the first sampling probability of the first sampling can be set to 0, the second sampling probability can be set to 1, the first sampling probability of the second sampling can be set to 0.1, the second sampling probability can be set to 0.9, the first sampling probability of the third sampling can be set to 0.2, the second sampling probability can be set to 0.8... The first sampling probability of the nth sampling can be set to 1, and the second sampling probability can be set to 0. It can be seen that the sum of the first sampling probability and the second sampling probability is 1. Among them, the value of n and the sampling step can be set according to actual needs.

[0062] Thus, the mapping relationship between the number of sampling steps and the first sampling probability and the second sampling probability in each sampling cycle can be recorded. Thus, at each sampling, the first sampling probability and the second sampling probability corresponding to the current sampling can be obtained from the above mapping relationship according to the number of sampling steps between the current sampling and the first sampling of the current sampling cycle.

[0063] In addition, the above mapping relationships of all sampling periods may adopt the same mapping relationship or different mapping relationships, which is not limited in the embodiment of the present invention.

[0064] Although the first sampling probability and the second sampling probability obtained by the above implementation can also meet the needs of obtaining a sample set and the training needs of the model to distinguish global features and local features of a face, in some embodiments, in order to obtain a more reasonable sample set so that the trained face recognition model has a stronger ability to distinguish global features and local features of a face, the embodiment of the present invention further provides another way to determine the first sampling probability and the second sampling probability, that is, in step S300, according to the set sampling period, the step of determining the first sampling probability and the second sampling probability used for each sampling in the sampling period may include:

[0065] In step S310, a first sampling probability corresponding to the current sampling is calculated according to the set first probability and second probability, the number of sampling steps between the current sampling and the first sampling in the current sampling period, and the total number of sampling steps in the current sampling period;

[0066] In step S320, the difference between 1 and the first sampling probability is used as the second sampling probability corresponding to the current sampling.

[0067] In the above, the first probability and the second probability can be set according to experience or actual needs. The first probability can represent the sampling start probability in a sampling period, and the second probability can represent the sampling end probability in a sampling period.

[0068] In some embodiments, for step S310, a calculation method for calculating the first sampling probability using the first probability, the second probability, the number of sampling steps, and the total number of sampling steps may be:

[0069]

[0070] In formula (1), P HD (t) is the first sampling probability of the current sampling, α is the first probability, β is the second probability, t is the number of sampling steps between the current sampling and the first sampling in the current sampling cycle, and n is the total number of sampling steps in the previous sampling cycle.

[0071] In some other embodiments, for step S310, another calculation method for calculating the first sampling probability using the first probability, the second probability, the number of sampling steps, and the total number of sampling steps may be:

[0072] That is, in step S310, the step of calculating the first sampling probability corresponding to the current sampling according to the set first probability and second probability, the number of sampling steps between the current sampling and the first sampling in the current sampling period, and the total number of sampling steps in the current sampling period includes:

[0073] In step S311, the remainder of the number of sampling steps and the total number of sampling steps is calculated;

[0074] In step S312, the difference between the second probability and the first probability is calculated; the second probability is greater than the first probability;

[0075] In step S313, the ratio of the product of the remainder and the difference to the total number of sampling steps is calculated;

[0076] In step S314, the sum of the first probability and the ratio is used as the first sampling probability corresponding to the current sampling.

[0077] The calculation process of steps S311 to S314 can be expressed as the following formula:

[0078]

[0079] The meaning of each letter in formula (2) can be found in the relevant records in formula (1), and will not be repeated here.

[0080] After calculating the first sampling probability and the second sampling probability of each sampling through any of the above embodiments, step S400 can be performed to achieve that in each sampling, the first sampling probability and the second sampling probability corresponding to the sampling are used to obtain a set number of sample images from the high-definition data set and the fuzzy data set respectively to obtain the sample set of the sampling. Among them, in the sample set of the sampling, the proportion of fuzzy face images matches the second sampling probability corresponding to the sampling, and the proportion of high-definition face images matches the first sampling probability corresponding to the sampling. In this way, n sample sets can be obtained after each sampling cycle.

[0081] After multiple sample sets are obtained through step S400, all sample sets may be used as training samples, or all sample sets may be divided into training sets, test sets, and validation sets in sequence according to a sample set quantity ratio of 7:2:1.

[0082] After obtaining the sample set, step S500 can be executed to train the initial network model using the sample set until the model converges, thereby obtaining a face recognition model. The initial network model can be a recognition network model for identifying image features, for example, it can be an AcrFace network model. In addition, the training principle of the model can refer to the training principle of the neural network model in the related art, which will not be described in detail here.

[0083] Although the above-mentioned initial network model adopts the recognition network model, it can also achieve the construction of a face recognition model with good effect, and can be applied to face-changing tasks. However, in the face-changing task, since it is still necessary to restore the target face based on the face features extracted by the face recognition model, in order to further improve the consistency between the restored face and the target face based on the global face features extracted by the face recognition model and the representative local face features, in some embodiments, the face recognition model construction method provided in the embodiment of the present invention also improves the structure of the initial network model, that is, the initial network model includes a recognition network and a generation network. And the training process of the model is adaptively adjusted, that is, in step S500, the initial network model is trained using the sample set to obtain the step of the face recognition model, including:

[0084] In step S510, the sample set is input into the recognition network to extract facial features;

[0085] In step S530, the facial features are input into the generation network through the recognition network to obtain a generated face image;

[0086] In step S540, a loss value is calculated based on the generated face image and the sample image corresponding to the generated face image in the sample set by using a set loss function;

[0087] In step S550, the parameters of the recognition network and the generation network are adjusted respectively using the loss value to obtain the recognition network and the generation network after parameter update;

[0088] In step S560, return to the step of inputting the sample set into the recognition network until the currently calculated loss value meets the expectation, thereby obtaining a trained recognition network and a generation network;

[0089] In step S570, the trained recognition network is used as a face recognition model.

[0090] Therefore, by improving the structure of the initial network model and continuously adjusting the parameters of the recognition network and the generation network by using steps S510 to S570, until the loss value between the face generation image output by the generation network and the corresponding sample image, that is, the source image meets the expectation, the model reaches a converged state, and the recognition network obtained at this time can be used as a face recognition model. It can be seen that the generation network is used to restore the face based on the face features extracted by the recognition network, and the restored face generation image and the source image are compared to calculate the loss value. The loss value is not only used to adjust the parameters of the generation network, but also used to adjust the parameters of the recognition network, so as to use the output of the generation network to assist in adjusting the parameters of the recognition network, so that the features extracted by the recognition network are more in line with the requirements of face restoration consistency.

[0091] In some embodiments, in order to improve the extraction accuracy of effective features and improve the accuracy and efficiency of the output of the generation network, the initial network model may also include a feature dimension conversion network; the feature dimension conversion network is connected between the recognition network and the generation network. Accordingly, before inputting the facial features extracted by the recognition network into the generation network, in step S500, the step of training the initial network model using the sample set to obtain a facial recognition model also includes:

[0092] In step S520, the facial features extracted by the recognition network are input into the feature dimension conversion network to obtain a 512-dimensional feature vector.

[0093] Correspondingly, in step S530, the step of inputting facial features to the generation network through the recognition network is adjusted to: inputting a 512-dimensional feature vector to the generation network through the feature dimension conversion network.

[0094] In some embodiments, the feature dimension conversion network may be composed of a map2style network.

[0095] In some embodiments, see Figure 3 , Figure 3 It is a network structure block diagram of an initial network model provided by an embodiment of the present invention, wherein the recognition network may be an AcrFace network, the generation network may be a StyleGAN network, and the generation network is also configured with an input terminal for accessing constants or fixed parameters. Among them, by configuring the generation network with an input terminal for accessing constants or fixed parameters to input constants or fixed parameters to the generation network model, the cost of model training can be reduced and model performance can be improved. The constants or fixed parameters may include, but are not limited to, one or more of the following: learning rate, momentum factor, weight decay coefficient.

[0096] In some embodiments, in order to improve the performance of the recognition network and the generation network, when the generation network is a StyleGAN network, the two loss functions, stylegan loss and weak feature loss, can be configured in addition to the original configuration of the StyleGAN network, and the loss values ​​of the two loss functions are fed back to the recognition network and the generation network. The technical principles of the two loss functions, stylegan loss and weak feature loss, can be found in the relevant technology and will not be described in detail here.

[0097] After constructing a face recognition model through any of the above embodiments, the face recognition model can be applied to face recognition tasks, or to face swapping tasks. For the application scenario of face swapping tasks, the embodiment of the present invention also provides a portrait face swapping method, please refer to Figure 4 , Figure 4 A flowchart of a portrait face-changing method provided by an embodiment of the present invention includes:

[0098] In step S10, the original face image is processed to obtain a first feature image of a non-face area in the original face image;

[0099] In step S20, the target face image is processed by a pre-built face recognition model to obtain a second feature image of the face area in the target face image;

[0100] In step S30, the first feature image and the second feature image are fused to obtain a face replacement image; the face replacement image is an image in which the face area in the original face image is replaced with the face area in the target face image.

[0101] The face-changing method provided in the embodiment of the present invention can be installed in a software with a video live broadcast function, an image editing function, a video editing function, or a face feature in the form of a program product function, or can be used as a separate program product. Figure 1 In the electronic device shown, the electronic device may be a mobile phone, a tablet, or a computer, which is not limited here.

[0102] Therefore, when the portrait face-changing function is activated, the camera of the electronic device can be called, or the original face image and the target face image can be obtained from the photo album of the electronic device according to user instructions, and then steps S10 and S20 are executed. It can be understood that steps S10 and S20 can be executed in parallel or one after another, but the order of execution is not limited.

[0103] During the execution of step S10, the feature extraction process of the non-face area of ​​the original face image is performed, thereby obtaining a first feature image of the non-face area in the original face image. In some examples, the original face image can be processed by a pre-trained non-face feature extraction model to obtain the first feature image. The non-face feature extraction model can be a neural network model, and the training principle of the model can be referred to in the relevant technology, which will not be described in detail here.

[0104] In the process of executing step S20, the face recognition model performs feature extraction processing on the face area of ​​the target face image, and then obtains a second feature image of the face area in the target face image. The face recognition model is constructed by the face recognition model construction method in any of the above embodiments of the present invention, so the technical principle of extracting the second feature image can be found in the relevant records above, which will not be repeated here.

[0105] After obtaining the first feature image and the second feature image, step S30 can be executed to fuse the first feature image and the second feature image to obtain a face replacement image in which the face area in the original face image is replaced with the face area in the target face image.

[0106] After obtaining the face replacement image, it can be displayed on the display screen of the electronic device.

[0107] It is worth noting that the technical features or technical solutions in any of the above embodiments of the present invention can be combined with each other as long as there is no combination contradiction.

[0108] In order to execute the corresponding steps in the embodiment and various possible methods of the above-mentioned face recognition model construction method, a method for implementing a face recognition model construction device is given below. Optionally, the face recognition model construction device can adopt the above-mentioned Figure 1 For further information, please refer to Figure 5 , Figure 5 A functional module diagram of a face recognition model construction device provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the face recognition model construction device provided in this embodiment are the same as those of the embodiment of the face recognition model construction method described above. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the embodiment of the face recognition model construction method described above. The face recognition model construction device 500 includes:

[0109] The definition processing module 510 is configured to: process the definition of each source face image to obtain a blurred face image and a high-definition face image corresponding to each source face image;

[0110] The storage module 520 is configured to: store all blurred face images into a blurred data set, and store all high-definition face images into a high-definition data set;

[0111] The probability determination module 530 is configured to: determine, according to a set sampling period, a first sampling probability and a second sampling probability used for each sampling in the sampling period; wherein the first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a blurred face image each time; in each sampling period, the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps;

[0112] The sampling module 540 is configured to: in each sampling, according to the corresponding first sampling probability and second sampling probability and the set number of samples, obtain a sample image from the high-definition data set and the fuzzy data set to obtain a sample set for each sampling;

[0113] The training module 550 is configured to: use the sample set to train the initial network model to obtain a face recognition model.

[0114] In some embodiments, the process of the probability determination module 530 determining the first sampling probability and the second sampling probability adopted for each sampling in the sampling period according to the set sampling period is configured as follows:

[0115] Calculate the first sampling probability corresponding to the current sampling according to the set first probability and second probability, the number of sampling steps between the current sampling and the first sampling in the current sampling period, and the total number of sampling steps in the current sampling period;

[0116] The difference between 1 and the first sampling probability is used as the second sampling probability corresponding to the current sampling.

[0117] In some embodiments, the process of the probability determination module 530 calculating the first sampling probability corresponding to the current sampling according to the set first probability and second probability, the number of sampling steps between the current sampling and the first sampling in the current sampling period, and the total number of sampling steps in the current sampling period is configured as follows:

[0118] Calculating the remainder of the number of sampling steps and the total number of sampling steps;

[0119] Calculating a difference between the second probability and the first probability; the second probability is greater than the first probability;

[0120] Calculating the ratio of the product of the remainder and the difference to the total number of sampling steps;

[0121] The sum of the first probability and the ratio is used as the first sampling probability corresponding to the current sampling.

[0122] In some embodiments, the initial network model includes a recognition network and a generation network. Accordingly, the training module 550 uses the sample set to train the initial network model to obtain a face recognition model, which is configured as follows:

[0123] Inputting the sample set into the recognition network to extract facial features;

[0124] Inputting facial features into the generation network through the recognition network to obtain a generated face image;

[0125] Calculating a loss value based on the generated face image and a sample image in the sample set corresponding to the generated face image using a set loss function;

[0126] Using the loss value, respectively adjust the parameters of the recognition network and the generation network to obtain the recognition network and the generation network after parameter update;

[0127] Returning to the step of inputting the sample set into the recognition network until the currently calculated loss value meets the expectation, a trained recognition network and a generation network are obtained;

[0128] The trained recognition network is used as the face recognition model.

[0129] In some embodiments, the initial network model further includes a feature dimension conversion network; the feature dimension conversion network is connected between the recognition network and the generation network. Accordingly, the training module 550 uses the sample set to train the initial network model to obtain a face recognition model, which is configured as follows:

[0130] Inputting the sample set into the recognition network to extract facial features;

[0131] Inputting the facial features extracted by the recognition network into the feature dimension conversion network to obtain a 512-dimensional feature vector;

[0132] Inputting a 512-dimensional feature vector into the generation network through the feature dimension conversion network to obtain a face generation image;

[0133] Calculating a loss value based on the generated face image and a sample image in the sample set corresponding to the generated face image using a set loss function;

[0134] Using the loss value, respectively adjust the parameters of the recognition network and the generation network to obtain the recognition network and the generation network after parameter update;

[0135] Returning to the step of inputting the sample set into the recognition network until the currently calculated loss value meets the expectation, a trained recognition network and a generation network are obtained;

[0136] The trained recognition network is used as the face recognition model.

[0137] In some embodiments, the recognition network is an AcrFace network, the generation network is a StyleGAN network, and the generation network is further configured with an input terminal for accessing constants or fixed parameters. Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory shown in the figure or solidified in the operating system (OS) of the electronic device, and can be Figure 1 Meanwhile, the data and program codes required for executing the above modules can be stored in the memory.

[0138] In order to execute the corresponding steps in the above-mentioned embodiments of the portrait face-changing method and various possible methods, a method for implementing a portrait face-changing device is provided below. Optionally, the portrait face-changing device can adopt the above-mentioned Figure 1 For further information, please refer to Figure 6 , Figure 6 This is a functional module diagram of a portrait face-changing device provided by an embodiment of the present invention. It should be noted that the basic principle and technical effects of the portrait face-changing device provided by this embodiment are the same as those of the embodiment of the portrait face-changing method described above. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the embodiment of the portrait face-changing method described above. The portrait face-changing device 600 includes:

[0139] The original image processing module 610 is configured to: process the original face image to obtain a first feature image of a non-face area in the original face image;

[0140] The target image processing module 620 is configured to: process the target face image by using a pre-trained face recognition model to obtain a second feature image of the face area in the target face image; the face recognition model is constructed by the face recognition model construction device according to claim 8;

[0141] The replacement processing module 630 is configured to: fuse the first feature image and the second feature image to obtain a face replacement image; the face replacement image is an image in which the face area in the original face image is replaced with the face area in the target face image.

[0142] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory shown in the figure or solidified in the operating system (OS) of the electronic device, and can be Figure 1 Meanwhile, the data and program codes required for executing the above modules can be stored in the memory.

[0143] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0145] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a face recognition model, characterized in that: include: Processing the clarity of each source face image to obtain a blurred face image and a high-definition face image corresponding to each source face image; All blurred face images are stored in the blurred dataset, and all high-definition face images are stored in the high-definition dataset; According to the set sampling period, determine the first sampling probability and the second sampling probability used for each sampling in the sampling period; wherein the first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a blurred face image each time; in each sampling period, the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps; In each sampling, according to the corresponding first sampling probability and second sampling probability and the set number of samples, a sample image is obtained from the high-definition data set and the fuzzy data set to obtain a sample set for each sampling; The sample set is used to train the initial network model to obtain a face recognition model.

2. The method according to claim 1, characterized in that The step of determining the first sampling probability and the second sampling probability adopted for each sampling in the sampling period according to the set sampling period includes: Calculate the first sampling probability corresponding to the current sampling according to the set first probability and second probability, the number of sampling steps between the current sampling and the first sampling in the current sampling period, and the total number of sampling steps in the current sampling period; The difference between 1 and the first sampling probability is used as the second sampling probability corresponding to the current sampling.

3. The method according to claim 2, characterized in that The step of calculating the first sampling probability corresponding to the current sampling according to the set first probability and second probability, the number of sampling steps between the current sampling and the first sampling in the current sampling period, and the total number of sampling steps in the current sampling period comprises: Calculate the remainder of the number of sampling steps and the total number of sampling steps; Calculating a difference between the second probability and the first probability; the second probability is greater than the first probability; Calculating the ratio of the product of the remainder and the difference to the total number of sampling steps; The sum of the first probability and the ratio is used as the first sampling probability corresponding to the current sampling.

4. The method according to any one of claims 1 to 3, characterized in that The initial network model includes a recognition network and a generation network; The step of using the sample set to train the initial network model to obtain a face recognition model includes: Inputting the sample set into the recognition network to extract facial features; Inputting facial features into the generation network through the recognition network to obtain a generated face image; Calculate a loss value based on the generated face image and a sample image in the sample set corresponding to the generated face image using a set loss function; Using the loss value, respectively adjust the parameters of the recognition network and the generation network to obtain the recognition network and the generation network after parameter update; Returning to the step of inputting the sample set into the recognition network until the currently calculated loss value meets the expectation, a trained recognition network and a generation network are obtained; The trained recognition network is used as the face recognition model.

5. The method according to claim 4, characterized in that The initial network model also includes a feature dimension conversion network; the feature dimension conversion network is connected between the recognition network and the generation network; Before inputting the facial features extracted by the recognition network into the generation network, the step of training the initial network model using the sample set to obtain the facial recognition model further includes: Inputting the facial features extracted by the recognition network into the feature dimension conversion network to obtain a 512-dimensional feature vector; Accordingly, the step of inputting facial features to the generation network through the recognition network is adjusted to: A 512-dimensional feature vector is input into the generation network through the feature dimension conversion network.

6. The method according to claim 5, characterized in that The recognition network is an AcrFace network, the generation network is a StyleGAN network, and the generation network is also configured with an input terminal for accessing constants or fixed parameters.

7. A method for face replacement of a portrait, characterized in that: include: Processing the original face image to obtain a first feature image of a non-face area in the original face image; The target face image is processed by a pre-constructed face recognition model to obtain a second feature image of the face area in the target face image; the face recognition model is constructed by the face recognition model construction method according to any one of claims 1 to 6; The first feature image and the second feature image are fused to obtain a face replacement image; the face replacement image is an image in which the face area in the original face image is replaced with the face area in the target face image.

8. A face recognition model construction device, characterized in that: include: The clarity processing module is configured to: process the clarity of each source face image to obtain a blurred face image and a high-definition face image corresponding to each source face image; A storage module is configured to: store all blurred face images into a blurred data set, and store all high-definition face images into a high-definition data set; The probability determination module is configured to: determine, according to a set sampling period, a first sampling probability and a second sampling probability adopted for each sampling in the sampling period; wherein the first sampling probability is used to characterize the probability of sampling a high-definition face image each time, and the second sampling probability is used to characterize the probability of sampling a blurred face image each time; in each sampling period, the first sampling probability is positively correlated with the number of sampling steps in the sampling period, and the second sampling probability is negatively correlated with the number of sampling steps; The sampling module is configured to: in each sampling, according to the corresponding first sampling probability and second sampling probability and the set number of samples, obtain a sample image from the high-definition data set and the fuzzy data set to obtain a sample set for each sampling; The training module is configured to: use the sample set to train the initial network model to obtain a face recognition model.

9. A face-changing device, characterized in that: include: The original image processing module is configured to: process the original face image to obtain a first feature image of a non-face area in the original face image; A target image processing module is configured to: process a target face image by using a pre-trained face recognition model to obtain a second feature image of a face region in the target face image; the face recognition model is constructed by the face recognition model construction device according to claim 8; The replacement processing module is configured to: fuse the first feature image and the second feature image to obtain a face replacement image; the face replacement image is an image in which the face area in the original face image is replaced with the face area in the target face image.

10. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the face recognition model construction method described in any one of claims 1 to 6, and / or the portrait face replacement method described in claim 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a face recognition model as described in any one of claims 1 to 6 and / or the method for portrait face replacement as described in claim 7 are implemented.

Images