Training method, device and computer equipment for face image generation model

By using pre-trained face recognition model and codec model to train the face image generation model, the problem that the Gaussian distribution cannot match the face features is solved, the accuracy of face image generation is improved, and identity recognition and coding are realized accuracy.

CN118675215BActive Publication Date: 2024-11-05TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411164873.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-05
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing face image generation model generates face images based on Gaussian distribution inaccurate and cannot fully match face features, resulting in inaccuracy in the generated images.

Method used

By obtaining the pre-trained face recognition model and the codec model to be trained, the face sample image is converted from pixel space to feature space, and the features are restored to pixel space, the features are extracted, and the codec model parameters are adjusted until the training is met Stop the condition and then train the face image generation model.

Benefits of technology

Improve the accuracy of the generated face image, so that the identity recognition ability of the face recognition model can be moved to the codec model. The trained codec model can accurately identify, encode and decode the identity in the face image, and the generated The content is more accurate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675215B_ABST
    Figure CN118675215B_ABST
Patent Text Reader

Abstract

The present application relates to a training method, device and computer equipment for a face image generation model. The method comprises: obtaining a pre-trained face recognition model, a codec model to be trained and a first face sample image; calling the codec model to be trained to convert the first face sample image from pixel space to feature space, and restoring the features in the feature space to pixel space to obtain a face restoration image; calling the pre-trained face recognition model to extract the features of the first face sample image and the face restoration image respectively, and obtaining the first sample image features and the restoration image features; training the codec model to be trained according to the first sample image features and the restoration image features; obtaining the face image generation model to be trained, and calling the trained codec model to train the face image generation model to be trained. The use of the present application can improve the accuracy of the generated image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, apparatus, computer equipment and storage medium for a facial image generation model. Background Art

[0002] Nowadays, face image generation technology is used in many scenarios. For example, images generated by face image generation technology can be used to enrich image library content. Face image generation usually relies on machine model implementation. The face image generation model is trained so that the trained face image generation model can generate face images.

[0003] Existing face image generation models generally use a sampling automatic variational encoder to encode face images into a Gaussian distribution, then sample from the Gaussian distribution and generate face images based on the sampled features. However, since the Gaussian distribution cannot completely match the face features, the face images generated by the face image generation model based on the Gaussian distribution are inaccurate. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a facial image generation model that can improve the accuracy of the generated facial images in order to solve the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for training a face image generation model, the method comprising:

[0006] Obtain a pre-trained face recognition model, a codec model to be trained, and a first face sample image;

[0007] Calling the codec model to be trained to convert the first face sample image from a pixel space to a feature space, and restoring the features in the feature space to the pixel space to obtain a face restoration image;

[0008] Calling the pre-trained face recognition model to extract features of the first face sample image and the face restoration image respectively, to obtain first sample image features and restoration image features;

[0009] Training the codec model to be trained according to the first sample image feature and the restored image feature until a first training stop condition is met;

[0010] A facial image generation model to be trained is obtained, and a trained encoding and decoding model is called to train the facial image generation model to be trained until a second training stop condition is met; wherein the trained facial image generation model is used to generate facial images.

[0011] In a second aspect, the present application also provides a training device for a face image generation model, the device comprising:

[0012] An image generation module is used to obtain a pre-trained face recognition model, a codec model to be trained and a first face sample image; call the codec model to be trained to convert the first face sample image from pixel space to feature space, and restore the features in the feature space to the pixel space to obtain a face restoration image.

[0013] The first training module is used to call the pre-trained face recognition model to extract the features of the first face sample image and the face restoration image respectively, so as to obtain the first sample image features and the restoration image features; and train the codec model to be trained according to the first sample image features and the restoration image features until the first training stop condition is met.

[0014] The second training module is used to obtain a facial image generation model to be trained, and call a trained encoding and decoding model to train the facial image generation model to be trained until a second training stop condition is met; wherein the trained facial image generation model is used to generate a facial image.

[0015] In one embodiment, the training of the facial image generation model also includes a third training module, which is used to obtain a second facial sample image and a sample label corresponding to the second facial sample image; perform feature extraction on the second facial sample image to obtain second sample image features; and determine, based on the second sample image features, a probability value that the second facial sample image belongs to each preset category in a preset identity category set; and train the facial recognition model to be trained based on the probability values ​​and the sample label corresponding to the second facial sample image until a third training stop condition is met.

[0016] In one embodiment, the third training module is also used to obtain a class center matrix, wherein the class center matrix includes the class center of each preset category in a preset identity category set; determine a first similarity between the second sample image feature and each class center in the class center matrix, and based on the first similarity, determine the probability value of the second face sample image belonging to each predicted category in the preset identity category set.

[0017] In one of the embodiments, the image generation module is also used to call the encoder in the codec model to be trained to perform multiple rounds of downsampling processing on the first face sample image to obtain the encoding features of the first face sample image; and call the decoder in the codec model to be trained to perform multiple rounds of upsampling processing on the encoding features of the first face sample image to obtain a face restoration image.

[0018] In one embodiment, the codec model to be trained includes a feature storage network; the feature storage network is used to store a feature dictionary; the feature dictionary refers to a dictionary storing features of each first face sample image in the first face sample image set; the image generation module is also used to obtain a feature dictionary, the feature dictionary stores features obtained by extracting features from each first face sample image in the first face sample image set; determine a second similarity between the encoding features of the first face sample image and the features in the feature dictionary, and filter out similar features similar to the encoding features of the first face sample image from the feature dictionary based on the second similarity; call the decoder to be trained to perform multiple rounds of upsampling on the similar features to obtain a face restoration image.

[0019] In one embodiment, the features in the feature space include the encoding features of the first face sample image; the first training module is also used to determine a first loss based on the difference between the face restoration image and the first face sample image; determine a second loss based on the difference between the first sample image features and the restoration image features; determine a third loss based on the difference between the encoding features of the first face sample image and similar features; the similar features are features similar to the encoding features of the first face sample image selected from a feature dictionary; and adjust the parameters of the codec model to be trained based on the first loss, the second loss and the third loss.

[0020] In one embodiment, the third loss includes a dictionary update loss and an encoder update loss; the dictionary update loss is used to fix the encoder in the codec model to be trained during the training process and update the features in the feature dictionary; the encoder update loss is used to fix the feature dictionary during the training process and update the parameters of the encoder in the codec model to be trained.

[0021] In one embodiment, the second training module is also used to obtain a third face sample image, call the encoder in the trained codec model to convert the third face sample image from pixel space to feature space, and obtain the encoding features of the third face sample image; input the encoding features of the third face sample image into the face image generation model to be trained, and call the face image generation model to be trained to generate predicted face features; call the decoder in the trained codec model to restore the predicted face features to pixel space to obtain a predicted face image; based on the predicted face image and the predicted face features, train the face image generation model to be trained until the second training stop condition is met.

[0022] In one embodiment, the second training module is also used to call the face recognition model to extract features of the predicted face image and the third face sample image to obtain predicted image features and third sample image features; determine a fourth loss based on the difference between the predicted image features and the third sample image features; obtain a preset original loss function and determine the original loss based on the original loss function; and adjust the model parameters of the face image generation model to be trained based on the original loss and the fourth loss.

[0023] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps in the training method of any facial image generation model provided in the embodiments of the present application.

[0024] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the training method of any facial image generation model provided in the embodiments of the present application are implemented.

[0025] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in the training method of any facial image generation model provided in the embodiments of the present application.

[0026] The training method, device, computer equipment, storage medium and computer program product of the above-mentioned face image generation model can obtain a pre-trained face recognition model, and can train the codec model based on the pre-trained face recognition model to obtain a trained codec model. By obtaining the trained codec model, the face image generation model can be trained based on the trained codec model, so that a trained face image generation model can be obtained. Since the codec model is trained by the face recognition model, the identity recognition ability of the face recognition model can be transferred to the codec model, and then the trained codec model can accurately identify, encode and decode the identity identifiable features in the face image. Since the trained codec model can accurately identify, encode and decode the identity identifiable features in the face image, the content output by the face image generation model obtained by training based on the codec model can also be more accurate.

[0027] Based on this, it is necessary to provide a facial image generation method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of the generated facial images in order to address the above technical problems.

[0028] In a first aspect, the present application provides a method for generating a face image, the method comprising:

[0029] Get the face image generation model and decoder;

[0030] Calling the facial image generation model to generate predicted facial features;

[0031] Calling the decoder to decode the predicted facial features to obtain a facial image;

[0032] Wherein, the face image generation model and the decoder are trained by the above-mentioned face image generation model training method.

[0033] In a second aspect, the present application also provides a training device for a face image generation model, the device comprising:

[0034] The model acquisition module is used to obtain the face image generation model and decoder.

[0035] The feature generation module is used to call the face image generation model to generate predicted face features.

[0036] The decoding module is used to call the decoder to decode the predicted facial features to obtain a facial image.

[0037] In one of the embodiments, the feature generation module is also used to obtain a scene description text, wherein the scene description text is a text used to describe the characteristics of a facial image to be generated; the scene description text is encoded to obtain text features of the scene description text; the text features are input into the facial image generation model, the facial image generation model is called to determine an original noise map corresponding to the text features, and multiple rounds of noise removal processing are performed based on the original noise map to obtain predicted facial features that match the scene description text.

[0038] In one of the embodiments, the facial image generation device also includes an update training module, which is used to generate a sample label corresponding to the facial image based on the scene description text; and update the facial recognition model based on the facial image and the sample label corresponding to the facial image.

[0039] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the facial image generation methods provided in the embodiments of the present application are implemented.

[0040] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any facial image generation method provided in the embodiments of the present application are implemented.

[0041] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in any facial image generation method provided in the embodiments of the present application.

[0042] The above-mentioned facial image generation method can output predicted facial features based on the facial image generation model by acquiring a facial image generation model and a decoder, and decode the predicted facial features based on the decoder to obtain a facial image. Since the facial image generation model is trained based on a pre-trained codec, and the codec trained based on the face recognition model can accurately identify, encode and decode the identity-recognizable features in the facial image, the facial image generation model trained based on the pre-trained codec can output more accurate predicted facial features. And since the decoder can accurately decode, the decoded facial image is also more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A diagram of an application environment of a training method for a facial image generation model in one embodiment;

[0044] Figure 2 A schematic diagram of a flow chart of a method for training a face image generation model in one embodiment;

[0045] Figure 3 is a schematic diagram of a coding and decoding model in an embodiment;

[0046] Figure 4 A schematic diagram of determining identity probability in one embodiment;

[0047] Figure 5 A schematic diagram of the overall training process of a face recognition model in one embodiment;

[0048] Figure 6 A schematic diagram of training a codec model in one embodiment;

[0049] Figure 7 A schematic diagram of the overall training process of a coding and decoding model in one embodiment;

[0050] Figure 8 A schematic diagram of generating a predicted face image in one embodiment;

[0051] Fig. 9A schematic diagram of a training process of a face image generation model in one embodiment;

[0052] Fig.10 Schematic diagram of a process of generating a facial image in one embodiment;

[0053] Fig.11 A schematic diagram of generating a face image based on a scene description text in one embodiment;

[0054] Fig.12 A schematic diagram of a flow chart of a method for training a facial image generation model in a specific embodiment;

[0055] Fig.13 A schematic diagram of an overall training framework of a face image generation model in one embodiment;

[0056] Fig.14 A schematic diagram of a specific framework for training a facial image generation model in one embodiment;

[0057] Fig.15 A structural block diagram of a training device for a facial image generation model in one embodiment;

[0058] Fig.16 is a structural block diagram of a facial image generating device in one embodiment;

[0059] Fig.17 is an internal structure diagram of a computer device in one embodiment;

[0060] Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] The training method of the face image generation model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Both the terminal 102 and the server 104 can be used alone to execute the training method of the face image generation model provided in the embodiment of the present application. The terminal 102 and the server 104 can also be used in collaboration to execute the training method of the face image generation model provided in the embodiment of the present application. Taking the example that the terminal 102 and the server 104 can be used in collaboration to execute the training method of the face image generation model provided in the embodiment of the present application, the user can initiate the training instruction of the face image generation model through the terminal 102, and then the server 104 can train the codec model based on the first face sample image and the face recognition model, and then train the face image generation model based on the trained codec model to obtain a trained face image generation model, wherein the trained face image generation model is used to generate a face image. The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0063] It should be noted that the words "first", "second" and similar terms used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Unless the context clearly indicates otherwise, the singular form "one", "an" or "the" and similar terms do not indicate a quantity restriction, but rather indicate the presence of at least one. The quantities of "multiple" or "multiple copies" mentioned in the various embodiments of the present application all refer to the quantity of "at least two", for example, "multiple" refers to "at least two", and "multiple copies" refers to "at least two copies".

[0064] In one embodiment, Figure 2 As shown, a training method for a face image generation model is provided, and the method is described by using a computer device as an example. The computer device can be Figure 1 The training method of the face image generation model includes the following steps:

[0065] Step 202: Obtain a pre-trained face recognition model, a codec model to be trained, and a first face sample image.

[0066] The face recognition model refers to a machine learning model used to recognize faces. The face recognition model can be trained in advance to obtain a trained face recognition model, which is the pre-trained face recognition model mentioned above. The codec model to be trained refers to the codec model to be trained. Figure 3 , the encoding and decoding model may include an encoder, a feature storage network and a decoder. The encoder is used to convert the image from the pixel space to the feature space, the feature storage network is used to store the feature dictionary, and the decoder is used to restore the features in the feature space to the pixel space. The pixel space is also called the RGB space, which is a space composed of pixels in the image. The feature space is a space composed of feature vectors. Figure 3 A schematic diagram of a coding and decoding model in implementation is shown.

[0067] Specifically, when a computer device receives a training task for a face image generation model, the computer device can obtain a pre-trained face recognition model, a first face sample image, and a codec model to be trained, so that the codec model to be trained can be trained based on the pre-trained face recognition model and the first face sample image, and then the face image generation model to be trained can be trained based on the trained codec model.

[0068] Step 204, calling the codec model to be trained to convert the first face sample image from the pixel space to the feature space, and restore the features in the feature space to the pixel space to obtain a face restoration image.

[0069] Specifically, the computer device can input the first face sample image into the codec model to be trained, convert the first face sample image from pixel space to high-dimensional feature space through the encoder in the codec model to be trained, and decode the high-dimensional features in the feature space through the decoder in the codec model to be trained to restore the high-dimensional features to the pixel space to obtain a face restoration image.

[0070] In one embodiment, the function of the encoder in the encoding and decoding model is to extract features from the input image and compress the image from the RGB space to a high-dimensional space. The feature map outputted by the encoder retains the semantics and representation information of the original image, but its computational dimension has been greatly reduced. The encoder usually has a structure of a convolutional neural network (CNN), which includes operations such as convolution calculation, nonlinear activation function (Relu) calculation, and pooling calculation. The vector size of the encoder output is k×n. Where k is the dimension size of each feature vector, and n is the number of feature vectors.

[0071] In one embodiment, the function of the decoder in the encoding and decoding model is to decode the high-dimensional features in the feature space and restore the high-dimensional information to the RGB space. The decoder usually has a convolutional neural network (CNN) structure, which includes operations such as convolution calculation, nonlinear activation function (ReLU) calculation, and pooling calculation.

[0072] Step 206, calling the pre-trained face recognition model to extract features of the first face sample image and the face restoration image respectively, to obtain first sample image features and restoration image features.

[0073] Specifically, after obtaining the face restoration image and the first face sample image, the computer device can input the face restoration image and the first face sample image into a pre-trained face recognition model, extract features of the face restoration image through the pre-trained face recognition model to obtain restoration image features, and extract features of the first face sample image through the pre-trained face recognition model to obtain first sample image features.

[0074] In one embodiment, the pre-trained face recognition model can extract spatial features from the first sample image features and the restored image features, and output the first sample image features and the restored image features that retain the image spatial structure information. That is, the first sample image features and the restored image features can include the mutual spatial position relationship or relative position relationship between multiple targets segmented from the corresponding image.

[0075] In one embodiment, taking the extraction of features of the first face sample image as an example, the reference face recognition model can first determine the location of the face in the first face sample image. For example, the face recognition model can first perform face detection, for example, use a pre-trained convolutional neural network to extract features, and then input the extracted features into multiple parallel branches, each branch has two fully connected layers for predicting faces of different sizes. The face recognition model outputs a matrix based on the predicted faces of different sizes. Each cell of the matrix predicts the probability that a face may exist in a certain coordinate area and the center coordinates, width and height of the face. Then, the overlapping prediction frames are merged by the non-maximum suppression method to obtain the final location of the face in the first face sample image. After the location of the face is obtained, the face recognition model can perform face alignment to transform the faces to a uniform angle or posture. For example, key point detection can be performed on the face at the location of the face, and the key points obtained by the detection are used to transform the face to a standard template face through similarity transformation to obtain the aligned face after transformation. When the aligned face is obtained, the face recognition model can extract the first sample image feature from the aligned face. Similarly, the face recognition model can also extract the restored image feature from the face restored image through the above process.

[0076] Step 208: training the codec model to be trained according to the first sample image feature and the restored image feature until a first training stop condition is met.

[0077] Specifically, the computer device may train the codec model to be trained based on the first sample image feature and the restored image feature until the first training stop condition is met, thereby obtaining the trained codec model. The first training stop condition may be freely set according to demand, for example, after reaching a preset training round, it may be determined that the first training stop condition is met, or after determining that the output result reaches a preset target, it may be determined that the first training stop condition is met.

[0078] In one of the embodiments, the computer device may train the codec model to be trained in the direction of reducing the difference between the features of the first sample image and the features of the restored image, so that the face restoration image output by the trained codec model can be close to the first face sample image. That is, the identity of the face in the face restoration image output by the trained codec model is the same as the identity of the face in the first face sample image. The codec model is trained based on the sampled face recognition model, and the face recognition capability of the face recognition model can be transferred to the codec model, so that the features output by the encoder in the codec model can maintain the identity identifiable features of the first face sample image, and then the face restoration image output by the decoder in the codec model based on the features containing the identity identifiable features can maintain the same identity as the first face sample image, so that the identity of the picture output by the decoder can be controlled.

[0079] Step 210, obtaining a facial image generation model to be trained, and calling a trained encoding and decoding model to train the facial image generation model to be trained until a second training stop condition is met; wherein the trained facial image generation model is used to generate facial images.

[0080] Specifically, after obtaining the trained codec model, the computer device can train the face image generation model to be trained through the trained codec model. For example, the computer device can obtain a third face sample image for training the face image generation model, and convert the third face sample image from the pixel space to the feature space based on the encoder in the trained codec model, and input the features converted to the feature space into the face image generation model to be trained, and then the face image generation model can perform face prediction based on the input features to obtain the face features of the predicted face. For the convenience of description, the face features of the predicted face output by the face image generation model are referred to as predicted face features. The computer device can input the predicted face features into the decoder in the trained codec model, and then the decoder can restore the predicted face features from the feature space to the pixel space to obtain the predicted restored image. Further, the computer device trains the face image generation model to be trained based on the third face sample image and the predicted restored image until the second training stop condition is met. Among them, the second training stop condition can be freely set according to needs. For example, when the difference between the third face sample image and the predicted restored image is less than or equal to a preset threshold, it is determined that the second training stop condition is met.

[0081] In one of the embodiments, after the trained face image generation model is obtained, the face image can be generated based on the trained face image generation model.

[0082] In the training method of the above-mentioned face image generation model, by obtaining a pre-trained face recognition model, the codec model can be trained based on the pre-trained face recognition model to obtain a trained codec model, and by obtaining the trained codec model, the face image generation model can be trained based on the trained codec model, so that a trained face image generation model can be obtained. Since the codec model is trained by the face recognition model, the identity recognition capability of the face recognition model can be transferred to the codec model, and then the trained codec model can accurately identify, encode and decode the identity identifiable features in the face image. Since the trained codec model can accurately identify, encode and decode the identity identifiable features in the face image, the content output by the face image generation model obtained by training based on the codec model can also be more accurate.

[0083] In addition, in the constraint process of the conventional codec model, the training process of the codec model is mainly constrained by using a picture classification model, so the codec model cannot maintain the identity-recognizable features of the face picture, which leads to the uncontrollable identity of the image output by the decoder. The present application uses a face recognition model to constrain the training process of the codec model, which can enhance the degree of preservation of identity-recognizable features during the conversion process from pixel space to feature space and from feature space back to pixel space. Since the codec model trained based on the face recognition model can maintain the identity-recognizable features of the picture during the picture conversion process, the encoder in the trained codec model can retain the identity-recognizable features of the picture, and the decoder in the trained codec model can accurately decode the features output by the encoder to obtain a face picture with controllable identity, and then the face image generation model trained based on the trained codec model can also accurately control the identity of the generated face.

[0084] In addition, since the face recognition model is used to train the encoding and decoding model, this means that the prior knowledge of the encoder in the encoding and decoding model is an identity feature extractor obtained after large-scale data training. This increases the encoder's ability to express face images and makes the encoding more accurate.

[0085] In one of the embodiments, before calling a pre-trained face recognition model to respectively extract features of the first face sample image and the face restoration image, the method further includes: obtaining a second face sample image and a sample label corresponding to the second face sample image; performing feature extraction on the second face sample image to obtain second sample image features; determining, based on the second training image features, a probability value that the second face sample image belongs to each preset category in a preset identity category set; and training the face recognition model to be trained based on the probability value and the sample label corresponding to the second face sample image until a third training stop condition is met.

[0086] Specifically, before calling the pre-trained face recognition model to extract features of the first face sample image and the face restoration image, the face recognition model can also be trained. When the face recognition model needs to be trained, the computer device can obtain a second face sample image. The second face sample image has a corresponding sample label, and the sample label is used to mark the identity of the face in the second face sample image.

[0087] The computer device inputs the second face sample image into the face recognition model, extracts the features of the second face sample image through the face recognition model, and obtains the features of the second sample image. For example, the face recognition model to be trained can perform convolution processing on the second face sample image to obtain the features of the second sample image. The face recognition model to be trained can determine the probability value of the second face sample image belonging to each category in the preset identity category set based on the features of the second sample image. Among them, the preset identity category set records a plurality of preset categories, each category represents an identity, for example, the second face sample image set includes the face image of user A, the face image of user B, the face image of user C, and the face image of user D, then the preset identity category set includes the identity identifier of user A, the identity identifier of user B, the identity identifier of user C, and the identity identifier of user D. Furthermore, the face recognition model to be trained can determine the identity corresponding to the second face sample image according to the probability value that the second face sample image belongs to each preset category, and train the face recognition model to be trained based on the identity and sample label corresponding to the second face sample image until the third training stop condition is met, so that the trained face recognition model can accurately identify the identity of the face in the image. The third training stop condition can be freely set according to needs.

[0088] In the above embodiment, the face recognition model is trained so that the trained face recognition model can accurately extract features that characterize the identity of the face image, and then the subsequent encoding and decoding model trained based on the features of the identity of the face image can generate a face restoration image that is consistent with the identity corresponding to the identity features, thereby avoiding the probability of the codec model randomly generating face restoration images of other identities.

[0089] In one embodiment, based on the second sample image feature, determining the probability value that the second face sample image belongs to each preset category in the preset identity category set includes: obtaining a class center matrix, wherein the class center matrix includes the class center of each preset category in the preset identity category set; determining a first similarity between the second sample image feature and each class center in the class center matrix, and based on the first similarity, determining the probability value that the second face sample image belongs to each predicted category in the preset identity category set.

[0090] Specifically, the face recognition model to be trained can obtain a class center matrix, wherein the class center matrix includes the class center of each preset category in the preset identity category set. The class center refers to the feature center of each identity. For example, if an identity has dozens of pictures, then the class center is the feature center extracted from these dozens of pictures. After obtaining the class center matrix, the face recognition model to be trained can determine the similarity between the second sample image feature and each class center in the class center matrix. For the convenience of description, the similarity between the second sample image feature and the class center is referred to as the first similarity below. After obtaining the first similarity between the second sample image feature and each class center, the face recognition model to be trained can determine the probability value of the second face sample image belonging to each predicted category in the preset identity category set based on the first similarity.

[0091] In one embodiment, the second sample image set includes multiple second sample images under different identities, for example, multiple second sample images of identity A, multiple second sample images of identity B, and so on. After obtaining the second sample image set, for each identity in the second sample image set, the features of each second sample image under the targeted identity can be extracted, and the features under the same feature dimension can be averaged to obtain the mean features under the feature dimension, and the mean features under each feature dimension can be combined to obtain the class center corresponding to the targeted identity. For example, for identity A, the computer device can extract the features of each second sample image under identity A, and average the features of each second sample image under identity A to obtain the class center corresponding to identity A.

[0092] In one embodiment, the shape of the class center matrix is ​​(d×m), where d is the feature dimension and m is the number of identities included in the second sample image set.

[0093] In one embodiment, reference Figure 4, the second sample image feature and the class center matrix can be subjected to matrix operation to obtain the first similarity between the second sample image feature and each class center, and each first similarity is normalized to obtain the probability value of the second face sample image belonging to each predicted class in the preset identity class set. The higher the similarity between the second sample image feature and the class center, the greater the probability that the second face sample image belongs to the identity corresponding to the class center. Figure 4 A schematic diagram of determining identity probability in one embodiment is shown.

[0094] In one embodiment, after obtaining the probability values ​​of the second sample image feature belonging to each preset category of × in the preset identity category set, the identity category corresponding to the maximum probability value can be used as the predicted label of the second sample face image, and then the face recognition model is trained based on the annotated label and predicted label of the second sample face image. The objective function of the training can be a classification function, such as a softmax function, and various softmax functions with margin types.

[0095] In the above embodiment, since the class center matrix stores the class centers corresponding to each identity, by determining the similarity between the second sample image feature and each class center, the probability value that the second sample image feature belongs to the identity corresponding to the class center can be accurately determined based on the similarity, thereby improving the accuracy of the determined probability value.

[0096] In one embodiment, reference Figure 5 , Figure 5The overall training process diagram of the face recognition model in one embodiment is shown. Step 501, during the training process, the second face sample image is read, and the read second face sample image is sent to the deep network unit for processing. Step 502, the spatial features of the second face sample image are extracted, and the output second sample image features retain the spatial structure information of the second face sample image. Step 503, obtain a class center matrix, whose shape is (d×m), where d is the feature dimension and m is the number of identity categories in the second face sample set. The class center matrix is ​​matrix-operated with the second sample image features to obtain the probability value of the second face sample image belonging to each preset category in the preset identity category set. Step 504, the probability value of the second face sample image belonging to each preset category in the preset identity category set obtained by performing matrix multiplication between the second sample image features and the centers of each category, and the label information of the two face sample images are used as input to calculate the objective function value. The objective function can select a classification function (such as softmax, various types of softmax plus margin type), or other types of objective functions can be used. Step 505, based on the gradient descent method (such as stochastic gradient descent, stochastic gradient descent with momentum term, adam, adamagrd), the entire face recognition model is trained and optimized. Repeat (step 501) to (step 505) during training, and determine whether the training stop condition is met. Step 506, if the training stop condition is met, stop training. The condition for terminating model training is generally set to the number of iterations meeting the set value, or the loss calculated by the face recognition objective function is less than the set value to complete the model training. Step 507, if the training stop condition is not met, continue training.

[0097] Since it is necessary to use a face recognition model to constrain the training of the encoding and decoding model in order to transfer the identity feature extraction capability of the face recognition model to the encoder and obtain an encoder that can effectively extract the features of face images, it is first necessary to obtain a face recognition model with a high recognition accuracy. Through the above steps 501 to 507, a face recognition model with a high recognition accuracy can be obtained.

[0098] In one of the embodiments, the codec model to be trained is called to convert the first face sample image from pixel space to feature space, and the features in the feature space are restored to pixel space to obtain a face restoration image, including: calling the encoder in the codec model to be trained to perform multiple rounds of downsampling processing on the first face sample image to obtain the encoding features of the first face sample image; calling the decoder in the codec model to be trained to perform multiple rounds of upsampling processing on the encoding features of the first face sample image to obtain the face restoration image.

[0099] Specifically, the encoder in the codec model to be trained may perform multiple rounds of downsampling processing on the first face sample image to convert the first face sample image in the pixel space to the feature space to obtain the encoding features in the feature space. And, the decoder in the codec model to be trained may perform multiple rounds of upsampling processing on the encoding features to restore the decoded features from the feature space to the pixel space to obtain the face restoration image. The multiple rounds may be greater than two rounds.

[0100] In one embodiment, the encoder is composed of a series of convolutional layers, activation functions (such as ReLU) and maximum pooling layers. Each downsampling process extracts features of the first face sample image and reduces the spatial dimension of the first face sample image. As the model goes deeper, the extracted features become more advanced and abstract. For example, the encoder may include input module 1, contraction module 1, contraction module 2, contraction module 3 and contraction module 4. The input module includes a convolutional layer and an activation layer for obtaining the first feature map. Each contraction module includes a maximum pooling layer, a convolutional layer and an activation layer. The output of input module 1 can be used as the input of contraction module 1, and the output of contraction module 1 can be used as the input of contraction module 2, and so on. The contraction module is used to reduce the size of the feature map and increase the abstraction level of the feature. Each time a contraction module is passed, it can be considered as a downsampling process. After each downsampling, the number of feature channels will double to capture more contextual information, and the feature size will be reduced, thereby reducing computing resources and memory usage, while increasing the abstraction level of the feature.

[0101] In one embodiment, the decoder portion is composed of upsampling layers and convolutional layers, the purpose of which is to restore the spatial size of the image and enhance the semantic information of the feature map. Specifically, each layer of the decoder includes the following steps: Upsampling: The size of the encoded feature is increased by an upsampling layer (such as deconvolution or bilinear interpolation) to restore it to the same spatial size as the original input image or the corresponding layer in the encoder. The purpose of this step is to restore the spatial details of the image because the encoder lost these details during the compression process. Feature splicing: After upsampling, the decoder will splice the upsampled feature map with the feature map of the corresponding size in the encoder. This splicing operation combines the low-level features (containing rich spatial information) and the high-level features (containing rich semantic information), thereby achieving the complementarity of spatial information and semantic information. Convolutional layer: The spliced ​​feature map is processed by a convolutional layer to extract higher-level features. These convolutional layers usually use smaller convolution kernels (such as 3x3) and use activation functions (such as ReLU) to increase the nonlinearity of the network. Through the above steps, the decoder can gradually restore the spatial size and details of the image while maintaining rich semantic information, and finally output a face restoration image of the same size as the first face sample image.

[0102] In the above embodiment, by upsampling and downsampling the first face sample image, a face restoration image of the same size as the first face sample image can be obtained, and then the codec model to be trained can be subsequently trained based on the first face sample image and the face restoration image of the same size as the first face sample image.

[0103] In one of the embodiments, the codec model to be trained includes a feature storage network; the feature storage network is used to store a feature dictionary; the feature dictionary refers to a dictionary that stores features of each first face sample image in the first face sample image set; the above method also includes: obtaining a feature dictionary, the feature dictionary stores features obtained by feature extraction of each first face sample image in the first face sample image set; determining a second similarity between the encoding features of the first face sample image and the features in the feature dictionary, and screening out similar features similar to the encoding features of the first face sample image from the feature dictionary based on the second similarity; calling the decoder to be trained to perform multiple rounds of upsampling processing on the encoding features to obtain a face restoration image, including: calling the decoder to be trained to perform multiple rounds of upsampling processing on the similar features to obtain a face restoration image.

[0104] Specifically, in addition to an encoder and a decoder, the codec model also includes a feature storage network, which stores a feature dictionary, and the feature dictionary stores features extracted from each first face sample image in the first face sample image set. After the encoder in the codec model to be trained extracts the coding features, it can determine the second similarity between the coding features of the first face sample image and the features in the feature dictionary, and filter out the features with the highest similarity to the coding features of the first face sample image from the feature dictionary. For the convenience of description, the filtered features will be referred to as similar features below. Furthermore, the computer device can call the decoder in the codec model to be trained to perform multiple rounds of upsampling on the similar features to obtain a face restoration image.

[0105] In one embodiment, the size of the feature dictionary is k×T, where k is the dimension of each feature vector and T is the number of feature vectors. After the encoder outputs n coding vectors, for each coding vector, a vector that is the nearest neighbor to the targeted coding vector can be found in the feature dictionary. For example, a matrix operation can be performed on the targeted coding vector and the feature vectors in the feature dictionary to obtain the second similarity between the targeted coding vector and each feature vector in the feature dictionary, and the feature vector with the largest second similarity is used as the feature vector that is the nearest neighbor to the targeted coding vector. The feature vector of the nearest neighbor is the similar feature mentioned above. The feature vectors of the nearest neighbors of the n coding vectors are used as the output of the feature storage network, and then the decoder can decode the output of the feature storage network to obtain a face restoration image.

[0106] In one embodiment, the encoder may extract features from each first face sample image in the first face sample set to obtain a feature dictionary. It is easy to understand that the feature dictionary in the initial stage of training may be an inaccurate feature dictionary. During the training of the codec model, the feature dictionary will be continuously updated, and finally a feature dictionary containing a standard feature vector will be obtained. The standard feature vector refers to the accurate feature of the first face sample image.

[0107] In the above embodiment, the feature dictionary can be regarded as a code book of a face image. Since the model training will eventually obtain a feature dictionary containing standard features, the code obtained by the encoder will also obtain a standard feature vector after being translated through the code book, so that the decoder can output a more accurate face image based on the standard feature vector.

[0108] In one of the embodiments, the codec model to be trained is trained according to the first sample image features and the restored image features, including: determining a first loss according to the difference between the face restored image and the first face sample image; determining a second loss according to the difference between the first sample image features and the restored image features; determining a third loss according to the difference between the encoding features of the first sample image and the similar features; the similar features are features similar to the encoding features of the first sample image selected from the feature dictionary; and adjusting the parameters of the codec model to be trained according to the first loss, the second loss and the third loss.

[0109] Specifically, after the first sample image features and the restored image features are extracted through the pre-trained face recognition network, the codec model can be trained based on the face restoration image, the first face sample image, the first sample image features, the restored image features, the encoding features and the similar features of the first sample image. For example, the computer device can determine the image difference between the face restoration image and the first face sample image, and determine the first loss based on the image difference between the face restoration image and the first face sample image; the computer device can determine the feature difference between the first sample image features and the restored image features, and determine the second loss based on the feature difference between the first sample image features and the restored image features; and the computer device can determine the feature difference between the encoding features and the similar features of the first sample image, and determine the third loss based on the feature difference between the encoding features and the similar features of the first sample image. Further, the computer device adjusts the model parameters of the codec model to be trained through the first loss, the second loss and the third loss.

[0110] In one embodiment, the model parameters of the codec model may be adjusted in the direction of reducing the image difference between the face restoration image and the first face sample image, reducing the feature difference between the first sample image feature and the restoration image feature, and reducing the feature difference between the encoded feature and the similar feature. The adjusted model parameters may include parameters of the encoder, parameters of the decoder, and feature vectors in the feature dictionary.

[0111] In the above embodiment, the codec model is trained by comprehensively combining the face restoration image, the first face sample image, the first sample image features, the restoration image features, the encoding features and the similar features, so as to achieve the purpose of adjusting the parameters of the encoder, the decoder and the feature dictionary in the codec model.

[0112] In one embodiment, the third loss includes a dictionary update loss and an encoder update loss; the dictionary update loss is used to fix the encoder in the codec model to be trained during the training process and update the features in the feature dictionary; the encoder update loss is used to fix the feature dictionary during the training process and update the parameters of the encoder in the codec model to be trained.

[0113] Specifically, refer to Figure 6 , the first loss and the second loss are used to train the decoder so that the decoder outputs a more accurate face restoration image. The third loss includes a dictionary update loss and an encoder update loss, wherein the dictionary update loss is used to train the feature dictionary to obtain a more accurate feature dictionary, and the encoder update loss is used to train the encoder so that the encoder outputs more accurate coding features. During the training process, the dictionary update loss can be used to fix the parameters of the encoder and only update the feature vectors in the feature dictionary, and the encoder loss is used to fix the feature dictionary and only update the parameters of the encoder. Figure 6 A schematic diagram of training a codec model in one embodiment is shown.

[0114] In one embodiment, the objective function for training the codec model may be the following formula:

[0115]

[0116] in, For the first loss, For the second loss, Update the loss for the dictionary, is the encoder update loss. x is the input first face sample image, e is the similarity feature, Decoder, For the encoder, is the first sample image feature, To restore image features, To stop the gradient operation, It is a hyperparameter and is generally set to 0.5.

[0117] Based on the objective function, the codec model is trained. This module optimizes the entire codec model based on the gradient descent method (such as stochastic gradient descent, stochastic gradient descent with momentum, adam, adagard) until the training result meets the training termination condition. The training termination condition is generally set to complete the training of the codec model when the number of iterations meets the set value or the loss calculated by the codec objective function is less than the set value.

[0118] In the above embodiment, by fixing the feature dictionary, similar features can be screened out from the feature dictionary with fixed features, thereby achieving the purpose of adjusting the model parameters of the encoder based on the screened similar vectors. By fixing the encoder, the encoding features can be output based on the encoder with fixed parameters, thereby achieving the purpose of updating the feature dictionary based on the encoding features. By adjusting the parameters of the encoder and updating the feature dictionary, the encoding features output by the encoder can be made more and more accurate, and the feature vectors stored in the feature dictionary can be made more and more accurate.

[0119] In one embodiment, reference Figure 7 , Figure 7The overall flow diagram of the training of the encoding and decoding model in one embodiment is shown. Step 701, read the first face sample image, and send the read first face sample image combination into the encoding and decoding model for processing. The size of the first face sample image is transformed to 256*256, and then the image pixel value is normalized to between [-1,1]. Step 702, the encoder in the encoding and decoding model extracts features from the first face sample image, compresses the first face sample image from the RGB space to the high-dimensional space, and the feature map outputted by it retains the semantics and representation information of the original image, but its calculation dimension has been greatly reduced. The encoder usually has a structure of a convolutional neural network (CNN), which includes operations such as convolution calculation, nonlinear activation function (Relu) calculation, and pooling calculation. Step 703, the feature storage network in the codec stores a feature dictionary, which is the feature vector extracted by the encoder for the first face sample image. After the n vectors output by the encoder, the n nearest neighbor vectors can be found in the feature dictionary. The closest n vectors are used as the input of the decoder. The feature dictionary can be regarded as a codebook for a face image. The code obtained by the encoder is translated through the codebook to obtain a standard feature vector. Step 704, the decoder decodes the high-dimensional features output in step 703, and restores and decodes the high-dimensional information into the RGB space. The decoder usually has a structure of a convolutional neural network (CNN), which includes operations such as convolution calculation, nonlinear activation function (Relu) calculation, and pooling calculation. Step 705, obtain a face recognition model, the input of which is the face restoration image and the first face sample image output by the decoder. The face recognition model extracts features from the face restoration image and the first face sample image respectively to obtain the first sample image features and the restoration image features. Step 706, the first face sample image, the face restoration image, the first sample image features, and the restoration image features are used as input to calculate the objective function value. Step 707, based on the gradient descent method (such as stochastic gradient descent, stochastic gradient descent with momentum term, adam, adagard), the entire codec model is trained and optimized. Repeat (step 701) ~ (step 707) during training, and determine whether the training stop condition is met. Step 708, if the training stop condition is met, stop training. The condition for terminating model training is generally set to the number of iterations to meet the set value, or the loss calculated by the codec objective function is less than the set value to complete the model training. Step 709, if the training stop condition is not met, continue training.

[0120] In this step, the parameters of the face recognition model are not updated and are only used to provide gradients.

[0121] When using the face image generation model to generate pictures, it is necessary to use multiple iterations to generate them. If the original picture size is used to train the diffusion model, the amount of calculation in the generation process will be too large. Therefore, a coding model can be used to train the face image generation model in a high-dimensional space, which can reduce the amount of calculation and resource consumption in the generation process. In the training of using the face image generation model to generate pictures, it is first necessary to use the coding model to encode the pictures into a high-dimensional space, and at the same time compress the dimension of the picture data, and then pass through a face image generation model, and finally decode the data generated by the face image generation model into the pixel space using the decoding model. Therefore, before training the face image generation model, it is necessary to train a coding model. This is achieved through the above steps 701 to 709.

[0122] In one of the embodiments, a trained codec model is called to train a face image generation model to be trained, including: obtaining a third face sample image; calling an encoder in the trained codec model to convert the third face sample image from pixel space to feature space to obtain encoding features of the third face sample image; inputting the encoding features of the third face sample image into the face image generation model to be trained, and calling the face image generation model to be trained to generate predicted face features; calling a decoder in the trained codec model to restore the predicted face features to pixel space to obtain a predicted face image; based on the predicted face image and the predicted face features, training the face image generation model to be trained until a second training stop condition is met.

[0123] Specifically, after obtaining the trained codec model, the facial image generation model can be trained based on the trained codec model. For example, the computer device can obtain a third facial sample image, and input the third facial sample image into the trained codec model, so as to encode the third facial sample image through the encoder in the trained codec model to obtain the encoding features of the third facial sample image. Furthermore, the computer device can input the encoding features of the third facial sample image into the facial image generation model to be trained, and output the predicted facial features through the facial image generation model to be trained. The computer device can input the predicted facial features into the decoder in the trained codec model, decode the predicted facial features through the decoder, obtain the predicted facial image, and then adjust the model parameters of the facial image generation model to be trained based on the predicted facial image and the predicted facial features.

[0124] In one embodiment, during the training of the face generation model, the encoder encodes the third face sample image into a high-dimensional feature space, and the face image generation model generates and trains the encoded features of the third face sample image in the high-dimensional feature space. The decoding network decodes the high-dimensional features generated by the face image generation model into the image space.

[0125] In one embodiment, since the trained codec model may include a feature storage network, and the feature storage network stores a feature dictionary, after obtaining the encoding features of the third face sample image, the similar features that are the nearest neighbors to the encoding features of the third face sample image can be searched in the feature dictionary, and the similar features can be input into the face image generation model to be trained, so as to output the predicted face features through the face image generation model. Since the feature dictionary stores standard feature vectors, the similar features obtained by converting the encoding features of the third face sample image through the feature dictionary are also standard feature vectors, and the accuracy of the predicted face features generated by the face image generation model can be improved based on the standard feature vectors.

[0126] In the above embodiment, the face recognition model is used to supervise the training of the codec model, and the identity feature extraction capability of the face recognition model can be transferred to the codec model, thereby obtaining an encoder that can effectively extract the features of the face image, and a decoder that can accurately decode the features. Since an encoder that can effectively extract the features of the face image and a decoder that can accurately decode the features can be obtained, the content output by the face image generation model obtained by training based on the encoder and decoder can also be more accurate.

[0127] In addition, if the original image is directly input into the face image generation model, the amount of calculation in the face image generation model to generate predicted facial features will be too large. Therefore, by using the encoder in the codec model to convert the image from pixel space to feature space, and then inputting the features in the feature space into the face image generation model, the amount of calculation and resource consumption of the face image generation model can be reduced, thereby improving training efficiency.

[0128] In one of the embodiments, the first face sample image used for training the encoding and decoding model is included in the first face sample image set, the second face sample image used for training the face recognition model is included in the second face sample image set, and the third face sample image used for training the face image generation model is included in the third face sample image set. The first face sample image set, the second face sample image set, and the third face sample image set may be the same image set or different image sets. The images in the first face sample image set, the second face sample image set, and the third face sample image set may be face images of real users collected after authorization, or may be images generated by a machine learning model.

[0129] In one of the embodiments, a face image generation model to be trained is trained based on the predicted face image and the predicted face features, including: calling a face recognition model to extract features of the predicted face image and a third face sample image to obtain predicted image features and third sample image features; determining a fourth loss based on the difference between the predicted image features and the third sample image features; obtaining a preset original loss function, determining the original loss based on the original loss function, and adjusting the model parameters of the face image generation model to be trained based on the original loss and the fourth loss.

[0130] Specifically, the computer device inputs the predicted face image and the third face sample image into the face recognition model, so that the face recognition model can extract the features of the predicted face image to obtain the predicted image features, and extract the features of the third face sample image to obtain the third sample image features. Further, the computer device determines the difference between the predicted image features and the third sample image features, and determines the fourth loss based on the difference between the predicted image features and the third sample image features. The computer device can obtain the original loss function pre-set for the face image generation model, determine the original loss based on the original loss function, and adjust the model parameters of the face image generation model to be trained based on the original loss and the fourth loss, through the original loss and the fourth loss, and based on the gradient descent method.

[0131] In one of the embodiments, a cosine distance between the predicted image feature and the third sample image feature may be determined, and the cosine distance may be used as the fourth loss.

[0132] In one embodiment, the formula Determine the fourth loss, where To predict image features, is the third sample image feature.

[0133] In one embodiment, the facial image generation model may be a diffusion model. Figure 8After the encoded features of the third face sample image are input into the face image generation model to be trained, the first round of multiple rounds of iterative generation of predicted face features is In the process of generating predicted facial features in multiple rounds, the face image generation model can perform noise prediction in the first round to obtain the predicted noise in the first round, and remove the predicted noise in the first round from the encoded features of the third face sample image to obtain the predicted facial features output in the first round. In the face image generation model, the noise prediction of the current round can be performed, and the prediction noise of the current round can be obtained. The prediction noise of the current round can be removed from the encoding features of the third face sample image to obtain the predicted face features outputted in the current round. The output predicted face features are used as the predicted face features finally output by the face recognition model. The decoder in the encoding and decoding model can decode the finally output predicted face features to obtain a predicted face image. Figure 8 A schematic diagram of generating a predicted face image in one embodiment is shown.

[0134] In one embodiment, the original loss function may be a target function of the diffusion model, for example, a cross entropy loss function, and the predicted noise and the actual noise may be input into the cross entropy loss function to obtain the original loss. After the fourth loss is superimposed on the original loss, the model parameters of the face image generation model to be trained are adjusted.

[0135] In one embodiment, reference Fig. 9, step 901, read the third face sample image, and send the read third face sample image combination to the encoding and decoding model for processing. Step 902, the encoder in the encoding and decoding model encodes the third face sample image to obtain the encoding features of the third face sample image, and inputs the encoding features of the third face sample image into the face image generation model. Step 903, the face image generation model can output the predicted face features, and input the predicted face features into the decoder and the original loss calculation module in the encoding and decoding model. Step 904, the decoder decodes the predicted face features to obtain the predicted face image, and inputs the predicted face image into the face recognition model. Step 905, the face recognition model extracts the features of the predicted face image and the third face sample image to obtain the predicted image features and the third sample image features. Step 906, the original loss calculation module performs loss calculation based on the predicted face features to obtain the original loss. Step 907, the fourth loss calculation model performs a fourth loss calculation based on the predicted image features and the third sample image features to obtain the fourth loss. Step 908, calculate the total loss based on the original loss and the fourth loss. Step 909, perform training based on the total loss, and determine whether the training stop condition is met. Step 910, if the training stop condition is met, stop training. The condition for terminating model training is generally set to complete the model training when the number of iterations meets the set value, or the total loss is less than the set value. Step 911, if the training stop condition is not met, continue training. Fig. 9 A schematic diagram of the training process of a face image generation model in one embodiment is shown.

[0136] In the above embodiment, by training the face image generation model, the face images generated by the face image generation model can become more and more accurate.

[0137] In one embodiment, if Fig.10 As shown, a method for generating a face image is provided, and the method is described by taking a computer device as an example. The computer device can be Figure 1 The face image generation method comprises the following steps:

[0138] Step 1002, obtaining a facial image generation model and a decoder.

[0139] Specifically, when a face image needs to be generated, a pre-trained face image generation model and a pre-trained decoder can be obtained, wherein the pre-trained face image generation model can be trained by the training method of the face image generation model. The pre-trained decoder is a decoder in a pre-trained codec model, and the pre-trained codec model can also be trained by the training method of the face image generation model.

[0140] Step 1004, calling the facial image generation model to generate predicted facial features.

[0141] Specifically, when a facial image needs to be generated, the computer device can generate a facial image generation task and send the facial image generation task to the facial image generation model, so that the facial image generation model can generate predicted facial features based on the facial image generation task.

[0142] Step 1006: Call a decoder to decode the predicted facial features to obtain a facial image.

[0143] Specifically, after the predicted facial features output by the facial image generation model are obtained, the predicted facial features can be input into a decoder, and the predicted facial features can be decoded by the decoder to obtain a facial image.

[0144] The above-mentioned facial image generation method can output predicted facial features based on the facial image generation model by acquiring a facial image generation model and a decoder, and decode the predicted facial features based on the decoder to obtain a facial image. Since the facial image generation model is trained based on a pre-trained codec, and the codec trained based on the face recognition model can accurately identify, encode and decode the identity-recognizable features in the facial image, the facial image generation model trained based on the pre-trained codec can output more accurate predicted facial features. And since the decoder can accurately decode, the decoded facial image is also more accurate.

[0145] In one of the embodiments, a facial image generation model is called to generate predicted facial features, including: obtaining a scene description text, wherein the scene description text is text used to describe the characteristics of a facial image to be generated; encoding the scene description text to obtain text features of the scene description text; inputting the text features into the facial image generation model, calling the facial image generation model to determine an original noise map corresponding to the text features, and performing multiple rounds of noise removal processing based on the original noise map to obtain predicted facial features that match the scene description text.

[0146] The scene description text is a text used to describe the characteristics of the face image to be generated, for example, a text used to describe the content, style, color, scene, etc. of the face image to be generated. The scene description text can be a paragraph, a sentence, or a word. Fig.11 , the video description text may be "generate a girl's face with a cartoon style", then the face image finally generated based on the scene description text may be a cartoon-style image of a girl's face. Fig.11 A schematic diagram of generating a face image based on a scene description text in an embodiment is shown.

[0147] Specifically, the computer device can obtain the scene description text and encode the video description text to obtain the text features of the scene description text. Further, the computer device can retrieve image features that match the text features in a preset image feature library, and input the retrieved image features into a face image generation model, and gradually add noise to the retrieved image features through the face image generation model to obtain an original noise map that matches the scene description text. Further, the face image generation model can gradually denoise the original noise map to output predicted face features that match the scene description text. For example, the face image generation model can be specifically a diffusion model. The diffusion model (Diffusion Modles) is a generation model. The diffusion model can be composed of multiple denoising networks connected in series. The first denoising network can generate the prediction noise of the first round, and subtract the prediction noise of the first round from the original noise map to obtain the predicted face features output by the first round; the second denoising network can generate the prediction noise of the second round, and subtract the prediction noise of the second round from the predicted face features output by the first round to obtain the predicted face features output by the second round, and so on. The predicted face features output by the last denoising network are used as the predicted face features that are finally output by the face image generation model and match the scene description text. Then the decoder can decode the predicted face features that match the scene description text to obtain a face image that matches the scene description text.

[0148] In the above embodiment, only the scene description text needs to be set to generate a face image that matches the scene description text, which greatly reduces the threshold for generating a face image under a preset scene.

[0149] In one of the embodiments, the method further includes: generating sample labels corresponding to the face image based on the scene description text; and updating and training the face recognition model based on the face image and the sample labels corresponding to the face image.

[0150] Specifically, due to the diversity of face recognition model deployment scenarios, the face recognition model needs to be adapted to the data in different scenarios. Therefore, it is necessary to collect data in different scenarios, which will consume a lot of resources. In addition, there will be the disadvantage of manual labeling errors when quickly annotating the collected data. Therefore, scene description text and face image generation model can be used to generate face pictures in specific scenes, and sample labels corresponding to face images are generated based on the scene description text. Based on the face image and the sample labels corresponding to the face image, the face recognition model is trained.

[0151] Exemplarily, since the training images of the face recognition model need to maintain the consistency of the identity within the class and the diversity of the pictures within the class, multiple face images under at least one identity can be generated based on the face image generation model and the decoder. For example, after the face image generation model is trained with the third face sample image under identity A, the third face sample image under identity B, and the third face sample image under identity C, the face image generation model has the ability to generate face images of identity A, identity B, and identity C. The face image generation model and the decoder can generate multiple face images under identity A when the scene description text is "please generate a face image of identity A", and use "identity A" in the scene description text as the sample label of the generated face image. In this way, the diversity of face images of identity A can be increased, and since the generated images are all face images of identity A, the consistency of identity within the class is also maintained, and the face recognition model can be trained based on face images with identity consistency within the class and diversity of pictures within the class.

[0152] In one specific embodiment, reference Fig.12 , Fig.12 A schematic flow chart of a method for training a facial image generation model in a specific embodiment is shown.

[0153] Step 1202, obtaining a second face sample image and a sample label corresponding to the second face sample image; performing feature extraction on the second face sample image to obtain second sample image features.

[0154] Step 1204, obtaining a class center matrix, wherein the class center matrix includes the class center of each preset category in the preset identity category set; determining a first similarity between the second sample image feature and each class center in the class center matrix, and based on the first similarity, determining the probability value of the second face sample image belonging to each predicted category in the preset identity category set.

[0155] Step 1206: Train the face recognition model to be trained according to the probability value and the sample label corresponding to the second face sample image until a third training stop condition is met.

[0156] Step 1208, obtain a pre-trained face recognition model, a codec model to be trained, and a first face sample image; call the encoder in the codec model to be trained to perform multiple rounds of downsampling processing on the first face sample image to obtain the encoding features of the first face sample image.

[0157] Step 1210, obtaining a feature dictionary, the feature dictionary storing features obtained by extracting features from each first face sample image in the first face sample image set; determining a second similarity between the encoded features of the first face sample image and the features in the feature dictionary, and screening out similar features similar to the encoded features of the first face sample image from the feature dictionary based on the second similarity.

[0158] Step 1212, calling the decoder to be trained to perform multiple rounds of upsampling processing on similar features to obtain a restored face image.

[0159] Step 1214, calling the pre-trained face recognition model to extract features of the first face sample image and the face restoration image respectively, to obtain first sample image features and restoration image features.

[0160] Step 1216, determining a first loss based on the difference between the face restoration image and the first face sample image; determining a second loss based on the difference between the first sample image feature and the restoration image feature; determining a third loss based on the difference between the encoding feature and the similar feature of the first face sample image; the similar feature is a feature similar to the encoding feature of the first face sample image selected from the feature dictionary; adjusting the parameters of the codec model to be trained based on the first loss, the second loss and the third loss.

[0161] Step 1218, obtaining a third face sample image; calling the encoder in the trained codec model to convert the third face sample image from pixel space to feature space, and obtaining the encoding features of the third face sample image.

[0162] Step 1220, input the encoded features of the third face sample image into the face image generation model to be trained, and call the face image generation model to be trained to generate predicted face features; call the decoder in the trained codec model to restore the predicted face features to pixel space to obtain a predicted face image.

[0163] Step 1222, calling the face recognition model to extract features of the predicted face image and the third face sample image, obtaining predicted image features and third sample image features; and determining the fourth loss based on the difference between the predicted image features and the third sample image features.

[0164] Step 1224, obtain a preset original loss function, and determine the original loss according to the original loss function; adjust the model parameters of the face image generation model to be trained according to the original loss and the fourth loss.

[0165] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0166] The present application also provides an application scenario, which applies the above-mentioned training method for generating a facial image model. Specifically, the training method for generating a facial image model is applied in the application scenario as follows:

[0167] When a face image generation model is needed, the face image generation model can be trained in advance. Fig.13 , Fig.13 A schematic diagram of the overall training framework of a face image generation model in one embodiment is shown. The entire training process may include a model acquisition stage and a model application stage. In the model acquisition stage, the face recognition model may be first trained to obtain a trained face recognition model, and then the codec model may be trained based on the trained face recognition model to obtain a trained codec model. In the model application stage, the face image generation model may be trained by the trained codec model to obtain a trained face image generation model. Among them, the face image generation model and the decoder in the codec model may be used together to generate a face image.

[0168] More specifically, refer to Fig.14 , Fig.14 A specific framework schematic diagram of the training of a face image generation model in one embodiment is shown. The face recognition model can be first trained using a second face sample image and a sample label corresponding to the second face sample image, so that the trained face recognition model can accurately identify the identity of the face. Then, the encoding and decoding model is trained using the trained face recognition model and the first face sample image to transfer the face recognition capability of the face recognition model to the encoding and decoding model, so that the encoding and decoding model can perform more accurate encoding and decoding. Finally, the face image generation model is trained using a third face sample image, the trained face recognition model, and the trained encoding and decoding model, so that the trained face image generation model can output more accurate predicted face features.

[0169] The present application also provides an application scenario, which applies the above-mentioned face image generation method. Specifically, the face image generation method is applied in the application scenario as follows:

[0170] When it is necessary to increase sample images for training the machine learning model, the face image generation model and decoder can be called to output face images, and then the machine learning model to be trained can be trained based on the output face images.

[0171] The above application scenarios are only for illustrative purposes, and it is understood that the application of the schedule synchronization method provided in each embodiment of the present application is not limited to the above scenarios. For example, in a game scenario, when it is necessary to generate a virtual photo in the game scenario, the face image generation model and decoder can be called to output the face image, and the output face image can be displayed as a virtual photo in the game scenario.

[0172] Based on the same inventive concept, the embodiment of the present application also provides a training device for a facial image generation model for implementing the training method for the facial image generation model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more facial image generation models provided below can refer to the limitations of the training method for the facial image generation model above, and will not be repeated here.

[0173] In one embodiment, Fig.15 As shown, a training device 1500 for a face image generation model is provided, comprising: an image generation module 1502, a first training module 1504 and a second training module 1506, wherein:

[0174] The image generation module 1502 is used to obtain a pre-trained face recognition model, a codec model to be trained and a first face sample image; call the codec model to be trained to convert the first face sample image from pixel space to feature space, and restore the features in the feature space to the pixel space to obtain a face restoration image.

[0175] The first training module 1504 is used to call the pre-trained face recognition model to extract the features of the first face sample image and the face restoration image respectively, and obtain the first sample image features and the restoration image features; train the codec model to be trained according to the first sample image features and the restoration image features until the first training stop condition is met.

[0176] The second training module 1506 is used to obtain the face image generation model to be trained, and call the trained encoding and decoding model to train the face image generation model to be trained until the second training stop condition is met; wherein the trained face image generation model is used to generate a face image.

[0177] In one embodiment, the training device 1500 for the face image generation model also includes a third training module 1508, which is used to obtain a second face sample image and a sample label corresponding to the second face sample image; perform feature extraction on the second face sample image to obtain the second sample image feature; and determine the probability value of the second face sample image belonging to each preset category in a preset identity category set based on the second sample image feature; and train the face recognition model to be trained based on the probability value and the sample label corresponding to the second face sample image until a third training stop condition is met.

[0178] In one embodiment, the third training module 1508 is also used to obtain a class center matrix, wherein the class center matrix includes the class center of each preset category in a preset identity category set; determine a first similarity between the second sample image feature and each class center in the class center matrix, and based on the first similarity, determine the probability value of the second face sample image belonging to each predicted category in the preset identity category set.

[0179] In one embodiment, the image generation module 1502 is also used to call the encoder in the codec model to be trained to perform multiple rounds of downsampling processing on the first face sample image to obtain the encoding features of the first face sample image; call the decoder in the codec model to be trained to perform multiple rounds of upsampling processing on the encoding features of the first face sample image to obtain a face restoration image.

[0180] In one embodiment, the codec model 1500 to be trained includes a feature storage network; the feature storage network is used to store a feature dictionary; the feature dictionary refers to a dictionary that stores features of each first face sample image in the first face sample image set; the image generation module 1502 is also used to obtain a feature dictionary, the feature dictionary stores features obtained by extracting features from each first face sample image in the first face sample image set; determine the second similarity between the encoding features of the first face sample image and the features in the feature dictionary, and filter out similar features similar to the encoding features of the first face sample image from the feature dictionary based on the second similarity; call the decoder to be trained to perform multiple rounds of upsampling on the similar features to obtain a face restoration image.

[0181] In one embodiment, the features in the feature space include the encoding features of the first face sample image; the first training module 1504 is also used to determine a first loss based on the difference between the face restoration image and the first face sample image; determine a second loss based on the difference between the first sample image features and the restoration image features; determine a third loss based on the difference between the encoding features of the first face sample image and similar features; the similar features are features similar to the encoding features of the first face sample image selected from a feature dictionary; and adjust the parameters of the codec model to be trained based on the first loss, the second loss and the third loss.

[0182] In one embodiment, the third loss includes a dictionary update loss and an encoder update loss; the dictionary update loss is used to fix the encoder in the codec model to be trained during the training process and update the features in the feature dictionary; the encoder update loss is used to fix the feature dictionary during the training process and update the parameters of the encoder in the codec model to be trained.

[0183] In one embodiment, the second training module 1506 is also used to obtain a third face sample image, call the encoder in the trained codec model to convert the third face sample image from pixel space to feature space, and obtain the encoded features of the third face sample image; input the encoded features of the third face sample image into the face image generation model to be trained, and call the face image generation model to be trained to generate predicted face features; call the decoder in the trained codec model to restore the predicted face features to pixel space to obtain a predicted face image; based on the predicted face image and the predicted face features, train the face image generation model to be trained until the second training stop condition is met.

[0184] In one embodiment, the second training module 1506 is also used to call the face recognition model to extract features of the predicted face image and the third face sample image to obtain predicted image features and third sample image features; determine a fourth loss based on the difference between the predicted image features and the third sample image features; obtain a preset original loss function and determine the original loss based on the original loss function; and adjust the model parameters of the face image generation model to be trained based on the original loss and the fourth loss.

[0185] Based on the same inventive concept, the embodiment of the present application also provides a facial image generation device for implementing the facial image generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more facial image generation device embodiments provided below can refer to the limitations on the facial image generation method above, and will not be repeated here.

[0186] In one embodiment, Fig.16 As shown, a facial image generation device 1600 is provided, comprising: a model acquisition module 1602, a feature generation module 1604 and a decoding module 1606, wherein:

[0187] The model acquisition module 1602 is used to obtain a facial image generation model and a decoder.

[0188] The feature generation module 1604 is used to call the face image generation model to generate predicted face features.

[0189] The decoding module 1606 is used to call the decoder to decode the predicted facial features to obtain a facial image.

[0190] In one embodiment, the feature generation module 1604 is also used to obtain scene description text, wherein the scene description text is text used to describe the characteristics of the facial image to be generated; encode the scene description text to obtain text features of the scene description text; input the text features into the facial image generation model, call the facial image generation model to determine the original noise map corresponding to the text features, and perform multiple rounds of noise removal processing based on the original noise map to obtain predicted facial features that match the scene description text.

[0191] In one embodiment, the facial image generating device 1600 further includes an updating training module 1606, which is used to generate a sample label corresponding to the facial image according to the scene description text; and to update the facial recognition model according to the facial image and the sample label corresponding to the facial image.

[0192] Each module in the training device of the face image generation model and the face image generation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0193] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.17 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store training data and face image generation data of a face image generation model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a face image generation model and a face image generation method are implemented.

[0194] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.18 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method for a face image generation model and a face image generation method are implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0195] Those skilled in the art will understand that Figure 17 to Figure 18The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0196] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0197] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0198] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a facial image generation model, characterized in that: The method comprises: Obtain a pre-trained face recognition model, a codec model to be trained, and a first face sample image; Calling the codec model to be trained to convert the first face sample image from a pixel space to a feature space, and restoring the features in the feature space to the pixel space to obtain a face restoration image; Calling the pre-trained face recognition model to extract features of the first face sample image and the face restoration image respectively, to obtain first sample image features and restoration image features; Training the codec model to be trained according to the first sample image feature and the restored image feature until a first training stop condition is met; Obtaining a face image generation model to be trained, and calling a trained encoding and decoding model to train the face image generation model to be trained until a second training stop condition is met; wherein the trained face image generation model is used to generate a face image; The calling of the trained encoding and decoding model to train the face image generation model to be trained includes: Obtain a third person's face sample image; Calling the encoder in the trained encoding and decoding model to convert the third face sample image from pixel space to feature space to obtain encoding features of the third face sample image; Inputting the encoded features of the third face sample image into a face image generation model to be trained, and calling the face image generation model to be trained to generate predicted face features; Calling the decoder in the trained encoding and decoding model to restore the predicted facial features to pixel space to obtain a predicted facial image; Based on the predicted facial image and the predicted facial features, the facial image generation model to be trained is trained.

2. The method according to claim 1, characterized in that Before obtaining the pre-trained face recognition model, the method further includes: Obtaining a second face sample image and a sample label corresponding to the second face sample image; Extracting features from the second face sample image to obtain second sample image features; Determining, based on the second sample image feature, a probability value that the second face sample image belongs to each preset category in a preset identity category set; The face recognition model to be trained is trained according to the probability value and the sample label corresponding to the second face sample image until a third training stop condition is met.

3. The method according to claim 2, characterized in that The step of determining, based on the second sample image feature, a probability value that the second face sample image belongs to each preset category in a preset identity category set includes: Obtaining a class center matrix, wherein the class center matrix includes the class center of each preset category in the preset identity category set; Determine a first similarity between the second sample image feature and each class center in the class center matrix, and determine a probability value of the second face sample image belonging to each predicted class in a preset identity class set based on the first similarity.

4. The method according to claim 1, characterized in that: The calling of the codec model to be trained to convert the first face sample image from a pixel space to a feature space, and restoring the features in the feature space to the pixel space to obtain a face restoration image, includes: Calling the encoder in the encoding and decoding model to be trained to perform multiple rounds of downsampling processing on the first face sample image to obtain encoding features of the first face sample image; The decoder in the encoding and decoding model to be trained is called to perform multiple rounds of upsampling processing on the encoding features of the first face sample image to obtain a face restoration image.

5. The method according to claim 4, characterized in that The encoding and decoding model to be trained includes a feature storage network; The feature storage network is used to store a feature dictionary; The method further comprises: Acquire a feature dictionary, wherein the feature dictionary stores features obtained by extracting features from each first face sample image in the first face sample image set; Determining a second similarity between the encoding feature of the first face sample image and the feature in the feature dictionary, and screening out similar features similar to the encoding feature of the first face sample image from the feature dictionary according to the second similarity; The calling of the decoder in the codec model to be trained to perform multiple rounds of upsampling processing on the coding features of the first face sample image to obtain a face restoration image includes: The decoder to be trained is called to perform multiple rounds of upsampling processing on the similar features to obtain a face restoration image.

6. The method according to claim 1, characterized in that The features in the feature space include the encoding features of the first face sample image; and the training of the codec model to be trained according to the first sample image features and the restored image features includes: determining a first loss according to a difference between the restored face image and the first face sample image; determining a second loss according to a difference between the first sample image feature and the restored image feature; determining a third loss according to a difference between the encoding feature of the first face sample image and a similar feature, wherein the similar feature is a feature similar to the encoding feature of the first face sample image selected from a feature dictionary; According to the first loss, the second loss and the third loss, the parameters of the codec model to be trained are adjusted.

7. The method according to claim 6, characterized in that The third loss includes a dictionary update loss and an encoder update loss; The dictionary update loss is used to fix the parameters of the encoder in the codec model to be trained during the training process, and to update the features in the feature dictionary; The encoder update loss is used to fix the feature dictionary during the training process and update the parameters of the encoder in the codec model to be trained.

8. The method according to claim 1, characterized in that The step of training the face image generation model to be trained based on the predicted face image and the predicted face features includes: Calling the pre-trained face recognition model to extract features of the predicted face image and the third face sample image respectively, to obtain predicted image features and third sample image features; determining a fourth loss according to a difference between the predicted image feature and the third sample image feature; Obtaining a preset original loss function, and determining an original loss according to the original loss function; According to the original loss and the fourth loss, the model parameters of the face image generation model to be trained are adjusted.

9. A method for generating a face image, characterized in that: The method comprises: Get the face image generation model and decoder; Calling the facial image generation model to generate predicted facial features; Calling the decoder to decode the predicted facial features to obtain a facial image; Wherein, the facial image generation model and the decoder are trained by the method described in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that The calling of the facial image generation model to generate predicted facial features includes: Acquire a scene description text, wherein the scene description text is a text used to describe the characteristics of the face image to be generated; Encoding the scene description text to obtain text features of the scene description text; The text features are input into the face image generation model, the face image generation model is called to determine the original noise map corresponding to the text features, and multiple rounds of noise removal processing are performed based on the original noise map to obtain predicted face features that match the scene description text.

11. The method according to claim 10, characterized in that The method further comprises: Generating a sample label corresponding to the face image according to the scene description text; The face recognition model is updated and trained according to the face image and the sample label corresponding to the face image.

12. A training device for a facial image generation model, characterized in that: The device comprises: An image generation module is used to obtain a pre-trained face recognition model, a codec model to be trained, and a first face sample image; call the codec model to be trained to convert the first face sample image from a pixel space to a feature space, and restore the features in the feature space to the pixel space to obtain a face restoration image; A first training module is used to call the pre-trained face recognition model to extract features of the first face sample image and the face restoration image respectively, to obtain first sample image features and restoration image features; and train the codec model to be trained according to the first sample image features and the restoration image features until a first training stop condition is met; The second training module is used to obtain a face image generation model to be trained, and call the trained codec model to train the face image generation model to be trained until the second training stop condition is met; wherein, the trained face image generation model is used to generate a face map; wherein, calling the trained codec model to train the face image generation model to be trained includes: obtaining a third face sample image; calling the encoder in the trained codec model to convert the third face sample image from pixel space to feature space, and obtaining the encoding features of the third face sample image; inputting the encoding features of the third face sample image into the face image generation model to be trained, and calling the face image generation model to be trained to generate predicted face features; calling the decoder in the trained codec model to restore the predicted face features to pixel space, and obtaining a predicted face image; based on the predicted face image and the predicted face features, training the face image generation model to be trained.

13. The device according to claim 12, characterized in that The training device of the face image generation model also includes a third training module, which is used to obtain a second face sample image and a sample label corresponding to the second face sample image; perform feature extraction on the second face sample image to obtain second sample image features; determine, based on the second sample image features, the probability value that the second face sample image belongs to each preset category in a preset identity category set; and train the face recognition model to be trained based on the probability value and the sample label corresponding to the second face sample image until a third training stop condition is met.

14. The device according to claim 13, characterized in that The third training module is also used to obtain a class center matrix, wherein the class center matrix includes the class center of each preset category in a preset identity category set; determine a first similarity between the second sample image feature and each class center in the class center matrix, and based on the first similarity, determine the probability value of the second face sample image belonging to each predicted category in the preset identity category set.

15. The device according to claim 12, characterized in that The image generation module is further used to call the encoder in the encoding and decoding model to be trained to perform multiple rounds of downsampling processing on the first face sample image to obtain the encoding features of the first face sample image; The decoder in the encoding and decoding model to be trained is called to perform multiple rounds of upsampling processing on the encoding features of the first face sample image to obtain a face restoration image.

16. The device according to claim 15, characterized in that The encoding and decoding model to be trained includes a feature storage network; The feature storage network is used to store a feature dictionary; the image generation module is further used to obtain a feature dictionary, the feature dictionary storing features obtained by extracting features from each first face sample image in the first face sample image set; determining a second similarity between the coded features of the first face sample image and the features in the feature dictionary, and screening out similar features similar to the coded features of the first face sample image from the feature dictionary according to the second similarity; The decoder to be trained is called to perform multiple rounds of upsampling processing on the similar features to obtain a face restoration image.

17. The device according to claim 12, characterized in that The features in the feature space include the encoding features of the first face sample image; the first training module is further used to determine a first loss according to the difference between the face restoration image and the first face sample image; determine a second loss according to the difference between the first sample image feature and the restoration image feature; determine a third loss according to the difference between the encoding features of the first face sample image and similar features; the similar features are features similar to the encoding features of the first face sample image selected from the feature dictionary; According to the first loss, the second loss and the third loss, the parameters of the codec model to be trained are adjusted.

18. The device according to claim 17, characterized in that The third loss includes a dictionary update loss and an encoder update loss; the dictionary update loss is used to fix the parameters of the encoder in the codec model to be trained during the training process, and to update the features in the feature dictionary; The encoder update loss is used to fix the feature dictionary during the training process and update the parameters of the encoder in the codec model to be trained.

19. The device according to claim 12, characterized in that The second training module is also used to call the pre-trained face recognition model to extract the features of the predicted face image and the third face sample image respectively, to obtain predicted image features and third sample image features; determine the fourth loss based on the difference between the predicted image features and the third sample image features; obtain a preset original loss function, and determine the original loss based on the original loss function; adjust the model parameters of the face image generation model to be trained based on the original loss and the fourth loss.

20. A facial image generating device, characterized in that: The device comprises: A model acquisition module is used to obtain a facial image generation model and a decoder; A feature generation module, used for calling the face image generation model to generate predicted face features; A decoding module is used to call the decoder to decode the predicted facial features to obtain a facial image, wherein the facial image generation model and the decoder are trained by the method described in any one of claims 1 to 8.

21. The device according to claim 20, characterized in that The feature generation module is also used to obtain a scene description text, wherein the scene description text is a text used to describe the characteristics of a facial image to be generated; encode the scene description text to obtain text features of the scene description text; input the text features into the facial image generation model, call the facial image generation model to determine an original noise map corresponding to the text features, and perform multiple rounds of noise removal processing based on the original noise map to obtain predicted facial features that match the scene description text.

22. The device according to claim 21, characterized in that The facial image generation device also includes an update training module, which is used to generate a sample label corresponding to the facial image according to the scene description text; and update the facial recognition model according to the facial image and the sample label corresponding to the facial image.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.