Digital human generation method and device, small sample model training method and device

Abstract

The present disclosure provides a method and device for generating a digital human, which relates to the field of artificial intelligence technology, specifically natural language processing, computer vision, deep learning and other technical fields. The specific implementation scheme is: obtaining an input image and a style template input by a user; cropping the face area of ​​the input image to obtain a face image; based on the face image, obtaining a small sample model and an intermediate image corresponding to the small sample model; based on the intermediate image and the style template, obtaining a digital human image corresponding to the digital human. This implementation improves the efficiency of generating digital humans.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as natural language processing, computer vision, and deep learning, and in particular to a method and device for generating a digital human, a method and device for training a small sample model, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] With the upgrade of AI (Artificial Intelligence) photo-generating technology, users are no longer satisfied with portraits taken by cameras (mobile phones), and more and more users are beginning to try to create their own digital images online.

[0003] In traditional technology, facial images are generally mapped to fixed templates based on traditional mathematical models or simple machine learning models to obtain digital humans. This method makes it difficult to effectively extract high-level features from the original images, which limits the accuracy and robustness of the algorithm. Summary of the invention

[0004] Provided are a digital human generation method and device, a small sample model training method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, a method for generating a digital human is provided, the method comprising: obtaining an input image and a style template input by a user; cropping the face area of ​​the input image to obtain a face image; based on the face image, obtaining a small sample model and an intermediate image corresponding to the small sample model, the small sample model being used to characterize the correspondence between the face image and the intermediate image; based on the intermediate image and the style template, obtaining a digital human image corresponding to the digital human.

[0006] According to a second aspect, a small sample model training method is provided, the method comprising: obtaining a preset sample set, the sample set comprising at least one sample, the sample comprising: a sample image and a sample text for the sample image; obtaining a pre-established digital human network, the digital human network comprising: a base model and a small sample network connected in parallel with the base model, the base model being used to characterize the correspondence between text and digital human images, and the small sample network being used to characterize the correspondence between images and image features; performing the following training steps: selecting a sample from the sample set; inputting the sample into the digital human network to obtain a digital human image output by the digital human network; in response to the small sample network satisfying a training completion condition, obtaining a small sample model corresponding to the small sample network.

[0007] According to a third aspect, a digital human generation device is provided, which includes: an information acquisition unit, configured to acquire an input image and a style template input by a user; a cropping unit, configured to crop the face area of ​​the input image to obtain a face image; an image acquisition unit, configured to obtain a small sample model and an intermediate image corresponding to the small sample model based on the face image, wherein the small sample model is used to characterize the correspondence between the face image and the intermediate image; and a result acquisition unit, configured to obtain a digital human image corresponding to the digital human based on the intermediate image and the style template.

[0008] According to a fourth aspect, a small sample model training device is provided, which includes: a set acquisition unit, configured to acquire a preset sample set, the sample set includes at least one sample, the sample includes: a sample image and a sample text for the sample image; a network acquisition unit, configured to acquire a pre-established digital human network, the digital human network includes: a base model and a small sample network connected in parallel with the base model, the base model is used to characterize the correspondence between text and digital human image, and the small sample network is used to characterize the correspondence between image and image features; a selection unit, configured to select a sample from the sample set; an input unit, configured to input the sample into the digital human network to obtain a digital human image output by the digital human network; a model acquisition unit, configured to obtain a small sample model corresponding to the small sample network in response to the small sample network satisfying a training completion condition.

[0009] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0011] According to a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any implementation of the first aspect or the second aspect.

[0012] The digital human generation method and device provided by the embodiments of the present disclosure first obtain the input image and style template input by the user; secondly, perform face area cropping on the input image to obtain the face image; thirdly, based on the face image, obtain a small sample model and an intermediate image corresponding to the small sample model, the small sample model is used to characterize the correspondence between the face image and the intermediate image; finally, based on the intermediate image and the style template, obtain the digital human image corresponding to the digital human. Thus, the small sample model is obtained through the face image input by the user, and the features of the input image can be effectively extracted based on the small sample model; the intermediate image is fused with the style template, so that the generated digital human image can fully present the features of the input image, thereby improving the effect of digital sub-human image generation.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a flow chart of an embodiment of a method for generating a digital human according to the present disclosure;

[0016] Figure 2 is a schematic diagram of the structure of a stable diffusion model in the digital human generation method disclosed in the present invention;

[0017] Figure 3 is a flowchart of an embodiment of a small sample model training method according to the present disclosure;

[0018] Figure 4 is a structural schematic diagram of an embodiment of a digital human generation device according to the present disclosure;

[0019] Figure 5 is a structural schematic diagram of an embodiment of a small sample model training device according to the present disclosure;

[0020] Figure 6 It is a block diagram of an electronic device used to implement the digital human generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In this embodiment, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0023] While ensuring the authenticity and aesthetics of the user's face, the background of the photo can also be generated by AI with a variety of styles. In traditional technology, a face photo is generally mapped to a fixed template based on a convolutional neural network or a traditional mathematical model to achieve face fusion. Compared with the generation of digital humans based on the diffusion model, the traditional face fusion algorithm has the following main problems:

[0024] Weak feature extraction capability: Traditional fusion algorithms are usually based on manually designed or simple machine learning models, which make it difficult to effectively extract high-level features from raw images, which limits the accuracy and robustness of the algorithm;

[0025] Insufficient processing capabilities for big data: Traditional fusion algorithms often suffer from problems such as low computational efficiency and inaccurate feature extraction when faced with large-scale high-dimensional data, and are unable to effectively handle complex problems in a big data environment;

[0026] Lack of automation and intelligence: Traditional fusion algorithms lack automation and intelligence and often require manual intervention and parameter adjustment, which not only increases the complexity and cost of the algorithm, but also limits its application scope and development prospects;

[0027] Limited ability to fuse multi-source data: Traditional fusion algorithms usually fuse a single type of data source and are unable to effectively handle the fusion problem between multi-source data, which limits the scope of the algorithm in practical applications;

[0028] Poor real-time performance: Traditional fusion algorithms have high computational complexity, require a lot of time and computing resources, have poor real-time performance, and are difficult to meet the requirements for speed and efficiency in practical applications.

[0029] The present invention provides a method for generating a digital human, which processes an input image through a small sample model to improve the effect of the digital human image. Figure 1 A process 100 of an embodiment of a method for generating a digital human according to the present disclosure is shown. The method for generating a digital human includes the following steps:

[0030] Step 101: Obtain an input image and a style template input by a user.

[0031] In this embodiment, the input image is an image that the user desires to be transformed into a digital human image, for example, the input image is a selfie of the user, or the input image is an image of another person input by the user. It should be noted that the input image input by the user is voluntary and does not involve the user's privacy.

[0032] In this embodiment, the style template is an image format with the style of the digital human. The style of the digital human can be determined through the style template. For example, the style template is an image format in the form of a comic, or the style template is an image format in the form of an oil painting.

[0033] like Figure 2 As shown, through the user input operation, the execution subject on which the digital human generation method runs can obtain the input image and the style template at the same time. The style of the digital human that the user expects to transform and the image that the user expects to transform can be determined through the input image and style template input by the user.

[0034] Step 102, cropping the face area of ​​the input image to obtain a face image.

[0035] In this embodiment, the input image is an image having a face region, and the input image is cropped to remove the image that does not belong to the face region of the input image, thereby obtaining a face image.

[0036] In this embodiment, the face image is the core area image of the digital human image. By fusing the face image with style images of different styles in the style template, the final digital human image can be a composite image of the style form of the style template and the face image.

[0037] Step 103: Based on the face image, a small sample model and an intermediate image corresponding to the small sample model are obtained.

[0038] In this embodiment, the small sample model is used to characterize the correspondence between the image features of the face image and the intermediate image, wherein the image features are features obtained by extracting the face image through the small sample model, and the intermediate image is an image obtained by extracting the image features from the small sample model.

[0039] In this embodiment, the small sample model can be a model obtained by real-time training of face images. Since it is obtained by training of face images, the small sample model can extract features of the face images, combine the extracted features, and then obtain the following by a module for processing the features (for example, a diffusion model or a neural network model that has been pre-trained). Figure 2 The intermediate image Z is shown.

[0040] Optionally, the intermediate image can also be an image directly obtained through a small sample model, and the small sample model can also be a model trained by user images belonging to the same user information as the face image (having the same user identity information as the face image). The small sample model can be used to obtain an intermediate image of the user image having the same identity information as the face image.

[0041] In this embodiment, the above step 103 includes: based on the face image, detecting whether there is an initial network (such as a neural network model) that is pre-set to complete the features of the user's face image extraction; in response to detecting that there is a preset initial network, training the initial network based on the face image, and when the initial network meets the training completion condition (for example, the number of initial network training iterations reaches a preset number), using the initial network as a small sample model.

[0042] Optionally, when the intermediate image is obtained directly from the small sample model, the above step 103 may also include: based on the face image, obtaining an identity identifier of the face image; based on the identity identifier, selecting a user image from a preset image set; detecting whether there is a small sample model that matches the user image; in response to detecting a small sample model that matches the user image, inputting the face image into the small sample model to obtain the intermediate image.

[0043] Optionally, when the intermediate image is obtained by the image features output by the small sample model, the small sample model may also be a model obtained by training a fine-tuning framework of a stable diffusion model, and step 103 may further include: determining a stable diffusion model based on a face image; determining whether there is a fine-tuning framework that matches the stable diffusion model and has been trained; in response to having a fine-tuning framework that matches the stable diffusion model and has been trained, obtaining a fine-tuning framework that matches the face image and obtains the small sample model; inputting the face image into the stable diffusion model corresponding to the small sample model to obtain an intermediate image output by the stable diffusion model.

[0044] Step 104: obtaining a digital human image corresponding to the digital human based on the intermediate image and the style template.

[0045] In this embodiment, Figure 2 As shown, based on the style template and the intermediate image Z, the digital human image S corresponding to the digital human is obtained by fusion of the face template. Among them, the intermediate image Z is the image output by the small sample model.

[0046] In this embodiment, the above step 104 includes: removing the face area of ​​the style template to obtain a style image with a blank face area; synthesizing the style image and the intermediate image to obtain a digital human image corresponding to the digital human. The above synthesizing the style image and the intermediate image includes: inputting the style image and the intermediate image into a pre-trained neural network model to obtain a digital human image corresponding to the digital human output by the neural network model.

[0047] In this embodiment, the digital human image corresponding to the digital human can represent the image of the digital human. Since there are multiple input images, there are also multiple digital human images. The multiple digital human images can be processed into a video to produce an animated digital human. Similarly, three-dimensional model processing based on the multiple digital human images can obtain a three-dimensional digital human image.

[0048] Optionally, when the small sample model is a fine-tuning framework of a stable diffusion model, the above step 104 may also include: removing the face area of ​​the style model to obtain a style image with a blank face area; inputting the style image and the intermediate image into the stable diffusion model to obtain a digital human image of the corresponding digital human output by the stable diffusion model.

[0049] In this optional implementation, the stable diffusion model is a deep learning model for image generation, which is characterized by being able to gradually generate realistic high-resolution images. In this embodiment, the stable diffusion model is used to realize the diffusion conditional generation from text to image, and the stable diffusion model includes: a picture-text matching model and a diffusion generation model; wherein the picture-text matching model is composed of a text encoder and an image encoder based on a Transformer structure, which respectively encodes the input text (related to the small sample model, automatically triggered according to the rules, and not open to users) and the style image (image related to the style template), and calculates the picture-text similarity based on the encoding representation. The diffusion generation model adopts a UNet (U-shaped structure network) network structure, and predicts the next digital human image based on the intermediate feature map representation of the current step in the stable diffusion model, and gradually generates the image result of the final digital human image. During the entire diffusion process, feature fusion guides the diffusion process based on the similarity between the diffusion intermediate map and the input text, so that the diffusion model generates an image that is consistent with the text semantics. The training process of the stable diffusion model will train the diffusion generation model and the picture-text matching model separately through a large number of pre-collected text and image data pairs.

[0050] This embodiment provides a digital human generation method, which combines a picture-text matching model and a diffusion generation model to achieve diffusion conditional generation from text to image. Users only need to upload their own photos (input images) to quickly generate personal digital human images.

[0051] The digital human generation method provided by the embodiment of the present disclosure first obtains the input image and style template input by the user; secondly, the face area of ​​the input image is cropped to obtain the face image; thirdly, based on the face image, a small sample model and an intermediate image corresponding to the small sample model are obtained, and the small sample model is used to characterize the correspondence between the face image and the intermediate image; finally, based on the intermediate image and the style template, a digital human image corresponding to the digital human is obtained. Thus, the small sample model is obtained through the face image input by the user, and the features of the input image can be effectively extracted based on the small sample model; the intermediate image is fused with the style template, so that the generated digital human image can fully present the features of the input image, thereby improving the effect of digital sub-human image generation.

[0052] In some embodiments of the present disclosure, the above-mentioned obtaining of a small sample model and an intermediate image corresponding to the small sample model based on a face image includes: based on the face image, determining whether the model library has a small sample model corresponding to the face image; in response to the judgment result that the model library has a small sample model corresponding to the face image, obtaining an intermediate image based on the face image and the small sample model.

[0053] In this optional implementation, the above-mentioned determination of whether there is a small sample model corresponding to the face image in the model library includes: obtaining a storage unit in the model library, the storage unit including a storage image and a storage model; comparing the face image with the storage image in the storage unit for similarity; in response to the similarity between the face image and the storage image being greater than a similarity threshold, using the storage model in the storage unit as a small sample model, and determining that the model library has a small sample model corresponding to the face image. The similarity threshold can be set based on the model requirements, for example, the similarity threshold is a value greater than 80%.

[0054] Optionally, in response to the similarity between the facial image and the stored images in each storage unit in the model library being less than or equal to a similarity threshold, it is determined that the model library does not have a small sample model corresponding to the facial image.

[0055] Optionally, the above-mentioned determination of whether the model library has a small sample model corresponding to the face image includes: obtaining a storage unit in the model library, the storage unit including storage text and storage model; converting the face image into image text, wherein the image text is text describing the input image and the format of the input image; performing similarity comparison between the image text and the storage text in the storage unit; in response to the similarity between the image text and the storage text being greater than a similarity threshold, using the storage model in the storage unit as a small sample model, and determining that the model library has a small sample model corresponding to the face image.

[0056] In this optional implementation, the above-mentioned obtaining of the intermediate image based on the face image and the small sample model includes: inputting the face image into the small sample model to obtain the intermediate image corresponding to the small sample model.

[0057] In this optional implementation, the small sample model is a model for extracting features from facial images. After extracting features from facial images using the small sample model, an intermediate image that fully represents the features of the facial image can be obtained.

[0058] Optionally, the small sample model may also be a feature extraction model generated based on a face image. Specifically, the face image or an image related to the face image is used as a sample to train a feature extraction network, thereby obtaining a small sample model.

[0059] Optionally, the small sample model can also be a model trained in combination with the diffusion model. For example, the small sample model is a fine-tuning framework for the diffusion model. When the small sample model is trained, the diffusion model is frozen and the small sample model is trained. It should be noted that after the small sample model training is completed, the small sample model can fine-tune the image of the diffusion model based on the diffusion model, thereby fine-tuning the diffusion model into the intermediate image desired by the small sample model.

[0060] In this optional implementation, the intermediate image is an image obtained after the small sample model performs feature learning on the face image.

[0061] The method for outputting an intermediate image provided in this embodiment first determines, based on the face image, whether there is a small sample model corresponding to the face image in the model library; after the small sample model corresponding to the face image is in the model library, the small sample model is directly used to output the intermediate image, thereby improving the convenience of obtaining the small sample model.

[0062] In some optional implementations of the present embodiment, the above-mentioned obtaining a small sample model and an intermediate image corresponding to the small sample model based on the face image also includes: in response to a judgment result that the model library does not have a small sample model corresponding to the face image, training a fine-tuning framework of a stable diffusion model based on the face image to obtain a small sample model; obtaining an intermediate image based on the face image and the small sample model; obtaining a digital human image corresponding to the digital human based on the intermediate image and the style template includes: intercepting the face area of ​​the style template to obtain a style image; inputting the style image and the intermediate image into the stable diffusion model to obtain a digital human image output by the stable diffusion model.

[0063] The digital human generation method provided by the embodiment of the present disclosure, after obtaining a face image and the model library does not have a small sample model corresponding to the face image, trains the fine-tuning framework of the stable diffusion model based on the face image, so that the trained small sample model is fully adapted to the face image, and the features of the face image can be extracted to obtain an intermediate image corresponding to the face image. It should be noted that when training the fine-tuning framework, the parameters of the part of the stable diffusion model that is not related to the small sample model are frozen and do not participate in the training.

[0064] In this optional implementation, the fine-tuning framework is a network that fine-tunes the stable diffusion model, and the fine-tuning framework is a module connected in parallel with the fine-tuning framework. When the fine-tuning framework is trained, the parameters of the stable diffusion model other than the fine-tuning framework are frozen and do not participate in the training. The fine-tuning framework is only trained based on the features of the input image, extracts the features of the input image, and obtains the intermediate image.

[0065] In this optional implementation, the fine-tuning framework can be a LoRA (Low-Rank Adaptation of LargeLanguage Models) model. The fine-tuning framework can be understood as a plug-in in the stable diffusion model, a model that can be trained with only a small amount of data. When generating digital human images, the LoRA model will be used in combination with the stable diffusion model to achieve adjustment of the image results output by the stable diffusion model.

[0066] In the case of fine-tuning of the stable diffusion model, the LoRA model can be applied to the cross-attention layer between the image representations related to the cues describing them. The advantages of LoRA model fine-tuning are: faster training; lower computational requirements; smaller training weights, because the original model is frozen and the new trainable layer is injected, the weights of the new layer can be saved as a file of about tens of MB in size, which is nearly a thousand times smaller than the original size of the UNet model.

[0067] In this optional implementation, when a digital human image is obtained, the style image and the intermediate image are input into the stable diffusion model, and the stable diffusion model fuses the style image and the intermediate image to obtain a digital human image corresponding to the digital human. The stable diffusion model can fully combine the style of the style image with the features of the intermediate image to obtain a realistic digital human image.

[0068] The digital human generation method provided in this embodiment, when there is no small sample model corresponding to the face image in the model library, trains the fine-tuning framework of the stable diffusion model based on the face image to obtain the small sample model, so that the small sample model can fully absorb the features of the face image, and combined with the parameters of the stable diffusion model, it can be suitable for the intermediate image of the stable diffusion model. When the digital human image is obtained, the stable diffusion model is used to synthesize the digital human image corresponding to the digital human, thereby improving the generation effect of the digital human image and ensuring the reliability of the digital human generation.

[0069] Optionally, the above-mentioned method of obtaining a small sample model and an intermediate image corresponding to the small sample model based on the face image includes: obtaining a small sample network; taking the face image as a sample of the small sample network, selecting images from the sample face image in turn, iteratively training the small sample network, and obtaining the small sample model in response to the small sample network satisfying a training completion condition; selecting an image with the highest clarity from the input images, inputting the image into the small sample model, and obtaining an intermediate image.

[0070] In some embodiments of the present disclosure, the above-mentioned digital human generation method also includes: based on the face image, training the fine-tuning framework of the stable diffusion model, and obtaining the small sample model includes: based on the face image, iteratively training the fine-tuning framework of the stable diffusion model for a set number of times to obtain multiple model parameters; based on the face image, determining the small sample model.

[0071] In this embodiment, the set number of times can be determined based on the training requirements, for example, the set number of times is 5,000. It should be noted that in the set number of iterative training of the fine-tuning framework, the sample image (there are multiple input images, and the sample image is an image selected from the input images) input to the fine-tuning framework each time can be the same image or different images.

[0072] In this embodiment, each time the fine-tuning framework performs an iterative training, it has a model parameter; when the fine-tuning framework performs a set number of iterative trainings, it has a set number of model parameters. It should be noted that the model parameters are parameters of the fine-tuning framework. During the training of the fine-tuning framework, the parameters of the stable diffusion model other than the fine-tuning framework are frozen and are unchanged parameters.

[0073] In this embodiment, the above-mentioned determination of the small sample model based on the face image, model parameters and the stable diffusion model includes: determining the image text corresponding to the face image based on the face image; for the stable diffusion models under different model parameters, inputting the same image text into the stable diffusion model respectively to obtain multiple result images output by the stable diffusion model; judging whether the multiple result images meet the preset image quality requirements; and using the fine-tuning framework under the stable diffusion model corresponding to the result image that meets the image quality requirements as the small sample model.

[0074] The digital human generation method provided by this optional implementation method obtains multiple model parameters by iterating the fine-tuning framework a set number of times; based on the model parameters and the stable diffusion model under each model parameter, a small sample model is obtained, providing a reliable implementation method for obtaining the small sample model.

[0075] Optionally, after obtaining the small sample model, the above method also includes: inputting the face image into the small sample model to obtain an intermediate image corresponding to the small sample model; detecting whether the intermediate image meets the image quality requirements, and if so, determining that the small sample model meets the requirements.

[0076] In some embodiments of the present disclosure, determining the small sample model based on the face image, model parameters and the stable diffusion model includes: determining multiple initial images based on the model parameters and the stable diffusion model; and determining the small sample model based on the initial images and the face image.

[0077] In this embodiment, the above-mentioned determination of multiple initial images based on model parameters and the stable diffusion model includes: determining the initial image output by the stable diffusion model under the conditions of each model parameter; the above-mentioned determination of the small sample model based on the initial image and the face image includes: comparing each initial image with the face image, and taking the model parameters corresponding to the initial image that best matches the face image as the parameters of the small sample model.

[0078] The digital human generation method provided by this optional implementation method determines multiple initial images through model parameters and a stable diffusion model; based on the initial images and face images, the parameters of the small sample model are determined, providing a reliable implementation method for determining the small sample model.

[0079] Considering that images uploaded by different target users have differences in portrait proportions, lighting, background, etc., resulting in different difficulties in training small sample models, in order to solve this problem, different from the method of selecting the same training iteration model parameters for all target digital human images in the traditional technology, the present disclosure proposes an automatic model optimization in the process of small sample model training to select small sample model parameters, such as Figure 2 As shown, the small sample model with the best performance can be obtained from models of different training iterations for the same target user.

[0080] In some optional implementations of the present disclosure, the above-mentioned determination of the small sample model based on the initial image and the face image includes: calculating the face similarity between the initial image and the face image respectively to obtain the similarity value corresponding to the initial image; calculating the average similarity corresponding to the initial image based on the number of initial images and the similarity value corresponding to the initial image; and selecting the model parameters of the iterative training with the largest average similarity under all iterative training as the parameters of the small sample model.

[0081] In this embodiment, the input image can be multiple, and the generated facial images are multiple, and there is no direct correspondence between the number of input images and the number of generated facial images; for multiple facial images and multiple initial images, it is necessary to calculate the average similarity between the initial image and the facial image, preferably, the average similarity between the initial image and the frontal image is calculated, wherein the frontal image is an image of the front face, with clear facial features, no obstruction, and good lighting among the multiple facial images; when there are multiple facial images, the above-mentioned facial similarity calculation of the initial image and the facial images respectively includes: selecting a frontal image from the multiple facial images, and calculating the similarity between the initial image and the frontal image.

[0082] The specific implementation process of this optional implementation method is as follows: First, the small sample models of all user input images are trained for N (N>1) iterations. Second, the trained model parameters of the N iterations are used respectively, and the method of fixed random seeds and prompt words is adopted to perform the Wensheng map based on the steady-state diffusion model, and K (K>1) initial images O of the relevant target digital person are obtained under each weight. At the same time, the input images uploaded by each user for training the small sample model are used to obtain the front image T, and the face similarity S between these K initial images and the front image is calculated respectively:

[0083] S ij =f(O ij , T),

[0084] Among them, f(·) represents the algorithm for calculating face similarity, i∈[1,N],j∈[1,K]. After obtaining the similarity between each initial image and the frontal image, the average similarity of all iterations is calculated by weighted average:

[0085]

[0086] And select the iteration with the highest average similarity as the parameter of the optimal small sample model, so as to obtain the optimal small sample model.

[0087] The method for determining the small sample model provided by this optional implementation uses the model parameters of the iterative training with the largest average similarity as the parameters of the small sample model, providing a reliable and accurate implementation method for the small sample model.

[0088] In another embodiment of the present disclosure, the above-mentioned cropping of the face area of ​​the input image to obtain the face image includes: scattering a set number of anchor points in the input image; determining the position of the face area based on the image area where each anchor point is located; and cropping the input image based on the position of the face area to obtain the face image.

[0089] In this embodiment, the anchor point is an image detection point, through which it can be determined whether the pixels around the anchor point belong to the preset pixels of the face; the above-mentioned determination of the face area position based on the image area where each anchor point is located includes: judging whether the image area where each anchor point is located is part of the face image through a pre-trained neural network; all anchor points belonging to the face image are grouped together to obtain the face area position in the image.

[0090] In this embodiment, the step of cropping the input image based on the face region position to obtain the face image includes: cropping the face region position in the input image to obtain the face image.

[0091] Optionally, the above-mentioned cropping of the input image based on the face region position to obtain the face image may also include: predicting the key point position of the cropped face based on the face region position using a pre-trained key point detection model. The key point position is calculated with the key point of the preset standard position of the face to obtain a transformation matrix, and the input image is affine transformed by the transformation matrix to obtain an aligned face image.

[0092] The method for obtaining a facial image provided in this embodiment first sets an anchor point, determines the facial region based on the image region where the anchor point is located, and crops the input image based on the facial region to obtain the facial image, thereby improving the accuracy of obtaining the facial image.

[0093] Optionally, the above-mentioned cropping of the face area of ​​the input image to obtain the face image includes: obtaining pre-labeled face key points; matching the face key points with each pixel point in the input image; in response to the matching value between the face key point and the pixel point in the input image being greater than a matching threshold (the matching threshold can be set based on matching requirements, for example, the matching threshold is 80%), taking the area where the pixel points matching the face key point are located as the face area, and cropping the face area in the input image to obtain the face image.

[0094] In one embodiment of the present disclosure, the digital human generation method further includes: performing data augmentation on the input image to obtain an augmented image; and replacing the input image with the augmented image.

[0095] In this embodiment, data augmentation of the input image refers to increasing the diversity of the input image. The purpose of increasing the diversity of the input image is achieved by performing color transformation, brightness, contrast, and saturation changes on the input image.

[0096] Optionally, the data augmentation of the input image further includes: performing a geometric transformation on the input image, wherein the geometric transformation includes: random cropping, rotation, translation, scaling and other processing.

[0097] Optionally, the data augmentation of the input image may further include: performing image synthesis on the input image.

[0098] The digital human generation method provided in this embodiment increases the diversity of input images by performing data augmentation processing on the input images, thereby improving the diversity of facial images. When training a small sample model, the small sample model can fully extract the features of the sample image through the input of diverse facial images, thereby improving the accuracy of the small sample model.

[0099] In some embodiments of the present disclosure, before cropping the face region of the input image, the digital human generation method further includes: removing blurred images in the input image, and preprocessing the input image to obtain a processed input image.

[0100] In this embodiment, the main purpose of preprocessing the input image is to eliminate irrelevant information in the input image, restore useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition. The preprocessing process generally includes steps such as digitization, geometric transformation, normalization, smoothing, restoration and enhancement.

[0101] In this embodiment, in order to better train the small sample model, the input images are generally multiple. Before the input image is cropped for the face image, the blurred image in the input image is first removed, so that the obtained face image is an image with relatively high clarity; preprocessing the input image can make the input image have more useful information, thereby improving the information content of the input image.

[0102] In some optional implementations of this embodiment, the digital human generation method further includes: removing blurred images in the digital human image.

[0103] In this embodiment, since there can be multiple input images, each input image can correspond to a digital human image, removing blurred images from multiple digital human images can make the digital human effect obtained by the user better, thereby improving the user experience.

[0104] The present invention provides a small sample model training method, which processes the input image through the small sample model to improve the effect of the digital human image. Figure 3 A process 300 of an embodiment of a small sample model training method according to the present disclosure is shown. The small sample model training method comprises the following steps:

[0105] Step 301: Obtain a preset sample set.

[0106] In this embodiment, the sample set includes at least one sample, and the sample includes: a sample image and a sample text for the sample image.

[0107] In this embodiment, the sample image is an image including a person's face, and the sample text is a text describing the person in the sample image, the scene around the person, and the image format. The sample text can enable the digital human network to fully understand the content described in the sample image.

[0108] Step 302: Acquire a pre-established digital human network.

[0109] In this embodiment, the digital human network includes: a base model and a small sample network connected in parallel with the base model. The base model is used to characterize the correspondence between text and digital human images, and the small sample network is used to characterize the correspondence between images and image features.

[0110] In this embodiment, the base model can be a large model, which is used to characterize the correspondence between text and digital human image, that is, the digital human image output by the base model can be obtained by inputting text into the base model. Optionally, the base model is a trained stable diffusion model. The small sample network is a network branch that can be trained separately based on the base model processing the image. The small sample network is used to extract image features from the image and input the image features into the base model. The base model analyzes the image features based on its own understanding of the text to obtain the digital human image output by the base model.

[0111] Optionally, in order to make the digital human have corresponding style features, the style image corresponding to the style features and the text and image features of the small sample network can be input into the base model together to obtain a digital human image with corresponding style features output by the base model.

[0112] In this embodiment, the loss functions of the base model and the small sample network can be the same. When training the small sample network, the parameters of the base model can be directly frozen, thereby achieving the purpose of training the small sample model alone. Optionally, the base model is an image feature processing module that does not require training. The image features output by the small sample model are input to the base model, and the base model processes the sample text and image features to obtain a digital human image.

[0113] Among them, the image features are features obtained by extracting face images through a small sample network, and the intermediate image is an image obtained by extracting image features through a small sample network.

[0114] Step 303: Select samples from the sample set.

[0115] In this embodiment, in this embodiment, the execution subject may select samples from the sample set obtained in step 301, and perform the training steps from step 303 to step 305. The method and number of sample selection are not limited in this application. For example, at least one sample may be randomly selected, or a sample with better clarity (i.e., higher pixel) may be selected.

[0116] Step 304: input the sample into the digital human network to obtain a digital human image output by the digital human network.

[0117] In this embodiment, the above step 304 includes: inputting the sample text in the sample into the digital human network, controlling the digital human network to obtain a predicted digital human image based on the input sample text; comparing the predicted digital human image with the sample image in the sample, and detecting whether the small sample network in the digital human network meets the training completion condition through the number of training iterations from step 303 to step 305, or the loss value of the loss function set for the small sample network. If the training completion condition is met, it is determined that the small sample network meets the training completion condition.

[0118] Step 305 , in response to the small sample network satisfying the training completion condition, a small sample model corresponding to the small sample network is obtained.

[0119] In this embodiment, the training completion condition includes at least one of the following: the number of training iterations reaches a predetermined iteration threshold, and the loss value is less than a predetermined loss value threshold. For example, the training iterations reach 5,000 times. The loss value is less than 0.05, and after the training is completed, only the small sample network is retained as the small sample model. It should be noted that when using the small sample model, the small sample model and the base model can be used in combination to obtain a digital human image that meets the requirements.

[0120] In this embodiment, if the small sample network does not meet the training completion condition, the relevant parameters in the small sample network are adjusted to make the loss value converge, and based on the adjusted small sample network, steps 303-304 are continued to be executed until step 305 is met.

[0121] The small sample model training method provided by the embodiment of the present disclosure first obtains a preset sample set; secondly, obtains a pre-established digital human network; thirdly, selects a sample from the sample set; thirdly, inputs the sample into the digital human network to obtain a digital human image output by the digital human network; finally, in response to the small sample network satisfying the training completion condition, obtains a small sample model corresponding to the small sample network. Thus, in the process of training the small sample network, the image output by the base model is gradually aligned with the text in the sample, thereby improving the training efficiency of the small sample model.

[0122] Further references Figure 4As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a digital human generation device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0123] like Figure 4 As shown, the digital human generation device 400 provided in this embodiment includes: an information acquisition unit 401, a cropping unit 402, an image acquisition unit 403, and a result acquisition unit 404. Among them, the above-mentioned information acquisition unit 401 can be configured to obtain an input image and a style template input by a user. The above-mentioned cropping unit 402 can be configured to crop the face area of ​​the input image to obtain a face image. The above-mentioned image acquisition unit 403 can be configured to obtain a small sample model and an intermediate image corresponding to the small sample model based on the face image, and the small sample model is used to characterize the correspondence between the face image and the intermediate image. The above-mentioned result acquisition unit 404 can be configured to obtain a digital human image corresponding to the digital human based on the intermediate image and the style template.

[0124] In this embodiment, the specific processing and technical effects of the information acquisition unit 401, the cropping unit 402, the image acquisition unit 403, and the result acquisition unit 404 in the digital human generation device 400 can be referred to in Figure 1 The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0125] In some optional implementations of the present embodiment, the image obtaining unit 403 is configured to: determine, based on the face image, whether the model library contains a small sample model corresponding to the face image; and in response to the determination result that the model library contains a small sample model corresponding to the face image, obtain an intermediate image based on the face image and the small sample model.

[0126] In some optional implementations of the present embodiment, the image obtaining unit 403 is configured to: in response to a judgment result that the model library does not have a small sample model corresponding to the face image, based on the face image, train a fine-tuning framework of the stable diffusion model to obtain the small sample model; based on the face image and the small sample model, obtain an intermediate image; the result obtaining unit 404 is configured to: perform a face area capture on the style template to obtain a style image; input the style image and the intermediate image into the stable diffusion model to obtain a digital human image output by the stable diffusion model.

[0127] In some optional implementations of this embodiment, the above-mentioned image acquisition unit 403 is configured to: based on the face image, iteratively train the fine-tuning framework of the stable diffusion model for a set number of times to obtain multiple model parameters; based on the model parameters and the stable diffusion model, determine the small sample model.

[0128] In some optional implementations of the present disclosure, the image obtaining unit 403 is configured to: determine a plurality of initial images based on model parameters and a stable diffusion model; and determine a small sample model based on the initial images and the face image.

[0129] In some optional implementations of the present disclosure, the above-mentioned image acquisition unit 403 is configured to: calculate the face similarity between the initial image and the face image respectively, and obtain the similarity value corresponding to the initial image; calculate the average similarity corresponding to the initial image based on the number of initial images and the similarity value corresponding to the initial image; select the model parameters of the iterative training with the largest average similarity under all iterative training as the parameters of the small sample model.

[0130] In some optional implementations of the present disclosure, the above-mentioned cropping unit 402 is configured to: scatter a set number of anchor points in the input image; determine the position of the face area based on the image area where each anchor point is located; and crop the input image based on the position of the face area to obtain a face image.

[0131] In some optional implementations of the present disclosure, the digital human generation device 400 further includes an augmentation unit (not shown in the figure). The augmentation unit is configured to: perform data augmentation on the input image to obtain an augmented image; and replace the input image with the augmented image.

[0132] In some optional implementations of the present disclosure, the digital human generation device 400 further includes: a pre-removal unit (not shown in the figure), which removes the blurred image in the input image and pre-processes the input image to obtain a processed input image.

[0133] In some optional implementations of the present disclosure, the digital human generation unit 400 further includes a post-removal unit (not shown in the figure), and the post-removal unit is configured to remove the blurred image in the digital human image.

[0134] The digital human generation device provided by the embodiment of the present disclosure, first, the information acquisition unit 401 acquires the input image and style template input by the user; secondly, the cropping unit 402 crops the face area of ​​the input image to obtain the face image; thirdly, the image acquisition unit 403 obtains the small sample model and the intermediate image corresponding to the small sample model based on the face image, and the small sample model is used to characterize the correspondence between the face image and the intermediate image; finally, the result acquisition unit 404 obtains the digital human image corresponding to the digital human based on the intermediate image and the style template. Thus, the small sample model is obtained through the face image input by the user, and the features of the input image can be effectively extracted based on the small sample model; the intermediate image is fused with the style template, so that the generated digital human image can fully present the features of the input image, thereby improving the effect of digital sub-human image generation.

[0135] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a small sample model training device. Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0136] like Figure 5 As shown, the small sample model training device 500 provided in this embodiment includes: a set acquisition unit 501, a network acquisition unit 502, a selection unit 503, an input unit 504, and a model acquisition unit 505. The above-mentioned set acquisition unit 501 can be configured to acquire a preset sample set, the sample set includes at least one sample, and the sample includes: a sample image and a sample text for the sample image. The above-mentioned network acquisition unit 502 can be configured to acquire a pre-established digital human network, the digital human network includes: a base model and a small sample network connected in parallel with the base model, the base model is used to characterize the correspondence between text and digital human image, and the small sample network is used to characterize the correspondence between image and image features. The above-mentioned selection unit 503 can be configured to select a sample from the sample set. The above-mentioned input unit 504 can be configured to input the sample into the digital human network to obtain a digital human image output by the digital human network. The above-mentioned model acquisition unit 505 can be configured to obtain a small sample model corresponding to the small sample network in response to the small sample network meeting the training completion condition.

[0137] In this embodiment, in the small sample model training device 500, the specific processing of the set acquisition unit 501, the network acquisition unit 502, the selection unit 503, the input unit 504, and the model acquisition unit 505 and the technical effects thereof can be referred to respectively. Figure 3 The relevant descriptions of step 301, step 302, step 303, and step 304 in the corresponding embodiment are not repeated here.

[0138] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0139] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0140] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0141] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0142] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a digital human generation method or a small sample model training method. For example, in some embodiments, the digital human generation method or the small sample model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or a communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the digital human generation method or the small sample model training method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the digital human generation method or the small sample model training method in any other appropriate manner (for example, by means of firmware).

[0144] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable digital human generation device or a small sample model training device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0149] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0150] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a digital human, the method comprising: Get the input image and style template entered by the user; Performing face region cropping on the input image to obtain a face image; Based on the face image, determining whether there is a small sample model corresponding to the face image in the model library, the small sample model being used to characterize the correspondence between the face image and the intermediate image; in response to the determination result that there is no small sample model corresponding to the face image in the model library, training a fine-tuning framework of a stable diffusion model based on the face image to obtain a small sample model; and obtaining an intermediate image based on the face image and the small sample model; Based on the intermediate image and the style template, obtaining a digital human image corresponding to the digital human, including: intercepting a face region of the style template to obtain a style image; The style image and the intermediate image are input into the stable diffusion model to obtain a digital human image output by the stable diffusion model.

2. The method according to claim 1, further comprising: In response to the judgment result that the model library has a small sample model corresponding to the face image, an intermediate image is obtained based on the face image and the small sample model.

3. The method according to claim 1, wherein: The fine-tuning framework of training the stable diffusion model based on the face image to obtain the small sample model includes: Based on the face image, iteratively train the fine-tuning framework of the stable diffusion model for a set number of times to obtain multiple model parameters; A small sample model is determined based on the face image, the model parameters and the stable diffusion model.

4. The method according to claim 3, wherein: The determining of the small sample model based on the face image, the model parameters and the stable diffusion model comprises: Based on the model parameters and the stable diffusion model, determining a plurality of initial images; A small sample model is determined based on the initial image and the face image.

5. The method according to claim 4, wherein: The determining of the small sample model based on the initial image and the face image comprises: Calculate the facial similarity between the initial image and the facial image to obtain a similarity value corresponding to the initial image; Based on the number of the initial images and the similarity values ​​corresponding to the initial images, calculating the average similarity corresponding to the initial images; The model parameters of the iterative training with the largest average similarity under all iterative training are selected as the parameters of the small sample model.

6. The method according to any one of claims 1 to 5, wherein: The step of cropping the face region of the input image to obtain the face image comprises: Spreading a set number of anchor points in the input image; Determine the location of the face area based on the image area where each anchor point is located; Based on the position of the face region, the input image is cropped to obtain a face image.

7. The method according to claim 6, further comprising: Performing data augmentation on the input image to obtain an augmented image; The input image is replaced by the augmented image.

8. The method according to claim 6, before cropping the face region of the input image, the method further comprises: The blurred image in the input image is removed, and the input image is preprocessed to obtain a processed input image.

9. The method according to claim 6, further comprising: The blurred image in the digital human image is removed.

10. A small sample model training method, the method comprising: Acquire a preset sample set, wherein the sample set includes at least one sample, and the sample includes: a sample image and a sample text for the sample image; Obtain a pre-established digital human network, the digital human network comprising: a base model and a small sample network connected in parallel with the base model, the base model is used to characterize the correspondence between text and digital human image, the small sample network is used to characterize the correspondence between image and image features, the base model is a stable diffusion model, the small sample network is a network branch that can be trained separately based on the base model processing the image, the small sample network is used to extract image features from the image and input the image features into the base model, and the small sample network is used as a fine-tuning framework of the stable diffusion model to train and obtain the small sample model; The following training steps are performed: selecting a sample from the sample set; inputting the sample into the digital human network to obtain a digital human image output by the digital human network; in response to the small sample network satisfying a training completion condition, obtaining a small sample model corresponding to the small sample network.

11. A digital human generation device, comprising: An information acquisition unit, configured to acquire an input image and a style template input by a user; A cropping unit, configured to crop the face region of the input image to obtain a face image; The image obtaining unit is configured to determine, based on the face image, whether there is a small sample model corresponding to the face image in the model library; in response to the determination result that there is no small sample model corresponding to the face image in the model library, train a fine-tuning framework of a stable diffusion model based on the face image to obtain a small sample model; and obtain an intermediate image based on the face image and the small sample model; The result obtaining unit is configured to obtain a digital human image corresponding to the digital human based on the intermediate image and the style template, including: intercepting a face region of the style template to obtain a style image; The style image and the intermediate image are input into the stable diffusion model to obtain a digital human image output by the stable diffusion model.

12. The device according to claim 11, wherein: The image obtaining unit is further configured to: in response to a judgment result that the model library has a small sample model corresponding to the face image, obtain an intermediate image based on the face image and the small sample model.

13. The device according to claim 11, wherein: The image acquisition unit is configured to: based on the face image, perform a set number of iterative training on the fine-tuning framework of the stable diffusion model to obtain multiple model parameters; based on the model parameters and the stable diffusion model, determine a small sample model.

14. The device according to claim 13, wherein: The image acquisition unit is configured to: determine a plurality of initial images based on the model parameters and the stable diffusion model; and determine a small sample model based on the initial images and the face image.

15. The device according to claim 14, wherein: The image acquisition unit is configured to: calculate the facial similarity between the initial image and the facial image respectively to obtain the similarity value corresponding to the initial image; calculate the average similarity corresponding to the initial image based on the number of the initial images and the similarity values ​​corresponding to the initial images; select the model parameters of the iterative training with the largest average similarity under all iterative training as the parameters of the small sample model.

16. The device according to any one of claims 11 to 15, wherein: The cropping unit is configured to: spread a set number of anchor points in the input image; determine the position of the face area based on the image area where each anchor point is located; and crop the input image based on the position of the face area to obtain a face image.

17. The apparatus according to claim 16, further comprising: Augmentation Unit; The augmentation unit is configured to perform data augmentation on the input image to obtain an augmented image; The input image is replaced by the augmented image.

18. The apparatus according to claim 16, further comprising: Pre-removal unit; The pre-image removal unit is configured to remove the blurred image in the input image and pre-process the input image to obtain a processed input image.

19. The apparatus according to claim 16, further comprising: Post-removal unit; The post-removal unit is configured to remove a blurred image in the digital human image.

20. A small sample model training device, the device comprising: A set acquisition unit is configured to acquire a preset sample set, wherein the sample set includes at least one sample, and the sample includes: a sample image and a sample text for the sample image; A network acquisition unit is configured to acquire a pre-established digital human network, wherein the digital human network includes: a base model and a small sample network connected in parallel with the base model, wherein the base model is used to characterize the correspondence between text and digital human image, and the small sample network is used to characterize the correspondence between image and image features, wherein the base model is a stable diffusion model, and the small sample network is a network branch that can be trained separately based on the base model processing the image, wherein the small sample network is used to extract image features from the image and input the image features into the base model, and the small sample network is used as a fine-tuning framework of the stable diffusion model and is used to train the small sample model; A selection unit, configured to select a sample from the sample set; An input unit is configured to input the sample into the digital human network to obtain a digital human image output by the digital human network, so as to train the fine-tuning framework of the stable diffusion model; The model obtaining unit is configured to obtain a small sample model corresponding to the small sample network in response to the small sample network satisfying a training completion condition.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images