A method and apparatus for generating hairstyle replacement videos based on a diffusion model

Abstract

This application belongs to the field of image processing technology and discloses a method and apparatus for generating hairstyle replacement videos based on a diffusion model. The method includes: acquiring a hairstyle image dataset and training a diffusion model to obtain a hairstyle generation model; acquiring a user-inputted image of a person and a target hairstyle prompt; using skeletal keypoint detection technology to identify the person image and obtain skeletal pose data; using an image segmentation algorithm to divide the hair region in the person image to obtain a hair mask; performing smoothing and feathering processing on the hair mask using Gaussian blur and feathering algorithms; inputting the hair mask and the target hairstyle prompt into the hairstyle generation model, and using the skeletal pose data as constraints for the hairstyle generation model to obtain a hairstyle replacement image; and generating a hairstyle replacement video based on the person image and the hairstyle replacement image. This application can improve the realism and naturalness of hairstyle replacement images and videos.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and relates to a hairstyle replacement video generation method and device based on a diffusion model. BACKGROUND

[0002] With the continuous development of artificial intelligence technology in the field of image / video generation, hairstyle generation and editing technology has gradually become a research hotspot. Although traditional hairstyle processing technologies such as generative adversarial networks (GAN), image editing technologies based on deep learning, and 3D modeling and rendering technologies have achieved certain results, they still face many challenges:

[0003] In the process of generating hairstyles by GAN, the details of the generated hairstyle may not be realistic, for example, the texture of the generated hair may not be fine enough, and there may be problems such as inaccurate structure in the generation of complex hairstyles (such as braids and buns). This is because the diversity of hairstyles in the training data may be limited, which makes the generator unable to learn all possible hairstyle details. Based on 2D image editing or 3D modeling and rendering back to 2D, when the generated or edited hairstyle is synthesized with the original face image, there may be problems such as unnatural boundaries and color incoordination. In addition, these existing technologies often operate based on a general hairstyle library when generating or editing hairstyles, and do not fully consider the user's body posture, which results in that the generated hairstyle image does not adapt to the current user's body posture and cannot accurately show the user the effect of the real hairstyle through the image.

[0004] In summary, the existing hairstyle processing technologies have problems such as insufficient details when generating complex hairstyles, the generated hairstyle does not adapt to the current user's posture, and the boundary between the generated hairstyle and the original image is unnatural and the colors are not coordinated. SUMMARY

[0005] The present application provides a hairstyle replacement video generation method and device based on a diffusion model, which can solve the problems of hard edges of hair and unnatural fusion with the original image, and improve the realism and naturalness of the hairstyle replacement image and video.

[0006] In a first aspect, the embodiments of the present application provide a hairstyle replacement video generation method based on a diffusion model, comprising:

[0007] Obtaining a hairstyle image dataset and training a diffusion model to obtain a hairstyle generation model;

[0008] Obtaining a user inputted character image and a target hairstyle prompt word;

[0009] Identifying the character image by using a skeleton key point detection technology to obtain skeleton posture data;

[0010] An image segmentation algorithm is used to divide the hair region in a person image to obtain a hair mask;

[0011] The hair mask is smoothed and feathered using Gaussian blur and feathering algorithms.

[0012] Input the hair mask and target hairstyle cue words into the hairstyle generation model, and use the skeletal pose data as a constraint for the hairstyle generation model to obtain the hairstyle replacement image;

[0013] Generate hairstyle replacement videos based on person images and hairstyle replacement images.

[0014] Furthermore, the hairstyle image dataset includes multiple images of different types of hairstyles and corresponding hairstyle prompts for each image; the hairstyle prompts include descriptions of hairstyle type, hairstyle style, and hairstyle length.

[0015] Furthermore, the above-mentioned acquisition of hairstyle image datasets and training of diffusion models result in a hairstyle generation model, including:

[0016] The training step involves randomly selecting hairstyle images from the hairstyle image dataset as input images.

[0017] Randomly select hairstyle prompts from the hairstyle image dataset as target hairstyle prompts;

[0018] Input the input image and the target hairstyle prompt into the diffusion model to obtain the output replacement image;

[0019] The parameters of the diffusion model are updated based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt.

[0020] Return to the training step until the diffusion model reaches the preset convergence condition to obtain the hairstyle generation model.

[0021] Furthermore, the method also includes:

[0022] Before training, the LoCon network is used to reduce the dimensionality of the convolutional layers in the diffusion model.

[0023] Furthermore, the method also includes:

[0024] In the process of updating the parameters of the diffusion model, a regularization method is used to constrain the parameters of the diffusion model.

[0025] Furthermore, the method also includes:

[0026] Before feeding the input image into the diffusion model, evaluate the current convergence status of the diffusion model;

[0027] Determine the Dropout rate based on the hairstyle image dataset and the current convergence status;

[0028] The output of neurons in the diffusion model is randomly set to 0 based on the Dropout rate.

[0029] Furthermore, the above-mentioned updating of the diffusion model parameters based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt includes: adjusting the first matrix and the second matrix based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt; multiplying the first matrix and the second matrix and adding them to the parameters of the diffusion model to update the parameters of the diffusion model.

[0030] Furthermore, the above-mentioned generation of hairstyle replacement videos based on person images and hairstyle replacement images includes:

[0031] Use the image of the person as the first image and the image of the hairstyle replacement as the keyframe image;

[0032] The displacement calculation step involves calculating the motion vector between the first image and the keyframe image based on the optical flow method.

[0033] The intermediate frame image is obtained based on the motion vector, the first image, and the keyframe image;

[0034] The intermediate frame image is used as the first image, and the displacement calculation step is returned until the preset number of frames is reached. The character image, each intermediate frame image and the key frame image are combined to obtain the hairstyle replacement video.

[0035] Furthermore, the above-mentioned calculation of the motion vector between the first image and the keyframe image based on the optical flow method includes:

[0036] Calculate the positive dense optical flow vector from the first image to the keyframe image;

[0037] Calculate the inverse dense optical flow vector from the keyframe image to the first image;

[0038] The forward dense optical flow vector and the reverse dense optical flow vector are used as motion vectors.

[0039] Furthermore, the process of obtaining the intermediate frame image based on the motion vector, the first image, and the keyframe image includes:

[0040] Determine the first moment of the first image and the second moment of the keyframe image;

[0041] Use the midpoint between the first and second time points as the interpolation time.

[0042] The forward prediction intermediate frame is obtained based on the forward dense optical flow vector, the interpolation time, and the first image;

[0043] The inverse prediction intermediate frame is obtained based on the inverse dense optical flow vector, the interpolation time, and the keyframe image;

[0044] The intermediate frame image is obtained by weighting the forward-predicted intermediate frame and the backward-predicted intermediate frame.

[0045] Furthermore, the process of obtaining the intermediate frame image based on the motion vector, the first image, and the keyframe image also includes:

[0046] The first weight of each pixel in the intermediate frame is obtained based on the first image;

[0047] The second weights of each pixel in the intermediate frame are obtained from the keyframe image for reverse prediction.

[0048] The intermediate frame image is obtained by weighting each pixel in the forward prediction intermediate frame with the second weight of each pixel in the backward prediction intermediate frame.

[0049] Furthermore, the first weights of each pixel in the forward prediction intermediate frame obtained from the first image include:

[0050] The intermediate frames of the forward prediction are divided into blocks to obtain multiple forward prediction blocks;

[0051] Determine the first block in the first image corresponding to each positive prediction block;

[0052] Calculate the spatial similarity and color similarity between each pixel in the first block and the center pixel;

[0053] The first weight of a pixel is determined based on its spatial similarity and color similarity.

[0054] The first weight is assigned to the pixel corresponding to the pixel in the positive prediction block.

[0055] Secondly, embodiments of this application provide a hairstyle replacement video generation device based on a diffusion model, comprising:

[0056] The model training module is used to acquire a dataset of hairstyle images and train a diffusion model to obtain a hairstyle generation model.

[0057] The acquisition module is used to acquire the user-inputted image of a person and the target hairstyle suggestion words;

[0058] The pose recognition module is used to identify human images using skeletal key point detection technology to obtain skeletal pose data.

[0059] The segmentation module is used to divide the hair region in a person image using an image segmentation algorithm to obtain a hair mask;

[0060] The smoothing module is used to smooth and feather the hair mask using Gaussian blur and feathering algorithms.

[0061] The hairstyle replacement module is used to input the hair mask and target hairstyle prompts into the hairstyle generation model, and to use the skeletal pose data as constraints for the hairstyle generation model to obtain the hairstyle replacement image;

[0062] The video generation module is used to generate hairstyle replacement videos based on person images and hairstyle replacement images.

[0063] Furthermore, the device also includes:

[0064] The dimensionality reduction module is used to perform dimensionality reduction operations on the convolutional layers in the diffusion model using the LoCon network before training.

[0065] Furthermore, the model training module specifically includes:

[0066] The first extraction unit is used to randomly extract hairstyle images from the hairstyle image dataset as input images;

[0067] The second extraction unit is used to randomly extract hairstyle prompt words from the hairstyle image dataset as target hairstyle prompt words;

[0068] The input unit is used to input the input image and the target hairstyle prompt into the diffusion model to obtain the output replacement image;

[0069] The update unit is used to update the parameters of the diffusion model based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt; it returns to the first extraction unit until the diffusion model reaches the preset convergence condition to obtain the hairstyle generation model.

[0070] Furthermore, the device also includes a constraint unit, used to constrain the parameters of the diffusion model using a regularization method during the process of updating the parameters of the diffusion model.

[0071] Furthermore, the device also includes:

[0072] The random exit module is used to evaluate the current convergence of the diffusion model; the Dropout rate is determined based on the hairstyle image dataset and the current convergence; and the neuron outputs of the diffusion model are randomly set to 0 based on the Dropout rate.

[0073] Furthermore, the video generation module specifically includes:

[0074] A keyframe unit is used to use the image of the person as the first image and the image of the hairstyle replacement as the keyframe image.

[0075] The displacement calculation unit is used to calculate the motion vector between the first image and the keyframe image based on the optical flow method.

[0076] The frame interpolation unit is used to obtain intermediate frame images based on motion vectors, the first image, and the keyframe image;

[0077] The frame rate unit is used to take the intermediate frame image as the first image and return it to the displacement calculation unit until the number of loops reaches the preset number of frames. The character image, each intermediate frame image and the key frame image are combined to obtain the hairstyle replacement video.

[0078] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the steps of the hairstyle replacement video generation method based on the diffusion model as described in any of the above embodiments.

[0079] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the hairstyle replacement video generation method based on the diffusion model as described in any of the above embodiments.

[0080] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0081] This application provides a method for generating hairstyle replacement videos based on a diffusion model. First, the method trains a hairstyle generation model using a diffusion model. Leveraging the diffusion model's stronger image semantic understanding and texture generation capabilities compared to GANs, the model can naturally and smoothly integrate the target hairstyle features into the hair region of a person's image based on the input hair mask and target hairstyle prompts. This ensures that the output hairstyle image presents a high degree of realism and coherence in visual effect, avoiding situations where the hairstyle lacks detail, has a chaotic structure, or does not match the current user's facial features. Second, the method performs smoothing and feathering processing on the hair mask before inputting it into the diffusion model, effectively solving the problems of harsh edges on the hair mask and unnatural integration with the original image, laying a solid foundation for accurate hairstyle replacement and natural integration. Finally, when generating hairstyle replacement images based on the diffusion model, the method uses the identified skeletal pose data as constraints to ensure the consistency between the person's pose and the person's image in the hairstyle replacement image, greatly improving the realism and naturalness of the hairstyle replacement image and video. Attached Figure Description

[0082] Figure 1 This is a flowchart illustrating a method for generating hairstyle replacement videos based on a diffusion model, as provided in one embodiment of this application.

[0083] Figure 2 This is a flowchart of the diffusion model training steps provided in one embodiment of this application.

[0084] Figure 3 This is a schematic diagram illustrating the implementation of the LoRA mechanism provided in one embodiment of this application.

[0085] Figure 4 This is a flowchart illustrating the steps for generating a hairstyle replacement video according to one embodiment of this application.

[0086] Figure 5 This is a structural diagram of a hairstyle replacement video generation device based on a diffusion model, provided in one embodiment of this application. Detailed Implementation

[0087] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0088] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0089] With the continuous development of artificial intelligence technology in the field of image / video generation, hairstyle generation and editing technology has gradually become a research hotspot. Traditional hairstyle processing technologies, such as those based on generative adversarial networks (GANs), deep learning-based image editing technologies, and 3D modeling and rendering technologies, have achieved certain results, but still face many challenges.

[0090] Generative Adversarial Networks (GANs) consist of a generator and a discriminator. The generator is responsible for generating new hairstyles based on the input face image. It attempts to synthesize realistic hairstyles by learning the distribution characteristics of a large amount of hairstyle data. The discriminator judges the input image (including hairstyle images generated by the generator and real hairstyle images), distinguishing between real and generated ones. Through adversarial training between the generator and the discriminator, the generator continuously optimizes the generated results, making them closer to real hairstyles.

[0091] Deep learning-based image editing techniques utilize convolutional neural networks (CNNs) for semantic segmentation of facial images, accurately separating the hair region from the face. After training on a large amount of labeled data, the network can recognize the boundaries and shapes of hair. For example, network architectures based on U-Net can achieve high-precision hair region segmentation. Then, based on deep neural networks, features are extracted from the content image (original hairstyle) and the style image (target hairstyle style), transferring the style features of the target hairstyle to the original hairstyle. For example, style features such as the curl and shine of curly hair can be transferred to straight hair, achieving hairstyle style transformation. A typical method is the style transfer algorithm based on the VGG network.

[0092] 3D modeling and rendering technology acquires 3D facial data using 3D scanning equipment or performs 3D reconstruction based on 2D images to construct a 3D model of the face. Then, hairstyles are designed and replaced on the 3D model, taking into account the three-dimensional effect and lighting variations. Finally, rendering technology is used to render the 3D hairstyle into a 2D image, making it look more realistic.

[0093] However, most existing technologies suffer from issues with the realism of generated hairstyles. While adversarial training can continuously improve the quality of hairstyles generated by GANs, the details still often lack realism. For example, the texture of the generated hair strands may not be fine enough, and structural inaccuracies can easily occur in the generation of complex hairstyles (such as braids and updos). This is because the diversity of hairstyles in the training data may be limited, preventing the generator from learning all possible hairstyle details. Image-editing-based methods, when using semantic segmentation and style transfer for hairstyle editing, are insufficient in handling the spatial three-dimensionality of hairstyles due to their 2D image manipulation. For instance, when transferring the style of one hairstyle to another, some spatial morphological features of the original hairstyle may be lost, resulting in a visually unnatural generated hairstyle.

[0094] In addition, existing technologies often rely on generic hairstyle libraries or styles when generating or editing hairstyles, lacking sufficient consideration for the user's individual facial features and style. For example, a generated hairstyle may look great on a fashion magazine model, but it may not suit a specific user's face shape, temperament, etc., resulting in a lack of personalization in the generated result. Furthermore, when compositing the generated or edited hairstyle with the original facial image, problems such as unnatural boundaries and color inconsistencies are prone to occur.

[0095] To address the above issues and generate highly realistic hairstyle replacement videos that match the user's style, please refer to [link / reference]. Figure 1 This application provides a method for generating hairstyle replacement videos based on a diffusion model, including:

[0096] Step S1: Obtain the hairstyle image dataset and train the diffusion model to obtain the hairstyle generation model.

[0097] The hairstyle image dataset includes multiple images of different types of hairstyles and corresponding hairstyle prompts for each image; all hairstyle images are in PNG format, with a uniform resolution of 768*1024, and the number of images is preferably 100.

[0098] Hairstyle tips include descriptions of hairstyle type, hairstyle style, and hairstyle length.

[0099] Among them, hairstyle type can be straight hair, curly hair, wavy hair, short hair, long hair, etc.; hairstyle style can include hair color and style descriptions such as retro style, modern style, Japanese style, European and American style, etc.

[0100] Furthermore, the Brushnet model is preferred for the Stable Diffusion diffusion model.

[0101] With its powerful image semantic understanding and texture generation capabilities, the Brushnet model can naturally and smoothly integrate the target hairstyle features corresponding to the target hairstyle prompts into the hair area of ​​the image based on the input image information and the target hairstyle prompts, ensuring that the redrawn hairstyle presents a high degree of realism and coherence in visual effect.

[0102] Furthermore, before training, the LoCon network is used to perform dimensionality reduction on the convolutional layers in the diffusion model.

[0103] Specifically, compared to the traditional LoRA, the significant improvement of the optimized network structure LoCon lies in its effective control over convolutional layers. In Stable Diffusion, convolutional layers are the key component for extracting image features. LoCon reduces the number of parameters while maintaining model performance by performing dimensionality reduction operations on the convolutional kernels. This dimensionality reduction operation not only improves the model's training efficiency but also optimizes its ability to extract and represent image features, enabling the model to more accurately capture subtle features and complex structures in hairstyle images, thus providing stronger support for hairstyle training tasks.

[0104] Step S2: Obtain the user-inputted image of the person and the target hairstyle prompt.

[0105] The image of the person must include at least the area of ​​the current user's head and hair. The target hairstyle prompts are consistent with the hairstyle prompts in the hairstyle image dataset and are used to describe the features of the target hairstyle that the user wants to replace.

[0106] Step S3: Use skeletal key point detection technology to identify human images and obtain skeletal pose data.

[0107] Specifically, Openpose is used to accurately identify skeletal pose data in human images. Openpose is an advanced human pose estimation technology that can accurately identify key points and pose information of the human body in images, including the torso, limbs, and face. By acquiring this information, it can be ensured that the body pose in the generated hairstyle replacement image is consistent with the original image during the subsequent hairstyle generation process, thereby improving the realism and naturalness of the generated effect.

[0108] Step S4: Use an image segmentation algorithm to divide the hair region in the person image to obtain a hair mask.

[0109] Step S5: Smooth and feather the hair mask using Gaussian blur and feathering algorithms.

[0110] Specifically, firstly, an image segmentation algorithm is used to segment the hair region in the image based on the differences in color, texture, and other features between the hair and the background, thus obtaining a hair mask; then, a Gaussian blur algorithm is used to smooth the edges of the hair mask to reduce jaggedness; finally, a feathering algorithm is used to make the edges of the hair mask transition naturally.

[0111] In the specific implementation process, OpenCV guided filter is used to feather the edges of the hair mask based on color similarity, which effectively solves the problem of stiff hair mask edges and unnatural integration with the original human image in the existing technology, laying a solid foundation for accurate replacement and natural integration of hairstyles in the future.

[0112] Step S6: Input the hair mask and target hairstyle prompt into the hairstyle generation model, and use the skeletal pose data as a constraint for the hairstyle generation model to obtain the hairstyle replacement image.

[0113] Specifically, the hair mask defines the hair area of ​​the person image, enabling the hairstyle generation model to generate the target hairstyle corresponding to the target hairstyle prompt based on the hair area of ​​the person image. At the same time, the hairstyle generation model adjusts the fusion effect between the target hairstyle and other parts of the body through a fusion loss function, so that the two are naturally connected. With skeletal pose data as a constraint, it ensures that the generated hairstyle replacement image is consistent with the user's pose in the person image and the hairstyle is naturally integrated into the new image.

[0114] Step S7: Generate a hairstyle replacement video based on the person image and the hairstyle replacement image.

[0115] It can be considered that the hairstyle replacement video shows the effect of gradually changing from the original image of a person to the image with the replaced hairstyle. By watching the hairstyle replacement video, users can intuitively feel the appearance of the target hairstyle before and after the prompt words for switching to the target hairstyle.

[0116] The above embodiments provide a method for generating hairstyle replacement videos based on a diffusion model. First, the hairstyle generation model obtained by training a diffusion model leverages the more powerful image semantic understanding and texture generation capabilities of the diffusion model compared to GANs. Based on the input hair mask and target hairstyle prompts, it can naturally and smoothly integrate the features of the target hairstyle into the hair region of the person's image, ensuring that the output hairstyle image presents a high degree of realism and coherence in visual effect, avoiding insufficient hairstyle details, structural chaos, or inconsistencies with the current user's facial features. Second, before inputting the hair mask into the diffusion model, the method performs smoothing and feathering processing, effectively solving the problems of harsh edges on the hair mask and unnatural integration with the original image, laying a solid foundation for accurate hairstyle replacement and natural integration. Finally, when generating hairstyle replacement images based on the diffusion model, the method uses the identified skeletal pose data as constraints to ensure the consistency between the person's pose and the person's image in the hairstyle replacement image, greatly improving the realism and naturalness of the hairstyle replacement image and video.

[0117] Please see Figure 2 The above-mentioned acquisition of hairstyle image datasets and training of diffusion models to obtain hairstyle generation models include:

[0118] Step S11, training step: randomly select hairstyle images from the hairstyle image dataset as input images.

[0119] Step S12: Randomly select hairstyle prompt words from the hairstyle image dataset as target hairstyle prompt words.

[0120] Step S13: Input the input image and the target hairstyle prompt into the diffusion model to obtain the output replacement image.

[0121] It can be assumed that hairstyle images and hairstyle prompts are randomly selected from the hairstyle image dataset (but it should be noted that the selected hairstyle prompts cannot be the same as the hairstyle prompts corresponding to the selected hairstyle images). Then, the hairstyle images are used as input images, and the hairstyle prompts are used as target hairstyle prompts for the hairstyles to be replaced. The diffusion model is then used to redraw the hairstyles in the hairstyle images as the hairstyles corresponding to the target hairstyle prompts, resulting in the output replacement image.

[0122] Step S14: Update the parameters of the diffusion model based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt.

[0123] Specifically, the parameters of the diffusion model are updated based on the difference between the output replacement image and the hairstyle image corresponding to the target hairstyle prompt. Furthermore, the above-mentioned updating of the parameters of the diffusion model based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt may also include: adjusting the first matrix and the second matrix based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt; multiplying the first matrix and the second matrix and adding them to the parameters of the diffusion model to update the parameters of the diffusion model.

[0124] In existing technologies, methods based on GANs and 3D modeling still suffer from computational resource consumption issues. The training process for GANs requires significant computational resources and time, especially when processing high-resolution images and complex hairstyles. Repeated iterations of the generator and discriminator require powerful GPU support, which is prohibitively expensive for ordinary users and some small applications. 3D modeling and rendering place even higher demands on hardware. Building high-precision 3D face and hairstyle models and performing realistic rendering requires substantial memory and computing power, resulting in slow software performance and limiting the widespread adoption of these technologies in real-time applications.

[0125] To this end, this application introduces low-rank adaptation (LoRA) technology in the training and fine-tuning stage of the hairstyle generation model.

[0126] The core innovation of LoRA technology lies in adjusting existing weights by introducing a rank-factored weight matrix. Specifically, without changing the overall model architecture, training is performed only on the newly added low-rank matrix. This significantly reduces memory consumption during training and dramatically improves the training efficiency of large models. Mathematically, LoRA utilizes the low-rank property of the matrix to effectively reduce the scale of parameters that need to be optimized. It can be considered that LoRA replaces styles by making small modifications at checkpoints, specifically in the Cross-Attention layer of the U-Net diffusion model. The Cross-Attention layer plays a central role in the overall style generation mechanism of the model; by finely adjusting it, the style features output by the model can be precisely controlled, thereby achieving diverse changes in hairstyle styles.

[0127] Please see Figure 3x is the input image; d is the feature dimension of the hidden layer of the diffusion model, which is an inherent value of the diffusion model; h is the output of the hidden layer of the diffusion model. The intermediate features obtained after the input image x is transformed by the weight matrix (pre-trained weights plus the low-rank residual matrix BA) are passed to subsequent layers for further calculation and finally participate in the output replacement image of the generation model; r is an adjustable parameter, with common values ​​of 16, 32, 64, and 128. The larger r is, the stronger the capacity of the residual matrix BA, but the number of parameters also increases, requiring multiple training sessions to determine the most suitable value; N(0,σ²) represents a normal distribution with a mean of 0 and a variance of σ², which is the initial value of the first matrix A before training; the implementation steps of the LoRA mechanism are: freezing the original model parameters (i.e., Figure 3 (Pre-trained weights of the diffusion model); during training, fine-tuning two small matrix parameters A and B, namely the first and second matrices mentioned above, can be understood as learning a residual, but expressed in the form of matrix decomposition; finally, multiplying the first and second matrices (BA) and adding it to the original model parameters, we get the updated diffusion model.

[0128] Step S15: Return to the training step until the diffusion model reaches the preset convergence condition to obtain the hairstyle generation model.

[0129] The preset convergence condition can be either the number of training iterations or the target condition of the loss function.

[0130] The above embodiments avoid direct adjustment of the original model parameters by introducing the LoRA mechanism, thereby reducing memory consumption.

[0131] In some embodiments, the method further includes:

[0132] In the process of updating the parameters of the diffusion model, a regularization method is used to constrain the parameters of the diffusion model.

[0133] Specifically, to enhance the generalization performance of the fine-tuned hairstyle generation model in hairstyle scenarios, this application also employs regularization techniques. Regularization imposes constraints on model complexity by introducing an additional regularization term into the loss function. The principle is that when the model complexity is too high, it is prone to overfitting on the training data, resulting in poor performance on unseen data.

[0134] The above embodiments punish the excessive complexity of the hairstyle generation model by regularization, which can enable the model to learn more general features, thereby effectively improving its prediction accuracy and generalization ability on unknown data.

[0135] In some embodiments, the method further includes:

[0136] Step S130: Before inputting the input image into the diffusion model, evaluate the current convergence of the diffusion model.

[0137] Step S131: Determine the Dropout rate based on the hairstyle image dataset and the current convergence status.

[0138] Step S132: Randomly set the neuron output of the diffusion model to 0 according to the Dropout rate.

[0139] Specifically, Dropout is another important strategy for improving the generalization ability of a model. During training, Dropout randomly sets the output of the model's neurons to 0 with a certain probability (Dropout rate), thus introducing randomness in each iteration. This randomness can effectively prevent the model from over-relying on specific neurons, avoid the model getting trapped in local optima, and thus improve the model's adaptability and robustness to different data distributions. This application, by incorporating the Dropout strategy into the training of the hairstyle generation model, enables the model to better cope with changes in various hairstyle features and enhances its generalization effect on different hairstyle image data.

[0140] First, this application evaluates the quality of the hairstyle image dataset and the convergence of the diffusion model: a hairstyle image dataset with fewer than 100 images is considered a small-scale dataset; otherwise, it is considered a large-scale dataset. When labeling the hairstyle prompt words corresponding to each hairstyle image, blurry images and hairstyle images with significant background interference are considered low-quality images. If the proportion of low-quality images in the entire hairstyle image dataset is greater than 20%, the dataset is considered low-quality; otherwise, it is considered high-quality. If the training loss of the diffusion model continuously decreases while the validation loss continuously increases, the current convergence is considered overfitting. If the training loss of the diffusion model is higher than 0.8 and the matching degree between the feature words of the generated output replacement image and the corresponding target occurrence prompt words is less than 90%, the current convergence is considered underfitting. The above evaluation criteria are one implementation method provided in this application. In specific implementation, the values ​​of the evaluation criteria can be fine-tuned according to requirements.

[0141] The initial value of the dropout rate is generally set to 0.2 by default. If the hairstyle image dataset is a small-scale or low-quality dataset, the initial value of the dropout rate should be increased to 0.25~0.35 to suppress overfitting to a limited number of samples and enhance the capture of general hairstyle features. If the hairstyle image dataset is a large-scale and high-quality dataset, the initial value of the dropout rate can be reduced to 0.10~0.15 to avoid excessive regularization leading to insufficient feature learning.

[0142] During the training of the diffusion model, after every 500 training iterations, the above-mentioned convergence evaluation operation is performed based on the training results of these 500 iterations to obtain the current convergence status. If the current convergence status is overfitting, the Dropout rate is increased by 0.05 to break neuronal co-adaptation and increase hairstyle diversity; if the current convergence status is underfitting, the Dropout rate is decreased by 0.05 to retain more low-rank update information and enhance hairstyle details.

[0143] Unlike traditional Dropout, which sets the neuron dropout probability globally, this application adaptively adjusts the Dropout rate based on the proportion and complexity of the hair region in the input image, thereby achieving stronger generalization learning of key regions.

[0144] Please see Figure 4 In one embodiment, the above-mentioned generation of a hairstyle replacement video based on a person image and a hairstyle replacement image includes:

[0145] Step S71: Use the image of the person as the first image and the image of the hairstyle replacement as the keyframe image.

[0146] Step S72, displacement calculation step, calculates the motion vector between the first image and the keyframe image based on the optical flow method.

[0147] Specifically, the above-mentioned calculation of the motion vector between the first image and the keyframe image based on the optical flow method includes:

[0148] Step S721: Calculate the forward dense optical flow vector from the first image to the keyframe image.

[0149] Step S722: Calculate the inverse dense optical flow vector from the keyframe image to the first image.

[0150] Step S723: The forward dense optical flow vector and the reverse dense optical flow vector are used as motion vectors.

[0151] First, it should be noted that optical flow calculation is mainly divided into two categories: sparse optical flow and dense optical flow. Considering the scenario where hairstyles need to be changed rapidly, this application adopts the Pyramidal Lucas-Kanade method for dense optical flow. This method combines Lucas-Kanade and image pyramid techniques. Its calculation process calculates motion vectors layer by layer from low resolution to high resolution. In low-resolution images, since the pixel displacement is relatively small, a rough motion estimate can be obtained quickly. Then, as the resolution gradually increases, the motion vectors are refined and refined, thereby effectively handling large displacement motions and improving the accuracy and robustness of optical flow calculation. Second, in order to achieve frame interpolation, the optical flow fields of the keyframe image and the first image need to be calculated. Specifically, this is obtained by calculating bidirectional optical flow, which includes a forward dense optical flow vector and a reverse dense optical flow vector. The forward dense optical flow vector is the dense optical flow from the first image to the keyframe image, which describes the position of each pixel moving from the previous frame to the next frame. By calculating the forward dense optical flow vector, we can understand the motion trajectory and displacement information of pixels in an image as they move forward in time. The reverse dense optical flow vector represents the dense optical flow from the keyframe image to the first image, indicating the position each pixel moves from the previous frame to the next. The forward and reverse dense optical flow vectors complement each other, providing more comprehensive pixel motion information for the frame interpolation process, which is especially important when dealing with complex motion and occlusion situations.

[0152] Step S73: Obtain the intermediate frame image based on the motion vector, the first image, and the keyframe image.

[0153] Specifically, obtaining the intermediate frame image based on the motion vector, the first image, and the keyframe image includes:

[0154] Step S731: Determine the first moment of the first image and the second moment of the keyframe image.

[0155] Step S732: Use the intermediate value between the first time point and the second time point as the frame interpolation time point.

[0156] Wherein, the first time t0 is the time point corresponding to the first image in the hairstyle replacement video, the second time t1 is the time point corresponding to the keyframe image in the hairstyle replacement video, and t is the frame interpolation time, that is, the time point corresponding to the intermediate frame image in the hairstyle replacement video. By using the forward dense optical flow vector and the reverse dense optical flow vector, the spatial position of the pixel at the frame interpolation time can be determined, which is the key step for accurately calculating the intermediate frame image in the subsequent process.

[0157] Step S733: Obtain the forward prediction intermediate frame based on the forward dense optical flow vector, the interpolation time, and the first image.

[0158] Step S734: Obtain the inverse prediction intermediate frame based on the inverse dense optical flow vector, the interpolation time, and the keyframe image.

[0159] Specifically, the forward dense optical flow vector and the reverse dense optical flow vector are first scaled to time t using the frame interpolation time:

[0160]

[0161]

[0162] in, It is the positive dense optical flow vector. The first image and the keyframe image are then inversely warped using the scaled optical flow vector to obtain the corresponding predicted intermediate frames.

[0163]

[0164]

[0165] in, To predict the pixel value at position p in the intermediate frame in a positive direction, To predict the pixel value at position p in the intermediate frame, For the first image, For keyframe images, Let p be the distance that the pixel at position p in the first image moves from the first time step to the interpolation time step. Similarly, Let p be the distance that a pixel at position p in the keyframe image moves backward from the second time step to the interpolation time step. From this, two candidate intermediate frame images can be obtained, one of which is a positively predicted intermediate frame derived from the distortion of the first image. One is caused by the distortion of keyframe images. .

[0166] Step S735: Perform a weighted average of the forward-predicted intermediate frame and the backward-predicted intermediate frame to obtain the intermediate frame image.

[0167] Finally, the pixel values ​​of the two predicted intermediate frames are weighted and averaged to generate the final intermediate frame image.

[0168] Step S74: Use the intermediate frame image as the first image, and return to the displacement calculation step until the number of loops reaches the preset number of frames. Combine the character image, each intermediate frame image and the key frame image to obtain the hairstyle replacement video.

[0169] In practical applications, hairstyle replacement videos are specifically displayed as a gradual transition from the user's image to keyframe images after the hairstyle change, allowing the user to intuitively perceive the change from the original image to the image after the hairstyle change.

[0170] Furthermore, the process of obtaining the intermediate frame image based on the motion vector, the first image, and the keyframe image also includes:

[0171] Step S736: Obtain the first weight of each pixel in the forward prediction intermediate frame based on the first image.

[0172] Specifically, the above-mentioned method of obtaining the first weight of each pixel in the forward prediction intermediate frame based on the first image includes:

[0173] Step S7361: Divide the forward prediction intermediate frame into blocks to obtain multiple forward prediction blocks.

[0174] Step S7362: Determine the first block corresponding to each positive prediction block in the first image.

[0175] Step S7363: Calculate the spatial similarity and color similarity between each pixel in the first block and the center pixel.

[0176] First, for each pixel p_t in the forward prediction intermediate frame, a rectangular region centered on p_t is defined. For example, a 5×5 region centered on p_t is taken as the forward prediction block. For each pixel in the forward prediction block, its corresponding source pixel in the first image is deduced using the prediction method described above. For example, the ideal corresponding point of the center pixel p_t in the first image is p0, and the ideal corresponding point of each pixel q_t in the first image is q0. Thus, p0 and each q0 in the first image form the first block corresponding to the forward prediction block. Then, the spatial distance and color difference between p0 and each q0 in the first block are calculated.

[0177] Step S7364: Determine the first weight of the pixel based on the spatial similarity and color similarity of the pixel.

[0178] Step S7365: Assign the first weight to the pixel corresponding to the pixel in the positive prediction block.

[0179] Specifically, if the spatial similarity and color similarity between p0 and a certain q0 are both higher than the corresponding preset thresholds, it means that p_t corresponding to p0 and q_t corresponding to q0 are likely to belong to the same object surface. In this case, a high first weight will be assigned to this q_t; otherwise, a low first weight will be assigned to this q_t.

[0180] It can be assumed that the first weight of a pixel that satisfies spatial-color consistency is greater than the first weight of a pixel that does not satisfy spatial-color consistency. Spatial-color consistency means that the spatial similarity between the original point corresponding to the pixel in the first image and the center pixel of the first block where the original point is located is greater than a preset spatial threshold and the color similarity is greater than a preset color threshold.

[0181] Step S737: Obtain the second weight of each pixel in the reverse prediction intermediate frame based on the keyframe image.

[0182] The calculation process for the second weight is the same as that for the first weight.

[0183] Step S738: The intermediate frame image is obtained by weighting the first weight of each pixel in the forward prediction intermediate frame and the second weight of each pixel in the reverse prediction intermediate frame.

[0184] The pixel values ​​of the final intermediate frame image are obtained by weighting the pixel values ​​of the two predicted intermediate frames with their corresponding weights:

[0185]

[0186] in, It is the location in the positive prediction block. The first weight of the pixel, It is a reverse prediction of the location in the block. The second weight of the pixel, the summation range of the above summation symbol is defined as follows: In the centered, forward / reverse prediction blocks, besides All pixels, Position in the intermediate frame image The pixel value of the pixel.

[0187] The above embodiments take into account that when fusing forward and backward prediction intermediate frames, a simple weighted average may lead to ghosting and blurring, especially in motion boundaries and occluded areas. Therefore, this application further employs bilateral interpolation to perform weighted fusion on the two positions during weighted fusion to improve interpolation accuracy. For pixels that are "consistent" on the motion path, the corresponding fusion weight is increased, while for pixels that are inconsistent (possibly from different objects or occluded backgrounds), the corresponding weight is decreased. Bilateral interpolation not only considers the spatial positional relationship of pixels but also the similarity of pixel values. It can better preserve the edge and detail information of the image when processing pixel interpolation in complex scenes, reduce interpolation errors caused by position inconsistency, and thus achieve smooth and adaptive fusion.

[0188] Furthermore, in the aforementioned frame interpolation process based on optical flow, multi-scale optical flow estimation, regularization constraints, and edge interpolation smoothing can be further optimized. Multi-scale optical flow estimation uses a pyramid hierarchical method, estimating optical flow layer by layer from low to high resolution. As the resolution gradually increases, the optical flow estimation is refined, enabling more accurate processing of detailed motion information in the image, thereby improving the accuracy and stability of optical flow calculation. Regularization constraints incorporate edge preservation or global smoothness constraints into the optical flow calculation. For example, in textured regions of an image, optical flow is prone to excessive jitter. By introducing edge preservation constraints, the optical flow can maintain continuity at object edges, avoiding erroneous motion estimation caused by texture changes. Simultaneously, global smoothness constraints make the entire optical flow field smoother, reducing the occurrence of local abnormal optical flow and improving the reliability and visual effect of optical flow calculation. Edge interpolation smoothing is a post-processing technique for edge artifacts appearing in the frame interpolation, using methods such as Gaussian smoothing or bilateral filtering. During frame interpolation, pixel interpolation operations may produce some artifacts at image edges, which affect the visual quality of the image. Edge artifacts can be effectively removed by edge interpolation smoothing techniques such as Gaussian smoothing or bilateral filtering, making the edges of the interpolated image more natural and smooth, and improving the overall visual effect of the video.

[0189] Please see Figure 5 Another embodiment of this application provides a hairstyle replacement video generation device based on a diffusion model, comprising:

[0190] The model training module 101 is used to acquire a dataset of hairstyle images and train a diffusion model to obtain a hairstyle generation model.

[0191] The acquisition module 102 is used to acquire the user-inputted image of a person and the target hairstyle prompt.

[0192] The pose recognition module 103 is used to recognize human images using skeletal key point detection technology to obtain skeletal pose data.

[0193] The segmentation module 104 is used to segment the hair region in a person image using an image segmentation algorithm to obtain a hair mask.

[0194] The smoothing module 105 is used to smooth and feather the hair mask using Gaussian blur and feathering algorithms.

[0195] The hairstyle replacement module 106 is used to input the hair mask and target hairstyle prompts into the hairstyle generation model, and use the skeletal pose data as constraints for the hairstyle generation model to obtain the hairstyle replacement image.

[0196] The video generation module 107 is used to generate a hairstyle replacement video based on the person image and the hairstyle replacement image.

[0197] Furthermore, the device also includes:

[0198] Dimensionality reduction module 100 is used to perform dimensionality reduction operations on the convolutional layers in the diffusion model using the LoCon network before training.

[0199] Furthermore, the model training module specifically includes:

[0200] The first extraction unit is used to randomly extract hairstyle images from the hairstyle image dataset as input images.

[0201] The second extraction unit is used to randomly extract hairstyle prompts from the hairstyle image dataset as target hairstyle prompts.

[0202] The input unit is used to input the input image and the target hairstyle prompt into the diffusion model to obtain the output replacement image.

[0203] The update unit is used to update the parameters of the diffusion model based on the output replacement image and the hairstyle image corresponding to the target hairstyle prompt; it returns to the first extraction unit until the diffusion model reaches the preset convergence condition to obtain the hairstyle generation model.

[0204] Furthermore, the device also includes a constraint unit, used to constrain the parameters of the diffusion model using a regularization method during the process of updating the parameters of the diffusion model.

[0205] Furthermore, the device also includes:

[0206] The random exit module is used to obtain the current convergence status of the evaluation diffusion model; determine the Dropout rate based on the hairstyle image dataset and the current convergence status; and randomly set the neuron output of the diffusion model to 0 based on the Dropout rate.

[0207] Furthermore, the video generation module specifically includes:

[0208] A keyframe unit is used to use the image of the person as the first image and the image of the hairstyle replacement as the keyframe image.

[0209] The displacement calculation unit is used to calculate the motion vector between the first image and the keyframe image based on the optical flow method.

[0210] The frame interpolation unit is used to obtain intermediate frame images based on motion vectors, the first image, and the keyframe image;

[0211] The frame rate unit is used to take the intermediate frame image as the first image and return it to the displacement calculation unit until the number of loops reaches the preset number of frames. The character image, each intermediate frame image and the key frame image are combined to obtain the hairstyle replacement video.

[0212] The specific limitations of the hairstyle replacement video generation device based on a diffusion model provided in the above embodiments can be found in the embodiments of the hairstyle replacement video generation method based on a diffusion model described above, and will not be repeated here. Each module in the above-described hairstyle replacement video generation device based on a diffusion model can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0213] This application provides a computer device that may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it causes the processor to perform the steps of a diffusion-based hairstyle replacement video generation method as described in any of the above embodiments.

[0214] The working process, working details, and technical effects of the computer device provided in this embodiment can be found in the embodiment above regarding a method for generating hairstyle replacement videos based on a diffusion model, and will not be repeated here.

[0215] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of a diffusion model-based hairstyle replacement video generation method as described in any of the above embodiments. The computer-readable storage medium refers to a data storage carrier, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The working process, details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiments regarding a diffusion model-based hairstyle replacement video generation method, and will not be repeated here.

[0216] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0217] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for generating hairstyle replacement videos based on a diffusion model, characterized in that, The method comprises the following steps: obtaining a hairstyle image dataset and training a diffusion model to obtain a hairstyle generation model; wherein the hairstyle image dataset comprises multiple different types of hairstyle images and corresponding hairstyle prompt words for each hairstyle image; the hairstyle prompt words comprise hairstyle type description, hairstyle style description and hairstyle length description; specifically, performing a training step, randomly selecting a hairstyle image in the hairstyle image dataset as an input image; randomly selecting a hairstyle prompt word in the hairstyle image dataset as a target hairstyle prompt word; inputting the input image and the target hairstyle prompt word into the diffusion model to obtain an output replacement image; updating the parameters of the diffusion model according to the output replacement image and the corresponding hairstyle image of the target hairstyle prompt word; returning to the training step until the diffusion model reaches a preset convergence condition to obtain the hairstyle generation model; obtaining a user inputted character image and a target hairstyle prompt word; using a skeleton key point detection technology to identify the character image to obtain skeleton posture data; using an image segmentation algorithm to divide the hair region in the character image to obtain a hair mask; performing smoothing and feathering processing on the hair mask through a Gaussian blur algorithm and a feathering algorithm; inputting the hair mask and the target hairstyle prompt word into the hairstyle generation model, and inputting the skeleton posture data as a constraint of the hairstyle generation model to obtain a hairstyle replacement image; generating a hairstyle replacement video based on the character image and the hairstyle replacement image. 2.The method of claim 1, wherein, Further comprising: Before training, performing dimension reduction operation on the convolution layer in the diffusion model using the LoCon network. 3.The method of claim 1, wherein, Further comprising: In the process of updating the parameters of the diffusion model, using a regularization method to constrain the parameters of the diffusion model. 4.The method of claim 1, wherein, Further comprising: Before inputting the input image into the diffusion model, evaluating the current convergence condition of the diffusion model; determining the Dropout rate according to the hairstyle image dataset and the current convergence condition; randomly setting the neuron output of the diffusion model to 0 according to the Dropout rate. 5.The method of claim 1, wherein, The step of updating the parameters of the diffusion model according to the output replacement image and the corresponding hairstyle image of the target hairstyle prompt word comprises the following steps: adjusting the first matrix and the second matrix according to the output replacement image and the corresponding hairstyle image of the target hairstyle prompt word; multiplying the first matrix and the second matrix and adding the result to the parameters of the diffusion model to update the parameters of the diffusion model. 6.The method of claim 1, wherein, The step of generating a hairstyle replacement video based on the character image and the hairstyle replacement image comprises the following steps: taking the character image as a first image and the hairstyle replacement image as a key frame image; a displacement calculation step for calculating the motion vector between the first image and the key frame image based on the optical flow method; obtaining an intermediate frame image according to the motion vector, the first image and the key frame image; taking the intermediate frame image as the first image and returning to the displacement calculation step until the number of cycles reaches a preset frame number, combining the character image, each intermediate frame image and the key frame image to obtain the hairstyle replacement video. 7.The method of claim 6, wherein, The step of calculating the motion vector between the first image and the key frame image based on the optical flow method comprises the following steps: calculating forward dense optical flow vectors from the first image to the key frame image; calculating reverse dense optical flow vectors from the key frame image to the first image; using the forward dense optical flow vectors and the reverse dense optical flow vectors as the motion vectors. 8.The method of claim 7, wherein, deriving an intermediate frame image according to the motion vectors, the first image and the key frame image, comprising: determining a first time point of the first image and a second time point of the key frame image; using an intermediate value of the first time point and the second time point as an interpolation time point; deriving a forward predicted intermediate frame according to the forward dense optical flow vectors, the interpolation time point and the first image; deriving a reverse predicted intermediate frame according to the reverse dense optical flow vectors, the interpolation time point and the key frame image; performing weighted average on the forward predicted intermediate frame and the reverse predicted intermediate frame to obtain the intermediate frame image. 9.The method of claim 8, wherein, deriving an intermediate frame image according to the motion vectors, the first image and the key frame image, further comprising: deriving first weights of each pixel point in the forward predicted intermediate frame according to the first image; deriving second weights of each pixel point in the reverse predicted intermediate frame according to the key frame image; performing weighted calculation according to the first weights of each pixel point in the forward predicted intermediate frame and the second weights of each pixel point in the reverse predicted intermediate frame to obtain the intermediate frame image.

10. The method of claim 9, wherein, deriving first weights of each pixel point in the forward predicted intermediate frame according to the first image, comprising: performing block division on the forward predicted intermediate frame to obtain a plurality of forward predicted blocks; determining a first block corresponding to each of the forward predicted blocks in the first image; calculating spatial similarity and color similarity of each pixel point of the first block with a center pixel point; determining first weights of each pixel point of the first block according to the spatial similarity and the color similarity of the pixel point; assigning the first weights to pixel points corresponding to the pixel point in the forward predicted block.

11. A hair replacement video generation apparatus based on a diffusion model, characterized by, comprising: a model training module configured to obtain a hairstyle image dataset and train a diffusion model to obtain a hairstyle generation model; wherein the hairstyle image dataset comprises a plurality of hairstyle images of different types and a hairstyle prompt corresponding to each of the hairstyle images; the hairstyle prompt comprises a hairstyle type description, a hairstyle style description and a hairstyle length description; the model training module specifically comprises: a first extraction unit configured to randomly extract a hairstyle image in the hairstyle image dataset as an input image; a second extraction unit configured to randomly extract a hairstyle prompt in the hairstyle image dataset as a target hairstyle prompt; an input unit configured to input the input image and the target hairstyle prompt into the diffusion model to obtain an output replacement image; an updating unit configured to update parameters of the diffusion model according to the output replacement image and the hairstyle image corresponding to the target hairstyle prompt; and returning to the first extraction unit until the diffusion model reaches a preset convergence condition to obtain the hairstyle generation model; an acquisition module configured to obtain a user-inputted character image and a target hairstyle prompt; The posture recognition module is configured to recognize the human image by using a skeleton key point detection technology to obtain skeleton posture data. The segmentation module is configured to divide a hair region in the human image by using an image segmentation algorithm to obtain a hair mask. The smoothing module is configured to perform smoothing processing and feathering processing on the hair mask by using a Gaussian blur algorithm and a feathering algorithm. The hairstyle replacement module is configured to input the hair mask and the target hairstyle prompt word into the hairstyle generation model, and input the skeleton posture data as a constraint of the hairstyle generation model to obtain a hairstyle replacement image. The video generation module is configured to generate a hairstyle replacement video based on the human image and the hairstyle replacement image. 12.The diffusion model based hairstyle replacement video generation apparatus of claim 11, wherein, The dimension reduction module is configured to perform dimension reduction operation on a convolution layer in the diffusion model by using a LoCon network before training. 13.The diffusion model based hairstyle replacement video generation apparatus of claim 11, wherein, The constraint unit is configured to constrain parameters of the diffusion model by using a regularization method during parameter updating of the diffusion model. 14.The diffusion model based hairstyle replacement video generation apparatus of claim 11, wherein, The random exit module is configured to evaluate a current convergence condition of the diffusion model, determine a Dropout rate according to the hairstyle image dataset and the current convergence condition, and randomly set neuron output of the diffusion model to 0 according to the Dropout rate. 15.The diffusion model based hairstyle replacement video generation apparatus of claim 11, wherein, The video generation module specifically includes: The key frame unit is configured to take the human image as a first image and take the hairstyle replacement image as a key frame image. The displacement calculation unit is configured to calculate a motion vector between the first image and the key frame image based on an optical flow method. The frame interpolation unit is configured to obtain an intermediate frame image according to the motion vector, the first image and the key frame image. The frame rate unit is configured to take the intermediate frame image as the first image, return to the displacement calculation unit until a preset frame number is reached, and combine the human image, each intermediate frame image and key frame image to obtain a hairstyle replacement video.

16. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program comprises computer program code configured to cause the processor to perform the method of any one of claims 1 to 15. The processor executes the computer program to implement the steps of the hairstyle replacement video generation method based on the diffusion model in any one of claims 1 to 10.

17. A computer readable storage medium storing a computer program, wherein the computer program comprises program instructions configured to cause a processor to perform the method according to any one of claims 1 to 16. The computer program is executed by the processor to implement the steps of the hairstyle replacement video generation method based on the diffusion model in any one of claims 1 to 10.

Images