Text generation model training method and device

Abstract

The present disclosure provides a text-to-image model training method and device. The method comprises: obtaining an image dataset, and extracting image descriptions of each person image in the image dataset; constructing a first training dataset using each person image and its image description; segmenting a person face image of each person image from each person image, and constructing a second training dataset using the person face image and the image description of each person image; combining the first training dataset and the second training dataset according to a preset ratio to obtain a third training dataset; constructing a plurality of LoRA branches for a U-Net network to obtain a denoising fine-tuning network, and constructing a text-to-image model using a text encoder in a CLIP network, a VAE network, and the denoising fine-tuning network; and training the text-to-image model using the third training dataset so that the text-to-image model learns the person image and the image description. The above technical means solves the problem that the existing text-to-image model generates images that are not as expected.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to a method and device for training a text-based graph model. Background Art

[0002] The text-to-graph model is a very popular large-scale model that can generate corresponding images based on input descriptions. However, in practice, it has been found that current text-to-graph models cannot consistently understand prompt words, resulting in abnormal image generation results. Summary of the Invention

[0003] In view of this, the embodiments of the present disclosure provide a method, device, electronic device and computer-readable storage medium for training a text-generated image model to solve the problem that the images generated by the Chinese text-generated image model in the prior art are not as expected.

[0004] In a first aspect of an embodiment of the present disclosure, a method for training a text-based graph model is provided, including: obtaining an image dataset, extracting an image description of each person image in the image dataset; constructing a first training dataset using each person image and its image description; segmenting each person image to obtain a person's facial image, and constructing a second training dataset using the person's facial image and image description of each person image; combining the first training dataset and the second training dataset in a preset ratio to obtain a third training dataset; constructing multiple LoRA branches for a U-Net network to obtain a denoising fine-tuning network, and constructing a text-based graph model using a text encoder, a VAE network, and a denoising fine-tuning network in a CLIP network; and training the text-based graph model using the third training dataset so that the text-based graph model learns the person images and image descriptions.

[0005] According to a second aspect of an embodiment of the present disclosure, a device for training a text-based graph model is provided, including: an acquisition module configured to acquire an image dataset and extract an image description of each person image in the image dataset; a first construction module configured to construct a first training dataset using each person image and its image description; a second construction module configured to segment each person image to obtain a person facial image of each person image, and construct a second training dataset using the person facial image and image description of each person image; a third construction module configured to combine the first training dataset and the second training dataset in a preset ratio to obtain a third training dataset; a modeling module configured to construct multiple LoRA branches for a U-Net network to obtain a denoising fine-tuning network, and construct a text-based graph model using a text encoder, a VAE network, and a denoising fine-tuning network in a CLIP network; and a training module configured to train the text-based graph model using the third training dataset, so that the text-based graph model learns the person images and image descriptions.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the disclosed embodiment has the following advantages: obtaining an image dataset, extracting image descriptions of each person image in the image dataset; constructing a first training dataset using each person image and its image description; segmenting each person image to obtain a person face image of each person image, and constructing a second training dataset using the person face image and image description of each person image; combining the first training dataset and the second training dataset in a preset ratio to obtain a third training dataset; constructing multiple LoRA branches for the U-Net network to obtain a denoising fine-tuning network, and constructing a text-generated graph model using the text encoder, VAE network, and denoising fine-tuning network in the CLIP network; and training the text-generated graph model using the third training dataset so that the text-generated graph model learns the person images and image descriptions. The above technical means can solve the problem that the image generation of the Chinese text-generated graph model in the prior art does not meet expectations, thereby improving the quality of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 1 is a flow chart of a method for training a text graph model according to an embodiment of the present disclosure;

[0011] Figure 2 It is a flowchart of a method for generating a text graph according to an embodiment of the present disclosure;

[0012] Figure 3 1 is a schematic diagram of the structure of a cultural graph model training device provided by an embodiment of the present disclosure;

[0013] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that other embodiments of the present disclosure may be implemented without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the present disclosure with unnecessary detail.

[0015] A method and apparatus for training a text-based graph model according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0016] Figure 1 The present invention provides a flowchart of a method for training a text-based graph model. Figure 1 The text graph model training method can be executed by a computer or server, or software on the computer or server.

[0017] like Figure 1 As shown, the text-based graph model training method includes:

[0018] S101, obtaining an image dataset and extracting image descriptions of each person image in the image dataset;

[0019] S102, constructing a first training data set using each person image and its image description;

[0020] S103, segmenting each person image to obtain a person's facial image of each person image, and constructing a second training data set using the person's facial image and image description of each person image;

[0021] S104, combining the first training data set and the second training data set according to a preset ratio to obtain a third training data set;

[0022] S105, construct multiple LoRA branches for the U-Net network to obtain a denoising fine-tuning network, and use the text encoder, VAE network and denoising fine-tuning network in the CLIP network to build a text-generated graph model;

[0023] S106 , using the third training data set to train the text-graph model, so that the text-graph model learns character images and image descriptions.

[0024] Each person image in the image dataset is a full-body image of each person. Algorithms such as Multi-Human-Parsing and Semantatic Human Matting can be used to segment each person image to obtain the person's facial image. A facial image only contains the person's face. Image captioning tools, such as GPT (Generative Pre-trained Transformer), are used to extract image descriptions of each person image. Image captioning tools are used to extract descriptions of the person's facial orientation, facial features, and body features. Facial orientation refers to the angle of the face in the image, facial features refer to the size and distribution of facial features, and body features refer to height, weight, shoulder width, and leg length.

[0025] A first training dataset and a second training dataset are constructed based on the image dataset. The first training dataset contains person images and their image descriptions, and the second training dataset contains person facial images and their image descriptions. The first training dataset and the second training dataset are combined in a preset ratio to obtain a third training dataset. The third training dataset contains multiple training images, each of which carries an image description and is a person image or a person facial image.

[0026] The CLIP (Contrastive Language-Image Pre-training) network includes a text encoder and an image encoder, a VAE network, i.e., a variational autoencoder (Variational Autoencoder), and a U-Net network, which is a network with a U-shaped symmetrical structure and is commonly used for semantic segmentation. In an embodiment of the present application, a plurality of LoRA branches are constructed for the U-Net network to obtain a denoising fine-tuning network, and the text encoder and the VAE network in the CLIP network are connected in parallel with the denoising fine-tuning network to obtain a text-generated graph model. The text-generated graph model is trained using a third training data set so that the text-generated graph model learns character images and image descriptions. Then, in the inference stage, the text-generated graph model can generate corresponding character images based on the text about the image description input by the user.

[0027] According to the technical solution provided by the embodiment of the present application, an image data set is obtained, and the image description of each character image in the image data set is extracted; a first training data set is constructed using each character image and its image description; facial images of each character image are segmented from each character image, and a second training data set is constructed using the facial images and image descriptions of each character image; the first training data set and the second training data set are combined in a preset ratio to obtain a third training data set; multiple LoRA branches are constructed for the U-Net network to obtain a denoising fine-tuning network, and a text-generated graph model is constructed using the text encoder, VAE network, and denoising fine-tuning network in the CLIP network; the text-generated graph model is trained using the third training data set so that the text-generated graph model learns the character images and image descriptions. The above technical means can solve the problem that the existing Chinese text-generated graph model generates images that do not meet expectations, thereby improving the quality of the generated images.

[0028] Furthermore, the third training data set is used to train the Vincent graph model so that the Vincent graph model learns character images and image descriptions, including: the third training data set contains multiple training images, each training image carries an image description, and each training image is a character image or a character face image; each training image and the image description it carries are input into the Vincent graph model, and within the Vincent graph model: the image description carried by each training image is encoded by the text encoder in the CLIP network to obtain the text features corresponding to each training image; each training image is encoded by the VAE network to obtain the image features corresponding to each training image, wherein the VAE network The network encodes each training image by adding noise; the denoising fine-tuning network decodes the image features corresponding to each training image according to the text features corresponding to each training image to obtain the restored image corresponding to each training image, wherein the denoising fine-tuning network decodes the image features corresponding to each training image by predicting the noise added by the VAE network and removing the noise based on the predicted noise; the divergence function is used to calculate the prediction loss value between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image; the model parameters of the text-based graph model are optimized according to the prediction loss value to complete the training of the text-based graph model.

[0029] Within the text-based graph model: the text encoder encodes each image description to obtain the text features corresponding to each training image. The VAE network encodes each training image to obtain the image features corresponding to each training image. The denoising fine-tuning network decodes the image features of each training image according to the text features corresponding to each training image to obtain the restored image corresponding to each training image. It should be noted that the VAE network encodes each training image by adding noise, and the denoising fine-tuning network decodes the image features corresponding to each training image by predicting the noise added by the VAE network and removing the noise based on the predicted noise. A divergence function is used to calculate the prediction loss between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image. The divergence function can be the KL divergence (Kullback-Leibler Divergence). The model parameters of the text-based graph model are optimized based on the predicted loss value to complete the training of the text-based graph model.

[0030] Furthermore, the denoising fine-tuning network is used to decode the image features corresponding to each training image according to the text features corresponding to each training image, so as to obtain the restored image corresponding to each training image, including: the denoising fine-tuning network includes a basic network, a cross-attention network and multiple LoRA branches; within the denoising fine-tuning network: the text features and image features corresponding to each training image are processed by the basic network; the text features and image features corresponding to each training image processed by the basic network are processed by the cross-attention network to obtain the cross-attention features corresponding to each training image; the cross-attention features corresponding to each training image are adjusted by multiple LoRA branches to obtain the restored image corresponding to each training image.

[0031] The base network is the most basic network in the U-Net network, including: convolutional layer, pooling layer, downsampling layer, upsampling layer and other networks. In the application of the U-Net network, in order to improve the effect of the U-Net network, a cross-attention network is often added to the U-Net network. As a classic structure of the U-Net network, the embodiment of the present application uses this U-Net network. Then the denoising fine-tuning network can be regarded as three parts, namely the base network, the cross-attention network and multiple LoRA branches. Inside the denoising fine-tuning network: for each training image, the base network processes the text features and image features. The cross-attention network processes the text features and image features processed by the base network to obtain cross-attention features. Multiple LoRA branches adjust the cross-attention features corresponding to each training image to obtain the restored image corresponding to each training image. Because the image itself is a feature matrix, the cross-attention features adjusted by multiple LoRA branches can also be directly regarded as images.

[0032] Further, the text features and the image features corresponding to each of the training images processed by the base network are processed by a cross-attention network to obtain cross-attention features corresponding to each of the training images, including: taking the text features corresponding to each of the training images processed by the base network as query vectors and value vectors corresponding to each of the training images, taking the image features corresponding to each of the training images processed by the base network as key vectors corresponding to each of the training images, processing the query vectors, the value vectors and the key vectors corresponding to each of the training images by the cross-attention network to obtain the cross-attention features corresponding to each of the training images.

[0033] The query vector is Query, the key vector is Key, and the value vector is Value. The query vector, the value vector and the key vector are input into the cross-attention network as a group of vectors, and the cross-attention feature is output.

[0034] Further, the cross-attention features corresponding to each of the training images are adjusted by a plurality of LoRA branches to obtain restored images corresponding to each of the training images, including: processing the cross-attention features corresponding to each of the training images by each of the LoRA branches to obtain adjustment features corresponding to each of the training images; and adding the cross-attention features corresponding to each of the training images and the plurality of adjustment features to obtain the restored images corresponding to each of the training images.

[0035] LoRA (Low-Rank Adaptation) is a low-rank adaptation technology for large language models (such as GPT series) and other multi-modal models (such as Stable Diffusion), which can be used as a branch to optimize the output of the original network. In the embodiments of the present application, the cross-attention features are processed by a plurality of LoRA branches to obtain adjustment features, and the cross-attention features and the plurality of adjustment features are added, and the result of the addition is taken as the restored image.

[0036] Further, the cross-attention features corresponding to each of the training images are adjusted by a plurality of LoRA branches to obtain restored images corresponding to each of the training images, including: the denoising fine-tuning network includes three LoRA branches; the cross-attention features corresponding to each of the training images are processed by the first LoRA branch to obtain first adjustment features corresponding to each of the training images; the cross-attention features corresponding to each of the training images are processed by the second LoRA branch to obtain second adjustment features corresponding to each of the training images; the cross-attention features corresponding to each of the training images are processed by the third LoRA branch to obtain third adjustment features corresponding to each of the training images; and the cross-attention features corresponding to each of the training images, the first adjustment features, the second adjustment features and the third adjustment features are added to obtain the restored images corresponding to each of the training images.

[0037] The embodiment of the present application uses three LoRA branches to optimize the cross-attention feature, and may also use two LoRA branches, four LoRA branches, or the like to optimize the cross-attention feature.

[0038] Furthermore, after using the divergence function to calculate the prediction loss value between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image, the method also includes: judging, by a discriminator, the probability that each training image is real under the condition of the image description carried by the training image; judging, by a discriminator, the probability that the restored image corresponding to each training image is real under the condition of the image description carried by the training image; calculating a judgment loss value based on the probability corresponding to each training image and the probability corresponding to the restored image corresponding to each training image; and optimizing the model parameters of the Vincent graph model based on the prediction loss value and the judgment loss value to complete the training of the Vincent graph model.

[0039] This embodiment of the application uses the discriminator in the adversarial network to optimize the model parameters of the GAN model (this embodiment of the application actually uses the GAN model as a generator). GANs contain a generator and a discriminator.

[0040] The probability that the discriminator determines that the training image x is real under the condition that it carries the image description y is recorded as D(x / y), and the probability that the discriminator determines that the restored image z corresponding to the training image x is real under the condition y is recorded as D(z / y). Then the judgment loss value LD can be calculated according to the following formula:

[0041] LD=-Ex[log(D(x / y))]-Ez[log(1-D(z / y))];

[0042] E is the expected value operator, log(D(x / y)) is the logarithm of D(x / y), Ex[log(D(x / y))] is used to calculate the average of the logarithm values ​​corresponding to all training images, log(1-D(z / y)) is the logarithm value of D(z / y), and Ez[log(1-D(z / y))] is used to calculate the average of the logarithm values ​​corresponding to all restored images.

[0043] Figure 2 This is a flow chart of a method for generating a text graph according to an embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0044] S201, obtaining text and images input by the user;

[0045] S202, inputting text and images into the text-graph model, within the text-graph model;

[0046] S203, encode the text through a text encoder in the CLIP network to obtain target text features;

[0047] S204, encode the image through a VAE network to obtain target image features;

[0048] S205, decode the target image features according to the target text features through a denoising fine-tuning network to obtain a target restored image.

[0049] The embodiment of the present application actually adjusts the image according to the text input by the user. The image can also be a template in the database, and the image in the database is adjusted according to the text input by the user to obtain an image meeting the requirements of the user.

[0050] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described one by one here.

[0051] The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiment of the present disclosure. For details not disclosed in the device embodiment of the present disclosure, please refer to the method embodiment of the present disclosure.

[0052] Figure 3 is a structural schematic diagram of a text-to-image model training device provided by an embodiment of the present disclosure. As shown in Figure 3 , the text-to-image model training device comprises:

[0053] The acquisition module 301 is configured to acquire an image data set and extract image descriptions of each person image in the image data set.

[0054] The first construction module 302 is configured to construct a first training data set by using each person image and its image description.

[0055] The second construction module 303 is configured to segment the person face image from each person image, and construct a second training data set by using the person face image and the image description.

[0056] The third construction module 304 is configured to combine the first training data set and the second training data set according to a preset proportion to obtain a third training data set.

[0057] The modeling module 305 is configured to construct a plurality of LoRA branches for a U-Net network to obtain a denoising fine-tuning network, and construct a text-to-image model by using a text encoder in a CLIP network, a VAE network and the denoising fine-tuning network.

[0058] The training module 306 is configured to train the text-to-image model by using the third training data set, so that the text-to-image model learns the person image and the image description.

[0059] According to the technical solution provided by the embodiment of the present application, an image data set is obtained, and the image description of each character image in the image data set is extracted; a first training data set is constructed using each character image and its image description; facial images of each character image are segmented from each character image, and a second training data set is constructed using the facial images and image descriptions of each character image; the first training data set and the second training data set are combined in a preset ratio to obtain a third training data set; multiple LoRA branches are constructed for the U-Net network to obtain a denoising fine-tuning network, and a text-generated graph model is constructed using the text encoder, VAE network, and denoising fine-tuning network in the CLIP network; the text-generated graph model is trained using the third training data set so that the text-generated graph model learns the character images and image descriptions. The above technical means can solve the problem that the existing Chinese text-generated graph model generates images that do not meet expectations, thereby improving the quality of the generated images.

[0060] In some embodiments, the training module 306 is further configured as follows: the third training data set includes multiple training images, each training image carries an image description, and each training image is a person image or a person face image; each training image and the image description it carries are input into the Vincent graph model, and inside the Vincent graph model: the image description carried by each training image is encoded by the text encoder in the CLIP network to obtain the text features corresponding to each training image; each training image is encoded by the VAE network to obtain the image features corresponding to each training image, wherein the VAE network is to generate the image features corresponding to each training image by adding noise. Each training image is encoded; the image features corresponding to each training image are decoded by the denoising fine-tuning network according to the text features corresponding to each training image to obtain a restored image corresponding to each training image, wherein the denoising fine-tuning network decodes the image features corresponding to each training image by predicting the noise added by the VAE network and removing the noise based on the predicted noise; the divergence function is used to calculate the prediction loss value between the noise added by the VAE network on each training image and the noise predicted by the denoising fine-tuning network; the model parameters of the text-based graph model are optimized according to the prediction loss value to complete the training of the text-based graph model.

[0061] In some embodiments, the training module 306 is further configured as a denoising fine-tuning network including a basic network, a cross-attention network and multiple LoRA branches; within the denoising fine-tuning network: the text features and image features corresponding to each training image are processed through the basic network; the text features and image features corresponding to each training image processed by the basic network are processed through the cross-attention network to obtain cross-attention features corresponding to each training image; the cross-attention features corresponding to each training image are adjusted through multiple LoRA branches to obtain restored images corresponding to each training image.

[0062] In some embodiments, the training module 306 is further configured to take the text features corresponding to each training image processed by the base network as the query vector and the value vector corresponding to each training image, take the image features corresponding to each training image processed by the base network as the key vector corresponding to each training image, and process the query vector, the value vector and the key vector corresponding to each training image through the cross-attention network to obtain the cross-attention features corresponding to each training image.

[0063] In some embodiments, the training module 306 is further configured to process the cross-attention features corresponding to each training image through each LoRA branch to obtain the adjustment features corresponding to each training image, and add the cross-attention features and the plurality of adjustment features corresponding to each training image to obtain the restored image corresponding to each training image.

[0064] In some embodiments, the training module 306 is further configured to take the cross-attention features corresponding to each training image processed by the first LoRA branch as the first adjustment features corresponding to each training image, take the cross-attention features corresponding to each training image processed by the second LoRA branch as the second adjustment features corresponding to each training image, and take the cross-attention features corresponding to each training image processed by the third LoRA branch as the third adjustment features corresponding to each training image, and add the cross-attention features, the first adjustment features, the second adjustment features and the third adjustment features corresponding to each training image to obtain the restored image corresponding to each training image.

[0065] In some embodiments, the training module 306 is further configured to determine the probability that each training image is real under the image description carried by the training image through the discriminator, determine the probability that the restored image corresponding to each training image is real under the image description carried by the training image through the discriminator, calculate the judgment loss value based on the probability corresponding to each training image and the probability corresponding to the restored image corresponding to each training image, and optimize the model parameters of the text-to-image model according to the prediction loss value and the judgment loss value to complete the training of the text-to-image model.

[0066] In some embodiments, the training module 306 is further configured to obtain the text and image input by the user, input the text and image into the text-to-image model inside the text-to-image model, encode the text through the text encoder in the CLIP network to obtain the target text feature, encode the image through the VAE network to obtain the target image feature, and decode the target image feature according to the target text feature through the denoising fine-tuning network to obtain the target restored image.

[0067] It should be understood that the sequence numbers of the steps in the above embodiments do not mean execution sequence, and the execution sequence of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0068] Figure 4 is a schematic diagram of an electronic device 4 according to an embodiment of the present disclosure. As shown in the figure, the electronic device 4 according to this embodiment includes a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. The processor 401 implements the steps in each of the above method embodiments when executing the computer program 403. Alternatively, the processor 401 implements the functions of each module / unit in each of the above device embodiments when executing the computer program 403. Figure 4

[0069] The electronic device 4 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The electronic device 4 can include but is not limited to the processor 401 and the memory 402. Those skilled in the art can understand that the electronic device 4 can include more or fewer components or different components. Figure 4 The electronic device 4 is merely an example and does not constitute a limitation on the electronic device 4, which can include more or fewer components or different components than those shown in the figure.

[0070] The processor 401 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc.

[0071] The memory 402 can be an internal storage unit of the electronic device 4, for example, a hard disk or a memory of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 402 can also include both internal storage units and external storage devices of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0072] ​Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the above-mentioned functional units and module divisions are used as examples for illustration. In actual applications, the above-mentioned functions can be distributed to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. In the embodiments, each functional unit and module can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the legislation and patent practice requirements in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0074] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for training a cultural graph model, characterized in that: include: Acquire an image dataset, and extract an image description of each person image in the image dataset; Constructing a first training data set using each person image and its image description; Segmenting each person image to obtain a person's facial image of each person image, and constructing a second training data set using the person's facial image and image description of each person image; Combining the first training data set and the second training data set in a preset ratio to obtain a third training data set; Construct multiple LoRA branches for the U-Net network to obtain a denoising fine-tuning network, and use the text encoder, VAE network and the denoising fine-tuning network in the CLIP network to construct a text-generated graph model; Training the text-graph model using the third training data set so that the text-graph model learns character images and image descriptions; Training the text-graph model using the third training data set so that the text-graph model learns character images and image descriptions includes: The third training data set includes a plurality of training images, each training image carries an image description, and each training image is a person image or a person face image; Each training image and its accompanying image description is input into the Wensheng graph model. Inside the Wensheng graph model: Encoding the image description carried by each training image through the text encoder in the CLIP network to obtain text features corresponding to each training image; Encoding each training image through the VAE network to obtain image features corresponding to each training image, wherein the VAE network encodes each training image by adding noise; Decoding the image features corresponding to each training image according to the text features corresponding to each training image through the denoising fine-tuning network to obtain a restored image corresponding to each training image, wherein the denoising fine-tuning network decodes the image features corresponding to each training image by predicting the noise added by the VAE network and removing the noise based on the predicted noise; Calculate the prediction loss between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image using a divergence function; Optimizing model parameters of the Wensheng graph model according to the predicted loss value to complete the training of the Wensheng graph model; After calculating the prediction loss between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image using a divergence function, the method further includes: The discriminator determines the probability that each training image is real given the image description carried by the training image; The discriminator determines the probability that the restored image corresponding to each training image is true under the condition of the image description carried by the training image; The loss value is calculated based on the probability corresponding to each training image and the probability corresponding to the restored image corresponding to each training image; The model parameters of the culture graph model are optimized according to the prediction loss value and the judgment loss value to complete the training of the culture graph model.

2. The method according to claim 1, characterized in that Decoding the image features corresponding to each training image according to the text features corresponding to each training image through the denoising fine-tuning network to obtain a restored image corresponding to each training image, including: The denoising fine-tuning network includes a base network, a cross-attention network and multiple LoRA branches; Inside the denoising fine-tuning network: Processing the text features and image features corresponding to each training image through the basic network; Processing the text features and image features corresponding to each training image processed by the basic network through the cross attention network to obtain the cross attention features corresponding to each training image; The cross-attention features corresponding to each training image are adjusted through multiple LoRA branches to obtain the restored images corresponding to each training image.

3. The method according to claim 2, characterized in that The text features and image features corresponding to each training image processed by the basic network are processed by the cross attention network to obtain the cross attention features corresponding to each training image, including: The text features corresponding to each training image processed by the basic network are used as the query vector and value vector corresponding to each training image, and the image features corresponding to each training image processed by the basic network are used as the key vector corresponding to each training image. The query vector, value vector and key vector corresponding to each training image are processed by the cross-attention network to obtain the cross-attention features corresponding to each training image.

4. The method according to claim 2, characterized in that The cross-attention features corresponding to each training image are adjusted through multiple LoRA branches to obtain the restored images corresponding to each training image, including: The cross-attention features corresponding to each training image are processed through each LoRA branch to obtain the adjustment features corresponding to each training image; The cross-attention features and multiple adjustment features corresponding to each training image are added together to obtain the restored image corresponding to each training image.

5. The method according to claim 2, characterized in that: The cross-attention features corresponding to each training image are adjusted through multiple LoRA branches to obtain the restored images corresponding to each training image, including: The denoising fine-tuning network includes three LoRA branches; The cross-attention features corresponding to each training image are processed by the first LoRA branch to obtain the first adjustment features corresponding to each training image; The cross-attention features corresponding to each training image are processed by the second LoRA branch to obtain the second adjustment features corresponding to each training image; The cross-attention features corresponding to each training image are processed by the third LoRA branch to obtain the third adjustment features corresponding to each training image; The cross-attention feature, the first adjustment feature, the second adjustment feature, and the third adjustment feature corresponding to each training image are added together to obtain a restored image corresponding to each training image.

6. A cultural graph model training device, characterized in that: include: an acquisition module configured to acquire an image dataset and extract an image description of each person image in the image dataset; A first construction module is configured to construct a first training data set using each person image and its image description; A second construction module is configured to segment each person image to obtain a person's facial image of each person image, and construct a second training data set using the person's facial image and image description of each person image; A third construction module is configured to combine the first training data set and the second training data set in a preset ratio to obtain a third training data set; A modeling module is configured to construct multiple LoRA branches for the U-Net network to obtain a denoising fine-tuning network, and use the text encoder, VAE network and the denoising fine-tuning network in the CLIP network to construct a text-generated graph model; a training module configured to train the text-graph model using the third training data set so that the text-graph model learns character images and image descriptions; Training the text-graph model using the third training data set so that the text-graph model learns character images and image descriptions includes: The third training data set includes a plurality of training images, each training image carries an image description, and each training image is a person image or a person face image; Each training image and its accompanying image description is input into the Wensheng graph model. Inside the Wensheng graph model: Encoding the image description carried by each training image through the text encoder in the CLIP network to obtain text features corresponding to each training image; Encoding each training image through the VAE network to obtain image features corresponding to each training image, wherein the VAE network encodes each training image by adding noise; Decoding the image features corresponding to each training image according to the text features corresponding to each training image through the denoising fine-tuning network to obtain a restored image corresponding to each training image, wherein the denoising fine-tuning network decodes the image features corresponding to each training image by predicting the noise added by the VAE network and removing the noise based on the predicted noise; Calculate the prediction loss between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image using a divergence function; Optimizing model parameters of the Wensheng graph model according to the predicted loss value to complete the training of the Wensheng graph model; After calculating the prediction loss between the noise added by the VAE network and the noise predicted by the denoising fine-tuning network on each training image using a divergence function, the method further includes: The discriminator determines the probability that each training image is real given the image description carried by the training image; The discriminator determines the probability that the restored image corresponding to each training image is true under the condition of the image description carried by the training image; The loss value is calculated based on the probability corresponding to each training image and the probability corresponding to the restored image corresponding to each training image; The model parameters of the culture graph model are optimized according to the prediction loss value and the judgment loss value to complete the training of the culture graph model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images