Image generation method, device, equipment and storage medium based on generative artificial intelligence

By using a separate face feature extractor and feature alignment module in face image generation, combined with face position information, the problem of insufficient similarity in face image generation in the prior art is solved, and higher quality and similarity are achieved. Face generation effect.

CN118553000BActive Publication Date: 2024-11-22SHENZHEN AIMALL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410996709.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-22
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

The prior art is difficult to maintain a high similarity in the generation of face images, especially the generation results in different races and angles have obvious deviations, making it difficult to ensure the similarity between the generated face and the real face.

Method used

A separate face feature extractor is used to extract face features, and the high-semantic features are fused into the diffusion space through the feature alignment module, and the embedding is adjusted and combined with face position information to ensure the directional control of face generation, and the use of decoupled crossover The attention layer and feature decoder structure perform image redrawing.

Benefits of technology

Effectively improve the similarity of generated faces, maintain a high-quality generated image, and reduce interference caused by other image information, provide a clearer direction of face generation control, and the faces generated from the generated image are more similar to the input face. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118553000B_ABST
    Figure CN118553000B_ABST
Patent Text Reader

Abstract

The present invention provides an image generation method, device, equipment and storage medium based on generative artificial intelligence, which inputs a text prompt word to be processed, obtains text features through text encoding according to the text prompt word; obtains face image information, the face image information includes image encoding and face feature extraction; inputs image feature prompts through a decoupled cross attention layer according to the image encoding, and embeds them into a diffusion space through an image cross attention module; extracts face features through an additional face feature extractor, and fuses features containing high semantics into the diffusion space through a feature alignment module as important prompts for face generation; while maintaining a high-quality generated image, the similarity of the generated face can be effectively improved, a separate face feature extractor is used to extract face features, and face feature fusion is performed in model sampling iterations, thereby ensuring the control of the generation direction of the face.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to technical fields such as deep learning, reinforcement learning and computer vision, and in particular to an image generation method, device, equipment and storage medium based on generative artificial intelligence. Background Art

[0002] Generative AI refers to the generation of highly complex output results through machine learning algorithms. It can help machines achieve autonomous creation and autonomous learning, bringing humans a more intelligent and personalized experience.

[0003] The field of image generation has always been a very popular research direction and topic. Generative AI has demonstrated strong capabilities in the fields of image generation and video production. The current mainstream technology is to generate images through Stable Diffusion (a large diffusion generative image model) text-generated images or image-generated images. Its advantages lie in the richness of generated scenes and the diversity of species. However, it also brings some problems. For example, in portrait generation, since there are many details that need to be portrayed, it is difficult for large models to obtain good results. Especially in terms of facial similarity, the generated faces often have some differences from the real faces, and it is difficult to ensure that they are photos of the same person.

[0004] Publication No. CN116049523B, the invention name is: A system for intelligently generating ancient poetry context videos by AI and its working method, which is a Chinese invention. It publicly constructs an AI intelligent system for generating ancient poetry context videos. The system automatically generates context videos corresponding to the content based on the ancient poetry text, which can provide a large number of video learning resources for teachers' teaching and students' self-study activities, reduce the difficulty of understanding, provide a new way to generate ancient poetry learning videos, and promote the intelligent generation of ancient poetry teaching resources;

[0005] Among the existing technologies, Stable Diffusion XL combined with IP Adapter can maintain a certain degree of facial similarity. A facial feature extraction module is trained in IP Adapter, and the Text Embedding of the prompt word is distinguished from the Face Embedding of the face. A Cross-Attention module is added to UNet to introduce facial features, so as to achieve the purpose of reshaping the fixed IP of the face.

[0006] In terms of technical implementation, some large models are usually used in conjunction with some large model components, such as IP Adapter, to separate image and text features, and use the features of facial images as input for visual cues to ensure that the same face is generated.

[0007] However, the faces generated by this method are often unstable, with obvious deviations in different angles for different races. Generating AI face images with high facial similarity is still a big challenge. To this end, a method for image generation based on generative artificial intelligence is proposed. Summary of the invention

[0008] In view of this, the present invention provides an image generation method, device, equipment and storage medium based on generative artificial intelligence. The present invention makes corresponding optimizations and improvements on the basis of generative technology, and can effectively improve the similarity of generated faces while maintaining a high-quality generated image. The Face Embedding embedding of faces in Stable Diffusion XL is optimized, and face features are extracted by a separate face feature extractor. Face feature fusion is performed in model sampling iterations, so as to ensure the control of face generation direction, solve or alleviate the technical problems existing in the prior art, and at least provide a beneficial choice.

[0009] The technical solution of the embodiment of the present invention is achieved as follows:

[0010] In a first aspect, the present invention provides an image generation method based on generative artificial intelligence, comprising the following steps:

[0011] Input a text prompt word to be processed, and obtain text features by encoding the text prompt word;

[0012] Acquiring facial image information, wherein the facial image information includes image coding and facial feature extraction;

[0013] According to the image encoding, the image feature prompt is input through the decoupled cross attention layer, and embedded into the diffusion space through the image cross attention module; according to the face feature extraction, the face area is encoded separately to obtain the face feature;

[0014] Acquire a face and head control image, wherein the face and head control image includes face position information of no less than five key points;

[0015] Constructing a control network model, the control network model inputs the face position information of the key points, and combines the embedding of face features by the feature alignment module to adjust and align the face features and the face position information of the key points and embed them into the diffusion space through the residual method;

[0016] The diffusion space denoises the random noise several times according to the prompt information, inputs the face image information and text features into the diffusion space for post-processing, and obtains an information feature matrix;

[0017] The information feature matrix is ​​redrawn through the feature decoder structure to obtain a generated image.

[0018] Further preferably, the inputting of image feature prompts through a decoupled cross attention layer according to the image encoding comprises:

[0019] Encode the input face image information, remove the last two linear layers used for encoding classification tasks, so as to extract the image features;

[0020] The projection network of the linear layer plus the normalization layer projects the image into a feature sequence of length N, so that the dimension of the image feature is consistent with the dimension of the text feature;

[0021] Decoupled Cross-Attention Layer: Image features and text features are decoupled through the cross-attention layer.

[0022] Further preferably, the cross attention layer includes a cross attention embedding module, and the calculation expression of the cross attention embedding module is:

[0023] ;

[0024] in, are the query, key, and value matrices of the text cross-attention operation, respectively. is the key and value matrix of the image cross attention;

[0025] is a controllable parameter;

[0026] represents text cross attention, Cross attention for images.

[0027] Further preferably, the diffusion space denoises the random noise several times according to the prompt information, inputs the face image information and text features into the diffusion space for post-processing, and obtains an information feature matrix, including:

[0028] Forward process: generating intermediate data by adding the random noise to the facial image information, text information, and facial feature information, wherein the intermediate data is a Markov chain, wherein the random noise is added to the intermediate data at each step;

[0029] Inverse process: Diffusion space learning recovers intermediate data from the random noise data;

[0030] Loss function: Train the model and restore intermediate data by adding the loss function;

[0031] The denoising specifically comprises the following steps:

[0032] Initialize random noise: Sample an initial random noise image from a diffuse spatial distribution ;

[0033] Stepwise Denoising: Model using the inverse process Step-by-step denoising to generate intermediate data;

[0034] Generate the final image: The last step is to generate the denoised image .

[0035] Further preferably, the text features of the input text prompt words to be processed specifically include:

[0036] Text encoding: The CLIP model encodes text prompt words and generates text feature vectors;

[0037] Feature Embedding: The text feature vector is embedded into the diffusion space through a separate cross-attention layer.

[0038] Further preferably, the facial features and the facial position information of the key points are adjusted and aligned and embedded into the diffusion space by means of residuals as follows:

[0039] ;

[0040] in, is the UNet module, x is random noise, Represents the frozen weights of the UNet model, and Z represents the weighted or Zero convolution calculation, It controls the weight value of the network model.

[0041] Further preferably, the image encoding includes performing edge detection on the image, and the edge detection includes gradient calculation, edge extraction and gradient amplitude calculation.

[0042] A second aspect: A generative artificial intelligence-based image generation device, applied to a generative artificial intelligence-based image generation method, comprising:

[0043] An input module, used for inputting the text prompt words to be processed, and obtaining text features through text encoding according to the text prompt words;

[0044] A facial image acquisition module is used to acquire the facial image information, wherein the facial image information includes image coding and facial feature extraction;

[0045] An image encoder for inputting image feature prompts through a decoupled cross-attention layer according to the image encoding and embedding into a diffusion space through an image cross-attention module;

[0046] A face feature extractor, used for extracting face features by encoding face regions separately to obtain face features;

[0047] A face head control image input module is used to provide the face head control image including face position information of no less than five key points, and to specify the image area where the face is generated when generating;

[0048] Constructing a control network module, which is used for the control network model to input the face position information of the key points, and combining the embedding of face features by the feature alignment module, adjusting and aligning the face features and the face position information of the key points and embedding them into the diffusion space through residuals;

[0049] An information feature matrix module is used to denoise the random noise of the diffusion space several times according to the prompt information, input the face image information and text features into the diffusion space for post-processing, and obtain an information feature matrix;

[0050] The image generation module redraws the information feature matrix through the feature decoder structure to obtain a generated image.

[0051] A third aspect: A computer device comprising:

[0052] processor;

[0053] A memory for storing executable instructions;

[0054] Among them, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the image generation method based on generative artificial intelligence.

[0055] A fourth aspect: A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the image generation method based on generative artificial intelligence.

[0056] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0057] First, according to the technical route of Stable Diffusion XL, the present invention extracts facial features through an additional facial feature extractor, and fuses the features containing high semantics into the diffusion space through a feature alignment module as an important hint for face generation;

[0058] While maintaining a high quality of generated images, the similarity of generated faces can be effectively improved. The Face Embedding in Stable Diffusion XL is optimized, and a separate face feature extractor is used to extract face features. Face feature fusion is performed in model sampling iterations to ensure the control of face generation direction.

[0059] Second, the facial coding features are extracted separately and aligned with the input. Due to the improvement, there are high-semantic facial information prompts in the diffusion process, which makes it possible to better prompt and compare facial information during multiple rounds of sampling and denoising. Therefore, in the diffusion direction, there is a clearer control direction for the facial prompt generation process, which makes the face of the generated image more similar to the input face.

[0060] Third, the present invention performs separate feature encoding on the face area image, and then embeds it into the latent space (diffusion space) through feature alignment. Such an operation can greatly maintain the generation similarity of the face. Compared with the IP Adapter component that only uses Clip Encode to encode the image, it can not only reduce the interference caused by other information in the image, but also can flexibly prompt input with the decoupled and separated text features.

[0061] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A flowchart of the image generation method based on generative artificial intelligence of the present invention;

[0064] Figure 2 It is a schematic diagram of the structure of the image generation device based on generative artificial intelligence of the present invention;

[0065] Figure 3 This is a schematic diagram of the computer device structure of Embodiment 4 of the present invention.

[0066] In the figure, 10 is a computer device; 1002 is a processor; 1004 is a memory; and 1006 is a transmission device. DETAILED DESCRIPTION

[0067] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0068] Explanation of key terms and technical abbreviations (the English abbreviations appearing below provide the corresponding English full names and Chinese translations, or detailed explanations of Chinese professional terms):

[0069] Stable Diffusion: A large diffusion-generated image model.

[0070] Stable Diffusion XL: An improved version based on Stable Diffusion;

[0071] IP Adapter: Image prompt adapter, a plug-in that can be used with large image models;

[0072] Text Embedding: text feature information;

[0073] Face Embedding: facial feature information;

[0074] UNet: A network model with an encoding-decoding structure that plays a diffusion (denoising) role in the large model.

[0075] Cross-Attention: Cross-attention structure;

[0076] Clip: data (text, image) encoder;

[0077] ControlNet: Module that accepts conditional inputs that guide the image generation process.

[0078] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0079] Embodiment 1

[0080] A method for generating images based on generative artificial intelligence, such as Figure 1 As shown, the following steps are included:

[0081] Input a text prompt word to be processed, and obtain text features by encoding the text prompt word;

[0082] Acquiring facial image information, wherein the facial image information includes image coding and facial feature extraction;

[0083] According to the image encoding, the image feature prompt is input through the decoupled cross attention layer, and embedded into the diffusion space through the image cross attention module; according to the face feature extraction, the face area is encoded separately to obtain the face feature;

[0084] Acquire a face and head control image, wherein the face and head control image includes face position information of no less than five key points;

[0085] Constructing a control network model, the control network model inputs the face position information of the key points, and combines the embedding of face features by the feature alignment module to adjust and align the face features and the face position information of the key points and embed them into the diffusion space through the residual method;

[0086] The diffusion space denoises the random noise several times according to the prompt information, inputs the face image information and text features into the diffusion space for post-processing, and obtains an information feature matrix;

[0087] The information feature matrix is ​​redrawn through the feature decoder structure to obtain a generated image.

[0088] Specifically: the step of inputting image feature prompts through a decoupled cross attention layer according to the image encoding includes:

[0089] Encode the input face image information, remove the last two linear layers used for encoding classification tasks, so as to extract the image features;

[0090] The projection network of the linear layer plus the normalization layer projects the image into a feature sequence of length N, so that the dimension of the image feature is consistent with the dimension of the text feature;

[0091] Decoupled Cross-Attention Layer: Image features and text features are decoupled through the cross-attention layer.

[0092] The cross attention layer includes a cross attention embedding module, and the calculation expression of the cross attention embedding module is:

[0093] ;

[0094] in, are the query, key, and value matrices of the text cross-attention operation, respectively. is the key and value matrix of the image cross attention;

[0095] is a controllable parameter;

[0096] represents text cross attention, For image cross attention, controllable parameters are used To flexibly adjust the embedding feature value of each iteration.

[0097] In this embodiment, the facial features and facial position information are adjusted and aligned and embedded into the diffusion space by means of residuals. The calculation expression is:

[0098] ;

[0099] in, is the UNet module, x is random noise, Represents the frozen weights of the UNet model, and Z represents the weighted or Zero convolution calculation, It controls the weight value of the network model.

[0100] For text prompt words, the original route of Stable Diffusion XL encodes the text features through the Clip text encoder and embeds the text features through a separate cross-attention layer.

[0101] For face images, the original single route is divided into two. The first branch uses a lightweight IPAdapter to encode the entire image, which is the image encoder module in the process. The image features obtained here are then prompted for input through the decoupled cross-attention layer, and are input separately from the text features, which can flexibly adjust the image condition constraint information.

[0102] On the second branch of the face image, the face feature extractor is an abbreviation, which includes two modules: face detection and feature extraction. Through this module, the face area is encoded separately to obtain face features with strong semantics.

[0103] In terms of face generation position control, an additional face head control image input is added. This image provides the position information of the five key points of the face and is used to specify the image area where the face is generated during generation.

[0104] The control network model uses the spatial information of key points as the input of the controllable module, and then combines it with the feature alignment module to embed facial features, adjust the facial features to align with the UNet module (diffusion space), increase facial semantic hints, and control the effect of face generation.

[0105] The overall goal is to embed the facial feature values ​​combined with the spatial position information of the key points into the original UNet module through the residual method.

[0106] The diffusion space denoises the random noise several times according to the prompt information, inputs the face image information and text features into the diffusion space for post-processing, and obtains the information feature matrix, including:

[0107] Forward process: generating intermediate data by adding the random noise to the facial image information, text information, and facial feature information, wherein the intermediate data is a Markov chain, wherein the random noise is added to the intermediate data at each step;

[0108] Inverse process: Diffusion space learning recovers intermediate data from the random noise data;

[0109] Loss function: Train the model and restore intermediate data by adding the loss function;

[0110] The denoising specifically comprises the following steps:

[0111] Initialize random noise: Sample an initial random noise image from a diffuse spatial distribution ;

[0112] Stepwise Denoising: Model using the inverse process Step-by-step denoising to generate intermediate data;

[0113] Generate the final image: The last step is to generate the denoised image Finally, the information feature matrix is ​​redrawn through the VEA (Variational AutoEncoder) feature decoder structure, and the operation result of the latent space is decoded into the actual picture dimension to obtain the generated image.

[0114] In the entire structure, the facial area image is encoded with separate features and then embedded into the latent space (diffusion space) through feature alignment. This operation can greatly maintain the similarity of face generation. Compared with the IPAdapter component that only uses Clip Encode to encode the image, it can not only reduce the interference caused by other image information, but also provide flexible prompt input with the decoupled and separated text features.

[0115] In this embodiment, the facial features and the facial position information of the key points are adjusted and aligned and embedded into the diffusion space by means of residuals. The calculation expression is:

[0116] ;

[0117] in, is the UNet module, x is random noise, Represents the frozen weights of the UNet model, and Z represents the weighted or Zero convolution calculation, It controls the weight value of the network model.

[0118] In this embodiment: the image encoding includes edge detection on the image, and the edge detection includes gradient calculation, edge extraction and gradient amplitude calculation, wherein the gradient calculation is to calculate the image gradient using the Sobel operator;

[0119] For the gradient of the image in the x direction :

[0120] ;

[0121] For the gradient in the y direction :

[0122] ;

[0123] Gradient amplitude calculation:

[0124] ;

[0125] in, and At the pixel point The gradient strength in the x and y directions calculated at ;

[0126] Edge extraction: Set the threshold to T, and consider pixels with gradient magnitude greater than T as edges:

[0127] ;

[0128] Among them, Edge represents the image edge set;

[0129] Edge contains all pixel points (x, y) that satisfy the condition G(x, y)>T, where G(x, y) is the gradient magnitude of the image at the pixel point (x, y).

[0130] After the face image information and text features are input into the diffusion space, the diffusion space then denoises the random noise several times according to the prompt information to obtain the processed information feature matrix, including:

[0131] Forward process: generate intermediate data by adding random noise to facial image information, text information, and facial feature information. The intermediate data is a Markov chain, in which random noise is added to the intermediate data at each step:

[0132] ;

[0133] Among them, X t represents the data at time step t, β tis a predefined random noise scheduling parameter that controls the amount of random noise added at each step, and N represents Gaussian distribution;

[0134] Inverse process: Diffusion space learning recovers intermediate data from random noise data:

[0135] ;

[0136] Among them, X t−1 represents the data at time step t−1, and are the mean and covariance of the model parameterization, learned by the neural network; P represents the inverse process formula, Represents a set of weights and bias parameters of a neural network;

[0137] Loss function: Train the model and restore intermediate data by adding the loss function;

[0138] ;

[0139] Among them, D KL represents the Kullback-Leibler divergence, which measures the difference between two distributions;

[0140] is the true posterior distribution of the forward process.

[0141] Specifically: the denoising comprises the following steps:

[0142] Initialize random noise: Sample an initial random noise image from a diffuse spatial distribution ;

[0143] Stepwise Denoising: Model using the inverse process Step-by-step denoising to generate intermediate data;

[0144] Generate the final image: The last step is to generate the denoised image .

[0145] Embodiment 2

[0146] Figure 2 A generative artificial intelligence-based image generation device is shown, which is applied to a generative artificial intelligence-based image generation method, including:

[0147] An input module, used for inputting the text prompt words to be processed, and obtaining text features through text encoding according to the text prompt words;

[0148] A facial image acquisition module is used to acquire the facial image information, wherein the facial image information includes image coding and facial feature extraction;

[0149] An image encoder for inputting image feature prompts through a decoupled cross-attention layer according to the image encoding and embedding into a diffusion space through an image cross-attention module;

[0150] A face feature extractor, used for extracting face features by encoding face regions separately to obtain face features;

[0151] A face head control image input module is used to provide the face head control image including face position information of no less than five key points, and to specify the image area where the face is generated when generating;

[0152] Constructing a control network module, which is used for the control network model to input the face position information of the key points, and combining the embedding of face features by the feature alignment module, adjusting and aligning the face features and the face position information of the key points and embedding them into the diffusion space through the residual method;

[0153] An information feature matrix module is used to denoise the random noise of the diffusion space several times according to the prompt information, input the face image information and text features into the diffusion space for post-processing, and obtain an information feature matrix;

[0154] The image generation module redraws the information feature matrix through the feature decoder structure to obtain a generated image.

[0155] The foregoing Figure 1 The various variations and specific examples of the image generation method based on generative artificial intelligence in Example 1 are also applicable to the image generation device based on generative artificial intelligence in this embodiment. Through the above detailed description of the image generation method based on generative artificial intelligence, those skilled in the art can clearly know the implementation method of the image generation device based on generative artificial intelligence in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here.

[0156] Embodiment 3

[0157] On the basis of Example 1, the facial coding features are extracted separately and aligned for input. Since the improved method provides highly semantic facial information prompts in the diffusion process, better prompts and comparisons of facial information are provided during multiple rounds of sampling and denoising. Therefore, in the diffusion direction, there is a clearer control direction for the facial prompt generation process, which results in an effect in which the face of the generated image is more similar to the input face.

[0158] Test sample input image: A.jpg;

[0159] Test sample face head position map: B.jpg;

[0160] Test sample prompt text: A 20-year-old girl in a kimono is standing under a cherry blossom tree with cherry blossoms scattered around her face, A ray of sunlight hit her face, realistic life style photography;

[0161] As the number of iteration steps (twice for each image) increases, the face redrawing gradually approaches the real face and keeps fitting the original input face image in details, which fully demonstrates the feasibility of the solution of the present invention.

[0162] Embodiment 4

[0163] An embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement an image generation method based on generative artificial intelligence as provided in the above method embodiment.

[0164] Figure 3 The hardware structure diagram of a device for implementing a method for generating an image based on generative artificial intelligence provided in an embodiment of the present application is shown. The device may participate in constituting or include the apparatus or system provided in an embodiment of the present application. Figure 3 As shown, the computer device 10 may include one or more processors 1002 (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0165] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0166] The memory 1004 can be used to store software programs and modules of application software, such as a program instruction / data storage device corresponding to a method for generating an image based on generative artificial intelligence in an embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, that is, implementing the above-mentioned method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The transmission device 1006 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device 10. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 1006 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0168] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer device 10 (or mobile device).

[0169] Embodiment 5

[0170] An embodiment of the present application also provides a computer-readable storage medium, which can be set in a server to store at least one instruction or at least one program related to an image generation method based on generative artificial intelligence in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the image generation method based on generative artificial intelligence provided in the above method embodiment.

[0171] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0172] Embodiment 6

[0173] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a method for generating an image based on generative artificial intelligence provided in the above various optional embodiments.

[0174] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0177] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

[0178] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating an image based on generative artificial intelligence, characterized in that: The following steps are involved: Input a text prompt word to be processed, and obtain text features by encoding the text prompt word; Acquiring facial image information, wherein the facial image information includes image coding and facial feature extraction; The image feature prompts are input through a decoupled cross-attention layer according to the image encoding, including: Encode the input face image information, remove the last two linear layers used for encoding classification tasks, so as to extract the image features; The projection network of the linear layer plus the normalization layer projects the image into a feature sequence of length N, so that the dimension of the image feature is consistent with the dimension of the text feature; Decoupled cross attention layer: The image features and text features are decoupled through the cross attention layer, and embedded into the diffusion space through the image cross attention module; according to the face feature extraction, the face area is encoded separately to obtain the face feature, the cross attention layer includes a cross attention embedding module, and the calculation expression of the cross attention embedding module is: ; in, are the query, key, and value matrices of the text cross-attention operation, respectively. is the key and value matrix of the image cross attention; is a controllable parameter; represents text cross attention, Cross attention for images; Acquire a face and head control image, wherein the face and head control image includes face position information of no less than five key points; Constructing a control network model, the control network model inputs the face position information of the key points, and combines the embedding of face features by the feature alignment module to adjust and align the face features and the face position information of the key points and embed them into the diffusion space through the residual method; The diffusion space denoises the random noise several times according to the prompt information, inputs the face image information and text features into the diffusion space for post-processing, and obtains an information feature matrix; The information feature matrix is ​​redrawn through a feature decoder structure to obtain a generated image.

2. The method according to claim 1, characterized in that: The diffusion space denoises the random noise several times according to the prompt information, inputs the face image information and text features into the diffusion space for post-processing, and obtains the information feature matrix, including: Forward process: generating intermediate data by adding the random noise to the facial image information, text information, and facial feature information, wherein the intermediate data is a Markov chain, wherein the random noise is added to the intermediate data at each step; Inverse process: Diffusion space learning recovers intermediate data from the random noise data; Loss function: Train the model and restore intermediate data by adding the loss function; The denoising specifically comprises the following steps: Initialize random noise: Sample an initial random noise image from a diffuse spatial distribution ; Stepwise denoising: Use the inverse process model to gradually denoise and generate intermediate data; Generate the final image: The last step is to generate the denoised image .

3. The method according to claim 1, characterized in that: The text features of the input text prompt words to be processed specifically include: Text encoding: The CLIP model encodes text prompt words and generates text feature vectors; Feature Embedding: The text feature vector is embedded into the diffusion space through a separate cross-attention layer.

4. The method according to claim 1, characterized in that: The calculation expression for adjusting and aligning the facial features and the facial position information of the key points and embedding them into the diffusion space through the residual method is: ; in, is the UNet module, x is random noise, Represents the frozen weights of the UNet model, and Z represents the weighted or Zero convolution calculation, It controls the weight value of the network model.

5. The method according to claim 1, characterized in that: The image encoding includes edge detection on the image, and the edge detection includes gradient calculation, edge extraction and gradient amplitude calculation.

6. A generative artificial intelligence-based image generation device, applied to the generative artificial intelligence-based image generation method according to any one of claims 1 to 5, characterized in that: include: An input module, used for inputting the text prompt words to be processed, and obtaining text features through text encoding according to the text prompt words; A facial image acquisition module is used to acquire the facial image information, wherein the facial image information includes image coding and facial feature extraction; An image encoder for inputting image feature prompts through a decoupled cross-attention layer according to the image encoding and embedding into a diffusion space through an image cross-attention module; A face feature extractor, used for extracting face features by encoding face regions separately to obtain face features; A face head control image input module is used to provide the face head control image including face position information of no less than five key points, and to specify the image area where the face is generated when generating; Constructing a control network module, which is used for the control network model to input the face position information of the key points, and combining the embedding of face features by the feature alignment module, adjusting and aligning the face features and the face position information of the key points and embedding them into the diffusion space through the residual method; An information feature matrix module is used to denoise the random noise of the diffusion space several times according to the prompt information, input the face image information and text features into the diffusion space for post-processing, and obtain an information feature matrix; The image generation module redraws the information feature matrix through a feature decoder structure to obtain a generated image.

7. A computer device, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the image generation method based on generative artificial intelligence as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the image generation method based on generative artificial intelligence as described in any one of claims 1 to 5.