Diffusion model-based role image generation method and related device

Abstract

The invention provides a role image generation method based on a diffusion model and a related device, the role image generation method is applied to a computing server of a role image generation system, the role image generation system further comprises a user terminal, the user terminal is in communication connection with the computing server, and the method comprises the following steps: obtaining a first image and a first text input by the user terminal, extracting three-dimensional character image parameters from the first image to obtain a first image parameter vector, extracting a first image semantic feature vector corresponding to the first image from the first image, and extracting a first text semantic feature vector corresponding to the first image from the first text to obtain a second image parameter vector; and performing dimension alignment on the first image parameter vector and the first image semantic feature vector to obtain a second image parameter vector, performing fusion to obtain a fused feature vector, and inputting the fused feature vector into a preset role image generation diffusion model to obtain a second image. Therefore, the fidelity of the generated role image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a role image generation method based on a diffusion model and related apparatuses. BACKGROUND

[0002] Portrait generation technology is a technology of automatically generating or editing a human face image with high realism by using a deep learning model, and it has wide applications in virtual reality, augmented reality, film and television production and other fields. At present, there are portrait generation methods based on a generative adversarial network (GAN) and portrait generation methods based on a diffusion model to generate a portrait of a person.

[0003] However, the portrait generation method based on the generative adversarial network cannot display the three-dimensional geometric structure of the modeled portrait of a person, and is prone to problems such as facial deformity and identity drift of the generated image; the portrait generation method based on the diffusion model mainly relies on two-dimensional appearance feature matching and lacks three-dimensional shape constraints, and is prone to problems such as loss of identity features and distortion of facial structure when generating an image, thereby resulting in low accuracy of the generated image.

[0004] Therefore, how to improve the fidelity of the generated image or the role image is an urgent problem to be solved. SUMMARY

[0005] Embodiments of the present application provide a role image generation method based on a diffusion model and related apparatuses, which achieve the improvement of the fidelity of the generated role image.

[0006] In a first aspect, the embodiments of the present application provide a role image generation method based on a diffusion model, applied to a computing server in a role image generation system, the role image generation system further comprising a user terminal, the user terminal being in communication connection with the computing server, and the method comprising: obtaining a first image and a first text input by the user terminal; extracting a three-dimensional role image parameter from the first image to obtain a first image parameter vector; extracting an image semantic feature corresponding to the first image from the first image, and extracting a text semantic feature corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector; aligning the first image parameter vector and the first image semantic feature vector in dimension to obtain a second image parameter vector; fusing the second image parameter vector, the first image semantic feature vector and the first text semantic feature vector to obtain a fused feature vector; input the fusion feature vector into a preset role image generation diffusion model to obtain a second image.

[0007] In a second aspect, an embodiment of the present application provides a role image generation device based on a diffusion model, applied to a computing server in a role image generation system, the role image generation system further comprising a user terminal, the user terminal being in communication connection with the computing server, and the device comprising: a data acquisition unit configured to acquire a first image and a first text input by the user terminal; a feature extraction unit configured to extract a three-dimensional role image parameter from the first image to obtain a first image parameter vector, extract an image semantic feature corresponding to the first image from the first image, and extract a text semantic feature corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector; a feature fusion unit configured to perform dimension alignment on the first image parameter vector and the first image semantic feature vector to obtain a second image parameter vector, and fuse the second image parameter vector, the first image semantic feature vector, and the first text semantic feature vector to obtain a fusion feature vector; an image generation unit configured to input the fusion feature vector into a preset role image generation diffusion model to obtain a second image.

[0008] In a third aspect, an embodiment of the present application provides a server, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs comprise instructions for performing steps in any method of the first aspect of the embodiments of the present application.

[0009] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to perform some or all of the steps described in any method of the first aspect of the embodiments of the present application.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform some or all of the steps described in any method of the first aspect of the embodiments of the present application. The computer program product can be a software installation package.

[0011] By implementing the embodiments of the present application, the following beneficial effects are achieved: The application provides a role image generation method based on a diffusion model and a related device, which is applied to a computing server of a role image generation system. The role image generation system also includes a user terminal in communication connection with the computing server. The method includes: obtaining a first image and a first text input by the user terminal, extracting a three-dimensional role image parameter from the first image to obtain a first image parameter vector, extracting a first image semantic feature vector corresponding to the first image from the first image, and extracting a first text semantic feature vector corresponding to the first image from the first text, performing dimension alignment on the first image parameter vector and the first image semantic feature vector to obtain a second image parameter vector, and performing fusion to obtain a fusion feature vector, inputting the fusion feature vector into a preset role image generation diffusion model to obtain a second image. In this way, the DECA is used to extract the face geometric parameter as the generation constraint of the diffusion model, and the CLIP is combined with the semantic identity feature, so that the three-dimensional perception and identity consistent portrait generation is realized without modifying the bottom structure of the diffusion model, and the fidelity of the portrait generation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 is an architecture diagram of a role image generation system based on a diffusion model provided by an embodiment of the application; Figure 2 is a structural schematic diagram of a server provided by an embodiment of the application; Figure 3 is a flowchart of a role image generation method based on a diffusion model provided by an embodiment of the application; Figure 4 is a method flowchart of role image generation diffusion model pre-training provided by an embodiment of the application; Figure 5 is a flowchart of another role image generation method based on a diffusion model provided by an embodiment of the application; Figure 6 is an architecture diagram of a role image generation method based on a diffusion model provided by an embodiment of the application; Figure 7 is an effect comparison diagram of different methods for generating a portrait provided by an embodiment of the application; Figure 8 is a functional module composition block diagram of a role image generation device based on a diffusion model provided by an embodiment of the application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0015] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0016] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, "multiple" refers to two or more.

[0017] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0018] In this application, the term "connection" refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices. This application does not impose any limitations on this.

[0019] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment. One of ordinary skill in the art will readily recognize from the disclosure herein a wide number of variations, alternatives, and equivalents in the application.

[0020] Currently, there are GAN-based portrait generation methods and diffusion model-based portrait generation methods to generate a portrait of a person. However, the GAN-based portrait generation method cannot display the three-dimensional geometric structure of the modeled human head, and is prone to problems such as facial deformity and identity drift of the generated image; the diffusion model-based portrait generation method mainly relies on two-dimensional appearance feature matching and lacks three-dimensional shape constraints, and is prone to problems such as loss of identity features and distortion of facial structure when generating an image, thereby resulting in low accuracy of the generated image.

[0021] To solve the above problems, the embodiment of the application provides a role image generation method and related device based on a diffusion model, which is applied to a computing server of a role image generation system. The role image generation system also includes a user terminal in communication connection with the computing server. The method includes: obtaining a first image and a first text input by the user terminal, extracting a three-dimensional role image parameter from the first image to obtain a first image parameter vector, extracting a first image semantic feature vector corresponding to the first image from the first image, and extracting a first text semantic feature vector corresponding to the first image from the first text, aligning the first image parameter vector and the first image semantic feature vector in dimension to obtain a second image parameter vector, and performing fusion to obtain a fusion feature vector, inputting the fusion feature vector into a preset role image generation diffusion model to obtain a second image. In this way, the fidelity of the generated role image can be improved.

[0022] The following will be described in combination with Figure 1 The system architecture of a role image generation method based on a diffusion model in the embodiment of the application is described, Figure 1 is an architecture diagram of a role image generation system based on a diffusion model provided by the embodiment of the application. The role image generation system 100 based on the diffusion model includes a user terminal 110 and a computing server 120.

[0023] The user terminal 110 is configured to send input information required for generating a character image to the computing server 120, and receive a character image result generated by the computing server 120. Specifically, the user terminal 110 comprises an input 111 and an output 112. The input 111 is configured to receive user input data, which at least comprises a first image representing a character appearance feature and first text information corresponding to the first image. The first image can be a portrait image, a face image or a character reference image uploaded by the user, and the first text can be text prompt information describing a character identity feature, an appearance attribute or a style feature. Through the input 111, the user can constrain the target character image in a combination of image and text, thereby providing a multi-modal conditional input for subsequent character image generation. The output 112 is configured to show or output the character image result generated by the computing server 120 to the user. The output result can be a generated character image, a character portrait or a corresponding visual result. Preferably, the output 112 can be a display screen, a graphical user interface or a file output interface, configured to display the generated second image, so that the user can intuitively obtain the character image generation result. The user terminal 110 can be a personal computer, a mobile terminal, a tablet device or other terminal device with image input and display capabilities. The user terminal 110 and the computing server 120 can interact through the Internet, a local area network or a dedicated communication network to realize uploading of input data and returning of the generation result.

[0024] The computing server 120 is configured to perform a role image generation computing task based on a diffusion model, and is a computing unit of the role image generation system 100 based on the diffusion model. The computing server 120 includes a computing module 121 and a communication module 122. The communication module 122 is configured to realize data communication between the computing server 120 and the user terminal 110, receive the first image and the first text sent by the user terminal 110, and return the generated role image result to the user terminal 110. The communication module 122 can use HTTP, WebSocket or other network communication protocols to ensure the reliability and real-time performance of multi-modal data transmission. The computing module 121 is configured to perform algorithm processing of role image generation on the received multi-modal input data. Specifically, the computing module 121 is configured to extract three-dimensional role image parameters from the first image, and extract image semantic features and text semantic features based on a contrastive language-image model. Further, the computing module 121 is configured to perform dimension alignment on the three-dimensional role image parameters and the image semantic features, and perform feature fusion on the aligned three-dimensional parameters, the image semantic features and the text semantic features to obtain a fusion feature vector. Then, the computing module 121 inputs the fusion feature vector as a condition into a preset role image generation diffusion model to generate a role image result with three-dimensional geometric constraints and semantic consistency. The computing module 121 can be deployed on a server device with high-performance computing capability, which can include a graphics processing unit (GPU) or a dedicated accelerator for accelerating the diffusion model and multi-modal feature processing. The role image generation diffusion model executed by the computing module 121 can be trained offline before the role image generation system 100 based on the diffusion model is run, and used for inference generation during system operation.

[0025] It can be seen that by completing the collection of multi-modal input on the user terminal 110 side and completing the extraction of three-dimensional role image parameters, the extraction of semantic features, the alignment of features, the fusion of features and the generation of diffusion models on the computing server 120 side, high-quality role image generation can be realized without increasing the computing resources of the user terminal. Further, the architecture of the role image generation system 100 based on the diffusion model introduces three-dimensional role image parameters, image semantic features and text semantic features into the condition input of the diffusion model, so that the generation process is not only constrained by two-dimensional appearance features, but also combines three-dimensional geometric structure information, thereby effectively alleviating the problems of identity drift and facial structure distortion in the role image generation process of traditional GAN or two-dimensional diffusion models.

[0026] The following will be described in combination with Figure 2 The electronic device in the embodiment of the present application is described, Figure 2 A structural schematic diagram of a server provided in the embodiment of the present application is shown in Figure 2As shown, the server includes a processor 210, a memory 220, a communication interface 230, and one or more programs 221, which are communicatively connected with each other by an internal communication bus.

[0027] The one or more programs 221 are stored in the memory 220 and configured to be executed by the processor 210, and the one or more programs 221 include instructions for performing any steps of the above method embodiments.

[0028] The processor 210 can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logical blocks, units, and circuits described in connection with the disclosure. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc. The communication unit can be a communication interface, a transceiver, a transceiver circuit, etc., and the storage unit can be a memory.

[0029] The memory 220 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0030] It can be understood that the server 200 can include more or less structural elements than the above structural block diagram, for example, including a power module, a physical button, a Wi-Fi module, a speaker, a Bluetooth module, a sensor, a display module, etc., which are not limited herein. It can be understood that the server 200 can be equipped with a role image generation system as described above. Figure 1 The architecture of the role image generation system based on the diffusion model.

[0031] After understanding the software and hardware architecture of the present application, the following will be combined with the Figure 3 A role image generation method based on a diffusion model in an embodiment of the present application is described, Figure 3 is a flowchart of a role image generation method based on a diffusion model provided by an embodiment of the present application. The method is applied to a computing server in a role image generation system, and the role image generation system further includes a user terminal. The user terminal is in communication connection with the computing server, and specifically includes the following steps: Step S310, obtaining a first image and a first text input by the user terminal.

[0032] The first image is image data uploaded by the user terminal containing a portrait of a person. The user terminal can be a camera that captures real-time images, or it can be an existing portrait image uploaded from local storage, covering image input needs in various scenarios such as daily shooting and professional photography. The first text is semantic description information associated with the first image. The text content can be a description of the identity of the person (such as facial feature details), posture and expression, such as "middle-aged man with black-framed glasses and a smile on his mouth," "woman with short hair and earrings on her side portrait." The first text description does not need to follow a strict fixed format and can be accurately expressed in natural language.

[0033] It should be noted that the server needs to perform preprocessing operations on the first image input by the user terminal to optimize data quality. The preprocessing process can be to use the dlib library to complete face detection, key point positioning and image alignment to ensure that the face region is in a standard position in the image, and to automatically filter out non-real face images (such as sculptures, paintings, cartoon images, etc.) to avoid invalid data interfering with subsequent model training and generation results. For the first text, if the user does not actively input, a basic semantic description can be automatically generated based on the first image; if the user input text is ambiguous or information is missing, text supplement prompts can be provided to guide the user to improve key feature descriptions and further improve the effectiveness of data input.

[0034] Step S320 extracts three-dimensional character image parameters from the first image to obtain a first image parameter vector.

[0035] The three-dimensional character image parameters are constructed based on a parameterized three-dimensional deformable face model (3DMM) and include head pose, face shape, and expression change dimensions, which are quantitative representations of three-dimensional spatial features of a portrait. The head pose parameter describes the spatial orientation of the person's head through rotation, translation, and other vectors, and can accurately correspond to front view, ±70° range side view, and pitch view, etc. The face shape parameter represents the inherent geometric features of the person's face, such as cheekbone height, nose shape, jaw contour, and other personalized structural information. The expression parameter represents dynamic changes caused by facial muscle movements, such as smiling, frowning, and opening the mouth.

[0036] Specifically, a pre-trained DECA (Detailed Expression Capture and Animation) model is used as the extraction network of the three-dimensional character image parameters. The DECA model is a high-precision three-dimensional face parameter extraction framework proposed by the YadiraF team in 2022, which can decouple pose, shape, expression and other three-dimensional parameters from a single two-dimensional face image while retaining detailed features such as skin texture, providing accurate geometric constraints for three-dimensional perception of character image generation. This model has important application value in the fields of computer vision, image generation and digital human modeling. In the extraction of three-dimensional character image parameters, the pre-processed (face detection, key point positioning and alignment) first image is first input into the DECA model. The model analyzes the facial texture, contour and light and shade changes in the image through multiple layers of neural networks, and decouples the independent parameter dimensions of pose, shape, expression and other parameters based on pre-trained three-dimensional face prior knowledge. After quantization and encoding, the extracted parameters of each dimension form an initial three-dimensional feature set. Then, through feature alignment processing, the scale difference between different images is eliminated, and finally a consistent-dimensional first image parameter vector is output.

[0037] It can be seen that the extraction of three-dimensional character image parameters from the first image and the obtaining of the first image parameter vector are the basic guarantee for subsequent realization of high-fidelity and three-dimensional perception of character portrait generation. On the one hand, the three-dimensional parameters of pose, shape, expression and other parameters extracted by the DECA model establish clear spatial geometric constraints, enabling the subsequent diffusion model to generate images that follow the three-dimensional structural characteristics of the character, effectively solving the geometric distortion problem of existing methods when the viewing angle changes, and significantly improving the structural rationality and realism of the generated images under different poses. On the other hand, the quantization representation characteristics of three-dimensional character image parameters provide a basis for realizing accurate and controllable generation of pose, expression and shape, enabling users to adjust the corresponding dimensions in the parameter vector to flexibly control the viewing angle direction, facial morphology and expression state of the generated portrait, meeting the controllability requirements of virtual reality, digital human and other application scenarios for portrait generation.

[0038] In one possible embodiment, the extraction of three-dimensional character image parameters from the first image and the obtaining of the first image parameter vector specifically include the following steps: 321. Perform character image recognition on the first image to obtain a first character image; 322. Process the first character image based on a preset perspective transformation method to obtain a second character image; 323. Extract pose parameters, shape parameters and expression parameters from the second character image based on a pre-set three-dimensional detailed expression capture model to obtain first pose parameters, first shape parameters and first expression parameters; 324、standardize the first pose parameter, the first shape parameter and the first expression parameter respectively to obtain a second pose parameter, a second shape parameter and a second expression parameter; 325、splice the second pose parameter, the second shape parameter and the second expression parameter according to a preset dimension splicing method to obtain the first image parameter vector.

[0039] Wherein, the three-dimensional role image parameter is a quantitative index set constructed based on a parameterized three-dimensional deformable face model (3DMM), including a pose parameter (describing a head space orientation), a shape parameter (characterizing a face inherent contour and bone characteristics) and an expression parameter (capturing dynamic changes caused by face muscle movement); the first image parameter vector is a unified dimension vector formed by splicing the three types of parameters after standardization.

[0040] Specifically, the role image recognition is to use a face detection algorithm to locate the target area of the input first image. Preferably, the face detection algorithm uses the face detection model in the dlib library. By recognizing the key feature points of the face (such as the corners of the eyes, the corners of the mouth, the wings of the nose, etc.), the image area containing the complete face is segmented, and the background, clothes, irrelevant objects and other interference information are removed. The first role image focuses only on the face of the character, ensuring that the subsequent parameter extraction process is only for the target area, avoiding the introduction of noise by irrelevant information. The perspective transformation method is a geometric correction technique that establishes a mapping relationship between a two-dimensional image and a three-dimensional coordinate system. The principle is to solve the perspective transformation matrix to correct the face perspective distortion (such as tilt, near-far distortion, etc.) in the first role image caused by the shooting angle, and convert the irregular face projection image into a regular image under the standard orthoview angle (i.e. the second role image), so that the proportions of each area of the face conform to the three-dimensional geometric rules. The three-dimensional detail expression capture model uses a pre-trained DECA model to analyze the texture distribution, contour features and light-dark changes of the second role image (i.e. the pre-processed image), and combines the pre-trained three-dimensional face prior knowledge to decouple the first pose parameter (such as head rotation angle, translation vector), the first shape parameter (such as cheekbone height, nose bridge width, jaw contour curvature, etc. Individualized features) and the first expression parameter (such as mouth up amplitude, frown degree, eye opening degree, etc. Dynamic features), realizing the fine extraction of three-dimensional geometric information. Then, the standardization processing is performed, and the Z-score standardization method is preferably used to calculate the mean and standard deviation of each parameter, and map the three types of parameters to the [0, 1] numerical interval, eliminating the scale difference between different parameter dimensions (such as the angle value of the pose parameter and the length value of the shape parameter), avoiding the dilution of some parameters in the subsequent splicing process, and ensuring that the three types of parameters have equal weight in the feature vector. The dimension splicing method can use the vector concatenation strategy, which can be horizontal splicing or vertical splicing, without limitation. According to the order of pose parameters-shape parameters-expression parameters, the three types of standardized parameters are connected head to tail to form the first image parameter vector, realizing the unified representation of three-dimensional geometric features.

[0041] It can be seen that through role image recognition and perspective transformation, the effectiveness and regularity of the input data are ensured, the fine extraction of pose, shape and expression parameters is realized based on the DECA model, and the standardization and splicing processing complete the unified representation of three-dimensional features; this process effectively alleviates the influence of original image interference, perspective distortion and other factors on parameter extraction, provides a basis for the subsequent deep fusion with semantic features, and further supports the diffusion model to generate a character portrait image with precise three-dimensional structure constraint and high fidelity, significantly alleviating the problem of weak three-dimensional perception and face structure distortion in the prior art.

[0042] Step S330, extracting image semantic features corresponding to the first image from the first image, and extracting text semantic features corresponding to the first image from the first text, to obtain a first image semantic feature vector and a first text semantic feature vector.

[0043] Among them, the image semantic features are high-level visual semantic information with recognition degree for the identity of the person in the first image, including facial texture details, facial feature ratio relationships, unique accessory features, and other visual representations, which are a high-dimensional abstraction of the bottom layer pixel features. The text semantic features are semantic description vectors related to the identity of the person parsed from the first text, corresponding to facial features, accessories, expression states, and other key information, achieving the supplement and strengthening of the image semantic features.

[0044] Specifically, a pre-trained contrastive language-image pre-training (CLIP) model is used as an image semantic and text semantic feature extraction network. The CLIP model was proposed by OpenAI in 2021, and the model has strong cross-modal alignment capability and semantic understanding capability, can process image and text data at the same time, and output feature vectors in a unified semantic space. In the image semantic feature extraction process, the preprocessed first image is input into the image encoder of the CLIP model, and the image is extracted and aggregated layer by layer through a multi-layer Transformer network, filtering irrelevant interference information such as background and light, and finally outputting a fixed-dimension first image semantic feature vector. In the text semantic feature extraction process, the first text is first standardized (such as word segmentation and removing redundant information), and then input into the text encoder of the CLIP model, and through word embedding, position encoding and semantic modeling, the natural language description is converted into a first text semantic feature vector in the same dimension space as the image semantic feature. To ensure the stability and effectiveness of feature extraction, the CLIP model is initialized with pre-trained weights, and is fine-tuned with the overall generation network in the subsequent two-stage training process, further improving the matching degree of image, text semantic features and identity information.

[0045] It can be seen that the CLIP model realizes the extraction of image and text semantic features, solving the limitations of traditional single-modal feature extraction. The image semantic features provide intuitive and concrete visual identity representation, and the text semantic features provide abstract semantic identity description, effectively improving the recognition and robustness of identity features. On the one hand, the image semantic features extracted by the CLIP model have perspective independence and identity recognition, which can accurately capture the core visual features of the face, effectively reducing the interference of factors such as posture, lighting, and background. On the other hand, the text semantic features further strengthen the uniqueness of identity information through semantic description of key features, complementing the image semantic features and reducing the risk of identity drift in cross-perspective and cross-expression generation scenarios.

[0046] In a possible embodiment, the extracting the image semantic feature corresponding to the first image from the first image and the text semantic feature corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector includes the following steps: 331, inputting the first image into a preset contrastive language-image model to obtain the first image semantic feature vector; 332, determining the text corresponding to the first image in the first text to obtain a second text; 333, inputting the second text into the contrastive language-image model to obtain a second text semantic feature vector; 334, determining the vector dimension of the first image semantic feature vector to obtain an image semantic vector dimension; 335, performing vector scaling processing on the dimension of the second text semantic feature vector according to the image semantic vector dimension to obtain the first text semantic feature vector.

[0047] The first image semantic feature vector and the first text semantic feature vector are standardized feature vectors formed after model extraction and dimension adaptation, and the dimensions of the two are consistent, which can be directly used for subsequent fusion operations.

[0048] Specifically, the preset contrastive language image model adopts a pre-trained CLIP model, which includes image encoder and text encoder core components. The image encoder is built based on a Transformer architecture and has strong visual feature extraction and semantic aggregation capabilities. During execution, the preprocessed first image is first input into the image encoder of the CLIP model. The CLIP model extracts bottom-level texture features, middle-level contour features, and high-level semantic features from the image layer by layer through multi-layer convolution and self-attention mechanism, filters out irrelevant interference information such as background and lighting, and finally outputs a high-dimensional semantic vector (such as 512-dimensional) through a fully connected layer, i.e., a first image semantic feature vector. This vector represents the core identity visual information of the character in the first image and has good identity recognition and perspective independence. If the first text is a character feature description directly input by the user, it is directly used as the second text. If the first text contains irrelevant content such as background description and scene explanation, the identity description content is filtered out through a keyword matching algorithm (such as TF-IDF-based text keyword extraction) to form the second text, ensuring that the text semantic feature extraction focuses on identity-related information. After standardizing the second text (such as word segmentation, removing stop words, and adding special markers), it is input into the text encoder of the CLIP model. The text encoder converts text words into low-dimensional word vectors through a word embedding layer, and then models the semantic features through position encoding and a Transformer encoder to convert the natural language description into a high-dimensional semantic vector, i.e., a second text semantic feature vector. Then, by reading the output layer configuration of the image encoder of the CLIP model, the fixed dimension (such as 512-dimensional) of the first image semantic feature vector is obtained, which is determined as the image semantic vector dimension. This dimension serves as a reference for adapting the dimension of the text semantic feature vector. Then, a lightweight fully connected layer (MLP) is used for vector scaling. The input dimension of the fully connected layer is the original dimension of the second text semantic feature vector, and the output dimension is set to the image semantic vector dimension. Through linear transformation and activation function combination, the dimension of the second text semantic feature vector is adjusted and the features are reconstructed. On the premise of keeping the core semantic information intact, the second text semantic feature vector is mapped to the same dimension space as the first image semantic feature vector, obtaining a first text semantic feature vector, ensuring the consistency of the two types of semantic feature vectors, and laying a foundation for subsequent fusion operations.

[0049] It can be seen that the dual-modal semantic feature extraction process realizes the accurate capture of visual and text identity information through the CLIP model. The text filtering step improves the relevance of semantic features, and the dimension scaling process ensures the compatibility of the two types of feature vectors. The process effectively integrates the concrete visual features of the image and the accurate semantic features of the text, significantly improves the recognition and robustness of the identity features, and can resist the interference of factors such as perspective change and image occlusion, reducing the risk of identity drift.

[0050] Step S340, dimensionally aligning the first image parameter vector with the first image semantic feature vector to obtain a second image parameter vector.

[0051] wherein the first image parameter vector is a three-dimensional character image parameter extracted by a DECA model, including posture, shape, expression and other geometric information, the dimension of which is determined by the structure of a parameterized three-dimensional deformable face model (3DMM); the first image semantic feature vector is extracted by a CLIP model image encoder, and the dimension thereof is a fixed semantic space dimension in the pre-training process of the CLIP model. Their dimensions may be different, and if directly fused, it will cause feature information dilution, space mapping imbalance, and further affect the precise use of conditional information by the subsequent model, so dimension alignment is needed to ensure the effectiveness of fusion.

[0052] Specifically, a fully connected layer (MLP) is used as the core neural network structure for dimension alignment, which maps the first image parameter vector to the dimension space where the first image semantic feature vector is located. In fact, the fully connected layer is essentially equivalent to an encoder, which performs dimension conversion and feature reconstruction on the first image parameter vector through linear transformation and combination of nonlinear activation functions. To ensure the semantic consistency of the aligned features, a double optimization objective is introduced during the training of the fully connected layer. One is the reconstruction loss, that is, the original three-dimensional image parameters are reconstructed by the aligned vector to ensure that the geometric information is not lost; the other is the similarity loss, which makes the aligned vector and the first image semantic feature vector maintain semantic similarity in the target space, ensuring the compatibility of the two types of features. Through this training method, the fully connected layer can accurately learn the mapping relationship between the three-dimensional geometric parameters and the identity semantic feature space, and output a second image parameter vector with the same dimension as the first image semantic feature vector. Preferably, the dimension alignment process can be performed in coordination with the two-stage training strategy of the diffusion model. In the first stage of training, the initial parameters of the fully connected layer are fixed, and only the mapping relationship between the not fully optimized aligned features and the image generation is learned by the diffusion model; when the diffusion model preliminarily establishes the association between the three-dimensional parameters and the generation result, the parameters of the fully connected layer are unfrozen in the later stage of the first stage of training, and are jointly fine-tuned combined with the reconstruction loss and the similarity loss to ensure the alignment accuracy adapts to the training progress of the model. In the second stage of training, multi-view data is introduced, and the parameters of the fully connected layer are fixed, and only the parameters of the diffusion model are fine-tuned, so that the aligned features can better adapt to the spatial consistency learning requirement. This collaborative training method not only ensures the accuracy of dimension alignment, but also avoids interference of the alignment process on the stability of the overall model training, further improving the effectiveness of feature fusion and the comprehensive performance of the generated results.

[0053] Step S350, fusing the second image parameter vector, the first image semantic feature vector and the first text semantic feature vector to obtain a fused feature vector.

[0054] The second image parameter vector, after dimension alignment, completely retains the three-dimensional geometric information of the posture, shape and expression of the character, providing bottom-layer spatial structure constraints for generating images; the first image semantic feature vector captures core visual identity information independent of the viewing angle, ensuring the embodiment of identity features; and the first text semantic feature vector enhances the identity uniqueness through semantic description, forming a supplement and verification of the visual features.

[0055] Specifically, based on an attention weight distribution mechanism, the three types of feature vectors are dynamically assigned weights, the correlation coefficients of the vectors and the generation task are calculated, a higher basic weight is assigned to the second image parameter vector, and adaptive weights are assigned to the first image semantic feature vector and the first text semantic feature vector. Then, the three types of weighted vectors are added at the element level to realize the fusion of basic information, and the unique semantic information of each vector is retained through feature splicing to avoid loss of details in the fusion process. Finally, the spliced vector is input into a light full connection layer for feature reconstruction, to eliminate the redundant information and semantic conflicts between different modal features, and output a unified dimension and condensed semantic fusion feature vector.

[0056] It can be seen that the fusion feature vector integrates the three-dimensional geometric information of the second image parameter vector, so that the subsequent diffusion model has clear spatial structure constraints when generating images, effectively solving the problems of face structure distortion and asymmetry in existing technologies when the posture changes, and significantly improving the structural rationality of the generated images under multiple viewing angles such as ±70° side view and elevation view. On the other hand, the integration of dual-modal semantic features enables the fusion vector to have strong identity recognition, which can maintain the identity features of the character when the three-dimensional parameters are dynamically adjusted (such as posture rotation and expression change), greatly reducing the risk of identity drift.

[0057] In one possible embodiment, the second image parameter vector, the first image semantic feature vector and the first text semantic feature vector are fused to obtain a fusion feature vector, specifically including the following steps: 351、obtain a channel attention weight and a spatial position attention weight of a preset spatial attention mechanism; 352、normalize the second image parameter vector, the first image semantic feature vector and the first text semantic feature vector to obtain a third image parameter vector, a second image semantic feature vector and a second text semantic feature vector; 353、multiply the third image parameter vector by the spatial position attention weight to obtain a spatial image feature vector; 354、multiply the channel attention weight with the second image semantic feature vector and the second text semantic feature vector respectively to obtain a target image semantic feature vector and a target text semantic feature vector; 355、fuse the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector based on a preset vector fusion formula to obtain the fusion feature vector.

[0058] wherein the spatial attention mechanism is an attention mechanism capable of capturing feature space position and channel importance, including channel attention weight (used to measure the importance of different feature channels) and spatial position attention weight (used to locate the key spatial position of the feature), so as to realize the strengthening of the key feature and the suppression of the redundant information. The preset vector fusion formula is a feature integration formula designed based on weighted summation, which can realize semantic cooperation and information complementation of multiple types of features.

[0059] Specifically, the spatial attention mechanism is based on pre-trained parameter initialization, and the channel attention weight is calculated through a squeeze-excitation (SE) module. The module adaptively learns the importance weight of each channel by global average pooling and nonlinear transformation of the feature channel, so as to highlight the feature channels related to three-dimensional structure perception and identity recognition. The spatial position attention weight is generated through a spatial attention module. The module captures the spatial dependence of the features by global pooling and convolution operation on the feature map, so as to locate the feature position weight corresponding to the key region of the face. The two types of weights are continuously iteratively optimized in the model training process to ensure the adaptability to the feature fusion task. Preferably, the normalization processing adopts the LayerNormalization method, which maps the three types of feature vectors to the [0, 1] interval by calculating the mean and variance of each vector, eliminates the scale difference between different features (such as the magnitude difference between the geometric quantization value of the second image parameter vector and the semantic representation value of the semantic feature vector), avoids the dilution of certain type of feature information in the fusion process, and ensures that the three types of features have equal weight expression ability. Then, the spatial positioning ability of the spatial position attention weight is used to weight and enhance the normalized third image parameter vector, so that the three-dimensional geometric features (pose, shape) are strengthened in the key spatial position of the face, forming a spatial image feature vector with geometric integrity and spatial targeting, and the channel attention weight is used to select and strengthen the double-modal semantic feature vector, retaining the semantic channel information strongly related to the identity feature and suppressing irrelevant channel interference such as background and noise, so that the target image semantic feature vector focuses more on the visual identity information of the face, and the target text semantic feature vector more accurately reflects the identity-related semantic description, improving the identity recognition of the two types of semantic features. The preset vector fusion formula is: fusion feature vector = a x spatial image feature vector + b x target image semantic feature vector + g x target text semantic feature vector, wherein a, b and g are preset weight coefficients (the initial values are respectively set to 0.5, 0.3 and 0.2, and are adaptively adjusted in the training process), and the semantic synergy of the three types of features is realized by weighted summation.

[0060] It can be seen that the spatial attention mechanism realizes the strengthening of the key features, the normalization processing eliminates the feature scale difference, and the weighted fusion formula realizes the complementary advantages of the multi-modal features, effectively integrating the spatial constraints of the three-dimensional geometric features and the identity recognition of the double-modal semantic features, significantly improving the information density and expression ability of the fusion feature vector, so that the subsequent diffusion model can simultaneously respond to the three-dimensional structure control and identity consistency requirements.

[0061] In one possible embodiment, the preset vector fusion formula is used to fuse the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector to obtain the fusion feature vector, specifically including the following steps: 3551、calculate the similarity between the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector, to obtain a first similarity, a second similarity and a third similarity; 3552、determine the weight coefficients corresponding to the first similarity, the second similarity and the third similarity, to obtain a first weight coefficient, a second weight coefficient and a third weight coefficient; 3553、determine the fusion feature vector according to the first weight coefficient, the second weight coefficient, the third weight coefficient and the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector.

[0062] wherein, the vector similarity refers to the correlation degree of the two types of feature vectors in the semantic space through quantitative calculation, reflecting the information redundancy and complementarity, which can be calculated by cosine similarity, the value range is [-1, 1], the closer to 1, the stronger the vector semantic correlation; the weight coefficient is the feature contribution weight based on the similarity result, used to adjust the influence degree of different feature vectors in the fusion process, to ensure the matching of information contribution and feature correlation.

[0063] Specifically, the first similarity is the similarity between the spatial image parameter vector and the target image semantic feature vector, used to measure the correlation degree of three-dimensional geometric features and visual identity features; the second similarity is the similarity between the spatial image parameter vector and the target text semantic feature vector, representing the semantic fit degree of three-dimensional geometric features and text identity description; the third similarity is the similarity between the target image semantic feature vector and the target text semantic feature vector, reflecting the information redundancy between the two modal identity features. In the calculation, the ratio of the dot product and the modulus product of the three groups of vectors is calculated by the cosine similarity formula respectively, to obtain the normalized similarity result; the determination of the weight coefficient is based on the complementary principle of high similarity and low weight, low similarity and high weight, that is, the higher the similarity of the two types of vectors, the greater the information redundancy, the lower the weight coefficient; the lower the similarity, the stronger the information complementarity, the higher the weight coefficient, while ensuring that the sum of the three types of weight coefficients is 1. In specific calculation, the three groups of similarities are first normalized in reverse, and then the reverse similarity is taken as the basis for weight allocation, and the first weight coefficient (weight of spatial image parameter vector), the second weight coefficient (weight of target image semantic feature vector) and the third weight coefficient (weight of target text semantic feature vector) are obtained by normalizing through the Softmax function. The preset vector fusion formula is: fusion feature vector = first weight coefficient x spatial image parameter vector + second weight coefficient x target image semantic feature vector + third weight coefficient x target text semantic feature vector. Then, the three groups of weight coefficients obtained are multiplied with the corresponding feature vectors respectively at the element level, and the product results are summed to obtain the fusion feature vector with unified dimension and semantic fusion.

[0064] It can be seen that by quantifying the feature correlation degree through vector similarity, adaptive allocation of weight coefficients is realized, effectively balancing the information contribution of three-dimensional geometric features and dual-mode identity features, highlighting the core role of three-dimensional structure constraint of spatial image parameter vector, and strengthening the identity recognition degree through complementary fusion of dual-mode identity features, avoiding information redundancy and weakening of core features, making the fusion feature vector have stronger adaptive ability and robustness, further improving the rationality of three-dimensional structure, identity consistency and visual reality of the generated image, effectively alleviating the problems of structure distortion and identity drift in the prior art.

[0065] Consistent with the above Figure 3 embodiments, please refer to Figure 4 , Figure 4 is a role image generation diffusion model pre-training method flowchart provided by the embodiment of the present application. It can be seen that before the fusion feature vector is input into the preset role image generation diffusion model to obtain a second image, the method specifically further includes the following steps: S401, acquiring an image data set; S402, inputting the image data set into a preset variational autoencoder model to obtain an image space feature vector; S403, inputting the image space feature vector into a preset image generation diffusion model for training to obtain a first image generation diffusion model; S404, fine-tuning the first image generation diffusion model according to the fusion feature vector to obtain the role image generation diffusion model.

[0066] The image dataset is a public dataset of human face images for pre-training of the model, preferably, the FFHQ public dataset is used, which contains 70,000 high-quality human face images with a resolution of 1024x1024, covering various diversity characteristics such as age, race, background, and accessories (such as glasses, hats, etc.). In the preprocessing stage, the dlib library is used for face detection, key point positioning, and image alignment, and an automatic filtering strategy is used to exclude non-real human face images (such as sculptures, paintings, etc.), to ensure the quality of the input data. The variational autoencoder model (VAE) is a generative model with feature extraction and dimension compression capabilities, preferably, the preset variational autoencoder model uses the VQ-VAE model (vector quantization variational autoencoder), which can map images to a low-dimensional latent space and generate discrete image space feature vectors. The image generation diffusion model is a basic generative network based on the implicit diffusion framework, which realizes the generation from noise vectors to images through an iterative denoising process, preferably, it can be a latent diffusion model (Latent Diffusion Model, LDM); the character image generation diffusion model is a special diffusion model with three-dimensional condition perception and identity feature preservation capabilities after fine-tuning of the fused feature vectors, which can respond to the geometric constraints and identity information in the fused features.

[0067] Specifically, the preset variational autoencoder model selects a pre-trained VQ-VAE model. The model maps image data from pixel space to low-dimensional hidden space through an encoder, quantizes and compresses image features, and generates image space feature vectors with uniform dimensions and concise semantics. The advantage of VQ-VAE is that the hidden space features have discreteness and interpretability through the vector quantization mechanism, which not only reduces the training complexity of the subsequent diffusion model, but also effectively preserves the details of the image (such as skin texture and hair texture), laying the foundation for generating high-fidelity images. The image generation diffusion model is built based on the implicit diffusion framework, and the training process is carried out in the hidden space of the VQ-VAE. During training, the image space feature vector is taken as the target, and by iteratively adding Gaussian noise and learning the denoising process, the model learns the mapping relationship from noise vector to high-quality image hidden features. The training optimization uses the Adam algorithm, with a learning rate of 1e-5. It is implemented on an NVIDIA RTX2080 Ti based on the PyTorch framework. Finally, the first image generation diffusion model with basic face generation capability is obtained. This model can synthesize high-resolution and visually realistic face images, but it still lacks three-dimensional perception and identity consistency control capability. The fine-tuning process uses the Lora (Low-Rank Adaptation) lightweight fine-tuning strategy, with the fusion feature vector as a conditional constraint, freezing most parameters of the first image generation diffusion model, and only training the low-rank matrix parameters related to conditional injection. The fine-tuning goal is to learn the relationship between the fusion feature vector (containing three-dimensional geometric parameters and dual-modal identity features) and image generation. Specifically, the fusion feature vector is injected into the attention layer of the model layer by layer through the spatial attention mechanism, so that the model can accurately respond to geometric constraints such as pose, shape, and expression, while maintaining the stability of the identity features. During fine-tuning, reconstruction loss and identity similarity loss are introduced as optimization objectives to ensure that the generated results meet the three-dimensional parameter constraints and maintain consistency with the identity features of the reference image. Finally, the role image generation diffusion model that meets the task requirements is obtained.

[0068] It can be seen that the selection and preprocessing of the FFHQ dataset guarantee the basic performance and generalization ability of the pre-trained model, the hidden space mapping of the VQ-VAE reduces the training complexity and preserves the detailed features, the pre-training of the diffusion model in the hidden space enables the model to master the ability of high-quality face generation, and the targeted fine-tuning of the fusion feature vector injects the key functions of three-dimensional condition perception and identity feature preservation into the model. Compared with the scheme of directly training a special diffusion model, the manner of the embodiment not only makes full use of the value of a large-scale general-purpose dataset and reduces the dependence on special labeled data, but also realizes the precise adaptation of the functions of the model through light-weight fine-tuning, guarantees the training efficiency and generation performance, and enables the final role image generation diffusion model to efficiently respond to the constraints of the fusion feature vector, thereby providing core model support for subsequent generation of a second image with three-dimensional perception, consistent identity and high fidelity.

[0069] In a possible embodiment, the role image generation diffusion model is obtained by fine-tuning a first image generation diffusion model according to the fusion feature vector, and specifically includes the following steps: A1, inputting the fusion feature vector into a preset multi-layer perception neural network model to obtain a conditional feature vector; A2, determining a parameter layer of the first image generation diffusion model that needs to participate in fine-tuning to obtain a target fine-tuning parameter layer; A3, inputting the image space feature vector and the conditional feature vector into the target fine-tuning parameter layer to obtain a fine-tuning parameter set; A4, updating corresponding parameters in the first image generation diffusion model according to the fine-tuning parameter set to obtain the role image generation diffusion model.

[0070] The multi-layer perception neural network model (MLP) is a deep learning model composed of fully connected layers and has feature conversion and dimension adaptation capabilities, and is used to map the fusion feature vector to a conditional feature suitable for the diffusion model. The target fine-tuning parameter layer refers to a key parameter layer related to conditional injection in the first image generation diffusion model, which can strengthen the response of the model to three-dimensional geometric constraints and identity features while preserving the basic generation ability through targeted fine-tuning. The fine-tuning parameter set is an optimized parameter set obtained by updating the target fine-tuning parameter layer during training.

[0071] Specifically, the preset multi-layer perception neural network model adopts a three-layer full connection structure, the input dimension is the dimension of the fusion feature vector, and the output dimension is consistent with the hidden space dimension of the first image generation diffusion model. During the training process, the multi-layer perception neural network model cooperates with the fine-tuning process to optimize, and by minimizing the identity difference loss of the generated result and the reference image, it is ensured that the conditional feature vector can accurately transmit the three-dimensional geometric constraint and identity feature information. The first image generation diffusion model is constructed based on an implicit diffusion framework, and the network structure thereof includes an encoder, a decoder, and multiple attention layers, wherein the attention layers (especially the spatial attention layers) are the core carriers of the conditional information injection. Therefore, the target fine-tuning parameter layer mainly includes the cross-attention layer, the spatial attention layer, and part of the full connection layer of the output layer in the diffusion model, and these parameter layers directly participate in the feature interaction and the decision process of the generated result. During the screening process, the basic feature extraction layer (such as the bottom convolutional layer of the encoder) of the frozen model is only kept for fine-tuning, which not only avoids the degradation of the basic generation capability, but also reduces the computational complexity of the fine-tuning, thereby realizing lightweight optimization. By using the Lora lightweight fine-tuning strategy, low-rank matrices are introduced into the target fine-tuning parameter layer to realize the cross-modal interaction between the conditional feature vector and the image spatial feature vector. Specifically, the image spatial feature vector is taken as the basic generation feature, and the conditional feature vector is taken as the constraint guide feature, which are input into the target fine-tuning parameter layer together, and the semantic correlation between the two is established through the spatial attention mechanism. During the training process, the reconstruction loss (which measures the structural similarity between the generated result and the reference image) and the identity similarity loss (which calculates the identity feature similarity between the generated result and the reference image based on the CLIP model) are taken as the optimization objectives, the Adam optimization algorithm is used to iteratively update the parameters of the target fine-tuning parameter layer, and finally the optimal fine-tuning parameter set is obtained. Then, the obtained fine-tuning parameter set is replaced with the original parameters of the target fine-tuning parameter layer in the first image generation diffusion model according to the parameter layer correspondence. During the updating process, the parameters of the non-target fine-tuning parameter layer remain unchanged, so as to ensure that the model retains the high-fidelity face generation capability learned in the pre-training stage. After the updating is completed, the model can simultaneously respond to the basic generation demand of the image spatial feature and the constraint demand of the conditional feature vector during the generation process, thereby realizing the target of three-dimensional structure controllability and identity feature consistency. In addition, the fine-tuned model generates in the VQ-VAE hidden space, thereby inheriting the efficiency and detail retention capability of the implicit diffusion model, and solving the problem of lacking three-dimensional perception and identity constraint of the basic model.

[0072] In step S360, the fusion feature vector is input into a preset role image generation diffusion model to obtain a second image.

[0073] The preset role image generation diffusion model is constructed based on an implicit diffusion framework, and a VQ-VAE model is used to map an image to a latent space for training and generation. This design can significantly improve the generation efficiency and detail retention capability. The fusion feature vector is used as the conditional input of the model, the second image parameter vector contained in the fusion feature vector provides a three-dimensional space structure constraint, the first image semantic feature vector and the first text semantic feature vector guarantee identity consistency, and the three types of information cooperatively guide the denoising process of the diffusion model, so that the generated second image not only follows the preset posture, shape and expression parameters, but also maintains the identity features consistent with the reference image.

[0074] Specifically, first, the fusion feature vector is injected into the attention layer of the diffusion model layer by layer through the spatial attention mechanism, and interacts with the noise vector in the latent space in a cross-modal manner. The spatial attention mechanism can accurately locate the three-dimensional geometric information and identity semantic information in the fusion feature vector, associate it with the local features of the image latent space, and enhance the model's perception of the three-dimensional structure. Then, the role image generation diffusion model performs an iterative denoising process in the VQ-VAE latent space. Based on the mapping relationship between the three-dimensional parameters learned in the pre-training process and the image generation, the clear latent features that meet the fusion feature constraints are gradually restored from the noise vector. During the denoising iteration, the fusion feature vector continuously provides constraints: the second image parameter vector ensures that the posture, shape and expression of the generated image meet the preset parameters, and the dual-modal semantic feature vector real-time calibrates the identity features to avoid identity drift or detail distortion during the generation process. Finally, after denoising for a preset number of iterations, the feature vector in the latent space is reconstructed into a pixel-level image through the decoder of the VQ-VAE, and a second image with three-dimensional perception, identity consistency and high fidelity is obtained. It should be noted that the diffusion model has completed pre-training through a two-stage training strategy: the first stage establishes the mapping relationship between the three-dimensional parameters and the image generation on the FFHQ single-view dataset, and the second stage introduces multi-view image data to fine-tune the model to ensure the spatial geometry consistency of the generated image.

[0075] As can be seen, through the combination of the implicit diffusion framework and the spatial attention mechanism, the fusion feature vector can accurately control the generation process. In addition, the introduction of the VQ-VAE latent space not only improves the generation efficiency, but also effectively retains the detail information such as skin texture and hair quality, solving the problem of lack of high-fidelity detail expression in traditional parameterized models.

[0076] For ease of understanding, please refer to Figure 5 , Figure 5is a flowchart of another role image generation method based on a diffusion model provided by the embodiment of the present application. As can be seen, the embodiment implements the generation of a high-fidelity, identity-consistent three-dimensional perceptual role image from a two-dimensional person image through the processes of data acquisition, three-dimensional parameter extraction, semantic feature acquisition, feature alignment and fusion, model generation, and quality evaluation, forms a standardized system from original data input to generated result quantitative evaluation, and can ensure the collaborative optimization of three-dimensional structure rationality and identity feature stability in the process of person portrait generation, thereby providing a high-quality role image generation scheme for the field of digital people, virtual reality, etc.

[0077] Specifically, first is step S1, obtaining character image data, this step collects high-quality image data containing complete facial features of the character, ensures that the image resolution is not less than 1024*1024 and there is no serious obstruction, blur and other interference, provides reliable data source for subsequent extraction of three-dimensional parameters and semantic features. After obtaining the character image data, step S2 is performed, three-dimensional face parameters of the character image are extracted based on the DECA model, the DECA model is a high-precision three-dimensional deformable face parameter extraction network, the texture, contour and shading changes of the input image are analyzed, the pose parameters, shape parameters and expression parameters are decoupled, respectively corresponding to the spatial orientation of the character's head, the inherent geometric features of the face and the dynamic expression state, realizing the quantitative conversion from two-dimensional image to three-dimensional geometric information. At the same time, step S3 is performed, the text embedding vector of the image is extracted based on the CLIP model, the CLIP model is a cross-modal pre-training model, the image encoder of which extracts a high-dimensional semantic vector from the character image, which includes facial texture details, facial feature ratios, unique accessories and other visual information strongly bound with identity, and has perspective independence, providing core semantic support for the stable preservation of identity features in the subsequent generation process. After completing the extraction of three-dimensional parameters and semantic features, step S4 is performed, aligning and fusing the three-dimensional parameters and the text embedding vector, this step maps the three-dimensional face parameters output by the DECA model to the dimension space of the text embedding vector of the CLIP model through a fully connected layer, realizing the dimension alignment of the two types of features, then using a spatial attention mechanism and a weighted fusion strategy, integrating the three-dimensional geometric parameters and the text embedding vector into a unified fusion feature vector, which not only preserves the three-dimensional structure constraint, but also strengthens the identity semantic information. Then, step S5 is entered, generating a character portrait image based on a diffusion model and performing a two-stage training strategy and fine-tuning the model, the fusion feature vector is injected into the preset role image generation diffusion model through a spatial attention mechanism, the model is optimized through a two-stage training strategy of pre-training and conditional fine-tuning: the first stage completes the pre-training of high-fidelity face generation capability on the FFHQ dataset, and the second stage is based on the fusion feature vector for light-weight fine-tuning, so that the model can accurately respond to three-dimensional geometric constraints and identity semantic information, and finally generate a character portrait image with three-dimensional perception ability and identity consistency. Finally, step S6 is performed, the reconstruction quality and rendering speed are quantitatively evaluated based on indicators such as PSNR, SSIM and LPIPS, the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) are used to measure the visual similarity between the generated image and the original image, the learned perceptual image block similarity (LPIPS) is used to evaluate the identity feature consistency, and the rendering speed is recorded to verify the practicability of the scheme, through multiple indicators to ensure the high fidelity of the generated results.

[0078] For ease of understanding, please refer to Figure 6 , Figure 6Figure 6 is an architecture diagram of a role image generation method based on a diffusion model provided by an embodiment of the present application. The architecture diagram 600 of the role image generation method based on the diffusion model shows the model architecture of the present method, as follows: Figure 6As shown, the architecture takes a Latent Diffusion Model as the core, and realizes the generation of role images with three-dimensional perception ability and semantic consistency by introducing three-dimensional face geometry parameters and cross-modal semantic features. Specifically, the input first image is first sent to the DECA three-dimensional face parameter extraction module. DECA is used to recover the corresponding three-dimensional face geometry information from the input portrait image of the character, and output the Flame parameters related to the face geometry. The Flame parameters are used to represent the three-dimensional shape, pose, and expression of the face, and are a parameterized description of the three-dimensional structure of the face. In this embodiment, the Flame parameters, as important constraint information of the role identity and geometry, are used to avoid identity drift or distortion of the face structure during the generation process of the diffusion model. Before entering the diffusion model, the Flame parameters are further input into a multi-layer perception MLP, which is used for encoding and mapping of the Flame parameters, converting the original three-dimensional geometry parameters into feature representations matching the conditional input space of the diffusion model. Through this mapping process, the three-dimensional geometry information can participate in the subsequent diffusion generation process in the form of conditional features, thereby introducing three-dimensional constraints into the role image generation without modifying the underlying structure of the diffusion model. At the same time, the input text description is sent to the CLIP cross-modal semantic extraction module, which is used for semantic encoding of the text to obtain the corresponding text semantic feature representation. In this embodiment, the text content "the woman's lips are slightly open, wearing lipstick, wearing earrings, and she is smiling" is given. This text is used to describe the appearance attributes and expression features of the target role, and through the CLIP model, the text information is mapped into a feature vector consistent with the image semantic space, so as to participate in the generation process of the diffusion model as a semantic condition. Further, the three-dimensional geometry condition features output by the MLP and the text semantic features output by the CLIP are jointly used as conditional inputs and sent to the latent diffusion model. Since the diffusion model is used to perform diffusion inversion in the latent space, it generates the latent space representation of the target role image through step-by-step denoising, and finally decodes to obtain the corresponding role image. Since the diffusion process is constrained by three-dimensional geometry conditions and semantic conditions, the generation result can maintain the consistency of the text description while maintaining the consistency of the character identity features and face structure. In terms of training strategy, different colors are used to distinguish the participation of each module in the training stage. Among them, participating in the first stage training means that the module participates in parameter updating in the first stage training process; fixed means that the module freezes the parameters during the training process; participating in all training means that the module participates in parameter updating in multiple training stages.For example, in the first stage of training, the diffusion model participates in training as a generative model, and the partial conditional coding module can remain fixed to enable the diffusion model to preferentially learn the basic face generation capability; in the subsequent training stage, the conditional features containing three-dimensional geometric constraints and semantic constraints are introduced to enable the model to gradually learn to generate role images that conform to the text description while maintaining identity consistency. Through the staged training strategy, the stability and generalization capability of the model under the conditions of multiple perspectives and multiple expressions can be effectively improved.

[0079] As can be seen, Figure 6 The architecture of the role image generation method based on the diffusion model shown in the figure achieves the collaborative constraint of three-dimensional geometric information and semantic information by jointly introducing the Flame parameters extracted by DECA and the text semantic features extracted by CLIP into the implicit diffusion model. Without changing the underlying network structure of the diffusion model, the architecture introduces additional conditional information, so that the generated role image is significantly improved in terms of structural rationality, identity consistency, and semantic matching degree, and can effectively overcome the problems of insufficient three-dimensional perception and unstable identity in the existing GAN or two-dimensional diffusion model in the generation of character portraits.

[0080] For ease of understanding, please refer to Figure 7 , Figure 7 is an effect comparison chart of generating character portraits by different methods provided by an embodiment of the present application. As can be seen, the first column to the fourth column in the effect comparison chart 700 of generating character portraits by different methods are the source image, the Zero-1-to-3 method, the InstanceID method, and the character portrait generation results under four conditions of the method of the present application, which are used to compare the differences in identity consistency, facial structure stability, and generation quality of different character portrait generation methods.

[0081] Specifically, the first column (Source) represents the input original portrait image of a person as a reference source image for various generation methods; the second column (Zero-1-to-3 method) represents the generation result obtained by using an existing method based on single-view to multi-view generation. It can be seen from the column that although the Zero-1-to-3 method can generate portraits of different perspectives or different forms to a certain extent, in some generation results, the face structure of the person has obvious distortion phenomenon, for example, the side profile is incomplete, the local area is missing or the overall shape is unstable, and it is difficult to maintain the identity features consistent with the source image. The third column (InstanceID method) represents the generation result of another portrait generation method based on identity constraint. It can be seen that the InstanceID method can better maintain the overall appearance style of the person in some samples, but under the condition of complex posture or expression change, there are still problems of weakening of identity features or inconsistency of local details, and in individual generation results, there are even cases of failure to successfully generate effective portraits, such as Figure 7 the result marked in the form of a placeholder ( Figure 7 in the second row and the third column). The fourth column (Ours) represents the generation result obtained by using the role image generation method based on the diffusion model provided in the embodiments of the present application. It can be seen that under different sample conditions, the portrait generated by the method has high consistency with the original person in the first column in terms of facial contour, facial feature structure and overall appearance, and can adapt to different posture and perspective changes, and the generation result is natural and coherent as a whole, without obvious face structure collapse or identity drift phenomenon. Further, from the comparison results of multiple groups, compared with the Zero-1-to-3 method and the InstanceID method, the embodiments of the present application introduce three-dimensional geometric constraints and semantic conditions into the diffusion model, so that the generation process not only depends on two-dimensional appearance feature matching, but also can combine the three-dimensional structure information of the person, so as to maintain the stability of the identity and face structure of the person under the condition of complex posture and perspective change.

[0082] Figure 7The qualitative comparison results of the generated images are shown. In order to more clearly show the effectiveness of the method proposed in the embodiment of the application, an ablation experiment is performed to evaluate from a quantitative aspect. Table 1 below records the results of each evaluation index in the ablation experiment. By comparing the numerical values of each evaluation index, it can be seen that the method of the embodiment of the application achieves higher effect than the baseline model on the FFHQ dataset. Regarding the ablation experiment of the text vector, the method of the embodiment of the application uses the text embedding vector to additionally provide the identity feature parameter. In order to evaluate the effectiveness of this scheme, a model that does not use the CLIP model is trained for comparison. It can be noted that the three indicators are all the lowest. Notably, removing the text condition does not significantly affect the generation quality in the naked eye, but significantly affects the maintenance of its identity feature, so it is the worst in each comparison. Ablation experiment of the two-step training strategy: a two-stage training strategy using Lora for fine-tuning. In order to evaluate its effectiveness, the model before the second stage training is compared with the model after the training. The quantitative analysis is shown in Table 1. It can be noted that the model without using the two-stage training has a significant gap in each indicator, which fully illustrates the necessity and rationality of the two-stage training of the embodiment of the application. Regarding the ablation experiment of the data enhancement of the two-stage training: a training method using multiple types of three-dimensional assets for data enhancement is proposed. In order to evaluate the effectiveness of this scheme, the method of the embodiment of the application trains a model without using this enhancement scheme in the second stage for comparison. The quantitative analysis is shown in Table 1. It can be noted that data enhancement is the least affected group of comparison experiments. This may be caused by the insufficient relevance between the data used for enhancement and the portrait of the person, but it still has some help, which reflects its effectiveness. The specific is shown in Table 1 as follows: Table 1 Quantitative comparison results of ablation experiment on FFHQ dataset

[0083] At the same time, in order to verify the superiority of the method of the embodiment of the application, the experimental results of the method of the embodiment of the application are compared with the experimental results of the existing method. As shown in Table 2, on the FFHQ dataset, the experimental results show that in the SSIM and LPIPS indicators, the method of the embodiment of the application is optimal in all image quality indicators. See Table 2 for details: Table 2 Quantitative comparison results of different methods on FFHQ dataset

[0084] It can be seen that by executing the role image generation method based on the diffusion model provided in the embodiment of the application, the fidelity of the generated role image can be improved.

[0085] The above describes the scheme of the embodiments of the present application mainly from the perspective of the process of executing the method. It can be understood that, in order to implement the above functions, the server comprises a hardware structure and / or a software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the unit examples and algorithm steps of the examples described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered as beyond the scope of the present application.

[0086] The embodiments of the present application can divide the functional units of the server according to the above method examples. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division. There can be another division method in actual implementation.

[0087] In the case of dividing each functional module corresponding to each function, Figure 8 is a functional module composition block diagram of a role image generation device based on a diffusion model provided by the embodiments of the present application, applied to a computing server in a role image generation system, the role image generation system further comprising a user terminal, the user terminal being in communication connection with the computing server, the role image generation device based on the diffusion model 800 comprising: a data acquisition unit 810 configured to acquire a first image and a first text input by the user terminal; a feature extraction unit 820 configured to extract a three-dimensional role image parameter from the first image to obtain a first image parameter vector, extract an image semantic feature corresponding to the first image from the first image, and extract a text semantic feature corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector; a feature fusion unit 830 configured to perform dimension alignment on the first image parameter vector and the first image semantic feature vector to obtain a second image parameter vector, and fuse the second image parameter vector, the first image semantic feature vector and the first text semantic feature vector to obtain a fused feature vector; an image generation unit 840 configured to input the fused feature vector into a preset role image generation diffusion model to obtain a second image.

[0088] In a possible embodiment, the feature extraction unit 820, in the aspect of extracting a three-dimensional character image parameter from the first image to obtain a first image parameter vector, specifically is configured to: perform character image recognition on the first image to obtain a first character image; perform processing on the first character image based on a preset perspective transformation method to obtain a second character image; extract a posture parameter, a shape parameter, and an expression parameter of a three-dimensional face parameter from the second character image based on a preset three-dimensional detailed expression capture model to obtain a first posture parameter, a first shape parameter, and a first expression parameter; perform standardization processing on the first posture parameter, the first shape parameter, and the first expression parameter respectively to obtain a second posture parameter, a second shape parameter, and a second expression parameter; splice the second posture parameter, the second shape parameter, and the second expression parameter according to a preset dimension splicing method to obtain the first image parameter vector.

[0089] In a possible embodiment, the feature extraction unit 820, in the aspect of extracting an image semantic feature corresponding to the first image from the first image and extracting a text semantic feature corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector, specifically is configured to: input the first image into a preset comparative language image model to obtain the first image semantic feature vector; determine a text corresponding to the first image in the first text to obtain a second text; input the second text into the comparative language image model to obtain a second text semantic feature vector; determine a vector dimension of the first image semantic feature vector to obtain an image semantic vector dimension; perform vector scaling processing on a dimension of the second text semantic feature vector according to the image semantic vector dimension to obtain the first text semantic feature vector.

[0090] In a possible embodiment, the feature fusion unit 830, in the aspect of fusing the second image parameter vector, the first image semantic feature vector, and the first text semantic feature vector to obtain a fused feature vector, specifically is configured to: obtain a channel attention weight and a spatial position attention weight of a preset spatial attention mechanism; normalize the second image semantic feature vector and the second text semantic feature vector respectively based on the channel attention weight, to obtain a target image semantic feature vector and a target text semantic feature vector; multiply the third image parameter vector and the spatial position attention weight to obtain a spatial image feature vector; multiply the channel attention weight and the second image semantic feature vector and the second text semantic feature vector respectively to obtain a target image semantic feature vector and a target text semantic feature vector; fuse the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector based on a preset vector fusion formula to obtain the fusion feature vector.

[0091] In a possible embodiment, the feature fusion unit 830 is specifically configured to: calculate the similarity between the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector to obtain a first similarity, a second similarity and a third similarity; determine the weight coefficients corresponding to the first similarity, the second similarity and the third similarity to obtain a first weight coefficient, a second weight coefficient and a third weight coefficient; determine the fusion feature vector according to the first weight coefficient, the second weight coefficient, the third weight coefficient and the spatial image parameter vector, the target image semantic feature vector and the target text semantic feature vector.

[0092] In a possible embodiment, the image generation unit 840 is specifically configured to: obtain an image data set; input the image data set into a preset variational autoencoder model to obtain an image space feature vector; input the image space feature vector into a preset image generation diffusion model for training to obtain a first image generation diffusion model; fine-tune the first image generation diffusion model according to the fusion feature vector to obtain the role image generation diffusion model.

[0093] In a possible embodiment, the image generation unit 840 is configured to fine-tune the first image generation diffusion model according to the fusion feature vector to obtain the character image generation diffusion model, and specifically configured to: input the fusion feature vector into a preset multi-layer perception neural network model to obtain a conditional feature vector; determine a parameter layer that needs to participate in fine-tuning in the first image generation diffusion model to obtain a target fine-tuning parameter layer; input the image space feature vector and the conditional feature vector into the target fine-tuning parameter layer to obtain a fine-tuning parameter set; update corresponding parameters in the first image generation diffusion model according to the fine-tuning parameter set to obtain the character image generation diffusion model.

[0094] It can be seen that the embodiment of the present application provides a character image generation device based on a diffusion model, which is applied to a computing server of a character image generation system. The character image generation system further includes a user terminal in communication connection with the computing server. The method includes: obtaining a first image and a first text input by the user terminal, extracting a three-dimensional character image parameter from the first image to obtain a first image parameter vector, extracting a first image semantic feature vector corresponding to the first image from the first image, and extracting a first text semantic feature vector corresponding to the first image from the first text. The first image parameter vector and the first image semantic feature vector are aligned in dimension to obtain a second image parameter vector, and are fused to obtain a fusion feature vector. The fusion feature vector is input into a preset character image generation diffusion model to obtain a second image. In this way, the fidelity of the generated character image can be improved.

[0095] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute part or all steps of any method described in the above method embodiments. The computer includes a server.

[0096] The embodiment of the present application further provides a computer program product, which includes a non-transitory computer readable storage medium storing a computer program. The computer program is operable to cause a computer to execute part or all steps of any method described in the above method embodiments. The computer program product can be a software installation package, and the computer includes a server.

[0097] It should be noted that, for the various embodiments described above, for the sake of simplicity, they are all described as a combination of a series of actions. Those skilled in the art should know that the present application is not limited by the order of the actions described, because some steps in the embodiments of the present application can be performed in other orders or at the same time. In addition, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions, steps, modules or units involved are not necessarily required in the embodiments of the present application.

[0098] In the above embodiments, the description of each embodiment of the present application has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0099] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: ROM or random access memory (RAM), magnetic disk or optical disk, and various program code storage media.

[0100] The steps of the method or algorithm described in the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, electrically EPROM (EEPROM), register, hard disk, mobile hard disk, compact disc read-only memory (CD-ROM), or any other form of storage medium well known in the art.

[0101] Each module / unit contained in each device / product described in the above embodiments can be a software module / unit, a hardware module / unit, or part of a software module / unit and part of a hardware module / unit. For example, for each device / product applied to or integrated into a chip, each module / unit contained therein can be implemented in the form of a circuit or other hardware, or at least part of the modules / units can be implemented in the form of a software program running on a processor integrated in the chip, and the remaining (if any) part of the modules / units can be implemented in the form of a circuit or other hardware.

[0102] The above detailed description of the specific implementation has further detailed the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above description is only a specific implementation of the embodiments of the present application and is not used to limit the protection scope of the embodiments of the present application. Any modification, equivalent replacement, improvement and the like made on the basis of the technical solutions of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A character image generation method based on a diffusion model, characterized in that, A computing server is used in a character image generation system, the character image generation system further includes a user terminal, the user terminal being communicatively connected to the computing server, the method comprising: Acquire the first image and the first text input by the user terminal; Extract the three-dimensional character image parameters from the first image to obtain the first image parameter vector; The image semantic features corresponding to the first image are extracted from the first image, and the text semantic features corresponding to the first image are extracted from the first text to obtain the first image semantic feature vector and the first text semantic feature vector; Align the first image parameter vector with the first image semantic feature vector in terms of dimensions to obtain the second image parameter vector. The second image parameter vector, the first image semantic feature vector, and the first text semantic feature vector are fused to obtain a fused feature vector. The fused feature vector is input into a preset character image generation diffusion model to obtain a second image.

2. The method as described in claim 1, characterized in that, The step of extracting three-dimensional character image parameters from the first image to obtain a first image parameter vector includes: Perform character image recognition on the first image to obtain the first character image; The first character image is processed based on a preset perspective transformation method to obtain the second character image; Based on the preset 3D detail expression capture model, the 3D face parameters of the second character image are extracted to obtain the first pose parameters, the first shape parameters and the first expression parameters. The first posture parameter, the first shape parameter, and the first expression parameter are standardized to obtain the second posture parameter, the second shape parameter, and the second expression parameter. The second pose parameter, the second shape parameter, and the second expression parameter are spliced ​​together according to a preset dimensional splicing method to obtain the first image parameter vector.

3. The method as described in claim 1, characterized in that, The step of extracting image semantic features corresponding to the first image from the first image and extracting text semantic features corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector includes: The first image is input into a preset contrastive language image model to obtain the semantic feature vector of the first image; Determine the text corresponding to the first image in the first text to obtain the second text; The second text is input into the contrastive language image model to obtain the semantic feature vector of the second text; Determine the vector dimension of the first image semantic feature vector to obtain the image semantic vector dimension; The first text semantic feature vector is obtained by scaling the dimension of the second text semantic feature vector according to the dimension of the image semantic vector.

4. The method according to any one of claims 1-3, characterized in that, The process of fusing the second image parameter vector, the first image semantic feature vector, and the first text semantic feature vector to obtain a fused feature vector includes: Obtain the channel attention weights and spatial location attention weights of the preset spatial attention mechanism; The second image parameter vector, the first image semantic feature vector, and the first text semantic feature vector are normalized respectively to obtain the third image parameter vector, the second image semantic feature vector, and the second text semantic feature vector. Multiply the third image parameter vector by the spatial location attention weight to obtain the spatial image feature vector; The channel attention weights are multiplied by the second image semantic feature vector and the second text semantic feature vector, respectively, to obtain the target image semantic feature vector and the target text semantic feature vector; The spatial image parameter vector, the target image semantic feature vector, and the target text semantic feature vector are fused based on a preset vector fusion formula to obtain the fused feature vector.

5. The method as described in claim 4, characterized in that, The process of fusing the spatial image parameter vector, the target image semantic feature vector, and the target text semantic feature vector based on a preset vector fusion formula to obtain the fused feature vector includes: Calculate the similarity between each pair of the spatial image parameter vector, the target image semantic feature vector, and the target text semantic feature vector to obtain the first similarity, the second similarity, and the third similarity; Determine the weight coefficients corresponding to the first similarity, the second similarity, and the third similarity to obtain the first weight coefficient, the second weight coefficient, and the third weight coefficient; The fused feature vector is determined based on the first weight coefficient, the second weight coefficient, the third weight coefficient, the spatial image parameter vector, the target image semantic feature vector, and the target text semantic feature vector.

6. The method according to any one of claims 1-3, characterized in that, Before inputting the fused feature vector into a preset character image generation diffusion model to obtain the second image, the method further includes: Obtain the image dataset; The image dataset is input into a preset variational autoencoder model to obtain image spatial feature vectors; The image spatial feature vector is input into a preset image generation and diffusion model for training to obtain a first image generation and diffusion model; The first image generation diffusion model is fine-tuned based on the fused feature vector to obtain the character image generation diffusion model.

7. The method as described in claim 6, characterized in that, The step of fine-tuning the first image generation diffusion model based on the fused feature vector to obtain the character image generation diffusion model includes: The fused feature vector is input into a preset multilayer perceptron neural network model to obtain a conditional feature vector; Determine the parameter layers in the first image generation diffusion model that need to be fine-tuned to obtain the target fine-tuning parameter layers; The image spatial feature vector and the conditional feature vector are input into the target fine-tuning parameter layer to obtain the fine-tuning parameter set; The corresponding parameters in the first image generation diffusion model are updated according to the fine-tuning parameter set to obtain the character image generation diffusion model.

8. A character image generation device based on a diffusion model, characterized in that, A computing server used in a character image generation system, the character image generation system further including a user terminal, the user terminal being communicatively connected to the computing server, the device comprising: The data acquisition unit is used to acquire the first image and the first text input by the user terminal; The feature extraction unit is used to extract three-dimensional character image parameters from the first image to obtain a first image parameter vector; extract image semantic features corresponding to the first image from the first image, and extract text semantic features corresponding to the first image from the first text to obtain a first image semantic feature vector and a first text semantic feature vector. The feature fusion unit is used to align the first image parameter vector with the first image semantic feature vector in terms of dimensions to obtain a second image parameter vector; and to fuse the second image parameter vector, the first image semantic feature vector and the first text semantic feature vector to obtain a fused feature vector. The image generation unit is used to input the fused feature vector into a preset character image generation diffusion model to obtain a second image.

9. A server, characterized in that, include: Processor, memory, communication interface, and one or more programs; The one or more programs are stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images