Image redrawing method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image restoration, and provides an image redrawing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: carrying out latent space feature extraction and noise addition processing on a to-be-redrawn image to obtain a noise feature of the to-be-redrawn image; performing visual feature extraction on the reference image to obtain visual features of the reference image, and performing semantic feature extraction on the prompt text to obtain semantic features of the prompt text; performing feature fusion on the semantic feature of the prompt text and the visual feature of the reference image to obtain a multi-modal feature and an adaptive feature of the reference image; diffusion processing is performed based on the noise feature and the multi-modal feature of the to-be-redrawn image and the adaptive feature of the reference image to obtain a target feature, and the target feature is decoded to obtain a target image corresponding to the to-be-redrawn image, so that the limitation of single-modal guidance during local redrawing of the image is solved, and the image redrawing efficiency is improved. And the diversity and quality of the generated target image are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image restoration, and in particular to an image redrawing method, device, electronic device and readable storage medium. Background Art

[0002] In the field of image restoration, image restoration technology based on diffusion models has gradually made significant progress and can be mainly divided into two categories: training-based methods and few-shot methods. The training-based image restoration method is achieved by training a diffusion model from image to image, using a pre-trained generative diffusion model and combined with control conditions, such as mask images, to optimize specific tasks. This method is fast in the inference stage, but requires a large-scale training data set and has difficulties in extending to guidance of different modalities. The few-shot image restoration method uses the prior knowledge of the pre-trained model to perform image restoration through model prior or guided sampling techniques without the need for additional training data, but it still focuses on the guidance of a single modality. The image redrawing in the existing technology can only be guided by a specific modality such as text, resulting in low controllability of the generated results. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide an image redrawing method, device, electronic device and readable storage medium to solve the problem of low controllability of the generated redrawn image caused by single modality guidance in the prior art.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image redrawing method, comprising: obtaining an image to be redrawn, a reference image corresponding to the image to be redrawn, and a prompt text; inputting the image to be redrawn, the reference image, and the prompt text into an image redrawing model, performing latent space feature extraction on the image to be redrawn to obtain the latent space feature of the image to be redrawn, and performing noise processing on the latent space feature of the image to be redrawn to obtain the noise feature of the image to be redrawn; performing visual feature extraction on the reference image to obtain the visual feature of the reference image, and performing semantic feature extraction on the prompt text to obtain the semantic feature of the prompt text; performing feature fusion on the semantic feature of the prompt text and the visual feature of the reference image to obtain the multimodal feature and the adaptation feature of the reference image, wherein the adaptation feature of the reference image is the intermediate feature generated in the feature fusion process; performing diffusion processing based on the noise feature of the image to be redrawn, the multimodal feature, and the adaptation feature of the reference image to obtain the target feature, and decoding the target feature to obtain the target image corresponding to the image to be redrawn.

[0005] According to a second aspect of the embodiments of the present disclosure, there is provided an image redrawing device, comprising: an acquisition module, configured to acquire an image to be redrawn, a reference image corresponding to the image to be redrawn, and a prompt text; a first feature extraction module, configured to input the image to be redrawn, the reference image, and the prompt text into an image redrawing model, perform latent space feature extraction on the image to be redrawn to obtain the latent space feature of the image to be redrawn, and perform noise processing on the latent space feature of the image to be redrawn to obtain the noise feature of the image to be redrawn; a second feature extraction module, configured to perform visual feature extraction on the reference image to obtain the visual feature of the reference image, and perform semantic feature extraction on the prompt text to obtain the semantic feature of the prompt text; a feature fusion module, configured to perform feature fusion on the semantic feature of the prompt text and the visual feature of the reference image to obtain the multimodal feature and the adaptation feature of the reference image, wherein the adaptation feature of the reference image is the intermediate feature generated in the feature fusion process; an image generation module, configured to perform diffusion processing based on the noise feature of the image to be redrawn, the multimodal feature, and the adaptation feature of the reference image to obtain the target feature, and decode the target feature to obtain the target image corresponding to the image to be redrawn.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the disclosed embodiment has the following beneficial effects: by acquiring the image to be redrawn, the reference image and the prompt text, the image area to be redrawn is determined, and the reference image and the prompt text for guiding the redrawing process are provided. The reference image can provide a reference for the visual content, and the prompt text can provide a specific description of the redrawn content. By introducing multimodal information (image + text), it is helpful to understand the required content to be redrawn from different angles, and enhance the understanding ability of the image redrawing model and the richness of the guiding information. The image to be redrawn, the reference image and the prompt text are input into the image redrawing model, and the latent space feature extraction of the image to be redrawn is performed to capture the key structure and detail information of the image to be redrawn, and the latent space feature of the image to be redrawn is obtained. The latent space feature of the image to be redrawn is subjected to noise processing, and the reverse process from noise to clear image is simulated, which helps to increase the generalization ability and the diversity of the generated results, and provides a flexible and plastic basis for the redrawing of the image. Visual features of the reference image and semantic features of the prompt text are extracted to obtain the visual features of the reference image and the semantic features of the prompt text, and multimodal feature information is obtained to provide rich visual and semantic information for image redrawing, which helps to make the redrawing process more accurate and controllable. The semantic features of the prompt text and the visual features of the reference image are fused to obtain the corresponding multimodal features and the adaptation features of the reference image, which helps to comprehensively consider the information of the prompt text and the reference image, thereby generating a redrawn image that meets the requirements. Diffusion processing is performed based on the noise features, multimodal features and adaptation features of the reference image to be redrawn, so as to gradually generate target features, and the target features are decoded to obtain the target image corresponding to the image to be redrawn. The multimodal features and the adaptation features of the reference image are used through diffusion processing to guide the generation process of the target features, so that the generated target image not only meets the style of the reference image but also contains the requirements of the prompt text. The decoding process maps the target features back to the image space to generate the final target image. By introducing multimodal information (prompt text + reference image) and combining it with the powerful generation capability of the diffusion model, the limitations of the single modality guidance faced by the existing local image redrawing method can be solved, thereby making the generated results richer and more controllable, improving the diversity and quality of the generated target images, and generating more accurate and attractive target images under multimodal guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1is a schematic diagram of an application scenario of an embodiment of the present disclosure;

[0011] Figure 2 is a flowchart of an image redrawing method provided by an embodiment of the present disclosure;

[0012] Figure 3 is a structural schematic diagram of an image redrawing model provided by an embodiment of the present disclosure;

[0013] Figure 4 is a structural schematic diagram of a diffusion model in an image redrawing model provided by an embodiment of the present disclosure;

[0014] Figure 5 is a structural schematic diagram of another image redrawing model provided by an embodiment of the present disclosure;

[0015] Figure 6 is a structural schematic diagram of a first upper diffusion submodule in a diffusion model provided by an embodiment of the present disclosure;

[0016] Figure 7 is a structural schematic diagram of a feature fusion module in an image redrawing model provided by an embodiment of the present disclosure;

[0017] Figure 8 is a structural schematic diagram of an image redrawing device provided by an embodiment of the present disclosure;

[0018] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0020] An image redrawing method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.

[0022] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the above electronic devices. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0023] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0024] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.

[0025] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0026] The user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, etc. Specifically, the server 4 obtains the image to be redrawn, the reference image corresponding to the image to be redrawn and the prompt text; inputs the image to be redrawn, the reference image and the prompt text into the image redrawing model, extracts the latent space features of the image to be redrawn, obtains the latent space features of the image to be redrawn, and performs noise processing on the latent space features of the image to be redrawn to obtain the noise features of the image to be redrawn; extracts the visual features of the reference image to obtain the visual features of the reference image, and extracts the semantic features of the prompt text to obtain the semantic features of the prompt text; performs feature fusion on the semantic features of the prompt text and the visual features of the reference image to obtain the multimodal features and the adaptation features of the reference image, and the adaptation features of the reference image are the intermediate features generated in the feature fusion process; performs diffusion processing based on the noise features of the image to be redrawn, the multimodal features and the adaptation features of the reference image to obtain the target features, and decodes the target features to obtain the target image corresponding to the image to be redrawn.

[0027] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.

[0028] Figure 2 It is a flowchart of an image redrawing method provided by an embodiment of the present disclosure. Figure 2 The image redrawing method can be Figure 1 Server 4 executes. Figure 2 As shown, the image redrawing method includes:

[0029] Step 201: Acquire the image to be redrawn, the reference image corresponding to the image to be redrawn, and the prompt text.

[0030] In some embodiments, the image to be redrawn is an original image that needs to be modified or partially redrawn. The image to be redrawn may contain parts that the user wants to change, or the user expects to add new content to these parts. The reference image can provide additional visual information and contextual information for the image to be redrawn. The reference image can contain the style, color, texture or other visual elements that the user wants to imitate. The reference image can provide a visual guide or framework for the redrawing process of the image to be redrawn. The prompt text can be a textual specific instruction or description provided by the user on how to modify the image to be redrawn. The prompt text can contain a description of the modified content of the image to be redrawn, the location of the modification, the effect after the modification, etc. The prompt text can convert the user's intention into an instruction that the computer can understand, which is used to guide the redrawing of the image to be redrawn. Specifically, the image to be redrawn can be a landscape image, the prompt text is "house", and the reference image is an image of a house. The local image redrawing method based on the diffusion model in the prior art relies on the input of a single modality, such as using only text prompts. The input of a single modality cannot fully express the effect that the user wants to achieve. Relying on one modality can easily cause the generated result to deviate from the user's expectations. The limitations of single-modal guidance can be solved by using reference images and prompt texts. Prompt texts and reference images provide complementary information so that the computer can understand the user's intentions from both a visual and linguistic perspective. Information from different modalities can be used to construct more complete guidance information, which helps to more accurately control the generation results.

[0031] Step 102, input the image to be redrawn, the reference image and the prompt text into the image redrawing model, extract the latent space features of the image to be redrawn to obtain the latent space features of the image to be redrawn, and perform noise processing on the latent space features of the image to be redrawn to obtain the noise features of the image to be redrawn.

[0032] In some embodiments, the image redrawing model proposed in the present disclosure can predict the redrawing of the image to be redrawn based on the input image to be redrawn, the reference image and the prompt text, and generate a target image corresponding to the image to be redrawn. The image redrawing model includes a feature extraction network and a diffusion model, and the feature extraction network includes a text encoder, a first image encoder and a latent space feature extraction model. The latent space feature extraction model can be a variational autoencoder (VAE). The image to be redrawn is input into the latent space feature extraction model of the image redrawing model, and the image to be redrawn is mapped to a latent space (latent space), capturing high-level information of the image to be redrawn, such as shape, structure, texture, etc., converting high-dimensional image data into low-dimensional feature representation, reducing the amount of calculation while retaining key information, and obtaining the latent space features of the image to be redrawn. And the latent space features of the image to be redrawn are subjected to noise processing, random noise can be added, and the number of times of adding random noise can be t times, which is helpful to learn the process of recovering the image from noise, and the generation of the target image corresponding to the image to be redrawn is achieved in the process of gradually adding noise and gradually removing noise.

[0033] Step 103: extract visual features from the reference image to obtain visual features of the reference image, and extract semantic features from the prompt text to obtain semantic features of the prompt text.

[0034] In some embodiments, the text encoder can be a text encoder of the CLIP model, and the first image encoder can be an image encoder of the CLIP model. The reference image is input into the first image encoder of the image redrawing model to extract visual features of the reference image, capture key visual features in the image, such as edges, textures, shapes, etc., and obtain visual features of the reference image. The visual features of the reference image can be represented in the form of a high-dimensional vector, which can provide important visual information for subsequent redrawing tasks. The prompt text is input into the text encoder of the image redrawing model to extract semantic features of the prompt text, perform semantic analysis on the prompt text, capture key semantic information in the prompt text, such as objects, and obtain semantic features of the prompt text. The semantic features of the prompt text can be represented in the form of a high-dimensional vector, which can provide necessary guidance and constraints for image redrawing. By extracting the visual features of the reference image and the semantic features of the prompt text at the same time, it is helpful to effectively fuse the information of these two different modalities in the future, and the multi-modal information fusion helps to more comprehensively understand the user's redrawing intention. The model considers the visual information of the reference image and the semantic information of the prompt text at the same time, which can more accurately guide the image redrawing process, help to generate local redrawing results that are more in line with user expectations, and improve the controllability of the generated results.

[0035] Step 104, feature fusion is performed on the semantic features of the prompt text and the visual features of the reference image to obtain multimodal features and adaptation features of the reference image, where the adaptation features of the reference image are intermediate features generated during the feature fusion process.

[0036] In some embodiments, the semantic features of the prompt text and the visual features of the reference image can be aligned and fused based on the attention mechanism to obtain the multimodal features and the adaptation features of the reference image, thereby achieving the effective fusion of the semantic features of the prompt text and the visual features of the reference image, which not only retains the important information of each modality, but also generates more expressive multimodal features. Subsequently, the multimodal features can be fused into each layer of features of the diffusion model through the cross-attention mechanism to ensure that the generation process is guided by the multimodal features and improve the controllability of the generation results. The intermediate features (i.e., the adaptation features of the reference image) generated during the feature fusion process are the result of adapting and adjusting the visual features of the reference image, making it more suitable for fusion with the text features. The generated adaptation features of the reference image can better guide the image redrawing process, so that the generated target image is more consistent with the reference image in content, and a target image that better meets user expectations is generated. By fusing the semantic features of the prompt text and the visual features of the reference image, multimodal features and adaptation features of the reference image are generated. The multimodal features and adaptation features of the reference image play an important role in enhancing the expressiveness of the image redrawing model, improving robustness, optimizing the generated results, and enhancing controllability. At the same time, the single-modal guidance limitation problem faced by the local image redrawing method based on the diffusion model in the existing technology is solved, and the ability of the image redrawing model to process multimodal information and the quality of the generated results are improved.

[0037] Step 105 , performing diffusion processing based on the noise features, multimodal features and adaptation features of the image to be redrawn, obtaining target features, and decoding the target features to obtain a target image corresponding to the image to be redrawn.

[0038] In some embodiments, the image redrawing model also includes a decoder, which can be a decoder of VAE. The noise features, multimodal features and adaptation features of the image to be redrawn are input into the diffusion model for diffusion processing, and the noise is gradually removed by the diffusion model. In each iteration, the noise distribution can be adjusted according to the adaptation features of the multimodal features and the reference image, so that the generated target features are gradually close to the features corresponding to the target image. After multiple diffusion iterations, the target features including the information of the image to be redrawn and the guidance of the adaptation features from the multimodal features and the reference image are obtained. The target features are the denoised latent space features generated by the diffusion model. The target features are input into the variational self-decoder for decoding and mapping back to the original space, and the target features are converted into visual image data to obtain the target image corresponding to the image to be redrawn. The method of obtaining the target image by diffusion processing and decoding based on the noise features, multimodal features and adaptation features of the image to be redrawn effectively solves the single modality guidance limitations and insufficient controllability of the generated results faced by the local image redrawing method based on the diffusion model in the prior art by fusing information of multiple modalities and introducing the constraints of the reference image, so that the generated target image is consistent with the input conditions in style and semantics.

[0039] In some embodiments, specifically, reference Figure 3The image redrawing model includes a feature extraction network 301, a feature fusion module 305, a diffusion model 306, a decoder 307, and a noise adding module 308. The feature extraction network 301 includes a latent space feature extraction model 302, a text encoder 303, and a first image encoder 304. The image to be redrawn is input into the feature extraction network 301 to extract the latent space features, and obtain the latent space features of the image to be redrawn. The prompt text is input into the text encoder 303 of the feature extraction network 301 to extract the semantic features of the prompt text, and obtain the semantic features of the prompt text. The reference image is input into the first image encoder 304 of the feature extraction network 301 to extract the visual features of the reference image, and obtain the visual features of the reference image; the latent space features of the image to be redrawn are input into the noise adding module 308 for noise adding processing, and obtain the noise features of the image to be redrawn. The semantic features of the prompt text and the visual features of the reference image are input into the feature fusion module 305 for feature fusion, and the multimodal features and the adaptation features of the reference image are obtained. The adaptation features of the reference image are the intermediate features generated during the feature fusion process. The noise features, multimodal features and adaptation features of the image to be redrawn are input into the diffusion model 306 for diffusion processing to obtain the target features, and the target features are input into the decoder 307 for decoding the target features to obtain the target image corresponding to the image to be redrawn. Through feature extraction, fusion and diffusion processing, information from multiple sources is integrated, and the powerful generation capability of the diffusion model is combined to make the generated results richer and more controllable, so that the generated target image not only conforms to the style of the reference image but also contains the requirements of the prompt text, solves the limitation of the single modality guidance faced by the local image redrawing method based on the diffusion model in the prior art, and improves the quality of the generated target image.

[0040] Based on the image redrawing method proposed in the present disclosure, by obtaining the image to be redrawn, the reference image and the prompt text, the image area that needs to be redrawn is determined, and the reference image and the prompt text for guiding the redrawing process are provided. The reference image can provide a reference for the visual content, and the prompt text can provide a specific description of the redrawn content. By introducing multimodal information (image + text), it is helpful to understand the required content to be redrawn from different angles, and enhance the understanding ability of the image redrawing model and the richness of the guiding information. The image to be redrawn, the reference image and the prompt text are input into the image redrawing model, and the latent space feature extraction of the image to be redrawn is performed to capture the key structure and detail information of the image to be redrawn, and the latent space feature of the image to be redrawn is obtained. The latent space feature of the image to be redrawn is subjected to noise processing, and the reverse process from noise to clear image is simulated, which helps to increase the generalization ability and the diversity of the generated results, and provides a flexible and plastic basis for image redrawing. Visual features of the reference image and semantic features of the prompt text are extracted to obtain the visual features of the reference image and the semantic features of the prompt text, and multimodal feature information is obtained to provide rich visual and semantic information for image redrawing, which helps to make the redrawing process more accurate and controllable. The semantic features of the prompt text and the visual features of the reference image are fused to obtain the corresponding multimodal features and the adaptation features of the reference image, which helps to comprehensively consider the information of the prompt text and the reference image, thereby generating a redrawn image that meets the requirements. Diffusion processing is performed based on the noise features, multimodal features and adaptation features of the reference image to be redrawn, so as to gradually generate target features, and the target features are decoded to obtain the target image corresponding to the image to be redrawn. The multimodal features and the adaptation features of the reference image are used through diffusion processing to guide the generation process of the target features, so that the generated target image not only meets the style of the reference image but also contains the requirements of the prompt text. The decoding process maps the target features back to the image space to generate the final target image. By introducing multimodal information (prompt text + reference image) and combining it with the powerful generation capability of the diffusion model, the limitations of the single modality guidance faced by the existing local image redrawing method can be solved, thereby making the generated results richer and more controllable, improving the diversity and quality of the generated target images, and generating more accurate and attractive target images under multimodal guidance.

[0041] In some embodiments, the image redrawing model includes a diffusion model, the diffusion model includes a plurality of upper diffusion submodules and a plurality of lower diffusion submodules, the number of diffusion processing is t times, and the diffusion processing is performed based on the noise characteristics, multimodal characteristics and adaptation characteristics of the image to be redrawn, and the target characteristics are obtained, including:

[0042] Perform k diffusion processing based on the noise characteristics of the image to be redrawn, and obtain the kth diffusion processing result;

[0043] The first upper diffusion submodule based on the diffusion model performs feature processing on the k-th diffusion processing result, the multimodal feature and the adaptation feature of the reference image to obtain the feature processing result of the first upper diffusion submodule;

[0044] The other upper diffusion submodules except the first upper diffusion submodule based on the diffusion model perform feature processing on the feature processing result of the first upper diffusion submodule, the multimodal feature and the adaptation feature of the reference image to obtain the feature processing results of the other upper diffusion submodules except the first upper diffusion submodule;

[0045] Based on each lower diffusion submodule of the diffusion model, feature processing is performed on the feature processing results of each upper diffusion submodule, the multimodal features and the adaptation features of the reference image to obtain the feature processing results of the corresponding multiple lower diffusion submodules, and the feature processing result of the last lower diffusion submodule is determined as the (k+1)th diffusion processing result;

[0046] Based on the (k+1)th diffusion processing result, the multimodal features and the adaptation features of the reference image, a (tk-1)th diffusion processing is performed to obtain the tth diffusion processing result, and the tth diffusion processing result is determined as the target feature.

[0047] In some embodiments, the diffusion model includes a plurality of diffusion submodules, the diffusion submodules are divided into upper diffusion submodules and lower diffusion submodules, and the number of upper diffusion submodules and lower diffusion submodules in the diffusion model is the same. Figure 4The diffusion model 306 may include 5 upper diffusion submodules and 5 lower diffusion submodules, and the diffusion model 306 includes: a first upper diffusion submodule 401, a second upper diffusion submodule 402, a third upper diffusion submodule 403, a fourth upper diffusion submodule 404, a fifth upper diffusion submodule 405, a first lower diffusion submodule 406, a second lower diffusion submodule 407, a third lower diffusion submodule 408, a fourth lower diffusion submodule 409, and a fifth lower diffusion submodule 410, wherein the first upper diffusion submodule 401 is the first upper diffusion submodule of the above diffusion model, and the second upper diffusion submodule 402, the third upper diffusion submodule 403, the fourth upper diffusion submodule 404, and the fifth upper diffusion submodule 405 are other upper diffusion submodules of the above diffusion model except the first upper diffusion submodule. Based on the noise features, multimodal features of the image to be redrawn, and the adaptation features of the reference image, multiple iterations (i.e., multiple diffusion processes) are performed, and the generated image features can be gradually optimized to make them closer to the features of the target image or meet specific task requirements. Each time the diffusion process is performed, the diffusion result output by the diffusion model is closer to the target feature. Diffusion processing is the process in which the diffusion model iteratively removes noise to generate image features during the generation process. The number of times t determines how many rounds of such iterations the diffusion model performs. In the process of diffusion processing, the random noise in the noise features of the image to be redrawn is gradually reduced, and a clear image is finally restored. Starting from the first diffusion process, each iteration is based on the result of the previous one, combined with the multimodal features and the adaptation features, and further optimized. After t iterations, the model outputs the final target features, and the target features are used to generate a target image corresponding to the image to be redrawn. In the present disclosure, the number of diffusion processes is t times. Specifically,

[0048] In the first diffusion process, the noise characteristics of the image to be redrawn are input into the diffusion model for diffusion processing to obtain the first diffusion processing result. The first diffusion processing result is input into the diffusion model for diffusion processing to obtain the second diffusion processing result. The iterative processing is performed with reference to this method. Each diffusion processing is based on the previous diffusion processing result to obtain the kth diffusion processing result. The denoising process (diffusion processing process) at this stage is unconditional denoising, that is, it does not depend on any specific conditional information.

[0049] Specifically, in the (k+1)th diffusion processing process, the kth diffusion processing result, the multimodal features and the adaptation features of the reference image are input into the first upper diffusion submodule 401 for feature processing to obtain the feature processing result of the first upper diffusion submodule; the feature processing result of the first upper diffusion submodule, the multimodal features and the adaptation features of the reference image are input into the second upper diffusion submodule 402 for feature processing to obtain the feature processing result of the second upper diffusion submodule; the feature processing result of the second upper diffusion submodule, the multimodal features and the adaptation features of the reference image are input into the third upper diffusion submodule 403 for feature processing. The feature processing is performed to obtain the feature processing result of the third upper diffusion submodule; the feature processing result of the third upper diffusion submodule, the multimodal feature and the adaptation feature of the reference image are input into the fourth upper diffusion submodule 404 for feature processing to obtain the feature processing result of the fourth upper diffusion submodule; the feature processing result of the fourth upper diffusion submodule, the multimodal feature and the adaptation feature of the reference image are input into the fifth upper diffusion submodule 405 for feature processing to obtain the feature processing result of the fifth upper diffusion submodule; the feature processing result of the fifth upper diffusion submodule is spliced ​​with the feature processing result of the fifth upper diffusion submodule , obtain a corresponding first stitching result, input the first stitching result, the multimodal feature and the adaptation feature of the reference image into the first lower diffusion submodule 406 for feature processing, and obtain the feature processing result of the first lower diffusion submodule; stitch the feature processing result of the first lower diffusion submodule with the feature processing result of the fourth upper diffusion submodule to obtain a corresponding second stitching result, input the second stitching result, the multimodal feature and the adaptation feature of the reference image into the second lower diffusion submodule 407 for feature processing, and obtain the feature processing result of the second lower diffusion submodule; stitch the feature processing result of the second lower diffusion submodule with the feature processing result of the fourth upper diffusion submodule The feature processing results of the three upper partial diffusion submodules are spliced ​​to obtain a corresponding third splicing result, and the third splicing result, the multimodal feature, and the adaptation feature of the reference image are input into the third lower partial diffusion submodule 408 for feature processing to obtain the feature processing result of the third lower partial diffusion submodule; the feature processing result of the third lower partial diffusion submodule and the feature processing result of the second upper partial diffusion submodule are spliced ​​to obtain a corresponding fourth splicing result, and the fourth splicing result, the multimodal feature, and the adaptation feature of the reference image are input into the fourth lower partial diffusion submodule 409 for feature processing to obtain the feature processing result of the fourth lower partial diffusion submodule;The feature processing result of the fourth lower diffusion submodule and the feature processing result of the first upper diffusion submodule are spliced ​​to obtain a corresponding fifth splicing result, the fifth splicing result, the multimodal feature and the adaptation feature of the reference image are input into the fifth lower diffusion submodule 410 for feature processing to obtain the feature processing result of the fifth lower diffusion submodule, and the feature processing result of the fifth lower diffusion submodule is determined as the (k+1)th diffusion processing result. ;

[0050] Referring to the (k+1)th diffusion process, a (tk-1)th diffusion process is performed based on the (k+1)th diffusion process result, the multimodal features, and the adaptation features of the reference image to obtain the tth diffusion process result. In the subsequent (tk)th diffusion process, the multimodal features and the adaptation features of the reference image are used as conditional guides, and the generation process is adjusted according to the features of the prompt text and the reference image, so that the target image finally generated matches the prompt text and the reference image.

[0051] The upper diffusion submodule of the diffusion model can perform preliminary feature extraction and fusion on the noise features, multimodal features and adaptation features of the input image to be redrawn, extract and integrate information from the input features, and each upper diffusion submodule further processes based on the output of the previous upper diffusion submodule, gradually deepening the fusion and abstraction of features. The upper diffusion submodule of the diffusion model can further adjust and optimize the image features based on the output of each upper submodule, combined with multimodal features and adaptation features, to generate a more refined feature representation. The output of the last lower submodule is used as the result of the current diffusion processing, that is, the feature representation of the optimal denoised image of this round. By simultaneously processing the noise features, multimodal features and adaptation features of the image to be redrawn, the information from multiple sources is fully utilized, and the limitations of single modality guidance are overcome. In the process of multiple rounds of diffusion processing, the output target features and target images can be gradually adjusted and optimized, which improves the controllability of the generated results. Users can adjust the reference image and prompt text to affect the specific content and style of the generated target image.

[0052] In some embodiments, before diffusion processing is performed based on the noise features, multimodal features and adaptive features of the image to be redrawn, the method further includes: obtaining a stick figure image, extracting visual features of the stick figure image, and obtaining visual features of the stick figure image; performing adaptive processing on the visual features of the stick figure image, and obtaining adaptive features of the stick figure image. Diffusion processing is performed (k-1) times based on the noise features of the image to be redrawn to obtain the (k-1)th diffusion processing result; the (k-1)th diffusion processing result is added to the adaptive features of the stick figure image to obtain corresponding fusion features, and diffusion processing is performed based on the fusion features to obtain the kth diffusion processing result; diffusion processing is performed (tk) times based on the kth diffusion processing result, the multimodal features and the adaptive features of the reference image to obtain the tth diffusion processing result, and the tth diffusion processing result is determined as the second target feature, and the second target feature is decoded to obtain a second target image corresponding to the image to be redrawn.

[0053] Specifically, refer to Figure 5, the feature extraction network 301 in the image redrawing model also includes a second image encoder 501, and the image redrawing model also includes a first adapter 502. The second image encoder 501 can be a vision transformer model (ViT), and the first adapter 502 can be a multilayer perceptron (MLP). The stick image can contain the basic outline and shape of the guiding content, which can be used as a guide for the subsequent generation process to provide visual information and context information for the image to be redrawn. For example, the stick image can be the outline of a house. The stick image is input into the second image encoder 501, and visual features are extracted from the stick image. Key visual elements in the stick image, such as edges, shapes and structures, are extracted to obtain visual features of the stick image. The visual features of the stick image are contour features extracted from the stick image, which can be used to enhance the detail expression of the generated second target image, especially in terms of edges and contours. The visual features of the stick image are input into the first adapter 502 to perform adaptation processing on the visual features of the stick image, and the visual feature level of the stick image is converted or adjusted to adapt to the subsequent generation task, so as to obtain the adaptation features of the stick image. The noise features of the image to be redrawn are input into the diffusion model 306 for (k-1) diffusion processing, and unconditional denoising is adopted. The features in the noise features of the image to be redrawn are gradually reduced in the (k-1) diffusion processing to obtain the (k-1)th diffusion processing result. The (k-1)th diffusion processing result is added to the adaptation features of the stick image, and the structural information of the stick image is integrated into the image features with reduced noise, so as to guide the further generation of the second target image, which helps to increase the details and edge clarity of the second target image finally generated, and obtain the corresponding fusion features. The fusion features are input into the diffusion model 306 for diffusion processing to obtain the kth diffusion processing result, and denoising is continued based on the fused features, which helps to further improve the quality of the second target image finally generated, so that it is closer to the original image or meets the expected results. The k-th diffusion processing result, the multimodal features and the adaptation features of the reference image are input into the diffusion model 306, and the diffusion processing is performed (tk) times to obtain the t-th diffusion processing result. The adaptation features of the reference image and the multimodal features are used as guidance, that is, the generation process is adjusted according to the prompt text description and the features of the reference image, so that the generated second target image matches the prompt text, the reference image and the sketch image. The t-th diffusion processing result is determined as the second target feature, and the second target feature is input into the decoder 307 for decoding. The final diffusion processing result is decoded into a visualized image to obtain a second target image corresponding to the image to be redrawn.

[0054] In some embodiments, the first upper diffusion submodule based on the diffusion model performs feature processing on the kth diffusion processing result, the multimodal features and the adaptation features of the reference image to obtain the feature processing result of the first upper diffusion submodule, including: inputting the kth diffusion processing result into the first upper diffusion submodule, performing self-attention processing on the noise features of the image to be redrawn, and obtaining a self-attention processing result; performing cross-attention processing on the self-attention processing result and the adaptation features of the reference image to obtain a first cross-attention processing result; performing cross-attention processing on the first cross-attention processing result and the multimodal features to obtain a second cross-attention processing result, and determining the second cross-attention processing result as the feature processing result of the first upper diffusion submodule.

[0055] In some embodiments, reference Figure 6The first upper diffusion submodule 401 includes a self-attention layer 601, a first cross-attention layer 602, and a second cross-attention layer 603. The k-th diffusion processing result is input into the self-attention layer 601 of the first upper diffusion submodule 401 for self-attention processing, focusing on the important area in the k-th diffusion processing result, while suppressing the unimportant part, enhancing the key features of the input features, and reducing the influence of noise to obtain the self-attention processing result. The self-attention processing result and the adapted features of the reference image are input into the first cross-attention layer 602, the self-attention processing result is used as a query vector (query), and the adapted features of the reference image are used as a key vector (key) and a value vector (value). Based on the query vector, the key vector and the value vector, cross-attention processing is performed, the association between the reference image and the self-attention processing result is learned, and the features of the reference image are integrated into the current processing result, which helps to better imitate the characteristics of the reference image, and obtain the first cross-attention processing result. The self-attention processing result and the adaptation feature of the reference image are cross-attention processed, that is, the useful information (such as style, layout, etc.) in the reference image is introduced into the image to be redrawn, which helps to integrate the features of the reference image while maintaining the basic content of the image to be redrawn. The first cross-attention processing result and the multimodal feature are input into the second cross-attention layer 603, the first cross-attention processing result is used as the query vector, the multimodal feature is used as the key vector and the value vector, and the cross-attention processing is performed based on the query vector, the key vector and the value vector, and the association between the multimodal feature and the first cross-attention processing result is learned, and the multimodal feature is integrated into the current processing result, which helps to better imitate the characteristics of the reference image and the prompt text, and obtain the second cross-attention processing result. The first cross-attention processing result and the multimodal feature are cross-attention processed, that is, the information in the multimodal feature (such as the information of the prompt text, the information of the reference image, etc.) is introduced into the image to be redrawn, which helps to integrate the features of the reference image and the prompt text while maintaining the basic content of the image to be redrawn, and at the same time, the image generation process is guided by visual and non-visual information, so as to improve the accuracy and diversity of the target image finally generated. After multi-step attention mechanism processing, a feature map integrating multiple information sources is obtained, that is, the second cross attention processing result. The second cross attention processing result is determined as the feature processing result of the first upper diffusion submodule, and as the input of the second upper diffusion submodule, it can provide a richer and more accurate feature representation for subsequent processing. Through the above-mentioned layered processing method (first processing noise features, then combining reference image features, and finally fusing multimodal features), combining multimodal features and reference image adaptation features, and using multiple types of information to guide target image generation, it overcomes the limitation of relying only on a single modality, helps to gradually build a more refined and accurate image representation, and obtain a more accurate target image.It should be noted that the structures of other diffusion submodules in the diffusion model of the present disclosure are the same as the structure of the first upper diffusion submodule 401 described above.

[0056] In some embodiments, feature fusion is performed on the semantic features of the prompt text and the visual features of the reference image to obtain multimodal features and adaptation features of the reference image, including: adapting the visual features of the reference image to obtain adaptation features of the reference image; performing self-attention processing on the adaptation features of the reference image to obtain self-attention processing results of the adaptation features of the reference image; obtaining initial query features, taking the initial query features as query vectors, and the self-attention processing results of the adaptation features of the reference image as key vectors and value vectors, performing cross-attention processing to obtain first interactive features; taking the initial query features as query vectors, and the semantic features of the prompt text as key vectors and value vectors, performing cross-attention processing to obtain second interactive features; adding the first interactive features and the second interactive features to obtain initial multimodal features; performing nonlinear transformation on the initial multimodal features to obtain multimodal features.

[0057] In some embodiments, reference Figure 7, the feature fusion module 305 includes a second adapter 701 and a deep fusion module 702, and the deep fusion module 702 can be a query former (QFormer) of a bidirectional encoder pre-training model 2 (Bidirectional Encoder Representations from Pre-training 2, BLIP2). The visual features of the reference image are obtained by extracting the visual features of the reference image, and are not suitable for direct use in subsequent multimodal feature fusion. The visual features of the reference image are input into the second adapter 701 for adaptation processing, and the adaptation processing may include operations such as feature dimensionality reduction, normalization, and feature selection to obtain the adaptation features of the reference image, so that the obtained adaptation features of the reference image are more matched with the semantic features of the prompt text in terms of dimension, distribution, etc., which is convenient for subsequent fusion. The adaptation features of the reference image and the semantic features of the prompt text are input into the deep fusion module 702, and the adaptation features of the reference image are self-attention processed. The self-attention processing is used to perform internal interaction, and the dependency between the adaptation features of the reference image is captured to generate a more expressive feature representation, that is, the self-attention processing result of the adaptation features of the reference image. The self-attention processing result of the adaptation features of the reference image is used as the input of the subsequent cross-attention processing, which helps to achieve the effective fusion of image features and text features. A set of initial query features is obtained, and the initial query features can be randomly initialized or obtained through learning. The initial query features are used as query vectors, and the self-attention processing results of the adaptation features of the reference image are used as key vectors and value vectors. Cross-attention processing is performed to capture the interactive information between the initial query features and the image features, and the key information of the reference image is further extracted to obtain the first interactive features. The initial query features are used as query vectors, and the semantic features of the prompt text are used as key vectors and value vectors. Cross-attention processing is performed to capture the interactive information between the initial query features and the text features, and the key information of the prompt text is further extracted to obtain the second interactive features. The first interactive feature and the second interactive feature are added together to integrate the information of the two modalities of image and text to obtain the initial multimodal feature. The initial multimodal feature can be input into the feedforward neural network for nonlinear transformation to learn more complex feature representations, improve the expressiveness of the output features, and obtain multimodal features. By adopting the feature processing steps of multimodal feature fusion, attention mechanism and nonlinear transformation, key information is extracted from the reference image and text description, and the information of the prompt text and the reference image is effectively fused to generate a multimodal feature representation that integrates visual and semantic information.

[0058] In some embodiments, the image redrawing model includes a diffusion model and a feature extraction network, and before the image to be redrawn, the reference image and the prompt text are input into the image redrawing model, the image redrawing model further includes:

[0059] Obtaining a training set, the training set includes multiple training samples, and the training samples include training text, training reference images, training sketch images, and training initial images;

[0060] Performing mask processing on the initial training image to obtain a masked initial training image, wherein the masked initial training image includes a masked area and a non-masked area;

[0061] Inputting the masked initial training image and the training text into the feature extraction network of the image redrawing model to be trained, performing latent space feature extraction on the masked initial training image to obtain the latent space feature of the masked initial training image, performing noise processing on the latent space feature of the masked initial training image to obtain the noise feature of the masked initial training image, and performing semantic feature extraction on the training text to obtain the semantic feature of the training text;

[0062] Inputting the semantic features of the training text and the noise features of the masked initial training image into the diffusion model to be trained, performing diffusion processing based on the semantic features of the training text and the noise features of the masked initial training image, and obtaining features of a first initial prediction image, wherein the features of the first initial prediction image include features of a non-masked area of ​​the first initial prediction image and features of a masked area of ​​the first initial prediction image;

[0063] Using the features of the non-masked area of ​​the masked training initial image to replace the features of the non-masked area of ​​the first initial prediction image to obtain the features of the first prediction image, and decoding the features of the first prediction image to obtain the first prediction image;

[0064] Calculating a first-stage loss value of the image redrawing model based on the first predicted image and the training initial image, and updating the parameters of the diffusion model according to the first-stage loss value to obtain a first-stage image redrawing model, wherein the first-stage image redrawing model includes a first-stage diffusion model;

[0065] Performing second-stage training on the first-stage image redrawing model based on the training set to obtain a second-stage image redrawing model;

[0066] Based on the training set and the second-stage image redrawing model, the third-stage training is performed to obtain a trained image redrawing model.

[0067] In some embodiments, obtaining a training set is the first step in training an image redrawing model. The training set includes multiple training samples, and the training samples are the basis for learning the image redrawing model. Each training sample includes training text, training reference images, training sketch images, and training initial images. The training initial image is the original image that needs to be redrawn or modified, and contains the part that needs to be replaced or repaired during the training process; the training text is a text description or instruction about the content of the training initial image after repairing the mask, which is very important for the image redrawing model to generate an image that conforms to the text description; the training reference image can be an image with a similar style to the predicted image corresponding to the training initial image, which can provide a visual reference for the masked training initial image during the image redrawing process. During the training process, the features, style or layout in the reference image can be learned to generate a higher quality predicted image; the training sketch image is a simplified image representation, which can be used to assist in understanding the structure or outline of the content to be redrawn, and can be used as a guide for generating predicted images. The training initial image is masked to simulate the image area that needs to be redrawn or modified in actual applications, and only certain specific areas of the training initial image are modified to obtain the masked training initial image, which includes the masked area and the non-masked area. The masked initial training image and training text are input into the feature extraction network of the image redrawing model to be trained, and the latent space feature extraction is performed on the masked initial training image to capture important features in the masked initial training image, such as shape, texture, color, etc. The high-dimensional image data (i.e., the masked initial training image) is converted into a low-dimensional feature representation containing key information, which helps to reduce the amount of calculation and improve the processing speed, and obtain the latent space feature of the masked initial training image. The latent space feature of the masked initial training image is subjected to noise processing, and random noise is added to the latent space feature, which can be Gaussian noise, to increase the uncertainty of the training process, and at the same time help the image redrawing model learn how to recover a clear predicted image from the noise. The semantic feature extraction is performed on the training text to extract the key semantic information in the training text, and the training text is converted into a numerical feature, i.e., the semantic feature of the training text, which is used to guide the noise feature of the masked initial training image to generate the corresponding target image in the diffusion process. The semantic features of the training text and the noise features of the masked training initial image are input into the diffusion model to be trained, and diffusion processing is performed based on the semantic features of the training text and the noise features of the masked training initial image, and noise is gradually added or removed to generate new data points, and the features of the first initial prediction image consistent with the description of the training text are generated. The features of the first initial prediction image include the features of the non-masked area of ​​the first initial prediction image and the features of the masked area of ​​the first initial prediction image, and the non-masked area is the part of the original image that is not planned to be modified. The features of the non-masked area of ​​the first initial prediction image are replaced with the features of the non-masked area of ​​the masked training initial image to obtain the features of the first prediction image.The parts of the original image that do not need to be changed are retained so that the first predicted image is consistent with the original image (training initial image) in the non-mask area, avoiding the introduction of unnecessary distortion or errors in the non-mask area, and mapping the features of the first predicted image back to the pixel space, thereby generating a complete image, that is, the first predicted image corresponding to the training initial image. The average value of the square of the pixel difference between the first predicted image and the training initial image can be calculated by the mean square error loss to obtain the first stage loss value of the image redrawing model. The first stage loss value can measure the gap between the first predicted image and the training initial image, and is used to guide the optimization process of the image redrawing model. The back propagation algorithm and the first stage loss value are used to update the parameters of the diffusion model in the image redrawing model to be trained, so that the image redrawing model can better generate a predicted image close to the training initial image. After the above training process, a preliminary image redrawing model, that is, the first stage image redrawing model, is obtained, which includes a first stage diffusion model that has been preliminarily trained. In the first stage training process, not only learning to generate content in the mask area, but also matching the generated content with the context of the training text, thereby generating a more natural and realistic predicted image.

[0068] During the second stage of training, the first stage image redrawing model is trained for the second stage based on the training set. The performance of the second stage image redrawing model is further optimized on the basis of the first stage image redrawing model, and image content coordinated with the training text, training reference image and training sketch image is generated in the mask area of ​​the predicted image. At the same time, the generated predicted image is consistent with the input conditions in style and semantics.

[0069] During the third stage of training, the second stage image redrawing model was trained based on the training set. Image consistency and aesthetic reward mechanisms were introduced to further improve the quality and aesthetic performance of the generated predicted images. This ensures that the generated predicted images not only retain content features but also meet aesthetic standards, and ultimately a trained image redrawing model is obtained.

[0070] In some embodiments, performing a second-stage training on the first-stage image redrawing model based on the training set to obtain the second-stage image redrawing model includes:

[0071] Inputting the masked initial training image, training text, training reference image and training stick figure image into the feature extraction network of the first-stage image redrawing model, performing latent space feature extraction on the masked initial training image to obtain the latent space feature of the masked initial training image, performing noise processing on the latent space feature of the masked initial training image to obtain the noise feature of the masked initial training image, performing visual feature extraction on the training reference image and the training stick figure image to obtain the visual feature of the training reference image and the visual feature of the training stick figure image, and performing semantic feature extraction on the training text to obtain the semantic feature of the training text;

[0072] Performing adaptation processing on the visual features of the training stick figure image to obtain adaptation features of the training stick figure image;

[0073] The semantic features of the training text and the visual features of the training reference image are fused to obtain the corresponding training multimodal features and the adaptation features of the training reference image. The adaptation features of the training reference image are the intermediate features generated in the multimodal feature fusion process.

[0074] Inputting the adapted features of the training reference image, the training multimodal features, the latent space features of the masked training initial image, and the adapted features of the training stick image into the first-stage diffusion model of the first-stage image redrawing model, performing diffusion processing based on the adapted features of the training reference image, the training multimodal features, the latent space features of the masked training initial image, and the adapted features of the training stick image to obtain features of the second predicted image and performing decoding processing on the features of the second predicted image to obtain a second predicted image;

[0075] The second stage loss value of the image redrawing model is calculated based on the second predicted image and the training initial image, and the parameters of the first stage image redrawing model are updated according to the second stage loss value to obtain the second stage image redrawing model.

[0076] In some embodiments, the masked initial training image, training text, training reference image and training stick image in each training sample are input into the feature extraction network of the first stage image redrawing model, the masked initial training image is subjected to latent space feature extraction, the image data is converted into a low-dimensional space feature more suitable for processing, and the latent space feature of the masked initial training image is obtained. Random noise is added to the latent space feature of the masked initial training image to simulate the change of the image under different conditions, and the noise feature of the masked initial training image is obtained; feature extraction is performed on the training reference image and the training stick image to obtain the visual features of the training reference image and the visual features of the training stick image, and the visual features of the training reference image and the visual features of the training stick image can be used to subsequently guide the model to generate a predicted image with a specific style or details; semantic feature extraction is performed on the training text to obtain the semantic features of the training text, and the semantic features of the training text can be used to guide the model to generate content that conforms to the description of the training text. The visual features of the stick image are adapted to obtain a feature representation suitable for fusion with other modal features, that is, the adaptation features of the training stick image. The semantic features of the training text and the visual features of the training reference image are fused with multimodal features to generate training multimodal features and adaptation features of the training reference image, so as to solve the limitation of single modality guidance in the prior art and enable the image redrawing model to process information of multiple modalities such as text and image at the same time. The adaptation features of the training reference image, the training multimodal features, the latent space features of the masked training initial image and the adaptation features of the training sketch image are input into the first stage diffusion model of the first stage image redrawing model for diffusion processing, and the various input features are used as conditions for stepwise denoising. The diffusion model generates a latent representation based on the input features, i.e., the features of the second predicted image, and decodes the features of the second predicted image to convert the features of the second predicted image into specific image data to obtain the second predicted image. The average of the squared pixel difference between the second predicted image and the initial training image can be calculated by the mean square error loss to obtain the second-stage loss value of the image redrawing model. The second-stage loss value can measure the gap between the second predicted image and the initial training image, and is used to guide the optimization process of the image redrawing model. The back propagation algorithm and the second-stage loss value are used to update the parameters of the diffusion model in the image redrawing model to be trained, so that the image redrawing model can better generate a predicted image close to the initial training image. After the second-stage training, the generation quality, controllability and diversity of the image redrawing model are improved through the integration and optimization of multimodal information, thereby overcoming the limitations of the existing technology based on single modality guidance and obtaining the second-stage image redrawing model.

[0077] In some embodiments, a third stage of training is performed on the second stage image redrawing model based on the training set to obtain a trained image redrawing model, including:

[0078] Inputting the masked initial training image, the training text, the training reference image and the training stick figure image into the second-stage image redrawing model, and performing a redrawing prediction on the masked initial training image based on the masked initial training image, the training text, the training reference image and the training stick figure image to obtain a third predicted image, wherein the third predicted image includes a masked area and a non-masked area;

[0079] Calculate the loss value of the first part of the third stage of the image repainting model based on the third predicted image and the training initial image;

[0080] Extracting visual features from the mask region of the third predicted image to obtain visual features of the corresponding mask region of the third predicted image;

[0081] Calculating the cosine similarity between the visual features of the mask area of ​​the third prediction image and the visual features of the training reference image, and determining the loss value of the second part of the third stage of the image redrawing model according to the cosine similarity;

[0082] Performing aesthetic scoring on the third predicted image and the initial training image respectively to obtain an aesthetic score of the third predicted image and an aesthetic score of the initial training image;

[0083] Calculate the third stage third part loss value of the image repainting model based on the aesthetic score of the third predicted image and the aesthetic score of the training initial image;

[0084] Based on the third-stage third-part loss value, the third-stage second-part loss value and the third-stage first-part loss value, the third-stage loss value of the image redrawing model is determined, and the parameters of the second-stage image redrawing model are updated according to the third-stage loss value to obtain the trained image redrawing model.

[0085] In some embodiments, the masked training initial image, training text, training reference image and training stick figure image are input into the second stage image redrawing model, and the masked training initial image is redrawn based on the masked training initial image, training text, training reference image and training stick figure image to obtain a third predicted image, which includes a masked area and a non-masked area.

[0086] Specifically, the average value of the square of the pixel difference between the third predicted image and the training initial image can be calculated by the mean square error loss to obtain the loss value L of the first part of the third stage of the image redrawing model: rebuild , the loss value of the first part of the third stage is L rebuild The gap between the third predicted image and the initial training image can be measured and used to guide the optimization process of the image repainting model.

[0087] Specifically, the mask region of the third predicted image can be input into ViT for visual feature extraction to obtain the visual feature f of the mask region of the corresponding third predicted image. gen , calculate the visual features f of the mask area of ​​the third predicted image gen and the visual features f of the training reference image ref The cosine similarity cos_sim(f gen ,f ref ), the cosine similarity cos_sim(f gen ,f ref ) is mapped to the interval [0,1] as the reward signal R consistency =σ(cos_sim(f gen ,f ref )), where σ can be a Sigmoid function or other similar nonlinear activation functions, which is fed back to the image redrawing model to guide the image redrawing model to optimize the retention of image features during the iteration process. The loss value of the second part of the third stage of the image redrawing model can be defined as the negative value of the reward signal, which encourages the image redrawing model to maximize the reward signal. The loss value of the second part of the third stage is L consistency =1-R consistency .

[0088] Specifically, a pre-trained aesthetic evaluation model can be used and fine-tuned through a human-annotated preference dataset so that the pre-trained aesthetic evaluation model can score the image. i And the training initial image x j Perform aesthetic scoring respectively to obtain the aesthetic score S of the third predicted image aethetic (x i ) and the aesthetic score S of the initial training image aethetic (x j ), the aesthetic score S of the third predicted image output by the aesthetic reward model aethetic (x i ) and the aesthetic score S of the initial training image aethetic (x j ) is used as feedback for the image redrawing model to enhance the visual effect of the output predicted image. The loss value of the third stage and the third part of the image redrawing model can be expressed as L aethetic =-(log(σ(S aethetic (x i )-S aethetic (x j )))), where it can be a Sigmoid function or other similar nonlinear activation functions. It should be noted that the above x i and x jDifferent third predicted images may also be generated for the same training initial image in different image redrawing processes.

[0089] Combining the third part loss value of the third stage, the second part loss value of the third stage and the first part loss value of the third stage, the third stage loss value of the image redrawing model is obtained by weighted sum, and the second stage image redrawing model is fine-tuned by using the third stage loss value, and the parameters of the second stage image redrawing model are updated to obtain the trained image redrawing model, so that the generated prediction image not only meets the content feature retention, but also meets the aesthetic standards. In the third stage training process, the consistency and aesthetic rewards are integrated into a multi-objective optimization problem, and the Pareto optimization method and other methods can be used to seek the best balance. In the training process, not only the image generation quality is paid attention to, but also the image consistency and aesthetic characteristics are considered, so as to achieve a more comprehensive optimization goal. At the same time, the weights of consistency and aesthetic rewards can be dynamically adjusted according to the quality of the generated prediction image, and a real-time feedback mechanism is established, so that the image redrawing can quickly adjust the generation strategy according to the reward signal.

[0090] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0091] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0092] Figure 8 Schematic diagram of an image redrawing device provided by an embodiment of the present disclosure. Figure 8 As shown, the image redrawing device includes:

[0093] An acquisition module 801 is configured to acquire an image to be redrawn, a reference image corresponding to the image to be redrawn, and a prompt text;

[0094] The first feature extraction module 802 is configured to input the image to be redrawn, the reference image and the prompt text into the image redrawing model, perform latent space feature extraction on the image to be redrawn to obtain the latent space features of the image to be redrawn, and perform noise processing on the latent space features of the image to be redrawn to obtain the noise features of the image to be redrawn;

[0095] The second feature extraction module 803 is configured to extract visual features from the reference image to obtain visual features of the reference image, and to extract semantic features from the prompt text to obtain semantic features of the prompt text;

[0096] The feature fusion module 804 is configured to perform feature fusion on the semantic features of the prompt text and the visual features of the reference image to obtain the multimodal features and the adaptation features of the reference image, where the adaptation features of the reference image are intermediate features generated during the feature fusion process;

[0097] The image generation module 805 is configured to perform diffusion processing based on the noise features, multimodal features and adaptation features of the image to be redrawn, obtain target features, and decode the target features to obtain a target image corresponding to the image to be redrawn.

[0098] According to the technical solution provided by the embodiment of the present disclosure, by obtaining the image to be redrawn, the reference image and the prompt text, the image area that needs to be redrawn is determined, and the reference image and the prompt text for guiding the redrawing process are provided. The reference image can provide a reference for the visual content, and the prompt text can provide a specific description of the redrawn content. By introducing multimodal information (image + text), it is helpful to understand the required content to be redrawn from different angles, and enhance the understanding ability of the image redrawing model and the richness of the guiding information. The image to be redrawn, the reference image and the prompt text are input into the image redrawing model, and the latent space feature extraction of the image to be redrawn is performed to capture the key structure and detail information of the image to be redrawn, and the latent space feature of the image to be redrawn is obtained. The latent space feature of the image to be redrawn is subjected to noise processing, and the reverse process from noise to clear image is simulated, which helps to increase the generalization ability and the diversity of the generated results, and provides a flexible and plastic basis for the redrawing of the image. Visual features of the reference image and semantic features of the prompt text are extracted to obtain the visual features of the reference image and the semantic features of the prompt text, and multimodal feature information is obtained to provide rich visual and semantic information for image redrawing, which helps to make the redrawing process more accurate and controllable. The semantic features of the prompt text and the visual features of the reference image are fused to obtain the corresponding multimodal features and the adaptation features of the reference image, which helps to comprehensively consider the information of the prompt text and the reference image, thereby generating a redrawn image that meets the requirements. Diffusion processing is performed based on the noise features, multimodal features and adaptation features of the reference image to be redrawn, so as to gradually generate target features, and the target features are decoded to obtain the target image corresponding to the image to be redrawn. The multimodal features and the adaptation features of the reference image are used through diffusion processing to guide the generation process of the target features, so that the generated target image not only meets the style of the reference image but also contains the requirements of the prompt text. The decoding process maps the target features back to the image space to generate the final target image. By introducing multimodal information (prompt text + reference image) and combining it with the powerful generation capability of the diffusion model, the limitations of the single modality guidance faced by the existing local image redrawing method can be solved, thereby making the generated results richer and more controllable, improving the diversity and quality of the generated target images, and generating more accurate and attractive target images under multimodal guidance.

[0099] In some embodiments, the image redrawing model includes a diffusion model, the diffusion model includes multiple upper diffusion submodules and multiple lower diffusion submodules, the number of diffusion processing is t times, the image generation module 805 is configured to perform k diffusion processing based on the noise characteristics of the image to be redrawn, and obtain the kth diffusion processing result; the first upper diffusion submodule based on the diffusion model performs feature processing on the kth diffusion processing result, multimodal features and adaptation features of the reference image to obtain the feature processing result of the first upper diffusion submodule; the other upper diffusion submodules except the first upper diffusion submodule based on the diffusion model perform feature processing on the feature processing result of the first upper diffusion submodule, multimodal features and adaptation features of the reference image The adaptation features of the image are subjected to feature processing to obtain feature processing results of the other upper diffusion submodules except the first upper diffusion submodule; the feature processing results of the feature processing of each upper diffusion submodule, the multimodal features and the adaptation features of the reference image are subjected to feature processing by each lower diffusion submodule based on the diffusion model to obtain feature processing results of the corresponding multiple lower diffusion submodules, and the feature processing result of the last lower diffusion submodule is determined as the (k+1)th diffusion processing result; the (tk-1)th diffusion processing is performed based on the (k+1)th diffusion processing result, the multimodal features and the adaptation features of the reference image to obtain the tth diffusion processing result, and the tth diffusion processing result is determined as the target feature.

[0100] In some embodiments, the image generation module 805 is configured to input the kth diffusion processing result into the first upper diffusion sub-module, perform self-attention processing on the noise features of the image to be redrawn, and obtain a self-attention processing result; perform cross-attention processing on the self-attention processing result and the adaptation features of the reference image to obtain a first cross-attention processing result; perform cross-attention processing on the first cross-attention processing result and the multimodal features to obtain a second cross-attention processing result, and determine the second cross-attention processing result as the feature processing result of the first upper diffusion sub-module.

[0101] In some embodiments, the feature fusion module 804 is configured to perform adaptation processing on the visual features of the reference image to obtain the adaptation features of the reference image; perform self-attention processing on the adaptation features of the reference image to obtain the self-attention processing results of the adaptation features of the reference image; obtain the initial query features, use the initial query features as the query vector, and the self-attention processing results of the adaptation features of the reference image as the key vector and the value vector, perform cross-attention processing to obtain the first interactive features; use the initial query features as the query vector, and the semantic features of the prompt text as the key vector and the value vector, perform cross-attention processing to obtain the second interactive features; perform addition processing on the first interactive features and the second interactive features to obtain the initial multimodal features; perform nonlinear transformation on the initial multimodal features to obtain the multimodal features.

[0102] In some embodiments, the image redrawing model includes a diffusion model and a feature extraction network. Before the image to be redrawn, the reference image and the prompt text are input into the image redrawing model, the image redrawing device is configured to obtain a training set, which includes multiple training samples, and the training samples include training text, training reference image, training stick figure image and training initial image; masking the training initial image to obtain a masked training initial image, and the masked training initial image includes a masked area and a non-masked area; the masked training initial image and the training text are input into the feature extraction network of the image redrawing model to be trained, latent space feature extraction is performed on the masked training initial image to obtain the latent space feature of the masked training initial image, the latent space feature of the masked training initial image is denoised to obtain the noise feature of the masked training initial image, and semantic feature extraction is performed on the training text to obtain the semantic feature of the training text; the semantic feature of the training text and the noise feature of the masked training initial image are input into the diffusion model to be trained. , based on the semantic features of the training text and the noise features of the masked training initial image, diffusion processing is performed to obtain the features of the first initial prediction image, and the features of the first initial prediction image include the features of the non-masked area of ​​the first initial prediction image and the features of the masked area of ​​the first initial prediction image; the features of the non-masked area of ​​the masked training initial image are used to replace the features of the non-masked area of ​​the first initial prediction image to obtain the features of the first prediction image, and the features of the first prediction image are decoded to obtain the first prediction image; based on the first prediction image and the training initial image, the first-stage loss value of the image redrawing model is calculated, and the parameters of the diffusion model are updated according to the first-stage loss value to obtain the first-stage image redrawing model, and the first-stage image redrawing model includes the first-stage diffusion model; based on the training set, the first-stage image redrawing model is trained in the second stage to obtain the second-stage image redrawing model; based on the training set, the second-stage image redrawing model is trained in the third stage to obtain the trained image redrawing model.

[0103] In some embodiments, the image redrawing device is configured to input the masked training initial image, training text, training reference image and training stick image into the feature extraction network of the first stage image redrawing model, perform latent space feature extraction on the masked training initial image to obtain the latent space feature of the masked training initial image, perform noise processing on the latent space feature of the masked training initial image to obtain the noise feature of the masked training initial image, perform visual feature extraction on the training reference image and the training stick image to obtain the visual feature of the training reference image and the visual feature of the training stick image, and perform semantic feature extraction on the training text to obtain the semantic feature of the training text; perform adaptation processing on the visual feature of the training stick image to obtain the adaptation feature of the training stick image; perform feature fusion on the semantic feature of the training text and the visual feature of the training reference image to obtain The corresponding training multimodal features and the adaptation features of the training reference image, the adaptation features of the training reference image are the intermediate features generated in the multimodal feature fusion process; the adaptation features of the training reference image, the training multimodal features, the latent space features of the masked training initial image and the adaptation features of the training stick image are input into the first-stage diffusion model of the first-stage image redrawing model, and diffusion processing is performed based on the adaptation features of the training reference image, the training multimodal features, the latent space features of the masked training initial image and the adaptation features of the training stick image to obtain the features of the second predicted image and decode the features of the second predicted image to obtain the second predicted image; the second-stage loss value of the image redrawing model is calculated based on the second predicted image and the training initial image, and the parameters of the first-stage image redrawing model are updated according to the second-stage loss value to obtain the second-stage image redrawing model.

[0104] In some embodiments, the image redrawing device is configured to input the masked training initial image, training text, training reference image and training stick figure image into the second stage image redrawing model, redraw the masked training initial image based on the masked training initial image, training text, training reference image and training stick figure image to obtain a third predicted image, the third predicted image including a masked area and a non-masked area; calculate the third stage first part loss value of the image redrawing model based on the third predicted image and the training initial image; extract visual features from the masked area of ​​the third predicted image to obtain the corresponding visual features of the masked area of ​​the third predicted image; calculate the visual features of the masked area of ​​the third predicted image and the training initial image; The method comprises the following steps: training the cosine similarity of the visual features of the reference image, and determining the second part loss value of the third stage of the image redrawing model according to the cosine similarity; performing aesthetic scoring on the third predicted image and the training initial image respectively to obtain the aesthetic score of the third predicted image and the aesthetic score of the training initial image; calculating the third part loss value of the third stage of the image redrawing model based on the aesthetic score of the third predicted image and the aesthetic score of the training initial image; determining the third stage loss value of the image redrawing model based on the third part loss value of the third stage, the second part loss value of the third stage and the first part loss value of the third stage, and updating the parameters of the second stage image redrawing model according to the third stage loss value to obtain the trained image redrawing model.

[0105] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0106] Fig. 9 Schematic diagram of an electronic device 9 provided in an embodiment of the present disclosure. Fig. 9 As shown, the electronic device 9 of this embodiment includes: a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 901 executes the computer program 903, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0107] The electronic device 9 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 9 may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will appreciate that Fig. 9 The electronic device 9 is merely an example and does not limit the electronic device 9 , and may include more or less components than those shown in the figure, or different components.

[0108] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0109] The memory 902 may be an internal storage unit of the electronic device 9, for example, a hard disk or memory of the electronic device 9. The memory 902 may also be an external storage device of the electronic device 9, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 9. The memory 902 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 902 is used to store computer programs and other programs and data required by the electronic device.

[0110] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0112] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image redrawing method, characterized in that: include: Acquire an image to be redrawn, a reference image corresponding to the image to be redrawn, and a prompt text; Inputting the image to be redrawn, the reference image and the prompt text into an image redrawing model, performing latent space feature extraction on the image to be redrawn to obtain latent space features of the image to be redrawn, and performing noise processing on the latent space features of the image to be redrawn to obtain noise features of the image to be redrawn; Extracting visual features of the reference image to obtain visual features of the reference image, and extracting semantic features of the prompt text to obtain semantic features of the prompt text; Performing feature fusion on the semantic features of the prompt text and the visual features of the reference image to obtain multimodal features and adaptation features of the reference image, wherein the adaptation features of the reference image are intermediate features generated during the feature fusion process; Diffusion processing is performed based on the noise feature of the image to be redrawn, the multimodal feature and the adaptation feature of the reference image to obtain a target feature, and the target feature is decoded to obtain a target image corresponding to the image to be redrawn.

2. The method according to claim 1, characterized in that The image redrawing model includes a diffusion model, the diffusion model includes a plurality of upper diffusion submodules and a plurality of lower diffusion submodules, the number of diffusion processing is t times, the diffusion processing is performed based on the noise feature of the image to be redrawn, the multimodal feature and the adaptation feature of the reference image to obtain the target feature, including: Perform k diffusion processing based on the noise characteristics of the image to be redrawn, and obtain a kth diffusion processing result; Based on the first upper diffusion submodule of the diffusion model, feature processing is performed on the k-th diffusion processing result, the multimodal feature and the adaptation feature of the reference image to obtain a feature processing result of the first upper diffusion submodule; Based on the other upper diffusion submodules except the first upper diffusion submodule of the diffusion model, feature processing is performed on the feature processing result of the first upper diffusion submodule, the multimodal feature and the adaptation feature of the reference image to obtain feature processing results of the other upper diffusion submodules except the first upper diffusion submodule; Based on each of the lower diffusion submodules of the diffusion model, feature processing is performed on the feature processing results of each of the upper diffusion submodules, the multimodal features, and the adaptation features of the reference image to obtain corresponding feature processing results of multiple lower diffusion submodules, and the feature processing result of the last lower diffusion submodule is determined as the (k+1)th diffusion processing result; Based on the (k+1)th diffusion processing result, the multimodal features and the adaptation features of the reference image, a (tk-1)th diffusion processing is performed to obtain a tth diffusion processing result, and the tth diffusion processing result is determined as the target feature.

3. The method according to claim 2, characterized in that The first upper diffusion submodule based on the diffusion model performs feature processing on the k-th diffusion processing result, the multimodal feature and the adaptation feature of the reference image to obtain the feature processing result of the first upper diffusion submodule, including: Inputting the k-th diffusion processing result into the first upper diffusion submodule, performing self-attention processing on the noise characteristics of the image to be redrawn, and obtaining a self-attention processing result; Performing cross attention processing on the self-attention processing result and the adaptation features of the reference image to obtain a first cross attention processing result; Cross-attention processing is performed on the first cross-attention processing result and the multimodal feature to obtain a second cross-attention processing result, and the second cross-attention processing result is determined as the feature processing result of the first upper diffusion sub-module.

4. The method according to claim 1, characterized in that: The step of fusing the semantic features of the prompt text and the visual features of the reference image to obtain multimodal features and adaptation features of the reference image includes: Performing adaptation processing on the visual features of the reference image to obtain adaptation features of the reference image; Performing self-attention processing on the adaptation features of the reference image to obtain a self-attention processing result of the adaptation features of the reference image; Acquire an initial query feature, use the initial query feature as a query vector, and use the self-attention processing result of the adaptation feature of the reference image as a key vector and a value vector, perform cross-attention processing, and obtain a first interactive feature; The initial query feature is used as a query vector, and the semantic feature of the prompt text is used as a key vector and a value vector, and a cross attention process is performed to obtain a second interactive feature; Adding the first interaction feature and the second interaction feature to obtain the initial multimodal feature; The initial multimodal features are subjected to nonlinear transformation to obtain the multimodal features.

5. The method according to claim 1, characterized in that The image redrawing model includes a diffusion model and a feature extraction network. Before inputting the image to be redrawn, the reference image and the prompt text into the image redrawing model, the image redrawing model further includes: Acquire a training set, wherein the training set includes a plurality of training samples, and the training samples include training text, training reference images, training stick figure images, and training initial images; Performing mask processing on the initial training image to obtain a masked initial training image, wherein the masked initial training image includes a masked area and a non-masked area; Inputting the masked initial training image and the training text into a feature extraction network of an image redrawing model to be trained, performing latent space feature extraction on the masked initial training image to obtain latent space features of the masked initial training image, performing noise processing on the latent space features of the masked initial training image to obtain noise features of the masked initial training image, and performing semantic feature extraction on the training text to obtain semantic features of the training text; Inputting the semantic features of the training text and the noise features of the masked initial training image into a diffusion model to be trained, performing diffusion processing based on the semantic features of the training text and the noise features of the masked initial training image, and obtaining features of a first initial prediction image, wherein the features of the first initial prediction image include features of a non-masked area of ​​the first initial prediction image and features of a masked area of ​​the first initial prediction image; Using the features of the non-masked area of ​​the masked training initial image to replace the features of the non-masked area of ​​the first initial predicted image to obtain the features of the first predicted image, and decoding the features of the first predicted image to obtain the first predicted image; Calculating a first-stage loss value of the image redrawing model based on the first predicted image and the training initial image, and updating the parameters of the diffusion model according to the first-stage loss value to obtain a first-stage image redrawing model, wherein the first-stage image redrawing model includes a first-stage diffusion model; Performing second-stage training on the first-stage image redrawing model based on the training set to obtain a second-stage image redrawing model; Based on the training set and the second-stage image redrawing model, a third-stage training is performed to obtain a trained image redrawing model.

6. The method according to claim 5, characterized in that The performing second-stage training on the first-stage image redrawing model based on the training set to obtain the second-stage image redrawing model includes: Inputting the masked initial training image, the training text, the training reference image and the training stick image into the feature extraction network of the first-stage image redrawing model, performing latent space feature extraction on the masked initial training image to obtain the latent space feature of the masked initial training image, performing noise processing on the latent space feature of the masked initial training image to obtain the noise feature of the masked initial training image, performing visual feature extraction on the training reference image and the training stick image to obtain the visual feature of the training reference image and the visual feature of the training stick image, and performing semantic feature extraction on the training text to obtain the semantic feature of the training text; Performing adaptation processing on the visual features of the training stick figure image to obtain adaptation features of the training stick figure image; Performing feature fusion on the semantic features of the training text and the visual features of the training reference image to obtain corresponding training multimodal features and adaptation features of the training reference image, wherein the adaptation features of the training reference image are intermediate features generated during the multimodal feature fusion process; Inputting the adapted features of the training reference image, the training multimodal features, the latent space features of the masked training initial image, and the adapted features of the training stick figure image into the first stage diffusion model of the first stage image redrawing model, performing diffusion processing based on the adapted features of the training reference image, the training multimodal features, the latent space features of the masked training initial image, and the adapted features of the training stick figure image to obtain features of a second predicted image, and performing decoding processing on the features of the second predicted image to obtain a second predicted image; A second-stage loss value of the image redrawing model is calculated based on the second predicted image and the training initial image, and the parameters of the first-stage image redrawing model are updated according to the second-stage loss value to obtain a second-stage image redrawing model.

7. The method according to claim 6, characterized in that The step of performing a third stage of training on the second stage image redrawing model based on the training set to obtain a trained image redrawing model includes: Inputting the masked initial training image, the training text, the training reference image and the training stick figure image into the second-stage image redrawing model, and performing a redrawing prediction on the masked initial training image based on the masked initial training image, the training text, the training reference image and the training stick figure image to obtain a third predicted image, wherein the third predicted image includes a masked area and a non-masked area; Calculate the first part of the loss value of the third stage of the image redrawing model based on the third predicted image and the training initial image; Extracting visual features from the mask area of ​​the third predicted image to obtain visual features of the corresponding mask area of ​​the third predicted image; Calculating the cosine similarity between the visual features of the mask area of ​​the third predicted image and the visual features of the training reference image, and determining the second part loss value of the third stage of the image redrawing model according to the cosine similarity; Performing aesthetic scoring on the third predicted image and the initial training image respectively to obtain an aesthetic score of the third predicted image and an aesthetic score of the initial training image; Calculating a third-stage third-part loss value of the image repainting model based on the aesthetic score of the third predicted image and the aesthetic score of the training initial image; Based on the third part loss value of the third stage, the second part loss value of the third stage and the first part loss value of the third stage, the third stage loss value of the image redrawing model is determined, and the parameters of the second stage image redrawing model are updated according to the third stage loss value to obtain the trained image redrawing model.

8. An image redrawing device, characterized in that: include: An acquisition module is configured to acquire the image to be redrawn, a reference image corresponding to the image to be redrawn, and a prompt text; A first feature extraction module is configured to input the image to be redrawn, the reference image and the prompt text into an image redrawing model, perform latent space feature extraction on the image to be redrawn to obtain latent space features of the image to be redrawn, and perform noise processing on the latent space features of the image to be redrawn to obtain noise features of the image to be redrawn; A second feature extraction module is configured to extract visual features from the reference image to obtain visual features of the reference image, and to extract semantic features from the prompt text to obtain semantic features of the prompt text; A feature fusion module is configured to perform feature fusion on the semantic features of the prompt text and the visual features of the reference image to obtain multimodal features and adaptation features of the reference image, wherein the adaptation features of the reference image are intermediate features generated during the feature fusion process; The image generation module is configured to perform diffusion processing based on the noise characteristics of the image to be redrawn, the multimodal characteristics and the adaptation characteristics of the reference image to obtain target characteristics, and decode the target characteristics to obtain a target image corresponding to the image to be redrawn.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images