Image fusion method, device, equipment, storage medium and program product

Abstract

The application discloses an image fusion method, device, equipment, storage medium and program product, relates to the technical field of image processing, and discloses the image fusion method. The disclosed image fusion method generates a first noise image of a fusion image of an original image and a target subject, fills random Gaussian noise in a background region corresponding to an editing region in the first noise image, generates a second noise image, respectively uses the first noise image, the first noise image combined with text prompt information and the second noise image combined with text prompt information, extracts a first feature vector, a second feature vector and a third feature vector, and obtains a target characteristic vector of a generated target fusion image. When the image is edited, the pre-trained SDM model can be directly applied to generate the target fusion image according to the target characteristic vector, and the technical problem of how to expand the application scene of the image editing method is solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image fusion method, apparatus, device, storage medium, and program product. Background Technology

[0002] In related technologies, the SDM (Stable Diffusion Model) can be used to achieve image editing based on reference images and / or text prompts. However, for different image editing scenarios, further fine-tuning and training of the pre-trained SDM model are required to achieve the corresponding editing effects. Model fine-tuning requires significant computational power and data support, limiting the application scenarios of image editing methods. Summary of the Invention

[0003] The main objective of this application is to provide an image fusion method, apparatus, device, storage medium, and program product, aiming to solve the technical problem of how to expand the application scenarios of image editing methods.

[0004] To achieve the above objectives, this application proposes an image fusion method, which includes:

[0005] Acquire the original image, reference image, and text prompt information, wherein the reference image includes the target subject, and the original image includes the editing area where the target subject is to be added;

[0006] A first noise image and a second noise image are generated. The first noise image is a noise image of the fused image obtained by fusing the target subject with the original image. The second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise. The background region is the region in the editing region other than the target subject.

[0007] The first feature vector, the second feature vector, and the third feature vector are extracted. The first feature vector is extracted from the first noisy image, the second feature vector is extracted from the first noisy image and the text prompt information, and the third feature vector is extracted from the second noisy image and the text prompt information.

[0008] The target feature vector is obtained by fusing the first feature vector, the second feature vector, and the third feature vector.

[0009] Generate a target fusion image based on the target feature vector.

[0010] In one embodiment, the step of extracting the first feature vector, the second feature vector, and the third feature vector includes:

[0011] The first noisy image is used as the first current noisy image, the first noisy image is used as the second current noisy image, and the second noisy image is used as the third current noisy image;

[0012] The first current feature vector, the second current feature vector, and the third current feature vector are extracted. The first current feature vector is extracted from the first current noisy image, the second current feature vector is extracted from the second current noisy image and the text prompt information, and the third current feature vector is extracted from the third current noisy image and the text prompt information.

[0013] The steps for fusing the first, second, and third feature vectors to obtain the target feature vector include:

[0014] The intermediate feature vector is obtained based on at least one of the first current feature vector, the second current feature vector, and the third current feature vector;

[0015] Before the step of generating the target fused image based on the target feature vector, the image fusion method also includes:

[0016] Based on the first current feature vector, obtain the first predicted noise image;

[0017] Based on the second current feature vector, obtain the second predicted noise image;

[0018] Based on the intermediate feature vector, the third predicted noise image is obtained;

[0019] Using the first predicted noise image, the first current noise image is denoised to obtain the first intermediate noise image;

[0020] Using the second predicted noise image, the second current noise image is denoised to obtain the second intermediate noise image;

[0021] Using the third predicted noise image, the third current noise image is denoised to obtain the third intermediate noise image;

[0022] The first intermediate noise image is used as the first current noise image, the second intermediate noise image is used as the second current noise image, and the third intermediate noise image is used as the third current noise image. Then, the steps of extracting the first current feature vector, the second current feature vector, and the third current feature vector are returned to be executed.

[0023] The process continues until the current iteration count reaches the preset maximum iteration count, at which point the intermediate feature vector is used as the target feature vector.

[0024] In one embodiment, the step of obtaining an intermediate feature vector based on at least one of a first current feature vector, a second current feature vector, and a third current feature vector includes:

[0025] If the current iteration number is less than or equal to the first preset iteration number, the third current feature vector is used as the intermediate feature vector;

[0026] If the current iteration number is greater than the first preset iteration number but less than the second preset iteration number, the first current feature vector, the second current feature vector, and the third current feature vector are merged to obtain the intermediate feature vector.

[0027] If the current iteration number is greater than or equal to the second preset iteration number, the second current feature vector and the third current feature vector are merged to obtain the intermediate feature vector.

[0028] In one embodiment, the step of extracting the first feature vector, the second feature vector, and the third feature vector includes:

[0029] The first noisy image is input into the first stable diffusion model to extract the first feature vector;

[0030] The first noisy image and text prompt information are input into the second stable diffusion model to extract the second feature vector;

[0031] The second noisy image and text prompt information are input into the third stable diffusion model to extract the third feature vector.

[0032] In one embodiment, the step of generating a first noisy image includes:

[0033] The target subject is extracted from the reference image;

[0034] The target subject is merged into the editing area of ​​the original image to obtain a merged image;

[0035] The fused image is converted into the first noisy image.

[0036] In one embodiment, the step of generating a second noisy image includes:

[0037] A mask image is generated based on the fused image of the original image and the target subject. The mask image is used to extract the background region in the fused image that corresponds to the editing region.

[0038] The first intermediate noise image is obtained by combining the mask image and the random noise image;

[0039] The second intermediate noise image is obtained by combining the inverse mask image of the mask image with the noise image of the fused image.

[0040] The first intermediate noise image and the second intermediate noise image are fused together to obtain the second noise image.

[0041] Furthermore, to achieve the above objectives, this application also proposes an image fusion apparatus, which includes:

[0042] The acquisition module is used to acquire the original image, the reference image, and the text prompt information. The reference image includes the target subject, and the original image includes the editing area where the target subject is to be added.

[0043] The first generation module is used to generate a first noise image and a second noise image. The first noise image is a noise image of a fused image obtained by fusing the target subject with the original image. The second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise. The background region is the region in the editing region other than the target subject.

[0044] The feature extraction module is used to extract a first feature vector, a second feature vector, and a third feature vector, wherein the first feature vector is extracted from the first noisy image, the second feature vector is extracted from the first noisy image and the text prompt information, and the third feature vector is extracted from the second noisy image and the text prompt information;

[0045] The feature fusion module is used to fuse the first feature vector, the second feature vector, and the third feature vector to obtain the target feature vector;

[0046] The second generation module is used to generate a target fusion image based on the target feature vector.

[0047] In addition, to achieve the above objectives, this application also proposes an image fusion apparatus, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image fusion method described above.

[0048] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image fusion method described above.

[0049] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image fusion method described above.

[0050] One or more technical solutions proposed in this application have at least the following technical effects:

[0051] This application provides an image fusion method, apparatus, device, storage medium, and program product. It generates a first noisy image by generating a fused image of the original image and the target subject. Random Gaussian noise is then filled into the background region corresponding to the editing area in the first noisy image to generate a second noisy image. First feature vectors, second feature vectors, and third feature vectors are extracted from the first noisy image, the first noisy image combined with text prompts, and the second noisy image combined with text prompts, respectively. These feature vectors are then fused to obtain a target characteristic vector for generating the target fused image. Since the first feature vector incorporates both background and subject features of the fused image, and the second feature vector adds text prompts to the first feature vector... The system can generate fused images that satisfy the semantics of the text as much as possible. The third feature vector not only integrates the background features, subject features, and text features of the text prompt information of the fused image, but also integrates random Gaussian noise in the background region. This allows the target subject in the generated target fused image to transition naturally with the surrounding background. Therefore, when editing images, the pre-trained SDM model can be directly applied to generate the target fused image based on the target feature vector. There is no need to further fine-tune and train the pre-trained SDM model for different application scenarios, which reduces the complex and time-consuming model fine-tuning process. It also eliminates the need to collect additional sample data and consume computing power, greatly expanding the application scenarios of image editing methods. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the first embodiment of the image fusion method of this application;

[0055] Figure 2 A schematic diagram of an exemplary image fusion process provided for the image fusion method of this application;

[0056] Figure 3 A schematic diagram illustrating an exemplary noisy image generation process provided for the image fusion method of this application;

[0057] Figure 4 A schematic diagram of the structure of the U-Net network in an exemplary SDM model provided for the image fusion method of this application;

[0058] Figure 5 This is a schematic diagram of the modules of the first embodiment of the image fusion apparatus of this application;

[0059] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the image fusion method in the embodiments of this application.

[0060] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0061] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0062] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an apparatus or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an apparatus or system. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the apparatus or system that includes that element.

[0063] If the embodiments of this application involve descriptions such as "first" or "second," such descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.

[0064] In this application, the suffixes such as "module," "component," or "unit" used to denote elements are used only for illustrative purposes and have no specific meaning in themselves. Therefore, "module," "component," or "unit" can be used interchangeably. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0065] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of this application embodiment is: to obtain an original image, a reference image, and text prompt information, wherein the reference image includes the target subject, and the original image includes the editing area where the target subject is to be added;

[0067] A first noise image and a second noise image are generated. The first noise image is a noise image of the fused image obtained by fusing the target subject with the original image. The second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise. The background region is the region in the editing region other than the target subject. A first feature vector, a second feature vector, and a third feature vector are extracted. The first feature vector is extracted from the first noise image, the second feature vector is extracted from the first noise image and the text prompt information, and the third feature vector is extracted from the second noise image and the text prompt information. The first feature vector, the second feature vector, and the third feature vector are fused to obtain the target feature vector. The target fused image is generated based on the target feature vector.

[0068] With the continuous advancement of deep learning technology, image editing and generation techniques have achieved remarkable success. Diffusion models, especially advanced models such as Stable Diffusion, can generate highly realistic images that match the text descriptions provided by users, greatly promoting the development of artistic creation and design.

[0069] In related technologies, the SDM model can use a reference image to modify the face image to be edited, so that the face in the modified target image is the same as the face in the reference image. That is, it relies on a single reference image to guide the image editing process.

[0070] The SDM model can also use input text prompts as content information conditions to edit video images, thereby obtaining a new video with the same structure as the original video and content consistent with the text prompts. In other words, it relies on a single reference text to guide the image editing process.

[0071] Therefore, it is evident that relevant image editing methods often rely on a single reference image or text prompts to guide the image editing process, which to some extent limits the diversity and flexibility of image editing methods.

[0072] Furthermore, for different image editing scenarios, the first encoder, second encoder, attention model, and decoder in the pre-trained SDM model used for image editing need further fine-tuning and training to achieve the corresponding editing effects. However, model fine-tuning requires significant computing power and data support, limiting the application scenarios of image editing methods.

[0073] Therefore, this application provides a solution that uses a first feature vector, a second feature vector, and a third feature vector to fuse and obtain a target feature vector for generating a target fused image. Since the first feature vector fuses the background and subject features of the fused image, and the second feature vector adds textual prompts to the first feature vector, a fused image that satisfies textual semantics as much as possible can be generated. The third feature vector not only fuses the background, subject, and textual features of the fused image, but also incorporates random Gaussian noise from the background region, allowing for a natural transition between the target subject and the surrounding background in the generated target fused image. Thus, during image editing, a pre-trained SDM model can be directly applied to generate the target fused image based on the target feature vector, eliminating the need for further fine-tuning and training of the pre-trained SDM model for different application scenarios. This reduces the complex and time-consuming model fine-tuning process and eliminates the need for additional sample data collection and computational costs, greatly expanding the application scenarios of image editing methods. Furthermore, since the target feature vector incorporates the subject features of the reference image and the textual features of the textual prompts, an image editing process guided by both the reference image and the textual prompts is achieved, increasing the diversity and flexibility of image editing methods.

[0074] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or image fusion device capable of performing the above functions. The following description uses an image fusion device as an example to illustrate this embodiment and the subsequent embodiments.

[0075] Based on this, embodiments of this application provide an image fusion method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image fusion method of this application.

[0076] In this embodiment, the image fusion method may include steps S100 to S500:

[0077] Step S100: Obtain the original image, reference image, and text prompt information.

[0078] The reference image includes the target subject, and the original image includes the editing area where the target subject is to be added.

[0079] It should be noted that, as Figure 2As shown, in image editing, the original image is the image to be edited, and the editing area is the region within the original image that needs to be edited. The position and size of the editing area can be adjusted according to editing needs, as long as the target subject from the reference image is added to the editing area and is located within the original image. The reference image is used to guide image editing and includes the target subject to be edited into the original image. Text prompts are the text used to guide image editing. For example, Figure 2 The subject in the reference image shown is a Chow Chow, and the text message reads "A photo of a Chow Chow lying on the beach".

[0080] Step S200: Generate a first noise image and a second noise image.

[0081] The first noise image is a noise image of the fused image obtained by fusing the target subject with the original image. The second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise. The background region is the region in the editing region other than the target subject.

[0082] It should be noted that by fusing the target subject into the edited area of ​​the original image to obtain a fused image, adding noise to the fused image yields the first noise image, and filling the background area corresponding to the edited area in the first noise image with random Gaussian noise yields the second noise image.

[0083] In one feasible implementation, such as Figure 3 As shown, step S100 may include: extracting the target subject from the reference image; fusing the target subject into the editing area of ​​the original image to obtain a fused image; and converting the fused image into a first noise image.

[0084] It should be noted that the target subject can be extracted from the reference image using a matting algorithm. The specific matting algorithm can be chosen according to actual needs, such as the Grab Cut algorithm. After extracting the target subject, it is placed in the editing area of ​​the original image to obtain the fused image. The first noisy image can be obtained by adding noise to the fused image using transformation algorithms such as DDIM (Denoising Diffusion Implicit Models) or DPM-Solver. Among them, DPM-Solver is a high-order ODE solver, often used in SDM models to generate noise, which can generate better latent representations.

[0085] In one feasible implementation, such as Figure 3As shown, step S100 may include: generating a mask image based on the fused image of the original image and the target subject, wherein the mask image is used to extract the background region corresponding to the editing region in the fused image; obtaining a first intermediate noise image based on the mask image and the random noise image; obtaining a second intermediate noise image based on the inverse mask image of the mask image and the noise image of the fused image; and fusing the first intermediate noise image and the second intermediate noise image to obtain a second noise image.

[0086] It should be noted that, as Figure 3 As shown, the mask image M reflects the background area within the edited region of the fused image, excluding the target subject.

[0087] The second noisy image can be obtained using the following formula: Formula 1 is:

[0088]

[0089] M is the second noisy image, N is the masked image, and Z is the random noise image. T The first noisy image is shown, and ⊙ represents the XOR operation. Random noise images can be generated randomly using a noise generator.

[0090] Step S300: Extract the first feature vector, the second feature vector, and the third feature vector.

[0091] The first feature vector is extracted from the first noisy image, the second feature vector is extracted from the first noisy image and the text prompt information, and the third feature vector is extracted from the second noisy image and the text prompt information.

[0092] It should be noted that the first feature vector integrates the background and subject features of the fused image; the second feature vector integrates not only the background and subject features of the fused image but also the text features of the text prompt, generating a target fused image that best satisfies the text semantics; and the third feature vector integrates not only the background, subject, and text features of the fused image but also random Gaussian noise in the background region, ensuring a natural transition between the target subject and the surrounding background in the generated target fused image. These feature vectors can be extracted using a pre-trained stable diffusion model.

[0093] Step S400: Fuse the first feature vector, the second feature vector, and the third feature vector to obtain the target feature vector.

[0094] Step S500: Generate a target fusion image based on the target feature vector.

[0095] It should be noted that feature fusion strategies can include directly fusing the first, second, and third feature vectors, or fusing them according to their respective preset feature weights. By inputting the target feature vector into the decoder, the target feature vector can be decoded into the edited target fused image.

[0096] Therefore, this embodiment provides an image fusion method. The first feature vector fuses the background and subject features of the fused image; the second feature vector fuses not only the background and subject features of the fused image but also the text features of the text prompt information, generating a target fused image that best satisfies the text semantics; and the third feature vector fuses not only the background, subject, and text features of the fused image but also random Gaussian noise in the background region, allowing for a natural transition between the target subject and the surrounding background in the generated target fused image. Thus, during image editing, a pre-trained SDM model can be directly applied to generate the target fused image based on the target feature vector, eliminating the need for further fine-tuning and training of the pre-trained SDM model for different application scenarios. This reduces the complex and time-consuming model fine-tuning process and eliminates the need for additional sample data collection and computational costs, greatly expanding the application scenarios of the image editing method. Furthermore, since the target feature vector incorporates the subject features of the reference image and the text features of the text prompt information, it achieves an image editing process guided by both the reference image and the text prompt information, increasing the diversity and flexibility of the image editing method.

[0097] In one feasible implementation, step S300 may include: inputting a first noise image into a first stable diffusion model to extract a first feature vector; inputting the first noise image and text prompt information into a second stable diffusion model to extract a second feature vector; and inputting the second noise image and text prompt information into a third stable diffusion model to extract a third feature vector.

[0098] It should be noted that the image fusion device can use a pre-trained SDM model to process the first noisy image, the second noisy image, and the text prompt information to extract the corresponding feature vectors.

[0099] like Figure 2 As shown, in this embodiment, feature extraction is performed using three pre-trained SDM models: the first stable diffusion model SDM-1, the second stable diffusion model SDM-2, and the third stable diffusion model SDM-3. SDM-1, SDM-2, and SDM-3 are the same pre-trained models.

[0100] During feature extraction, the first noisy image is input into SDM-1 to extract the first feature vector; the first noisy image and text prompt information are input into SDM-2 to extract the second feature vector; the second noisy image and text prompt information are input into SDM-2 to extract the third feature vector. The feature vectors can be extracted by the U-Net network in the SDM model, such as... Figure 4 As shown, the feature vector can be the self-attention feature of the intermediate layer CrossAttnBlock of the U-Net network.

[0101] Understandably, the SDM model, as a fundamental model for image generation, allows its decoder to generate a fused image of the target based on the target feature vector. Therefore, as... Figure 2 As shown, after obtaining the target feature vector, inputting the target feature vector into the SDM-3 decoder will generate the target fused image.

[0102] Therefore, this embodiment provides an image fusion method that can directly apply a pre-trained SDM model to extract the corresponding feature vector from the input image and generate the target fused image based on the target feature vector. This eliminates the need for further fine-tuning and training of the pre-trained SDM model for different application scenarios, reducing the complex and time-consuming model fine-tuning process. It also eliminates the need for additional sample data collection and computational power consumption, greatly expanding the application scenarios of image editing methods.

[0103] In one feasible implementation, step S300 may include: using a first noise image as a first current noise image, using the first noise image as a second current noise image, and using the second noise image as a third current noise image; extracting a first current feature vector, a second current feature vector, and a third current feature vector, wherein the first current feature vector is extracted from the first current noise image, the second current feature vector is extracted from the second current noise image and the text prompt information, and the third current feature vector is extracted from the third current noise image and the text prompt information.

[0104] Correspondingly, step S400 may include: obtaining an intermediate feature vector based on at least one of the first current feature vector, the second current feature vector, and the third current feature vector.

[0105] Before step S500, the image fusion method may further include: obtaining a first predicted noise image based on a first current feature vector; obtaining a second predicted noise image based on a second current feature vector; obtaining a third predicted noise image based on an intermediate feature vector; denoising the first current noise image using the first predicted noise image to obtain a first intermediate noise image; denoising the second current noise image using the second predicted noise image to obtain a second intermediate noise image; denoising the third current noise image using the third predicted noise image to obtain a third intermediate noise image; using the first intermediate noise image as the first current noise image, the second intermediate noise image as the second current noise image, and the third intermediate noise image as the third current noise image, and returning to execute the step of extracting the first current feature vector, the second current feature vector, and the third current feature vector; until the current iteration number is the preset maximum iteration number, and the intermediate feature vector is used as the target feature vector.

[0106] It should be noted that the pre-trained SDM model can include a U-Net network with T time steps, where T is the preset maximum number of iterations. The current feature vector for the current iteration is extracted by the U-Net network at time step t. During the first iteration, the three current feature vectors are extracted by the U-Net networks at the first time step of SDM-1, SDM-2, and SDM-3, respectively, from the first noisy image, the first noisy image and text prompt information, and the second noisy image and text prompt information. In subsequent iterations, the three current feature vectors are extracted by the U-Net networks at the t-th time step of SDM-1, SDM-2, and SDM-3, respectively, from the first current noisy image, the second current noisy image and text prompt information, and the third current noisy image and text prompt information.

[0107] Understandably, after each iteration and before moving to the next, the U-Net network can be used to denoise the current noisy image based on the predicted noise image determined by the current feature vector, obtaining an intermediate noisy image. This intermediate image serves as the input to the U-Net network for the next iteration. This process continues until t=T, at which point the target feature vector, containing T denoising iterations, is decoded into the target fused image.

[0108] In the specific implementation, the first noise image is used as the first current noise image, the second noise image is used as the second current noise image, and the third noise image is used as the third current noise image. The first current noise image is input into SDM-1, the second current noise image and the text prompt information are input into SDM-2, and the third current noise image and the text prompt information are input into SDM-3.

[0109] The first current feature vector is extracted from the first current noise image using SDM-1, and a first predicted noise image is obtained. The second current feature vector is extracted from the second current noise image and the text prompt information using SDM-2, and a second predicted noise image is obtained. The third current feature vector is extracted from the third current noise image and the text prompt information using SDM-2. An intermediate feature vector is obtained based on at least one of the first, second, and third current feature vectors. The third predicted noise image is obtained based on the intermediate feature vector using SDM-3.

[0110] Then, after denoising the first, second, and third current noise images respectively, the process returns to the step of inputting the first current noise image into SDM-1, the second current noise image and text prompt information into SDM-2, and the third current noise image and text prompt information into SDM-3. When the current iteration number is the preset maximum iteration number, the target feature vector is obtained. The target fused image is obtained by the decoder of SDM-3 based on the target feature vector.

[0111] Different feature fusion strategies can be employed at different stages of the image fusion process. An intermediate feature vector is obtained based on at least one of the first, second, and third current feature vectors. The fusion feature strategy for each stage can be determined according to the specific process characteristics.

[0112] Therefore, this embodiment provides an image fusion method that generates a target fused image by performing multiple denoising operations on the feature vectors of the noisy image and the text prompt information, thereby reducing the noise impact on the target fused image and improving the quality of the target fused image.

[0113] In one feasible implementation, the step "obtaining an intermediate feature vector based on at least one of the first current feature vector, the second current feature vector, and the third current feature vector" may include: if the current iteration number is less than or equal to a first preset iteration number, using the third current feature vector as the intermediate feature vector; if the current iteration number is greater than the first preset iteration number and less than the second preset iteration number, fusing the first current feature vector, the second current feature vector, and the third current feature vector to obtain the intermediate feature vector; and if the current iteration number is greater than or equal to the second preset iteration number, fusing the second current feature vector and the third current feature vector to obtain the intermediate feature vector.

[0114] It should be noted that the image fusion process can include a layout stage, a content generation stage, and a material / style generation stage. In the layout stage, the current iteration number is less than or equal to a first preset iteration number; in the content generation stage, the current iteration number is greater than the first preset iteration number and less than a second preset iteration number; in the material / style generation stage, the current iteration number is greater than or equal to the second preset iteration number and less than a preset maximum iteration number.

[0115] In this embodiment, the feature fusion strategy is as follows: Formula 2:

[0116]

[0117] Where t is the time step, T is the total time step, and α is the third current feature vector f. e Feature weights, β, the second current feature vector f p The feature weights, where γ is the first current feature vector f n Feature weights, layout stage t < τ²T, content generation stage t ∈ (τ a T,τ b T), Material / style generation stage t>τ b T,f BD This is the intermediate feature vector.

[0118] It is understandable that t < τ a T represents the layout stage, indicating the initial phase of the SDM model generating the target fused image from the noisy image. No operation is performed on the third current feature vector output from the intermediate layer of SDM-3. f is then used... e Generate the approximate layout of the target fused image.

[0119] t∈(τ a T,τ b T) represents the content generation stage, indicating the intermediate stage in the SDM model's generation of the target fused image from the noisy image. This stage requires combining three feature vectors to generate the content of the target fused image. Specifically, during the content generation stage, when setting the values ​​of α, β, and γ, it is necessary to consider f... p It simultaneously includes information from the fused image and text prompts, based on f p To generate the most suitable target fusion image, it is necessary to provide f. p Set a larger weight β. n f e As an auxiliary tool, it is used to supplement image content and smooth the transition between the target subject and its surrounding environment. The weights of each feature can be obtained by conducting comparative experiments using multiple sets of parameters and then determining the optimal parameters, for example, α = 0.25, β = 0.5, γ = 0.25.

[0120] t>τ bT represents the material / style generation stage, indicating the later stage of the SDM model generating the target fused image from the noisy image. This occurs after the content of the target fused image is largely generated, and then f... p f e Control the final stage of generation to adjust the material and style characteristics of the target fused image.

[0121] Therefore, this embodiment provides an image fusion method that divides the image fusion process into a layout stage, a content generation stage, and a material / style generation stage. The fusion weights of the feature vectors are dynamically adjusted at different stages, so that the generated target fused image can both retain the background of the original image and correctly embed the target subject in the reference image, thereby improving the quality of the target fused image.

[0122] This application provides an image fusion apparatus, such as... Figure 5 As shown, the image fusion apparatus may include:

[0123] The acquisition module 10 is used to acquire the original image, the reference image, and the text prompt information, wherein the reference image includes the target subject, and the original image includes the editing area where the target subject is to be added;

[0124] The first generation module 20 is used to generate a first noise image and a second noise image. The first noise image is a noise image of a fused image obtained by fusing the target subject with the original image. The second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise. The background region is the region in the editing region other than the target subject.

[0125] The feature extraction module 30 is used to extract a first feature vector, a second feature vector, and a third feature vector, wherein the first feature vector is extracted from the first noisy image, the second feature vector is extracted from the first noisy image and the text prompt information, and the third feature vector is extracted from the second noisy image and the text prompt information;

[0126] The feature fusion module 40 is used to fuse the first feature vector, the second feature vector, and the third feature vector to obtain the target feature vector;

[0127] The second generation module 50 is used to generate a target fusion image based on the target feature vector.

[0128] For more details on the specific implementation of the above image fusion device, please refer to the description of the specific implementation of the image fusion method in the above embodiments. For the sake of brevity, these details will not be repeated here.

[0129] This application provides an image fusion apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image fusion method in the first embodiment described above.

[0130] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an image fusion device suitable for implementing embodiments of this application. The image fusion device in the embodiments of this application may include, but is not limited to, mobile terminals such as laptops, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as desktop computers. Figure 6 The image fusion device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 6 As shown, the image fusion device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image fusion device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the image fusion device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image fusion devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0132] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0133] The image fusion device provided in this application, employing the image fusion method described in the above embodiments, can solve the technical problem of how to expand the application scenarios of image editing methods. Compared with related technologies, the beneficial effects of the image fusion device provided in this application are the same as those of the image fusion method described in the above embodiments, and other technical features of this image fusion device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0136] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image fusion method described in the above embodiments.

[0137] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0138] The aforementioned computer-readable storage medium may be included in the image fusion apparatus or may exist independently without being assembled into the image fusion apparatus.

[0139] The aforementioned computer-readable storage medium carries one or more programs. When the image fusion device executes the aforementioned one or more programs, the image fusion device causes the image fusion device to: acquire an original image, a reference image, and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area where the target subject is to be added; generate a first noise image and a second noise image, wherein the first noise image is a noise image of the fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling the background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is the area in the editing area other than the target subject; extract a first feature vector, a second feature vector, and a third feature vector, wherein the first feature vector is extracted from the first noise image, the second feature vector is extracted from the first noise image and the text prompt information, and the third feature vector is extracted from the second noise image and the text prompt information; fuse the first feature vector, the second feature vector, and the third feature vector to obtain a target feature vector; and generate a target fused image based on the target feature vector.

[0140] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0143] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image fusion method, thereby solving the technical problem of how to expand the application scenarios of image editing methods. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image fusion method provided in the above embodiments, and will not be repeated here.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image fusion method described above.

[0145] The computer program product provided in this application solves the technical problem of how to expand the application scenarios of image editing methods. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the image fusion method provided in the above embodiments, and will not be repeated here.

[0146] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image fusion method, characterized in that, The image fusion method includes: Acquire an original image, a reference image, and text prompt information, wherein the reference image includes the target subject, and the original image includes an editing area where the target subject is to be added; A first noise image and a second noise image are generated, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise, wherein the background region is the region in the editing region other than the target subject; A first feature vector, a second feature vector, and a third feature vector are extracted, wherein the first feature vector is extracted from the first noisy image, the second feature vector is extracted from the first noisy image and the text prompt information, and the third feature vector is extracted from the second noisy image and the text prompt information; The target feature vector is obtained by fusing the first feature vector, the second feature vector, and the third feature vector. A target fusion image is generated based on the target feature vector.

2. The image fusion method as described in claim 1, characterized in that, The steps for extracting the first feature vector, the second feature vector, and the third feature vector include: The first noise image is used as the first current noise image, the first noise image is used as the second current noise image, and the second noise image is used as the third current noise image; A first current feature vector, a second current feature vector, and a third current feature vector are extracted. The first current feature vector is extracted from the first current noise image, the second current feature vector is extracted from the second current noise image and the text prompt information, and the third current feature vector is extracted from the third current noise image and the text prompt information. The step of fusing the first feature vector, the second feature vector, and the third feature vector to obtain the target feature vector includes: An intermediate feature vector is obtained based on at least one of the first current feature vector, the second current feature vector, and the third current feature vector; Before the step of generating the target fused image based on the target feature vector, the image fusion method further includes: Based on the first current feature vector, a first predicted noise image is obtained; Based on the second current feature vector, a second predicted noise image is obtained; Based on the intermediate feature vector, a third predicted noise image is obtained; Using the first predicted noise image, the first current noise image is denoised to obtain a first intermediate noise image; Using the second predicted noise image, the second current noise image is denoised to obtain the second intermediate noise image; Using the third predicted noise image, the third current noise image is denoised to obtain the third intermediate noise image; The first intermediate noise image is used as the first current noise image, the second intermediate noise image is used as the second current noise image, and the third intermediate noise image is used as the third current noise image. Then, the process returns to the step of extracting the first current feature vector, the second current feature vector, and the third current feature vector. The intermediate feature vector is used as the target feature vector until the current iteration number reaches the preset maximum iteration number.

3. The image fusion method as described in claim 2, characterized in that, The step of obtaining an intermediate feature vector based on at least one of the first current feature vector, the second current feature vector, and the third current feature vector includes: If the current iteration number is less than or equal to the first preset iteration number, the third current feature vector is used as the intermediate feature vector; If the current iteration number is greater than the first preset iteration number and less than the second preset iteration number, the first current feature vector, the second current feature vector, and the third current feature vector are fused to obtain the intermediate feature vector; If the current iteration number is greater than or equal to the second preset iteration number, the second current feature vector and the third current feature vector are fused to obtain the intermediate feature vector.

4. The image fusion method as described in claim 1, characterized in that, The steps for extracting the first feature vector, the second feature vector, and the third feature vector include: The first noise image is input into the first stable diffusion model to extract the first feature vector. The first noise image and the text prompt information are input into the second stable diffusion model to extract the second feature vector; The second noise image and the text prompt information are input into the third stable diffusion model to extract the third feature vector.

5. The image fusion method according to any one of claims 1 to 4, characterized in that, The step of generating the first noisy image includes: The target subject is extracted from the reference image; The target subject is fused into the editing area of ​​the original image to obtain a fused image; The fused image is converted into a first noisy image.

6. The image fusion method according to any one of claims 1 to 4, characterized in that, The step of generating the second noisy image includes: A mask image is generated based on the fused image of the original image and the target subject, wherein the mask image is used to extract the background region in the fused image corresponding to the editing region; Based on the mask image and the random noise image, a first intermediate noise image is obtained; A second intermediate noise image is obtained by combining the inverse mask image of the mask image with the noise image of the fused image. The first intermediate noise image and the second intermediate noise image are fused together to obtain the second noise image.

7. An image fusion apparatus, characterized in that, The image fusion device includes: The acquisition module is used to acquire an original image, a reference image, and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area where the target subject is to be added; A first generation module is used to generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling the background region corresponding to the editing region in the first noise image with random Gaussian noise, wherein the background region is the region in the editing region other than the target subject; The feature extraction module is used to extract a first feature vector, a second feature vector, and a third feature vector, wherein the first feature vector is extracted from the first noisy image, the second feature vector is extracted from the first noisy image and the text prompt information, and the third feature vector is extracted from the second noisy image and the text prompt information; The feature fusion module is used to fuse the first feature vector, the second feature vector, and the third feature vector to obtain the target feature vector; The second generation module is used to generate a target fusion image based on the target feature vector.

8. An image fusion device, characterized in that, The image fusion device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image fusion method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image fusion method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the image fusion method as described in any one of claims 1 to 6.

Images