Image fusion method and device, equipment, storage medium and program product

Abstract

The invention discloses an image fusion method and device, equipment, a storage medium and a program product, and relates to the technical field of image processing. Filling random Gaussian noise in a background region corresponding to the editing region in the first noise image to generate a second noise image, and extracting to obtain a first feature vector, a second feature vector and a third feature vector by using the first noise image, the first noise image combined with the text prompt information and the second noise image combined with the text prompt information respectively, and a target characteristic vector for generating a target fusion image is obtained through fusion. When image editing is carried out, the pre-trained SDM model can be directly applied to generate the target fusion image according to the target feature vector, and the technical problem of how to expand the application scene of the image editing method is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, device, equipment, storage medium and program product. Background Art

[0002] In the related art, the SDM (Stable Diffusion Model) model can be used to implement image editing based on the guidance of reference images and / or text prompts. However, for different image editing scenarios, the pre-trained SDM model needs to be further fine-tuned and trained to achieve the corresponding editing effect. Model fine-tuning and model fine-tuning require a lot of computing power and data support, which limits the application scenarios of image editing methods. Summary of the invention

[0003] The main purpose of this application is to provide an image fusion method, device, equipment, storage medium and program product, aiming to solve the technical problem of how to expand the application scenarios of image editing methods.

[0004] To achieve the above purpose, the present application proposes an image fusion method, which includes:

[0005] Acquire an original image, a reference image, and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area to which the target subject is to be added;

[0006] Generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling a background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is an area other than the target subject in the editing area;

[0007] Extracting a first eigenvector, a second eigenvector, and a third eigenvector, wherein the first eigenvector is extracted from the first noise image, the second eigenvector is extracted from the first noise image and text prompt information, and the third eigenvector is extracted from the second noise image and text prompt information;

[0008] The first eigenvector, the second eigenvector and the third eigenvector are fused to obtain a target eigenvector;

[0009] Generate a target fusion image based on the target feature vector.

[0010] In one embodiment, the step of extracting the first eigenvector, the second eigenvector and the third eigenvector comprises:

[0011] Using the first noise image as a first current noise image, using the first noise image as a second current noise image, and using the second noise image as a third current noise image;

[0012] Extracting a first current feature vector, a second current feature vector, and a third current feature vector, wherein the first current feature vector is extracted from the first current noise image, the second current feature vector is extracted from the second current noise image and text prompt information, and the third current feature vector is extracted from the third current noise image and text prompt information;

[0013] The step of fusing the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector includes:

[0014] Obtaining an intermediate feature vector according to at least one of the first current feature vector, the second current feature vector, and the third current feature vector;

[0015] Before the step of generating a target fused image according to the target feature vector, the image fusion method further includes:

[0016] Obtaining a first predicted noise image according to the first current feature vector;

[0017] Obtaining a second predicted noise image according to the second current feature vector;

[0018] Obtaining a third predicted noise image according to the intermediate feature vector;

[0019] Using the first predicted noise image, denoising the first current noise image to obtain a first intermediate noise image;

[0020] Using the second predicted noise image, denoising the second current noise image to obtain a second intermediate noise image;

[0021] Using the third predicted noise image, denoising the third current noise image to obtain a third intermediate noise image;

[0022] Using the first intermediate noise image as the first current noise image, using the second intermediate noise image as the second current noise image, and using the third intermediate noise image as the third current noise image, returning to the step of extracting to obtain the first current feature vector, the second current feature vector, and the third current feature vector;

[0023] Until the current number of iterations reaches the preset maximum number of iterations, the intermediate feature vector is used as the target feature vector.

[0024] In one embodiment, the step of obtaining an intermediate feature vector according to at least one of the first current feature vector, the second current feature vector and the third current feature vector comprises:

[0025] When the current number of iterations is less than or equal to the first preset number of iterations, taking the third current feature vector as the intermediate feature vector;

[0026] When the current number of iterations is greater than the first preset number of iterations and less than the second preset number of iterations, the first current feature vector, the second current feature vector and the third current feature vector are fused to obtain an intermediate feature vector;

[0027] When the current number of iterations is greater than or equal to the second preset number of iterations, the second current feature vector and the third current feature vector are fused to obtain an intermediate feature vector.

[0028] In one embodiment, the step of extracting the first eigenvector, the second eigenvector and the third eigenvector comprises:

[0029] Inputting the first noise image into a first stable diffusion model to extract a first eigenvector;

[0030] Inputting the first noise image and the text prompt information into a second stable diffusion model to extract a second eigenvector;

[0031] The second noise image and the text prompt information are input into a third stable diffusion model to extract a third eigenvector.

[0032] In one embodiment, the step of generating a first noise image includes:

[0033] Extract the target subject from the reference image;

[0034] The target subject is fused into the editing area of ​​the original image to obtain a fused image;

[0035] Convert the fused image to the first noisy image.

[0036] In one embodiment, the step of generating the second noise image includes:

[0037] Generate a mask image according to the fused image of the original image and the target subject, wherein the mask image is used to extract the background area corresponding to the editing area in the fused image;

[0038] Obtaining a first intermediate noise image according to the mask image and the random noise image;

[0039] Obtaining a second intermediate noise image according to the inverse mask image of the mask image and the noise image of the fused image;

[0040] The first intermediate noise image and the second intermediate noise image are fused to obtain a second noise image.

[0041] In addition, to achieve the above purpose, the present application also proposes an image fusion device, which includes:

[0042] An acquisition module, used to acquire an original image, a reference image and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area to which the target subject is to be added;

[0043] A first generating module is used to generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling a background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is an area other than the target subject in the editing area;

[0044] A feature extraction module, used to extract a first feature vector, a second feature vector and a third feature vector, wherein the first feature vector is extracted from the first noise image, the second feature vector is extracted from the first noise image and text prompt information, and the third feature vector is extracted from the second noise image and text prompt information;

[0045] A feature fusion module, used to fuse the first feature vector, the second feature vector and the third feature vector to obtain a target feature vector;

[0046] The second generating module is used to generate a target fused image according to the target feature vector.

[0047] In addition, to achieve the above objectives, the present application also proposes an image fusion device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image fusion method as described above.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image fusion method as described above are implemented.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the image fusion method as described above are implemented.

[0050] One or more technical solutions proposed in this application have at least the following technical effects:

[0051] The present application provides an image fusion method, apparatus, device, storage medium and program product, which generate a first noise image of a fused image of an original image and a target subject, fill a background area corresponding to an editing area in the first noise image with random Gaussian noise, generate a second noise image, respectively use the first noise image, the first noise image combined with text prompt information, and the second noise image combined with text prompt information to extract a first feature vector, a second feature vector and a third feature vector to fuse to obtain a target feature vector for generating a target fused image; since the first feature vector fuses the background features and subject features of the fused image, the second feature vector adds the text prompt information on the basis of the first feature vector. Until then, a fusion image that satisfies the text semantics as much as possible can be generated. The third eigenvector not only integrates the background features, subject features and text features of the fusion image, but also integrates the random Gaussian noise of the background area, which can make the target subject and the surrounding background in the generated target fusion image transition naturally. Therefore, when editing the image, the pre-trained SDM model can be directly applied to generate the target fusion image according to the target feature vector. There is no need to further fine-tune and train the pre-trained SDM model for different application scenarios, which reduces the complex and time-consuming model fine-tuning process. There is no need to collect sample data and consume computing power separately, which greatly expands the application scenarios of the image editing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a flow chart of the first embodiment of the image fusion method of the present application;

[0055] Figure 2 A schematic diagram of an exemplary image fusion process provided for the image fusion method of the present application;

[0056] Figure 3 A schematic diagram of an exemplary noise image generation process provided for the image fusion method of the present application;

[0057] Figure 4 A schematic diagram of the structure of a U-Net network in an exemplary SDM model provided for the image fusion method of the present application;

[0058] Figure 5 This is a schematic diagram of the modules of the first embodiment of the image fusion device of the present application;

[0059] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the image fusion method in the embodiment of the present application.

[0060] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0062] In this application, the term "comprises", "includes" or any other variation thereof is intended to cover non-exclusive inclusion, so that a device or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a device or system. In the absence of more restrictions, an element defined by the sentence "includes..." does not exclude the presence of other identical elements in the device or system including the element.

[0063] If there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0064] In the present application, the suffixes such as "module", "component" or "unit" used to represent elements are used only to facilitate the description of the present application and have no specific meaning in themselves. Therefore, "module", "component" or "unit" can be used in a mixed manner. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, the technical solutions of the various embodiments can be combined with each other, but they must be based on the ability of those of ordinary skill in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0065] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of the embodiment of the present application is: obtaining an original image, a reference image and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area to which the target subject is to be added;

[0067] Generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing a target subject with an original image, and the second noise image is obtained by filling a background area corresponding to an editing area in the first noise image with random Gaussian noise, and the background area is an area other than the target subject in the editing area; extract a first eigenvector, a second eigenvector, and a third eigenvector, wherein the first eigenvector is extracted from the first noise image, the second eigenvector is extracted from the first noise image and text prompt information, and the third eigenvector is extracted from the second noise image and text prompt information; fuse the first eigenvector, the second eigenvector, and the third eigenvector to obtain a target eigenvector; and generate a target fused image according to the target eigenvector.

[0068] With the continuous advancement of deep learning technology, image editing and generation technology has achieved remarkable results. Diffusion models, especially advanced models such as Stable Diffusion, can generate highly realistic images that match the text description based on the text prompts provided by the user, greatly promoting the development of art creation and design.

[0069] In the related art, the SDM model can use a reference image to perform image modification on a face image to be edited so that the face in the modified target image is reflected as the face in the reference image, that is, relying on a single reference image to guide the image editing process.

[0070] The SDM model can also use the input text prompt as a content information condition to edit the video image, thereby obtaining a new video with the same structure as the original video and the same content as the text prompt, that is, relying on a single reference text to guide the image editing process.

[0071] It can be seen that in related image editing methods, a single reference image or text prompt is often relied on to guide the image editing process, which to a certain extent limits the diversity and flexibility of image editing methods.

[0072] In addition, for different image editing scenarios, the first encoder, second encoder, attention model and decoder used for image editing in the pre-trained SDM model need to be further fine-tuned and trained to achieve the corresponding editing effect. Model fine-tuning and model fine-tuning require a lot of computing power and data support, which limits the application scenarios of image editing methods.

[0073] Therefore, the present application provides a solution, using the first feature vector, the second feature vector and the third feature vector to fuse and obtain the target characteristic vector for generating the target fused image; since the first feature vector fuses the background features and the main features of the fused image, the second feature vector adds the text prompt information on the basis of the first feature vector until, a fused image that satisfies the text semantics as much as possible can be generated, and the third feature vector not only fuses the background features, the main features and the text features of the text prompt information of the fused image, but also fuses the random Gaussian noise of the background area, so that the target subject and the surrounding background in the generated target fused image can be naturally transitioned, so that when performing image editing, the pre-trained SDM model can be directly applied to generate the target fused image according to the target feature vector, and there is no need to further fine-tune and train the pre-trained SDM model for different application scenarios, which reduces the complex and time-consuming model fine-tuning process, and does not need to collect sample data and consume computing power separately, which greatly expands the application scenarios of the image editing method. In addition, since the main features of the reference image and the text features of the text prompt information are fused in the target feature vector, the image editing process guided by the reference image and the text prompt information is realized, which increases the diversity and flexibility of the image editing method.

[0074] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an image fusion device, etc. The following takes the image fusion device as an example to illustrate this embodiment and the following embodiments.

[0075] Based on this, the present application embodiment provides an image fusion method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the image fusion method of the present application.

[0076] In this embodiment, the image fusion method may include steps S100 to S500:

[0077] Step S100, obtaining an original image, a reference image and text prompt information.

[0078] The reference image includes the target subject, and the original image includes the editing area to which the target subject is to be added.

[0079] It should be noted that if Figure 2As shown, during the image editing process, the original image is the image to be edited, and the editing area is the area in the original image that needs to be edited. The position of the editing area can be adjusted according to the editing requirements, and the size of the editing area can also be adjusted according to the editing requirements. It is only necessary to ensure that after the target subject in the reference image is added to the editing area, the target subject is located inside the original image. The reference image is an image used to guide image editing, and the reference image includes the target subject that needs to be edited into the original image. The text prompt information is the text used to guide image editing. For example, Figure 2 The target subject in the reference image shown is a Chow Chow dog, and the text prompt information is "a photo of a Chow Chow dog lying on the beach."

[0080] Step S200: generating a first noise image and a second noise image.

[0081] Among them, the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling random Gaussian noise in the background area corresponding to the editing area in the first noise image, and the background area is the area outside the editing area except the target subject.

[0082] It should be noted that the target subject is fused into the edited area in the original image to obtain a fused image, and the fused image is noisy to obtain a first noise image. The background area corresponding to the edited area in the first noise image is filled with random Gaussian noise to obtain a second noise image.

[0083] In a possible implementation, Figure 3 As shown, step S100 may include: extracting a target subject from a reference image; fusing the target subject into an editing region of the original image to obtain a fused image; and converting the fused image into a first noise image.

[0084] It should be noted that the target subject can be extracted from the reference image by a cutout algorithm. The specific cutout algorithm can be selected according to actual needs, for example, the Grab Cut algorithm. After the target subject is extracted, the target subject is placed in the editing area of ​​the original image to obtain a fused image. The first noise image can be obtained by adding noise to the fused image using a conversion algorithm such as DDIM (DenoisingDiffusion Implicit Models) or DPM-Solver. Among them, DPM-Solver is a high-order ODE solver, which is often used to generate noise in SDM models and can generate better potential expressions.

[0085] In a possible implementation, Figure 3As shown, step S100 may include: generating a mask image according to a fused image of the original image and the target subject, wherein the mask image is used to extract a background area corresponding to the editing area in the fused image; obtaining a first intermediate noise image according to the mask image and the random noise image; obtaining a second intermediate noise image according to an inverse mask image of the mask image and the noise image of the fused image; and obtaining a second noise image by fusing the first intermediate noise image and the second intermediate noise image.

[0086] It should be noted that if Figure 3 As shown, the mask image M reflects the background area excluding the target subject in the editing area of ​​the fused image.

[0087] The second noise image can be obtained using the following formula 1:

[0088]

[0089] is the second noise image, M is the mask image, N is the random noise image, Z T is the first noise image, ⊙ represents an XOR operation. A random noise image can be randomly generated using a noise generator.

[0090] Step S300: extracting a first eigenvector, a second eigenvector and a third eigenvector.

[0091] The first feature vector is extracted from the first noise image, the second feature vector is extracted from the first noise image and the text prompt information, and the third feature vector is extracted from the second noise image and the text prompt information.

[0092] It should be noted that the first feature vector integrates the background features and subject features of the fused image, the second feature vector integrates not only the background features and subject features of the fused image, but also the text features of the text prompt information, and can generate a target fused image that satisfies the text semantics as much as possible, and the third feature vector integrates not only the background features, subject features and text features of the fused image, but also the random Gaussian noise of the background area, and can make the target subject and the surrounding background in the generated target fused image naturally transition. Among them, the feature vector can be extracted using a pre-trained stable diffusion model.

[0093] Step S400: fusing the first feature vector, the second feature vector and the third feature vector to obtain a target feature vector.

[0094] Step S500: generating a target fused image according to the target feature vector.

[0095] It should be noted that the feature fusion strategy may include directly fusing the first feature vector, the second feature vector and the third feature vector, or fusing the first feature vector, the second feature vector and the third feature vector according to the preset feature weights of the feature vectors. The target feature vector is input into the decoder, and the target feature vector can be decoded into the edited target fused image.

[0096] Therefore, this embodiment provides an image fusion method, because the first feature vector fuses the background features and main features of the fused image, the second feature vector not only fuses the background features and main features of the fused image, but also fuses the text features of the text prompt information, and can generate a target fused image that satisfies the text semantics as much as possible, and the third feature vector not only fuses the background features, main features and text features of the fused image, but also fuses the random Gaussian noise of the background area, so that the target subject and the surrounding background in the generated target fused image can be naturally transitioned, so that when performing image editing, the pre-trained SDM model can be directly applied to generate the target fused image according to the target feature vector, and there is no need to further fine-tune and train the pre-trained SDM model for different application scenarios, which reduces the complex and time-consuming model fine-tuning process, and does not need to collect sample data and consume computing power separately, which greatly expands the application scenarios of the image editing method. In addition, because the main features of the reference image and the text features of the text prompt information are fused in the target feature vector, the image editing process guided by the reference image and the text prompt information is realized, which increases the diversity and flexibility of the image editing method.

[0097] In a feasible implementation, step S300 may include: inputting the first noise image into a first stable diffusion model to extract a first eigenvector; inputting the first noise image and text prompt information into a second stable diffusion model to extract a second eigenvector; inputting the second noise image and text prompt information into a third stable diffusion model to extract a third eigenvector.

[0098] It should be noted that the image fusion device can use the pre-trained SDM model to process the first noise image, the second noise image and the text prompt information to extract the corresponding feature vector.

[0099] like Figure 2 As shown, in this embodiment, feature extraction is performed through three pre-trained SDM models, namely, the first stable diffusion model SDM-1, the second stable diffusion model SDM-2 and the third stable diffusion model SDM-3. SDM-1, SDM-2 and SDM-3 are the same pre-trained models.

[0100] In the feature extraction process, the first noise image is input into SDM-1 to extract the first feature vector; the first noise image and text prompt information are input into SDM-2 to extract the second feature vector; the second noise image and text prompt information are input into SDM-2 to extract the third feature vector. The feature vector can be extracted by the U-Net network in the SDM model, such as Figure 4 As shown, the feature vector can be the self-attention feature of the middle layer CrossAttnBlock of the U-Net network.

[0101] It can be understood that the SDM model is the basic model for image generation, and the decoder of the SDM model can generate the target fused image according to the target feature vector. Figure 2 As shown, after obtaining the target feature vector, the target feature vector is input into the decoder of SDM-3 to generate the target fused image.

[0102] Therefore, this embodiment provides an image fusion method, which can directly apply the pre-trained SDM model to extract the corresponding feature vector according to the input image, and generate the target fused image according to the target feature vector. There is no need to further fine-tune and re-train the pre-trained SDM model for different application scenarios, which reduces the complex and time-consuming model fine-tuning process, and there is no need to collect sample data and consume computing power separately, thereby greatly expanding the application scenarios of the image editing method.

[0103] In a feasible implementation, step S300 may include: taking the first noise image as the first current noise image, taking the first noise image as the second current noise image, and taking the second noise image as the third current noise image; extracting a first current feature vector, a second current feature vector, and a third current feature vector, wherein the first current feature vector is extracted from the first current noise image, the second current feature vector is extracted from the second current noise image and text prompt information, and the third current feature vector is extracted from the third current noise image and text prompt information.

[0104] Correspondingly, step S400 may include: obtaining an intermediate feature vector according to at least one of the first current feature vector, the second current feature vector and the third current feature vector.

[0105] Before step S500, the image fusion method may further include: obtaining a first predicted noise image according to the first current feature vector; obtaining a second predicted noise image according to the second current feature vector; obtaining a third predicted noise image according to the intermediate feature vector; using the first predicted noise image, denoising the first current noise image to obtain a first intermediate noise image; using the second predicted noise image, denoising the second current noise image to obtain a second intermediate noise image; using the third predicted noise image, denoising the third current noise image to obtain a third intermediate noise image; using the first intermediate noise image as the first current noise image, the second intermediate noise image as the second current noise image, and the third intermediate noise image as the third current noise image, and returning to the step of extracting the first current feature vector, the second current feature vector and the third current feature vector; until the current number of iterations reaches the preset maximum number of iterations, the intermediate feature vector is used as the target feature vector.

[0106] It should be noted that the pre-trained SDM model may include a U-Net network of T time steps, where T is the preset maximum number of iterations. The current feature vector of the current iteration number is extracted from the U-Net network of the tth time step. During the first iteration, the three current feature vectors are extracted from the first noise image, the first noise image and text prompt information, and the second noise image and text prompt information by the U-Net network of the 1st time step of SDM-1, SDM-2, and SDM-3, respectively. In the subsequent iterations, the three current feature vectors are extracted from the first current noise image, the second current noise image and text prompt information, and the third current noise image and text prompt information by the U-Net network of the tth time step of SDM-1, SDM-2, and SDM-3, respectively.

[0107] It can be understood that after each iteration, before entering the next iteration, the U-Net network can be used to denoise the current noise image based on the predicted noise image determined by the current feature vector to obtain an intermediate noise image as the input of the U-Net network at the next time step for the next iteration. Until t = T, the target feature vector containing T times of denoising is decoded into the target fused image.

[0108] In a specific implementation, the first noise image is used as the first current noise image, the first noise image is used as the second current noise image, and the second noise image is used as the third current noise image. The first current noise image is input into SDM-1, the second current noise image and text prompt information are input into SDM-2, and the third current noise image and text prompt information are input into SDM-3.

[0109] The first current feature vector is extracted from the first current noise image through SDM-1, and the first predicted noise image is obtained; the second current feature vector is extracted from the second current noise image and the text prompt information through SDM-2, and the second predicted noise image is obtained; the third current feature vector is extracted from the third current noise image and the text prompt information through SDM-2; the intermediate feature vector is obtained according to at least one of the first current feature vector, the second current feature vector and the third current feature vector; the third predicted noise image is obtained according to the intermediate feature vector through SDM-3.

[0110] Then, after denoising the first current noise image, the second current noise image and the third noise image respectively, the first current noise image is input into SDM-1, the second current noise image and the text prompt information are input into SDM-2, and the third current noise image and the text prompt information are input into SDM-3, and when the current number of iterations is the preset maximum number of iterations, the target feature vector is obtained. The target fused image is obtained according to the target feature vector by the decoder of SDM-3.

[0111] Different stages of the image fusion process can adopt different feature fusion strategies, and an intermediate feature vector is obtained according to at least one of the first current feature vector, the second current feature vector and the third current feature vector. The fusion feature strategies at different stages can be determined according to specific process characteristics.

[0112] Therefore, this embodiment provides an image fusion method, which generates a target fused image by performing multiple noise reduction on the feature vectors of the noise image and the text prompt information, thereby reducing the noise influence of the target fused image and improving the quality of the target fused image.

[0113] In a feasible implementation, the step of "obtaining an intermediate feature vector based on at least one of the first current feature vector, the second current feature vector and the third current feature vector" may include: when the current number of iterations is less than or equal to the first preset number of iterations, using the third current feature vector as the intermediate feature vector; when the current number of iterations is greater than the first preset number of iterations and less than the second preset number of iterations, fusing the first current feature vector, the second current feature vector and the third current feature vector to obtain the intermediate feature vector; when the current number of iterations is greater than or equal to the second preset number of iterations, fusing the second current feature vector and the third current feature vector to obtain the intermediate feature vector.

[0114] It should be noted that the image fusion process may include a layout phase, a content generation phase, and a material / style generation phase. In the layout phase, the current iteration number is less than or equal to the first preset iteration number; in the content generation phase, the current iteration number is greater than the first preset iteration number and less than the second preset iteration number; in the material / style generation phase, the current iteration number is greater than or equal to the second preset iteration number and less than the preset maximum iteration number.

[0115] In this embodiment, the feature fusion strategy is as follows:

[0116]

[0117] Among them, t is the time step, T is the total time step, and α is the third current feature vector f e The feature weight, β, of the second current feature vector f p The feature weight of γ is the first current feature vector f n The feature weights of the layout stage t<τ2T and the content generation stage t∈(τ a T,τ b T), material / style generation stage t>τ b T,f BD is the intermediate eigenvector.

[0118] It is understandable that t<τ a T is the layout stage, which means that the SDM model generates the target fusion image from the noise image in the early stage. The third current feature vector output by the intermediate layer of SDM-3 is not operated. e Generate a rough layout of the target fused image.

[0119] t∈(τ a T,τ b T) is the content generation stage, which means the middle stage of the SDM model generating the target fused image from the noise image. It is necessary to combine the three feature vectors to generate the content of the target fused image. In the content generation stage, when setting the values ​​of α, β, and γ, it is necessary to consider f p It also contains the fused image information and text prompt information, based on f p It can generate the target fusion image that meets the conditions to the greatest extent, so it is necessary to p Set a larger weight β. f n 、f e As an auxiliary, it is used to supplement the image content and make the transition between the target subject and the surrounding environment smooth. Among them, each feature weight can be obtained according to the optimal parameters after comparative experiments using multiple sets of parameters, for example, α=0.25, β=0.5, γ=0.25.

[0120] t>τ bT is the material / style generation stage, which indicates the late stage of the SDM model generating the target fusion image from the noise image. After the content of the target fusion image is roughly generated, it is generated by f p , f e Control the final stage of generation to adjust properties such as the material and style of the target fused image.

[0121] Therefore, this embodiment provides an image fusion method, which divides the image fusion process into a layout stage, a content generation stage, and a material / style generation stage, and dynamically adjusts the fusion weights of the feature vectors at different stages, so that the generated target fused image can not only retain the background of the original image, but also correctly embed the target subject in the reference image, thereby improving the quality of the target fused image.

[0122] The present application provides an image fusion device, such as Figure 5 As shown, the image fusion device may include:

[0123] The acquisition module 10 is used to acquire an original image, a reference image and text prompt information, wherein the reference image includes a target subject and the original image includes an editing area to which the target subject is to be added;

[0124] A first generating module 20 is used to generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling a background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is an area other than the target subject in the editing area;

[0125] A feature extraction module 30, configured to extract a first feature vector, a second feature vector and a third feature vector, wherein the first feature vector is extracted from the first noise image, the second feature vector is extracted from the first noise image and the text prompt information, and the third feature vector is extracted from the second noise image and the text prompt information;

[0126] A feature fusion module 40 is used to fuse the first feature vector, the second feature vector and the third feature vector to obtain a target feature vector;

[0127] The second generating module 50 is used to generate a target fused image according to the target feature vector.

[0128] For more implementation details in the specific implementation of the above-mentioned image fusion device, please refer to the description of the specific implementation of the image fusion method in the above-mentioned embodiment. For the sake of brevity of the description, they will not be repeated here.

[0129] The present application provides an image fusion device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image fusion method in the above-mentioned embodiment 1.

[0130] Reference below Figure 6 , which shows a schematic diagram of the structure of an image fusion device suitable for implementing the embodiment of the present application. The image fusion device in the embodiment of the present application may include but is not limited to mobile terminals such as laptop computers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), etc., and fixed terminals such as desktop computers, etc. Figure 6 The image fusion device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 6 As shown, the image fusion device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image fusion device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image fusion device to communicate with other devices wirelessly or by wire to exchange data. Although the image fusion device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0132] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0133] The image fusion device provided by the present application adopts the image fusion method in the above embodiment, which can solve the technical problem of how to expand the application scenarios of the image editing method. Compared with the related art, the beneficial effects of the image fusion device provided by the present application are the same as the beneficial effects of the image fusion method provided by the above embodiment, and the other technical features in the image fusion device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0136] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the image fusion method in the above-mentioned embodiment.

[0137] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0138] The computer-readable storage medium may be included in the image fusion device; or may exist independently without being assembled into the image fusion device.

[0139] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image fusion device, the image fusion device: obtains an original image, a reference image and text prompt information, wherein the reference image includes a target subject and the original image includes an editing area to which the target subject is to be added; generates a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling a background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is an area other than the target subject in the editing area; extracts a first eigenvector, a second eigenvector and a third eigenvector, wherein the first eigenvector is extracted from the first noise image, the second eigenvector is extracted from the first noise image and the text prompt information, and the third eigenvector is extracted from the second noise image and the text prompt information; fuses the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector; and generates a target fused image according to the target eigenvector.

[0140] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0143] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image fusion method, and can solve the technical problem of how to expand the application scenarios of the image editing method. Compared with the related art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the image fusion method provided in the above-mentioned embodiment, and will not be repeated here.

[0144] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image fusion method when executed by a processor.

[0145] The computer program product provided by this application can solve the technical problem of how to expand the application scenarios of the image editing method. Compared with the related art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the image fusion method provided by the above embodiment, which will not be repeated here.

[0146] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image fusion method, characterized in that: The image fusion method comprises: Acquire an original image, a reference image, and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area to which the target subject is to be added; Generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling a background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is an area in the editing area excluding the target subject; Extracting a first eigenvector, a second eigenvector, and a third eigenvector, wherein the first eigenvector is extracted from the first noise image, the second eigenvector is extracted from the first noise image and the text prompt information, and the third eigenvector is extracted from the second noise image and the text prompt information; fusing the first feature vector, the second feature vector and the third feature vector to obtain a target feature vector; A target fused image is generated according to the target feature vector.

2. The image fusion method according to claim 1, characterized in that: The step of extracting the first eigenvector, the second eigenvector and the third eigenvector comprises: Using the first noise image as a first current noise image, using the first noise image as a second current noise image, and using the second noise image as a third current noise image; Extracting a first current feature vector, a second current feature vector, and a third current feature vector, wherein the first current feature vector is extracted from the first current noise image, the second current feature vector is extracted from the second current noise image and the text prompt information, and the third current feature vector is extracted from the third current noise image and the text prompt information; The step of fusing the first feature vector, the second feature vector and the third feature vector to obtain a target feature vector comprises: Obtaining an intermediate feature vector according to at least one of the first current feature vector, the second current feature vector, and the third current feature vector; Before the step of generating a target fused image according to the target feature vector, the image fusion method further comprises: Obtaining a first predicted noise image according to the first current feature vector; Obtaining a second predicted noise image according to the second current feature vector; Obtaining a third predicted noise image according to the intermediate feature vector; Using the first predicted noise image, denoising the first current noise image to obtain a first intermediate noise image; Using the second predicted noise image, denoising the second current noise image to obtain a second intermediate noise image; Using the third predicted noise image, denoising the third current noise image to obtain a third intermediate noise image; Using the first intermediate noise image as the first current noise image, using the second intermediate noise image as the second current noise image, and using the third intermediate noise image as the third current noise image, returning to the step of extracting to obtain the first current feature vector, the second current feature vector, and the third current feature vector; Until the current number of iterations reaches the preset maximum number of iterations, the intermediate feature vector is used as the target feature vector.

3. The image fusion method according to claim 2, characterized in that: The step of obtaining an intermediate feature vector according to at least one of the first current feature vector, the second current feature vector and the third current feature vector comprises: When the current number of iterations is less than or equal to the first preset number of iterations, using the third current feature vector as the intermediate feature vector; When the current number of iterations is greater than the first preset number of iterations and less than the second preset number of iterations, fusing the first current feature vector, the second current feature vector and the third current feature vector to obtain the intermediate feature vector; When the current number of iterations is greater than or equal to the second preset number of iterations, the second current feature vector and the third current feature vector are fused to obtain the intermediate feature vector.

4. The image fusion method according to claim 1, characterized in that: The step of extracting the first eigenvector, the second eigenvector and the third eigenvector comprises: Inputting the first noise image into a first stable diffusion model to extract the first feature vector; Inputting the first noise image and the text prompt information into a second stable diffusion model to extract the second feature vector; The second noise image and the text prompt information are input into a third stable diffusion model to extract the third eigenvector.

5. The image fusion method according to any one of claims 1 to 4, characterized in that: The step of generating a first noise image comprises: Extracting the target subject from the reference image; Merging the target subject into the editing area of ​​the original image to obtain a fused image; The fused image is converted into a first noisy image.

6. The image fusion method according to any one of claims 1 to 4, characterized in that: The step of generating a second noise image comprises: Generate a mask image according to the fused image of the original image and the target subject, wherein the mask image is used to extract a background area corresponding to the editing area in the fused image; Obtaining a first intermediate noise image according to the mask image and the random noise image; Obtaining a second intermediate noise image according to the inverse mask image of the mask image and the noise image of the fused image; The first intermediate noise image and the second intermediate noise image are fused to obtain the second noise image.

7. An image fusion device, characterized in that: The image fusion device comprises: An acquisition module, used to acquire an original image, a reference image and text prompt information, wherein the reference image includes a target subject, and the original image includes an editing area to which the target subject is to be added; A first generating module is used to generate a first noise image and a second noise image, wherein the first noise image is a noise image of a fused image obtained by fusing the target subject with the original image, and the second noise image is obtained by filling a background area corresponding to the editing area in the first noise image with random Gaussian noise, and the background area is an area in the editing area excluding the target subject; A feature extraction module, configured to extract a first feature vector, a second feature vector, and a third feature vector, wherein the first feature vector is extracted from the first noise image, the second feature vector is extracted from the first noise image and the text prompt information, and the third feature vector is extracted from the second noise image and the text prompt information; A feature fusion module, used for fusing the first feature vector, the second feature vector and the third feature vector to obtain a target feature vector; The second generating module is used to generate a target fused image according to the target feature vector.

8. An image fusion device, characterized in that: The image fusion device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image fusion method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 6 are implemented.

Images