Image fusion method and device, storage medium and program product

Abstract

The invention relates to an image fusion method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a first to-be-fused image, and performing foreground extraction on the first to-be-fused image to obtain a target object foreground image; performing first preset processing on the target object foreground image to obtain a first object foreground image; performing intra-region fusion according to the first object foreground image and the target region of the second to-be-fused image to obtain a candidate fusion image; performing second preset processing on the candidate fusion image according to the first object foreground image and the candidate fusion image through a target diffusion model to obtain a second fusion image; and performing consistency processing on the second fusion image to obtain a target fusion image. According to the image fusion method, the generated shadow can be closer to the shadow in the original image in direction and brightness on the premise that information such as the light source direction is not needed.

Description

Technical Field

[0001] The present application relates to the technical field of image fusion, and in particular to an image fusion method, device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of image fusion technology, shadow generation technology has emerged in the image fusion process to make the fused image more three-dimensional and realistic. However, there are two main types of shadow generation methods before. One is based on 3D rendering, which requires additional information such as the direction of the light source, but this is often difficult to obtain in practical applications. The other is based on the generative adversarial network method, but the shadows generated by this method are often inconsistent with other shadows in the original image in terms of direction and brightness, resulting in an unrealistic visual effect of the fused image. Summary of the invention

[0003] Based on this, the present application provides an image fusion method, device, computer-readable storage medium and computer program product, which are used to make the generated shadow closer to the shadow in the original image in terms of direction and brightness without the need for information such as the direction of the light source.

[0004] On the one hand, the present application provides an image fusion method, comprising:

[0005] Acquire a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of the target object;

[0006] Performing a first preset processing on the target object foreground image to obtain a first object foreground image;

[0007] Performing regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image;

[0008] performing a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image;

[0009] Perform consistency processing on the second fusion image to obtain a target fusion image.

[0010] In one embodiment, performing a first preset process on the target object foreground image to obtain a first object foreground image includes:

[0011] Performing an expansion and corrosion operation on the target object foreground image to obtain an edge uncertainty region image;

[0012] The edge model is constructed to perform edge segmentation and refinement processing on the edge uncertainty region map to obtain the first object foreground map.

[0013] In one embodiment, performing intra-region fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image includes:

[0014] Performing coordination processing on the first object foreground image according to the target area to obtain a coordinated foreground image;

[0015] The coordinated foreground image is fused with the target area to obtain the candidate fused image.

[0016] In one embodiment, the target diffusion model includes: a target control network and a target stable diffusion model; the target diffusion model is used to perform a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image to obtain a second fused image, including:

[0017] Extracting features from the first object foreground image and the candidate fused image through a target control network to obtain target shadow condition features;

[0018] The target stable diffusion model is used to generate shadows for the candidate fused images according to the target shadow condition characteristics to obtain the second fused image.

[0019] In one embodiment, performing consistency processing on the second fusion image to obtain a target fusion image includes:

[0020] Performing color and texture analysis on the second fusion image through a consistency processing model to obtain adjustment analysis data;

[0021] The second fusion image is adaptively adjusted according to the adjustment analysis data to obtain the target fusion image.

[0022] In one embodiment, the method further comprises:

[0023] The original diffusion model is trained according to the basic training shadow samples to obtain a fixed basic diffusion model;

[0024] Acquire newly added shadow training samples, and perform model training on the original control network according to the newly added training samples to obtain a newly added control network;

[0025] Building a model based on the fixed basic diffusion model and the newly added control network to obtain a candidate diffusion model;

[0026] The candidate diffusion model is subjected to loss adjustment processing by using a target loss function to obtain the target diffusion model.

[0027] In one embodiment, the model training of the original control network according to the newly added training samples to obtain the newly added control network includes:

[0028] Obtaining updated weights by performing low-rank decomposition on the model weights of the original control network;

[0029] Performing learning and adaptation on the updated weight according to the newly added training samples to obtain a target weight;

[0030] The model of the original control network is updated according to the target weight to obtain the newly added control network.

[0031] In one embodiment, the model training of the original control network according to the newly added training samples to obtain the newly added control network includes:

[0032] Obtaining updated weights by performing low-rank decomposition on the model weights of the original control network;

[0033] Performing learning and adaptation on the updated weight according to the newly added training samples to obtain a target weight;

[0034] The model of the original control network is updated according to the target weight to obtain the newly added control network.

[0035] On the one hand, the present application also provides an image fusion device, comprising:

[0036] A foreground extraction module, used to obtain a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of a target object;

[0037] A first processing module, configured to perform a first preset processing on the target object foreground image to obtain a first object foreground image;

[0038] A fusion module, configured to perform regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image;

[0039] A second processing module, configured to perform a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image;

[0040] The consistency processing module is used to perform consistency tuning on the shadow fusion image to obtain a target fusion image.

[0041] In one aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] Acquire a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of the target object;

[0043] Performing a first preset processing on the target object foreground image to obtain a first object foreground image;

[0044] Performing regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image;

[0045] performing a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image;

[0046] Perform consistency processing on the second fusion image to obtain a target fusion image.

[0047] In one aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0048] Acquire a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of the target object;

[0049] Performing a first preset processing on the target object foreground image to obtain a first object foreground image;

[0050] Performing regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image;

[0051] performing a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image;

[0052] Perform consistency processing on the second fusion image to obtain a target fusion image.

[0053] The above-mentioned image fusion method, device, computer-readable storage medium and computer program product obtain a target object foreground image by acquiring a first image to be fused and performing foreground extraction on the first image to be fused; performing a first preset processing on the target object foreground image to obtain a first object foreground image; performing regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image; performing a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through the target diffusion model to obtain a second fused image; performing consistency processing on the second fused image to obtain a target fused image. Therefore, by using the target diffusion model, the propagation of light and the formation process of shadows can be simulated, and the generated shadows are similar to the real images in terms of direction, shape and brightness, so that the shadows of the synthesized images are more natural and real, and the overall realism and visual effect of the image are improved. Then, the uniform processing is used to ensure that the fusion between the foreground and the background is more harmonious and natural, and the abrupt parts are avoided, thereby improving the visual consistency and overall effect of the entire image. Therefore, this scheme can realize an image fusion method that can still make the generated shadows closer to the shadows in the original image in terms of direction and brightness without the need for information such as the direction of the light source. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] FIG1 is a diagram of an application environment of an image fusion method in one embodiment;

[0056] FIG2 is a schematic diagram of a flow chart of an image fusion method in one embodiment;

[0057] FIG3 is a schematic diagram of a first preset processing flow of an image fusion method in one embodiment;

[0058] FIG4 is a schematic diagram of a second preset processing flow of an image fusion method in one embodiment;

[0059] FIG5 is a comparison diagram of fusion effects of an image fusion method in one embodiment;

[0060] FIG6 is a schematic diagram of a flow chart of an image fusion method in another embodiment;

[0061] FIG7 is a structural block diagram of an image fusion device in one embodiment;

[0062] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The image fusion method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1. The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the first image to be fused, and extracts the foreground of the first image to be fused to obtain the foreground image of the target object; performs a first preset processing on the foreground image of the target object to obtain the first object foreground image; performs regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image; through the target diffusion model, the candidate fused image is subjected to a second preset processing according to the first object foreground image and the candidate fused image to obtain a second fused image; and the second fused image is subjected to consistency processing to obtain a target fused image. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0065] In an exemplary embodiment, as shown in FIG2 , an image fusion method is provided, which is described by taking the method applied to the server 104 in FIG1 as an example, and includes the following steps 202 to 210. Among them:

[0066] Step 202: Acquire a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of the target object.

[0067] The first image to be fused is a foreground image used to provide a foreground object, and foreground extraction refers to separating the target object and the background part in the image.

[0068] In some embodiments, the first image to be fused may be captured in real time by a camera device to perform foreground extraction on a real-time target object, or different types of target objects may be acquired by acquiring the first image to be fused from different types of target object data sets.

[0069] Among them, the target object can be a person, a commodity, a pet, etc.

[0070] In some embodiments, foreground extraction is performed on the first image to be fused using a foreground extraction model to obtain a target object foreground image containing the target object in the first image to be fused, so as to separate the target object and the background in the first image to be fused.

[0071] Among them, the foreground extraction model can use saliency detection technology. It should be noted that this saliency detection technology is based on a dual-branch structure of a neural network architecture (transformer) and a convolutional neural network (CNN) to extract complementary features. The upper branch uses a new visual neural network architecture (Swin transformer) as an encoder, and the lower branch uses a residual neural network (Resnet) as an encoder. The center uses a cross-model grafting module (Cross-Model GraftingModule) to fuse the features of the upper and lower branches. On the right side, a series of decoder blocks (Decoderblocks (DB1, DB2, DB3)) are used for decoding and generating outputs. The entire saliency detection technology network has multiple auxiliary outputs and loss functions (aux sup, AGLoss) for supervised training. In addition, the model structure is trained on a self-built dataset to support automatic foreground extraction of multiple semantic types, such as people, goods, pets, etc.

[0072] Step 204: Perform a first preset process on the target object foreground image to obtain a first object foreground image.

[0073] The first preset processing may be edge thinning processing, or other processing techniques may be used to perform thinning processing on the foreground image of the target object, but the process is not limited thereto.

[0074] The target object foreground image is a binary image containing target object information, and the edge refinement process is used to complete the hair details of the target object in the target object foreground image.

[0075] In some embodiments, a first preset processing is performed on the target object foreground map to obtain a first object foreground map, including: performing an expansion and corrosion operation on the target object foreground map to obtain an edge uncertainty area map; and performing edge segmentation and refinement processing on the edge uncertainty area map by constructing an edge model to obtain the first object foreground map.

[0076] The edge uncertainty region map may adopt a trimap map, in which there are clear foreground target objects and backgrounds, as well as an uncertainty region around the target object. The edge model may be constructed by using an edge refine module.

[0077] In some embodiments, a fixed kernel dilation and erosion operation is performed on the target object foreground image to obtain an edge uncertainty region map, and the edge model is constructed to refine the edge segmentation details of the edge uncertainty region map according to the similarity between the foreground and background and other detail features to obtain the first object foreground image.

[0078] Among them, the fixed kernel refers to a fixed kernel, the foreground image of the first object is a finely segmented binary image, and the specific edge refinement processing flow can be referred to Figure 3, wherein the "foreground mask B" in Figure 3 refers to the foreground image of the target object, the "image A" in Figure 3 is the first image to be fused, and the "finely segmented image mask C" in Figure 3 is the foreground image of the first object.

[0079] It should be further explained that the foreground image of the target object segmented by only using saliency detection technology will be rather rigid, especially the details of human or animal hair will be lost. Such loss of edge details will inevitably cause distortion of the fused image. Therefore, it is necessary to use the constructed edge model to supplement the hair details to improve the authenticity and restoration of the fused image.

[0080] Step 206 , performing intra-region fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image.

[0081] The second image to be fused is a background image used to provide a background environment, and the target area is determined by user needs.

[0082] In some embodiments, intra-region fusion is performed on the target area of ​​the first object foreground image and the second image to be fused to obtain a candidate fused image, including: coordinately optimizing the first object foreground image according to the target area to obtain a coordinated foreground image; and fusing the coordinated foreground image with the target area to obtain a candidate fused image.

[0083] Among them, the coordination optimization is an adaptive optimization adjustment process performed through the image harmony model.

[0084] In some embodiments, the properties between the target area and the first object foreground image are analyzed for coordination through an image harmony model to obtain adaptive adjustment data, and then the first object foreground image is coordinately optimized through the adaptive adjustment data to obtain a coordinated foreground image, and then the foreground elements of the coordinated foreground image are pasted onto the target area for fusion to obtain a candidate fused image.

[0085] Among them, the adaptive adjustment data may include adjustment data of attributes such as color, brightness, and contrast. The image harmony model may adopt the region-aware adaptive instance normalization technology for image harmony to solve the image harmony problem, explicitly calculate the visual style of the target area and then use the image style transfer algorithm (AdaIN) to act on the foreground image of the first object. The overall framework uses a residual neural network structure and a decoder to extract image features. The decoder part uses the AdaIN module to estimate the two offset scales (scale) and bias (bias) of the background to act on the image element features of the foreground image of the first object.

[0086] In this embodiment, the image elements of the inserted or edited foreground image of the first object are adjusted to make it visually consistent with the background scene of the target area, mainly including coordination in terms of color, lighting, texture, etc.

[0087] Step 208 , performing a second preset process on the candidate fused image according to the first object foreground image and the candidate fused image through the target diffusion model to obtain a second fused image.

[0088] Among them, the second preset processing can be used to perform shadow generation processing on the candidate fused image, or it can be other specific technical processing to improve the authenticity of the image. The second fused image can be a shadow fusion image. The target diffusion model is a pre-trained diffusion model. The target diffusion model can adopt ControlNet technology. It should be noted that ControlNet is a technology used in combination with a diffusion model, which is mainly used to more accurately control the image generation process.

[0089] In some embodiments, the target diffusion model includes: a target control network and a target stable diffusion model; through the target diffusion model, a second preset processing is performed on the candidate fused image according to the first object foreground image and the candidate fused image to obtain a second fused image, including: extracting features of the first object foreground image and the candidate fused image through the target control network to obtain target shadow condition features; through the target stable diffusion model, shadow generation is performed on the candidate fused image according to the target shadow condition features to obtain the second fused image.

[0090] Among them, the target control network can be a control image diffusion model (ControlNet), and the target stable diffusion model can be a stable diffusion model (stable diffusion). It should be noted that since ControlNet introduces the concept of "zero convolution" layer to achieve adaptive control, the diffusion model is allowed to ignore certain control signals when necessary, ensuring that the image generation quality will not be affected when the control signals are irrelevant.

[0091] In some embodiments, feature extraction is performed on the first object foreground image and the candidate fused image through the encoder network in the target control network to obtain target shadow condition features, which are fused with the intermediate feature map of the target stable diffusion model using operations such as addition or connection, thereby achieving fine control over the shadow generation process in image fusion.

[0092] The target shadow condition feature may be information about the region where the shadow is generated, as well as attributes such as the direction and shape of the generated shadow.

[0093] In some embodiments, the target shadow condition characteristics are converted into control signals at multiple stages, and the control signals are injected at multiple stages during the diffusion process of shadow generation of the candidate fused image by the target stable diffusion model to obtain a shadow fused image, thereby achieving multi-scale fine control.

[0094] In this embodiment, the target control network is used to make the target stable diffusion model consider the influence of conditional input at each step in the process of generating shadows by reverse diffusion, and gradually refine the output result to obtain the second fused image.

[0095] Step 210: Perform consistency processing on the second fusion image to obtain a target fusion image.

[0096] Among them, it should be noted that consistency processing refers to ensuring that the fusion between the foreground and background in the shadow fusion image is more harmonious and natural through adaptive adjustment of the background in the shadow fusion image, avoiding abrupt parts, thereby improving the visual consistency and overall effect of the entire shadow fusion image.

[0097] In some embodiments, consistency processing is performed on the second fusion image to obtain a target fusion image, including: performing color texture analysis on the second fusion image through a consistency processing model to obtain adjustment analysis data; and adaptively adjusting the second fusion image according to the adjustment analysis data to obtain the target fusion image.

[0098] Among them, the consistency tuning model can adopt image harmonization technology based on generative adversarial network (GAN).

[0099] In some embodiments, the second fusion image is subjected to color texture analysis through a consistency tuning model, and further analysis is performed to obtain adjustment analysis data that can be used to adaptively adjust the color texture of the second fusion image. The second fusion image is adaptively adjusted based on the adjustment analysis data to obtain a target fusion image in which the foreground and background are matched in color and fused in texture, thereby improving the overall consistency of the target fusion image.

[0100] In some embodiments, please refer to FIG. 4, which specifically shows the specific process of the second preset processing, wherein the target object background image is added as one of the bases for shadow generation in the shadow generation process of the second preset processing, and input into the target control network to make the generated shadow more reliable, and the subsequent specific process of noise processing by some diffusion models is also included, such as noise prediction and noise addition processes, but not limited to this. In order to more clearly show the actual effect of shadow generation, please refer to the effect display diagram of FIG. 5, wherein the left image of FIG. 5 is a fused image synthesized by direct mapping, and the right image of FIG. 5 is a target fused image after the shadow is generated by the diffusion model and the target object background image, and the overall consistency processing is performed.

[0101] In the above-mentioned image fusion method, a first image to be fused is obtained, and a foreground extraction is performed on the first image to be fused to obtain a foreground image of a target object; a first preset processing is performed on the foreground image of the target object to obtain a first object foreground image; a region fusion is performed according to the target region of the first object foreground image and the second image to be fused to obtain a candidate fused image; a second preset processing is performed on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image; and a consistency processing is performed on the second fused image to obtain a target fused image. Therefore, by using the target diffusion model, the propagation of light and the formation process of shadows can be simulated, and the generated shadows are similar to the real images in terms of direction, shape, brightness and other characteristics, so that the shadows of the synthesized images are more natural and real, and the overall realism and visual effect of the image are improved. Then, the consistency processing is used to ensure that the fusion between the foreground and the background is more harmonious and natural, and abrupt parts are avoided, thereby improving the visual consistency and overall effect of the entire image. Therefore, this scheme can realize an image fusion method that can still make the generated shadows closer to the shadows in the original image in terms of direction and brightness without the need for information such as the direction of the light source.

[0102] In an exemplary embodiment, as shown in FIG6 , the method further includes: Step 602 to Step 608. Wherein:

[0103] Step 602: Perform model training on the original diffusion model according to the basic training shadow samples to obtain a fixed basic diffusion model.

[0104] Among them, the basic training shadow samples are fixed sample data obtained from the shadow dataset.

[0105] In some embodiments, the original diffusion model is trained through basic training shadow samples to fix the parameters of the original diffusion model, thereby obtaining a fixed basic diffusion model with fixed parameters, so that the subsequent training process can omit the training process of the fixed basic diffusion model, saving a large amount of computer resources required for training.

[0106] Step 604: obtain newly added shadow training samples, and perform model training on the original control network according to the newly added training samples to obtain a newly added control network.

[0107] The newly added shadow training samples are used to represent some newly added training sample data with new image features or styles.

[0108] In some embodiments, model training is performed on the original control network according to the newly added training samples to obtain a newly added control network, including: obtaining update weights by performing low-rank decomposition on the model weights of the original control network; obtaining target weights by learning and adapting the update weights according to the newly added training samples; and obtaining a new control network by model updating the original control network according to the target weights.

[0109] Among them, the low-rank decomposition process adopts LoRA (Low-Rank Adaptation) technology.

[0110] In some embodiments, LoRA assumes that the model weight update of the original control network can be approximated by low-rank decomposition, that is, two small matrices A and B are introduced, and the updated weight is expressed as W+AB, where W refers to the model weight of the fixed basic diffusion model. The updated weight obtained by such assumption processing not only has a small error, but also greatly reduces the amount of parameters that need to be learned.

[0111] In some embodiments, in order to reduce the amount of parameters that need to be learned for training, the model weights of the fixed basic diffusion model are first frozen, and the weights of the original control network in the updated weights are learned and adapted only according to the newly added training samples, that is, the small matrices A and B in the updated weights are learned and adapted only according to the newly added training samples to obtain the target weights, and W+αAB is calculated during forward propagation, and the model of the original control network is updated according to the target weights to obtain the newly added control network. In this embodiment, by learning small matrices A and B and using LoRA technology, the key changes required for the diffusion model to adapt to new tasks can be effectively captured, and the low-rank property helps generalization and prevents overfitting.

[0112] Among them, α is a scaling factor used to adjust the update intensity. In the low-rank decomposition process, different ranks r will be used according to needs to affect the balance between the adaptability and efficiency of the diffusion model.

[0113] Step 606 , constructing a model based on the fixed basic diffusion model and the newly added control network to obtain a candidate diffusion model.

[0114] In some embodiments, a newly added control network is applied to the cross-attention layer and the self-attention layer of the fixed basic diffusion model to obtain a candidate diffusion model, so that the candidate diffusion model can learn new image features or styles without changing the model architecture.

[0115] Step 608: Perform loss adjustment processing on the candidate diffusion model using the target loss function to obtain the target diffusion model.

[0116] Among them, the loss adjustment processing can be a specific process of optimizing the loss of the candidate diffusion model. The target loss function is a specially designed loss function used to balance the generation quality and the degree of compliance with the shadow generation conditions. The user can set the target loss function according to the actual application requirements.

[0117] In some embodiments, the candidate diffusion model is subjected to loss adjustment processing by using a target loss function to obtain a target diffusion model that satisfies the balance requirements between generation quality and compliance with conditions.

[0118] It should be noted that in most cases, reasonable model adaptability training is required for each application scenario to improve the adaptability of the model, so as to make the fused image more suitable for the actual application scenario. However, if the full model training method is adopted for each application scenario, a large amount of computing resources and training time are required. Therefore, in this embodiment, by adopting LoRA and low-rank decomposition, only the diffusion model is trained on the newly added shadow samples to achieve efficient adaptation of the diffusion model to the application scenario. While maintaining the fixed basic diffusion model capability, the computing resource requirements and training time required for model training are reduced, and the rapid adaptation and personalization of large models such as diffusion models are excellent.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides an image fusion device for implementing the above-mentioned image fusion method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image fusion device embodiments provided below can refer to the limitations on the image fusion method above, and will not be repeated here.

[0121] In an exemplary embodiment, as shown in FIG. 7 , an image fusion device is provided, comprising: a foreground extraction module 702, a first processing module 704, a fusion module 706, a second processing module 708 and a consistency processing module 710, wherein:

[0122] A foreground extraction module 702 is used to obtain a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of a target object;

[0123] A first processing module 704 is used to perform a first preset processing on the target object foreground image to obtain a first object foreground image;

[0124] A fusion module 706, configured to perform regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image;

[0125] A second processing module 708 is used to perform a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image;

[0126] The consistency processing module 710 is used to perform consistency processing on the second fused image to obtain a target fused image.

[0127] In some embodiments, the first processing module 704 is further used to perform dilation and erosion operations on the target object foreground map to obtain an edge uncertainty region map; and to perform edge segmentation and refinement processing on the edge uncertainty region map by constructing an edge model to obtain a first object foreground map.

[0128] In some embodiments, the fusion module 706 is further configured to perform coordination processing on the first object foreground image according to the target region to obtain a coordinated foreground image; and fuse the coordinated foreground image with the target region to obtain a candidate fused image.

[0129] In some embodiments, the target diffusion model includes: a target control network and a target stable diffusion model; through the target diffusion model, the second processing module 708 is also used to extract features of the first object foreground image and the candidate fused image through the target control network to obtain target shadow condition features; through the target stable diffusion model, shadow generation is performed on the candidate fused image according to the target shadow condition features to obtain a second fused image.

[0130] In some embodiments, the consistency processing module 710 is further used to perform color texture analysis on the second fused image through a consistency processing model to obtain adjustment analysis data; and adaptively adjust the second fused image according to the adjustment analysis data to obtain a target fused image.

[0131] In some embodiments, the image fusion device also includes: a model training module, which is used to perform model training on the original diffusion model according to the basic training shadow samples to obtain a fixed basic diffusion model; obtain new shadow training samples, and perform model training on the original control network according to the new training samples to obtain a new control network; construct a model according to the fixed basic diffusion model and the new control network to obtain a candidate diffusion model; and perform loss adjustment processing on the candidate diffusion model through a target loss function to obtain a target diffusion model.

[0132] In some embodiments, the model training module is also used to obtain updated weights by performing low-rank decomposition on the model weights of the original control network; learn and adapt the updated weights according to the newly added training samples to obtain target weights; and update the model of the original control network according to the target weights to obtain a newly added control network.

[0133] In the above-mentioned image fusion device, a first image to be fused is obtained, and a foreground extraction is performed on the first image to be fused to obtain a foreground image of a target object; a first preset processing is performed on the foreground image of the target object to obtain a first object foreground image; a target region of the first object foreground image and the second image to be fused is subjected to intra-regional fusion to obtain a candidate fused image; a second preset processing is performed on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image; and a consistency processing is performed on the second fused image to obtain a target fused image. Therefore, by using the target diffusion model, the propagation of light and the formation process of shadows can be simulated, and the generated shadows are similar to the real images in terms of direction, shape, brightness and other characteristics, so that the shadows of the synthesized images are more natural and real, and the overall realism and visual effect of the image are improved. Then, the uniform processing is performed to ensure that the fusion between the foreground and the background is more harmonious and natural, and abrupt parts are avoided, thereby improving the visual consistency and overall effect of the entire image. Therefore, this scheme can realize an image fusion method that can still make the generated shadows closer to the shadows in the original image in terms of direction and brightness without the need for information such as the direction of the light source.

[0134] Each module in the above-mentioned image fusion device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0135] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a first image to be fused and a second image to be fused. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image fusion method is implemented.

[0136] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0141] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image fusion method, characterized in that: The method comprises: Acquire a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of the target object; Performing a first preset processing on the target object foreground image to obtain a first object foreground image; Performing regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image; performing a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image; Perform consistency processing on the second fusion image to obtain a target fusion image.

2. The method according to claim 1, characterized in that The step of performing a first preset process on the target object foreground image to obtain a first object foreground image includes: Performing an expansion and corrosion operation on the target object foreground image to obtain an edge uncertainty region image; The edge model is constructed to perform edge segmentation and refinement processing on the edge uncertainty region map to obtain the first object foreground map.

3. The method according to claim 1, characterized in that: The step of performing regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image comprises: Performing coordination processing on the first object foreground image according to the target area to obtain a coordinated foreground image; The coordinated foreground image is fused with the target area to obtain the candidate fused image.

4. The method according to claim 1, characterized in that The target diffusion model includes: a target control network and a target stable diffusion model; the target diffusion model is used to perform a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image to obtain a second fused image, including: Extracting features from the first object foreground image and the candidate fused image through a target control network to obtain target shadow condition features; The target stable diffusion model is used to generate shadows for the candidate fused images according to the target shadow condition characteristics to obtain the second fused image.

5. The method according to claim 1, characterized in that The performing consistency processing on the second fusion image to obtain a target fusion image includes: Performing color and texture analysis on the second fusion image through a consistency processing model to obtain adjustment analysis data; The second fusion image is adaptively adjusted according to the adjustment analysis data to obtain the target fusion image.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The original diffusion model is trained according to the basic training shadow samples to obtain a fixed basic diffusion model; Acquire newly added shadow training samples, and perform model training on the original control network according to the newly added training samples to obtain a newly added control network; Building a model based on the fixed basic diffusion model and the newly added control network to obtain a candidate diffusion model; The candidate diffusion model is subjected to loss adjustment processing by using a target loss function to obtain the target diffusion model.

7. The method according to claim 6, characterized in that The method of performing model training on the original control network according to the newly added training samples to obtain a newly added control network includes: Obtaining updated weights by performing low-rank decomposition on the model weights of the original control network; Performing learning and adaptation on the updated weight according to the newly added training samples to obtain a target weight; The model of the original control network is updated according to the target weight to obtain the newly added control network.

8. An image fusion device, characterized in that: The device comprises: A foreground extraction module, used to obtain a first image to be fused, and perform foreground extraction on the first image to be fused to obtain a foreground image of a target object; A first processing module, configured to perform a first preset processing on the target object foreground image to obtain a first object foreground image; A fusion module, configured to perform regional fusion according to the first object foreground image and the target area of ​​the second image to be fused to obtain a candidate fused image; A second processing module, configured to perform a second preset processing on the candidate fused image according to the first object foreground image and the candidate fused image through a target diffusion model to obtain a second fused image; The consistency processing module is used to perform consistency processing on the second fusion image to obtain a target fusion image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images