An image processing method, apparatus, device and storage medium

Abstract

The application provides an image processing method, device and equipment and a storage medium. The method comprises the following steps: obtaining a first image and a second image to be fused, the first image containing an image of a target object, and the second image not containing an image of the target object; fusing the image of the target object in the first image into the second image to obtain an initial fusion image; inputting the initial fusion image into a pre-trained image synthesis model to obtain a final target fusion image, wherein the image synthesis model comprises a guided filter layer. The application inputs the initial fusion image into the image synthesis model to obtain the final target fusion image. The guided filter layer is added in the image synthesis model to extract a feature image containing high-frequency information, which facilitates the subsequent recovery of the contour details of the target object replacement area, makes the target object and the surrounding area in the final target fusion image more smooth, and makes the image fusion effect more real and natural.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Currently, in fields such as virtual plastic surgery and image creation, it is often necessary to replace a target object in an image. For example, in the field of virtual plastic surgery, this might involve replacing single eyelids with double eyelids in a user's facial image, or replacing the nose in a facial image with a nose that has undergone rhinoplasty, and so on.

[0003] One related technology provides a method based on image segmentation and sharpness matching. This method uses a coarse target localization method combined with a U-net fully convolutional network to achieve target segmentation, and generates a target replacement image through an EDSR super-resolution model.

[0004] While the aforementioned techniques can achieve image-based target replacement, they do not consider the sharpness of the fused image or the excessive edge transitions of the fused region. This results in a poor image fusion effect, where the boundary between the replaced target and its surrounding area is sharp. Summary of the Invention

[0005] This application proposes an image processing method, apparatus, device, and storage medium that fuses an image containing a target object and an image not containing the target object to obtain an initial fused image. The initial fused image is then input into an image synthesis model including a guided filter layer to obtain a final target fused image. The guided filter layer yields a feature image containing high-frequency information, facilitating subsequent restoration of details in the target object's feature regions. This results in a smoother transition between the target object and its surrounding areas in the final target fused image, leading to a more realistic and natural image fusion effect.

[0006] The first aspect of this application provides an image processing method, including:

[0007] Obtain a first image and a second image to be fused, wherein the first image contains the target object and the second image does not contain the target object;

[0008] The image of the target object in the first image is fused into the second image to obtain an initial fused image;

[0009] The initial fused image is input into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guided filter layer.

[0010] In some embodiments of this application, fusing the image of the target object in the first image to the second image to obtain an initial fused image includes:

[0011] A preset fusion algorithm is used to extract the first low-frequency component and the first high-frequency component of the first image, and to extract the second low-frequency component and the second high-frequency component of the second image.

[0012] The first low-frequency component and the second low-frequency component are fused to obtain a low-frequency fused image;

[0013] The first high-frequency component and the second high-frequency component are fused to obtain a high-frequency fused image;

[0014] The low-frequency fused image and the high-frequency fused image are added together to obtain the initial fused image.

[0015] In some embodiments of this application, inputting the initial fused image into a pre-trained image synthesis model to obtain the final target fused image includes:

[0016] The high-frequency feature vector of the initial fused image is extracted by a first feature extraction module included in the pre-trained image synthesis model. The first feature extraction module includes a guided filter layer.

[0017] The image synthesis model extracts the conventional feature vector of the initial fused image through its second feature extraction module.

[0018] The image synthesis model fuses the conventional feature vector and the high-frequency feature vector through an image fusion layer to obtain a fused feature map.

[0019] The fused feature map and random noise information are input into the generator included in the image synthesis model to generate the final target fused image.

[0020] In some embodiments of this application, the first feature extraction module includes a first downsampling layer, a first convolutional layer, a guided filter layer, and a residual image transformation layer; the step of extracting the high-frequency feature vector of the initial fused image through the first feature extraction module included in the image synthesis model includes:

[0021] The low-frequency feature vector of the initial fused image is extracted through the first downsampling layer and the first convolutional layer;

[0022] Based on the initial fused image and the low-frequency feature vector, a residual image is generated through the residual image transformation layer;

[0023] Based on the initial fused image, the low-frequency feature vector, and the residual image, the high-frequency feature vector corresponding to the initial fused image is obtained through the guided filtering layer.

[0024] In some embodiments of this application, obtaining the high-frequency feature vector corresponding to the initial fused image through the guided filter layer based on the initial fused image, the low-frequency feature vector, and the residual image includes:

[0025] The low-frequency weighting coefficients are determined based on the low-frequency feature vector and the residual image.

[0026] The low-frequency weighting coefficients are upsampled to obtain the high-frequency weighting coefficients;

[0027] Based on the initial fused image and the high-frequency weight coefficients, a high-frequency feature vector corresponding to the initial fused image is generated.

[0028] In some embodiments of this application, determining the low-frequency weighting coefficients based on the low-frequency feature vector and the residual image includes:

[0029] Based on the low-frequency feature vector and the residual image, the low-frequency weight coefficient is determined by formula (1);

[0030]

[0031] In the formula (1), i is the pixel position, w k Let k be the filter window, and k be the label of the filter window. and The low-frequency weighting coefficient is... The low-frequency feature vector, The residual image is referred to here.

[0032] In some embodiments of this application, generating a high-frequency feature vector corresponding to the initial fused image based on the initial fused image and the high-frequency weight coefficients includes:

[0033] Based on the initial fused image and the high-frequency weight coefficients, the low-frequency feature vector corresponding to the initial fused image is generated by formula (2);

[0034] Subtracting the low-frequency feature vector from the initial fused image yields the high-frequency feature vector corresponding to the initial fused image.

[0035] O L =a H *I H +b H …(2)

[0036] In the above formula (2), OL Let I be the low-frequency feature vector. H For the initial fused image, a H and b H The high-frequency weighting coefficient is denoted as .

[0037] In some embodiments of this application, before inputting the initial fused image into a pre-trained image synthesis model, the method further includes:

[0038] Obtain a training set, which includes multiple pairs of sample images, each pair of sample images including an original sample image and its corresponding fused sample image, wherein the original sample image contains the target object;

[0039] Construct the model structure corresponding to the network model used for image synthesis;

[0040] The constructed model structure is trained based on the training set to obtain a trained image synthesis model.

[0041] In some embodiments of this application, obtaining the training set includes:

[0042] Obtain the original sample image containing the target object;

[0043] The original sample image is segmented into a first sub-image containing only the target object and a second sub-image not containing the target object by using a preset semantic segmentation model;

[0044] Noise is added to the first sub-image and the second sub-image;

[0045] A preset fusion algorithm is used to fuse the image of the target object in the first sub-image after adding noise into the second sub-image to obtain a fused sample image;

[0046] The original sample image and the fused sample image are combined to form a set of sample image pairs for the training set.

[0047] In some embodiments of this application, the model structure for constructing the network model for synthesizing images includes:

[0048] The first feature extraction module and the second feature extraction module are respectively connected to the image fusion layer, and the first feature extraction module includes a guided filter layer;

[0049] Connect the image fusion layer to the generator;

[0050] Connect the generator to the discriminator.

[0051] In some embodiments of this application, connecting the first feature extraction module and the second feature extraction module to the image fusion layer respectively includes:

[0052] The first downsampling layer, the first convolutional layer, and the guided filter layer are connected in sequence, and the residual image conversion layer is connected to the first convolutional layer and the guided filter layer respectively to obtain the first feature extraction module;

[0053] The second convolutional layer is connected to the second downsampling layer to obtain the second feature extraction module;

[0054] The second downsampling layer and the guided filter layer are respectively connected to the image fusion layer.

[0055] In some embodiments of this application, training the constructed model structure based on the training set to obtain a trained image synthesis model includes:

[0056] Obtain sample image pairs from the training set;

[0057] The first feature extraction module, the second feature extraction module, and the image fusion layer are used to generate a fusion feature map corresponding to the fusion sample image in the sample image pair.

[0058] The fusion feature map and random noise corresponding to the fusion sample image are input into the generator to generate the target fusion image corresponding to the fusion sample image;

[0059] The original sample image, the fused sample image, and the target fused image corresponding to the fused sample image in the sample image pair are input into the discriminator to obtain the discrimination result;

[0060] The overall loss value for the current training cycle is calculated based on the original sample image, the fused sample image, the target fused image corresponding to the fused sample image, and the discrimination result.

[0061] In some embodiments of this application, calculating the overall loss value for the current training period based on the original sample image, the fused sample image, the target fused image corresponding to the fused sample image, and the discrimination result includes:

[0062] Based on the discrimination result, the loss value of the discriminator is calculated;

[0063] Calculate the content loss value based on the target fused image and the fused sample image;

[0064] Based on the target fused image and the original sample image, calculate the smoothing loss value and the style loss value;

[0065] Calculate the total variation regularization loss value based on the target fused image;

[0066] The overall loss value for the current training cycle is calculated based on the discriminator's loss value, the content loss value, the smoothing loss value, the style loss value, and the total variation regularization loss value.

[0067] In some embodiments of this application, the method further includes:

[0068] Based on the current number of training epochs or the overall loss value of the current training epoch, determine whether the preset convergence condition is met.

[0069] Under the condition that the preset convergence condition is met, obtain the model parameters corresponding to the training cycle with the minimum overall loss value.

[0070] The parts of the model structure other than the discriminator, together with the model parameters, form a trained image synthesis model.

[0071] A second aspect of this application provides an image processing apparatus, comprising:

[0072] The acquisition module is used to acquire a first image and a second image to be fused, wherein the first image contains an image of the target object and the second image does not contain an image of the target object;

[0073] The fusion module is used to fuse the image of the target object in the first image into the second image to obtain an initial fused image;

[0074] An image synthesis module is used to input the initial fused image into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guided filtering layer.

[0075] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0076] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.

[0077] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0078] According to the image processing method provided in this application embodiment, an image containing a target object and an image not containing a target object can be fused to obtain an initial fused image. This initial fused image is then input into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model provided in this application embodiment includes a guided filtering layer. By adding the guided filtering layer, feature images containing high-frequency information are extracted. Based on the feature images containing high-frequency information, the final target fused image is generated, improving the restoration of contour details in the target object replacement region, making the transition between the target object and the surrounding region in the final target fused image smoother, and resulting in a more realistic and natural image fusion effect. This method avoids the drawbacks of manual intervention in traditional algorithms and can achieve automatic and efficient image fusion. Furthermore, this method allows users to preview local simulation effects, greatly improving the user experience.

[0079] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0080] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0081] In the attached diagram:

[0082] Figure 1 A flowchart of an image processing method provided in an embodiment of this application is shown;

[0083] Figure 2 A flowchart of a fusion method for an initial fused image provided in an embodiment of this application is shown;

[0084] Figure 3 A flowchart of a training sample creation method provided in an embodiment of this application is shown;

[0085] Figure 4 A schematic diagram of a model structure for training an image synthesis model is shown in one embodiment of this application;

[0086] Figure 5 This invention provides a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present application.

[0087] Figure 6 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0088] Figure 7A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0089] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0090] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0091] The following description, in conjunction with the accompanying drawings, describes an image processing method, apparatus, device, and storage medium according to embodiments of this application.

[0092] Currently, in fields such as virtual plastic surgery and image creation, it is often necessary to replace a target object in an image. For example, in the field of virtual plastic surgery, it may be necessary to replace an image of a specific facial feature. Related image-based facial feature replacement algorithms mainly include: constrained facial feature replacement algorithms, Poisson fusion-based facial feature editing algorithms, and facial feature image segmentation and sharpness matching methods. Among these, the constrained facial feature replacement algorithm automatically extracts facial feature points to calculate feature regions, clones the feature features onto the target image, and applies constraints to the blending process to eliminate ghosting. However, this algorithm requires user intervention to adjust weights, reducing its automation level, and ghosting exists in the blended feature region, resulting in uneven blending space. The Poisson fusion-based facial feature editing algorithm uses skin color detection combined with feature point methods for feature region extraction, grayscale image dilation to extract facial features, Poisson equation difference to achieve seamless fusion, and finally, optimal mask coverage to achieve facial feature replacement. However, the feature region localization method used in this approach cannot accurately extract the facial feature regions, and the skin color of the face is affected by lighting and shooting angle, impacting the accuracy of feature extraction. A facial feature image segmentation and sharpness matching method uses a coarse feature localization method combined with a U-net fully convolutional network to achieve facial feature segmentation, and generates replacement images of facial features using an EDSR super-resolution model. Although this method shows some improvement in feature segmentation and feature fusion compared to traditional methods, it does not consider the sharpness of the image after feature fusion or the transition of edges in the fused region.

[0093] Based on this, this application provides an image processing method that fuses images containing a target object and images not containing the target object to obtain an initial fused image. This initial fused image is then input into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model provided in this application includes a guided filter layer. By adding the guided filter layer, a feature image containing high-frequency information is obtained. Subsequently, the feature image extracted from the input image through convolution and the feature image containing high-frequency information are input into a generator. The generated fused image enables better restoration of the contour details of the replacement region of the target object, a smoother transition between the target object and the surrounding area, and a more realistic and natural image fusion effect. This method avoids the drawbacks of manual intervention in traditional algorithms and can achieve automatic and efficient image fusion. Furthermore, this method allows users to preview local simulation effects, greatly improving the user experience.

[0094] See Figure 1 The method specifically includes the following steps:

[0095] Step 101: Obtain the first image and the second image to be fused. The first image contains the target object, and the second image does not contain the target object.

[0096] In this embodiment, the execution entity is a server. A client compatible with the image processing services provided by the server is installed on the user's mobile phone or computer. When the user needs to use the image processing service, the user opens the client on the terminal, and the client displays an entry point for image processing.

[0097] The user submits a first image containing the target object through the aforementioned entry point displayed on the client. The first image can be an image captured by the terminal's camera of the target object, or it can be an image containing the target object selected by the user from the network or a local folder. To improve the accuracy of the target object replacement and enhance the smoothness of the boundary between the target object and surrounding areas in the subsequently obtained fused image, making the target object replacement more realistic and natural, the first image in this embodiment may contain only the image of the target object, while other areas can be blank images.

[0098] Users can also submit a third image through the aforementioned entry point displayed on the client. This third image includes an image of the target object and a non-blank background image. The target object in the third image may be exactly the same as or different from the target object in the first image. If they are different, the target object in the third image and the target object in the first image can be different objects with the same name. For example, the target object in the third image is the nose of person A, and the target object in the first image is the nose of person B. Alternatively, the target object in the third image is the head of person A, and the target object in the first image is the head of person B.

[0099] After receiving the first and third images submitted by the user, the client sends them to the server. Upon receiving the first and third images, the server identifies the target object from the first image, then performs target recognition on the third image to determine the bounding rectangle of the target object in the third image. Finally, the server removes the image region corresponding to the bounding rectangle from the third image to obtain a second image that does not contain the target object.

[0100] In another implementation, the second image can also be generated on the client side. Specifically, the user submits a first image and a third image on the client. The client performs target recognition on the third image, determines the bounding rectangle of the target object in the third image, and removes the image region corresponding to the bounding rectangle from the third image to obtain a second image that does not contain the target object. The client then sends the first image and the second image to the server.

[0101] The target object can be any object. In fields such as virtual cosmetic surgery, the target object can be any facial feature, such as the nose, eyes, mouth, eyebrows, or any combination of multiple facial features. The target object can also be any body part such as the head or chest.

[0102] In an exemplary scenario, a user wants to preview a partial simulation effect. For example, if a user wants to see how an existing rhinoplasty example looks on their own face, they can obtain a first image containing only the nose example and a second image not containing the nose.

[0103] Specifically, a user's facial image can be captured using a mobile phone, camera, or other imaging device. The Dlib library is then used to identify the corner coordinates of facial features such as the eyes, nose tip, mouth corner, and eyebrows, performing facial feature alignment. The HRnet semantic segmentation model is then used to perform semantic segmentation on the aligned facial image, extracting the image excluding the nose to obtain a second image without the target object. Next, a pre-stored facial image of a rhinoplasty example is obtained. The Dlib library is used to identify the corner coordinates of the eyes, nose tip, mouth corner, and eyebrows, performing facial feature alignment. The HRnet semantic segmentation model is then used to perform semantic segmentation on the aligned facial image, extracting the image containing only the nose to obtain a first image containing only the target object.

[0104] Step 102: Fuse the image of the target object in the first image into the second image to obtain the initial fused image.

[0105] Specifically, a preset fusion algorithm is used to extract the first low-frequency component and the first high-frequency component of the first image, and to extract the second low-frequency component and the second high-frequency component of the second image. The preset fusion algorithm may include wavelet transform fusion algorithm or Boson fusion algorithm, etc.

[0106] The following explanation uses wavelet transform fusion algorithm as an example. Wavelet transform is used to extract low-frequency and high-frequency components of the image. Specifically, the first image is decomposed using wavelet transform to obtain the first low-frequency component La(x,y) and the first high-frequency component Ta(x,y). The second image is then decomposed using wavelet transform to obtain the second low-frequency component Lb(x,y) and the second high-frequency component Tb(x,y). The low-frequency components reflect the background information of the image, while the high-frequency components reflect the overall contour and edge information of the image.

[0107] The first low-frequency component and the second low-frequency component are fused to obtain a low-frequency fused image. Specifically, the first low-frequency component and the second low-frequency component are fused according to the following formula to obtain the low-frequency fused image.

[0108] L(x,y)=a1La(x,y)+a2Lb(x,y)

[0109] Where L(x,y) represents the low-frequency fused image, and a1 and a2 are low-frequency wavelet coefficients. The low-frequency wavelet coefficients can be the average of the low-frequency wavelet coefficients of the two source images, and can be calculated using the median averaging method.

[0110] The first high-frequency component and the second high-frequency component are fused to obtain a high-frequency fused image. Specifically, the first high-frequency component and the second high-frequency component are fused according to the following formula to obtain a high-frequency fused image.

[0111] T(x,y)=b1Ta(x,y)+b2Tb(x,y)

[0112] Where T(x,y) represents the high-frequency fused image, and b1 and b2 represent the high-frequency wavelet coefficients, which can be calculated according to the principle of maximizing regional features.

[0113] The low-frequency wavelet coefficients a1 and a2, and the high-frequency wavelet coefficients b1 and b2, can be determined based on the principles of wavelet transform fusion. The principles of wavelet transform fusion include: calculating the energy and matching degree of the central region. The formula for calculating the energy of the central region is as follows:

[0114]

[0115] The formula for calculating the matching degree is as follows:

[0116]

[0117] The formula for calculating the regional center is as follows:

[0118]

[0119] Where D(x,y) is the high-frequency coefficient matrix of image K(M,N), j is the scale parameter, i is the horizontal, vertical and diagonal high-frequency components, and M and N are the width and height of image K, respectively. The energy of the central region of image K, For matching degree, The energy of the central region of image A, The energy of the central region of image B, The center of region K in image K.

[0120] In this embodiment of the application, a matching degree threshold is set; if the calculated matching degree... Components with a matching degree less than the matching degree threshold are considered low-frequency components, and their low-frequency wavelet coefficients are calculated using the median averaging principle. Components with a matching degree greater than or equal to the matching degree threshold are considered high-frequency components, and their high-frequency wavelet coefficients are calculated using the principle of maximizing regional features. The matching degree threshold can be set according to actual conditions, and this embodiment does not impose specific limitations.

[0121] After obtaining the low-frequency fused image and the high-frequency fused image using the above method, the two images are added together to obtain the initial fused image. Specifically, the initial fused image I is obtained by adding the low-frequency fused image and the high-frequency fused image according to the following formula. H (x,y):

[0122] I H (x,y)=L(x,y)+T(x,y)

[0123] Figure 2A flowchart of a fusion method for an initial fused image provided in an embodiment of this application is shown, as follows: Figure 2 As shown, a wavelet transform is performed on the first image to obtain a first low-frequency component and a first high-frequency component. A wavelet transform is then performed on the second image to obtain a second low-frequency component and a second high-frequency component. The first and second low-frequency components are then fused using a first fusion principle, which is the median averaging principle, to obtain a low-frequency fused image. The first and second high-frequency components are then fused using a second fusion principle, which is the principle of maximizing regional features, to obtain a high-frequency fused image. Finally, an inverse wavelet transform is performed on the low-frequency and high-frequency fused images to obtain an initial fused image.

[0124] In this embodiment, the low-frequency components obtained through wavelet decomposition are all positive transform values, reflecting the general appearance of the source image at that resolution. The low-frequency wavelet coefficients of the fused image can be taken as the average of the low-frequency wavelet coefficients of the two source images, using the median averaging method. The high-frequency components are all large transform values, representing salient features in the image, such as edges, bright lines, and region contours. This information reflects local detail contrast. High-frequency fusion is performed using the principle of maximizing region features.

[0125] In some other embodiments of this application, the Boson fusion algorithm can also be used to fuse the first image and the second image to obtain an initial fused image.

[0126] Step 103: Input the initial fused image into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guiding filter layer.

[0127] Before processing the initial fused image using an image synthesis model, the model must first be trained. The model training process includes:

[0128] Obtain a training set, which includes multiple pairs of sample images. Each pair of sample images includes an original sample image and its corresponding fused sample image. The original sample image contains the target object. Construct a model structure corresponding to the network model used for image synthesis. Train the constructed model structure based on the training set to obtain a trained image synthesis model.

[0129] The training set acquisition process begins by obtaining original sample images containing the target object. Using a pre-defined semantic segmentation model, these original sample images are segmented into a first sub-image containing only the target object and a second sub-image not containing the target object.

[0130] The preset semantic segmentation model can include semantic segmentation models such as HRnet and Unet. The preset semantic segmentation model determines the bounding rectangle of the target object in the original sample image. The image region of this bounding rectangle is extracted from the original sample image to obtain a first sub-image containing only the target object. The image region of this bounding rectangle in the original sample image is then filled with blank pixels to obtain a second sub-image that does not contain the target object.

[0131] Taking a face image as the original sample image, and facial features such as the eyes and nose as the target objects, this explanation is as follows: A batch of face images are acquired, and the Dlib library is used to identify the corner coordinates of the target objects such as the eyes, nose tip, mouth corners, and eyebrows in the face images. A preset standard face image is created as a standard facial feature alignment image. The corner coordinates of the aforementioned face image are aligned with the corner coordinates of the corresponding parts in the preset standard image. Then, a preset semantic segmentation model is used to perform semantic segmentation on the facial feature-aligned face images, extracting a first sub-image containing only the target objects and a second sub-image containing no target objects.

[0132] After obtaining the first and second sub-images corresponding to the original sample images using the above method, noise is added to both sub-images. Gaussian blur, pixel shift, and size scaling are used to add noise to the first and second sub-images so that the target object in the first sub-image does not completely match the image region in the second sub-image that does not contain the target object. This facilitates subsequent model training based on the first and second sub-images.

[0133] Furthermore, a preset fusion algorithm is used to fuse the image of the target object in the first sub-image (after adding noise) into the second sub-image, resulting in a fused sample image. The preset fusion algorithm can be a wavelet transform fusion algorithm or a Pascal fusion algorithm, etc. The original sample image and the aforementioned fused sample image are then combined to form a set of sample image pairs in the training set.

[0134] The above method is used to obtain multiple original sample images and corresponding fused sample images to form sample image pairs, and these multiple sample image pairs are used to form a training set.

[0135] To facilitate understanding of the process of creating the training set, the following explanation uses a face image as an example of the original sample image, with reference to the accompanying diagram. Figure 3 A flowchart of a training sample creation method provided in an embodiment of this application is shown, as follows: Figure 3As shown, firstly, a batch of face datasets is input, and the Dlib library is used to identify the corner coordinates of target objects such as eyes, nose tip, mouth corners, and eyebrows. A preset standard face image is created as a standard facial feature alignment image, and the input image is aligned with the preset standard image at five points. Then, the HRnet semantic segmentation model is used to detect feature points on the facial feature-aligned face image, extract facial feature features, and perform semantic segmentation to extract a first sub-image containing only the target object and a second sub-image containing no target object. Gaussian blur, pixel shift, and size scaling are used to add noise to the first and second sub-images. A preset fusion algorithm is used to fuse the image of the target object in the noisy first sub-image into the second sub-image to obtain a fused sample image. Finally, the original sample image and the fused sample image are combined into a set of sample image pairs for the training set to obtain the constructed training sample set, and the training sample image is output.

[0136] This application proposes a method for creating model samples, addressing the difficulty of sample creation in deep learning training. Existing sample creation methods typically involve manually synthesizing a label file from a first sample (extracting facial features) and a second sample (with facial features removed). This application separates the facial features of the same image into a first sub-image and a second sub-image, processes noise separately, and then fuses them using a fusion algorithm. Finally, the fused image and the original image are used as input samples. By performing alignment, target object extraction, noise processing, and fusion operations on the same image, the sample collection process is greatly simplified, significantly improving model training efficiency.

[0137] Before, after, or simultaneously with the creation of the training set using the methods described above, the model structure corresponding to the network model used for synthesizing images is constructed. Specifically, the first feature extraction module and the second feature extraction module are respectively connected to the image fusion layer. The first feature extraction module includes a guided filtering layer. The image fusion layer is connected to the generator, and the generator is connected to the discriminator.

[0138] The first feature extraction module is formed by sequentially connecting the first downsampling layer, the first convolutional layer, and the guided filter layer, and then connecting the residual image transformation layer to both the first convolutional layer and the guided filter layer. The second feature extraction module is formed by connecting the second convolutional layer and the second downsampling layer. Finally, the second downsampling layer and the guided filter layer are connected to the image fusion layer.

[0139] Figure 4 This application provides a schematic diagram of a model structure for training an image synthesis model according to an embodiment of the present application. Figure 4As shown, the model structure used for training includes a first feature extraction module and a second feature extraction module. The first feature extraction module includes a first downsampling layer, a first convolutional layer, a guided filter layer, and a residual image transformation layer. The second feature extraction module includes a second convolutional layer and a second downsampling layer. The second downsampling layer and the guided filter layer are respectively connected to the image fusion layer, the image fusion layer is connected to the generator, and the generator is connected to the discriminator.

[0140] In one possible implementation, the generator produces a clear image by preserving its structure and details. It employs a symmetric encoder-decoder structure similar to U-Net and ResNet. The encoder consists of convolutional layers and performs upsampling operations. During decoding, residual image features are mapped to corresponding layers. The decoder consists of convolutional layers and a non-linear space, and also performs upsampling operations. This application does not provide a detailed description of the generator's specific structure; however, please refer to the structural details of generators in GAN networks in related technologies.

[0141] The discriminator receives the generator's output and determines whether the generated image is a real and clear image. The discriminator uses a PatchGAN network structure, whose basic operations include convolution, batch normalization, and LeakyReLU rectified unit activation. The PatchGAN network structure outputs a predicted probability value for each region of the input image, determining whether the input region is real or fake. This application does not provide a detailed description of the discriminator's specific structure; however, please refer to the structural details of the discriminator in related GAN networks.

[0142] After obtaining the training set and constructing the model structure using the above method, the constructed model structure is trained based on the training set to obtain the trained image synthesis model.

[0143] Specifically, sample image pairs are obtained from the training set, and the original sample image and the fused sample image are input into the constructed model structure. Through a first feature extraction module, a second feature extraction module, and an image fusion layer, a fusion feature map corresponding to the fused sample image in the sample image pair is generated. The fusion feature map corresponding to the fused sample image and random noise are input into a generator to generate the target fused image corresponding to the fused sample image. The original sample image, the fused sample image, and the target fused image corresponding to the fused sample image in the sample image pair are input into a discriminator to obtain the discrimination result.

[0144] The generator learns the probability distribution of real data, while the discriminator determines the probability that the input sample is real data. The discriminator aims to maximize the probability that the input is a real sample and minimize the probability that the input is a generated sample, while the generator aims to maximize the probability that the discriminator will give an incorrect judgment.

[0145] After obtaining the discrimination result, the overall loss value of the current training cycle is calculated based on the original sample image, the fused sample image, the target fused image corresponding to the fused sample image, and the discrimination result.

[0146] Specifically, based on the discrimination results, the discriminator's loss value is calculated. For each input image x, the goal is to generate a G(x,z) that minimizes both content and style losses. The discriminator's loss function is shown below:

[0147]

[0148] in, Let G(x,z) be the loss value of the discriminator, x be the fused sample image, y be the original sample image, G(x,z) be the target fused image, D(x,y) be the probability value of the discriminator in distinguishing x and y, and D(x,G(x,z)) be the probability value of the discriminator in distinguishing x and G(x,z).

[0149] For comparison, a regular GAN is also trained, which only allows the discriminator to determine whether an image is real.

[0150]

[0151] Here, G attempts to minimize the objective adversary and attempts to maximize its adversary D, i.e., argminGmaxDLcGAN(G,D).

[0152] In this embodiment, to ensure the similarity between the input and output images, multiple loss functions are introduced. Different weighting coefficients are applied to these loss functions to form a new summary loss function, and total variation normalization is performed within the loss function. The introduced loss functions include a content loss function, a smoothing loss function, and a style loss function.

[0153] Specifically, the content loss value is calculated based on the target fused image and the fused sample image.

[0154]

[0155] in, Let Φ be the content loss value, x be the fused sample image, and G(x,z) be the target fused image. The feature image of x in the l-th layer of the network is defined as Φ. l (x), |Φ l (x)|=ClHlWl is Φ l The dimension of (x).

[0156] The style loss value is calculated based on the target fused image and the original sample image. G is called the gram matrix, which can be viewed as the eccentric covariance matrix between features. Once the gram matrix is ​​obtained, the positive and negative correlations of relevant features are compared to determine the style of the work. Further, by calculating the corresponding style loss function using the gram matrices of the style image and the generated image, the style differences between the two can be compared. It is the output of the style image at position (i,j,k) of layer l in the CNN, where (i,j,k) corresponds to the height, width and number of channels of the image.

[0157]

[0158]

[0159]

[0160] in, Let G(x,z) be the style loss value, G(x,z) be the target fused image, and y be the original sample image.

[0161] Calculate the smoothing loss value based on the target fused image and the original sample image.

[0162]

[0163] Calculate the total variation regularization loss value based on the target fused image.

[0164]

[0165] The overall loss value for the current training cycle is calculated based on the discriminator's loss value, content loss value, smoothing loss value, style loss value, and total variation regularization loss value.

[0166]

[0167] Among them, G * This represents the overall loss value for the current training cycle, with α, β, and γ being weighting coefficients.

[0168] The embodiments of this application add content loss, style loss, and smoothing loss to the original loss function to form a loss weight matrix, and perform total variation normalization on the final result, which greatly improves the training effect of the model.

[0169] In the training process described above, the input real image is first represented by y, and the feature image of the input image is represented by x, where x is the fused sample image obtained through a preset fusion algorithm. During training, pairs of images (y and x) are required. The high-frequency feature map of x is extracted by the first feature extraction module, and the regular feature map of x is extracted by the second feature extraction module. These two feature maps are fused and used as input to the generator G to obtain the generated target fused image G(x, z). Then, G(x, z) and x are merged based on the channel dimension.

[0170] The images G(x) and x are merged based on the channel dimension and then used as input to the discriminator D to obtain a predicted probability value. This predicted probability value indicates whether the input images G(x) and x are a pair of real images. The closer the probability value is to 1, the more certain the discriminator D is that the input images G(x) and x are a pair of real images. In addition, the real images y and x are also merged based on the channel dimension and used as input to the discriminator D to obtain the predicted probability values ​​of images y and x.

[0171] The training objective of the discriminator D is to output a small probability value when the input is not a pair of real images (x and G(x,z)), and a large probability value when the input is a pair of real images (x and y). The training objective of the generator G is to make the probability value output by the discriminator D as large as possible when the generated G(x,z) and x are used as inputs to the discriminator D, thus successfully deceiving the discriminator D.

[0172] Iterative training is performed based on the constructed training set and model structure in the manner described above, and the current convergence condition is determined based on the current number of training epochs or the overall loss value of the current training epoch.

[0173] Specifically, if the current number of training cycles reaches the preset number of training cycles, then the preset convergence condition is determined to be met. The preset number of cycles can be 200, 500, or 800, etc. This embodiment does not specifically limit the value of the preset number of cycles, and can be set according to the actual situation.

[0174] In other embodiments, if the overall loss value of the current training cycle is less than a preset loss value threshold, it is determined that the preset convergence condition is met. The embodiments of this application do not specifically limit the value of the preset loss value threshold; it can be set according to actual circumstances.

[0175] Under the condition of meeting the preset convergence, obtain the model parameters corresponding to the training cycle with the minimum overall loss value, combine the part of the model structure other than the discriminator with the model parameters to form the trained image synthesis model, and save the trained image synthesis model.

[0176] After obtaining the trained image synthesis model through the above method, step 103 is executed, in which the initial fused image obtained in step 102 is input into the pre-trained image synthesis model to obtain the final target fused image.

[0177] Specifically, the high-frequency feature vector of the initial fused image is extracted by the first feature extraction module of the pre-trained image synthesis model. The first feature extraction module includes a first downsampling layer, a first convolutional layer, a guided filter layer, and a residual image transformation layer. The first downsampling layer and the first convolutional layer extract the low-frequency feature vector of the initial fused image. Based on the initial fused image and the low-frequency feature vector, the residual image transformation layer generates a residual image. Based on the initial fused image, the low-frequency feature vector, and the residual image, the guided filter layer obtains the high-frequency feature vector corresponding to the initial fused image.

[0178] In the guided filtering layer, low-frequency weight coefficients are determined based on the low-frequency feature vector and the residual image; the low-frequency weight coefficients are upsampled to obtain high-frequency weight coefficients; and the high-frequency feature vector corresponding to the initial fused image is generated based on the initial fused image and the high-frequency weight coefficients.

[0179] Specifically, the initial fused image I H The low-frequency feature vector I is obtained by sequentially inputting the first downsampling layer and the first convolutional layer. L The initial fused image I H and low-frequency eigenvector I L Input the residual image transformation layer to obtain the residual image I R The initial fused image I H Low-frequency feature vector I L Residual Image I R The input is guided by a filter layer, resulting in a high-frequency feature vector O. H .

[0180] Specifically, the guided filter layer is based on the low-frequency feature vector I L and residual image I R The low-frequency weighting coefficient is determined by formula (1);

[0181]

[0182] In formula (1), i is the pixel position, w k Let k be the filter window, and k be the label of the filter window. and Low-frequency weighting coefficients For low-frequency feature vectors, This is the residual image.

[0183] For low-frequency weighting coefficients and Upsampling is performed to obtain the high-frequency weighting coefficient a. H and b H According to the initial fused image I H The low-frequency feature vector corresponding to the initial fused image is generated by formula (2) using high-frequency weight coefficients and formula (2).

[0184] O L =a H *I H +b H …(2)

[0185] In formula (2), O L For low-frequency feature vectors, I H For the initial fused image, a H and b H These are high-frequency weighting coefficients.

[0186] The initial fused image I H Subtract low-frequency feature vector O L The high-frequency feature vector O corresponding to the initial fused image is obtained. H That is: O H =I H –O L .

[0187] The high-frequency information contains contour information of features. The image synthesis model in this embodiment extracts the high-frequency feature vector of the initial fused image by adding a guided filter layer, which facilitates the subsequent restoration of contour details and can obtain a fused image with a natural transition.

[0188] The high-frequency feature vector O is obtained through the above method. H Simultaneously, the conventional feature vector of the initial fused image is extracted through the second feature extraction module included in the image synthesis model.

[0189] The second feature extraction module includes a second convolutional layer and a second downsampling layer. The initial fused image is sequentially input into the second convolutional layer and the second downsampling layer to obtain the conventional feature vector of the initial fused image.

[0190] Then, the image fusion layer in the image synthesis model fuses the regular feature vectors and high-frequency feature vectors to obtain a fused feature map. The fused feature map and random noise information are then input into the generator in the image synthesis model to generate the final target fused image.

[0191] The image processing method provided in this application can fuse an image containing a target object and an image not containing the target object to obtain an initial fused image. This initial fused image is then input into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guided filter layer. By adding the guided filter layer, feature images containing high-frequency information are extracted. Based on these feature images, the final target fused image is generated, improving the restoration of contour details in the target object replacement region. This results in a smoother transition between the target object and its surrounding area in the final target fused image, leading to a more realistic and natural image fusion effect. This method avoids the drawbacks of manual intervention in traditional algorithms and can achieve automatic and efficient image fusion. Furthermore, this method allows users to preview local simulation effects, greatly improving the user experience.

[0192] This application also provides an image processing apparatus for performing the image processing method provided in any of the above embodiments. For example... Figure 5 As shown, the device includes:

[0193] The acquisition module 201 is used to acquire a first image and a second image to be fused, wherein the first image contains the target object and the second image does not contain the target object;

[0194] The fusion module 202 is used to fuse the image of the target object in the first image into the second image to obtain an initial fused image;

[0195] The image synthesis module 203 is used to input the initial fused image into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guiding filter layer.

[0196] The fusion module 202 is used to extract the first low-frequency component and the first high-frequency component of the first image and the second low-frequency component and the second high-frequency component of the second image using a preset fusion algorithm; fuse the first low-frequency component and the second low-frequency component to obtain a low-frequency fused image; fuse the first high-frequency component and the second high-frequency component to obtain a high-frequency fused image; and add the low-frequency fused image and the high-frequency fused image to obtain an initial fused image.

[0197] The image synthesis module 203 is used to extract high-frequency feature vectors of the initial fused image through a first feature extraction module included in a pre-trained image synthesis model, the first feature extraction module including a guided filter layer; extract regular feature vectors of the initial fused image through a second feature extraction module included in the image synthesis model; fuse the regular feature vectors and high-frequency feature vectors through an image fusion layer included in the image synthesis model to obtain a fused feature map; and input the fused feature map and random noise information into a generator included in the image synthesis model to generate the final target fused image.

[0198] The image synthesis module 203 is used to extract the low-frequency feature vector of the initial fused image through the first downsampling layer and the first convolutional layer; generate a residual image through the residual image conversion layer based on the initial fused image and the low-frequency feature vector; and obtain the high-frequency feature vector corresponding to the initial fused image through the guided filtering layer based on the initial fused image, the low-frequency feature vector and the residual image.

[0199] The image synthesis module 203 is used to determine the low-frequency weight coefficients based on the low-frequency feature vector and the residual image; upsample the low-frequency weight coefficients to obtain the high-frequency weight coefficients; and generate the high-frequency feature vector corresponding to the initial fused image using the initial fused image and the high-frequency weight coefficients.

[0200] Image synthesis module 203 is used to determine low-frequency weight coefficients based on low-frequency feature vectors and residual images using formula (1);

[0201]

[0202] In formula (1), i is the pixel position, w k Let k be the filter window, and k be the label of the filter window. and Low-frequency weighting coefficients For low-frequency feature vectors, This is the residual image.

[0203] Image synthesis module 203 is used to generate a low-frequency feature vector corresponding to the initial fused image according to formula (2) based on the initial fused image and high-frequency weight coefficients.

[0204] Subtracting the low-frequency feature vector from the initial fused image yields the high-frequency feature vector corresponding to the initial fused image.

[0205] O L =a H *I H +b H …(2)

[0206] In formula (2), O L For low-frequency feature vectors, I H For the initial fused image, a H and b H These are high-frequency weighting coefficients.

[0207] The device also includes a model training module for acquiring a training set, which includes multiple pairs of sample images, each pair containing an original sample image and its corresponding fused sample image. The original sample image contains the target object. The module constructs a model structure corresponding to the network model used for synthesizing images. The constructed model structure is trained based on the training set to obtain a trained image synthesis model.

[0208] The model training module is used to acquire the original sample image containing the target object; using a preset semantic segmentation model, the original sample image is segmented into a first sub-image containing only the target object and a second sub-image not containing the target object; noise is added to the first sub-image and the second sub-image; a preset fusion algorithm is used to fuse the image of the target object in the first sub-image after adding noise into the second sub-image to obtain a fused sample image; the original sample image and the fused sample image are combined to form a set of sample image pairs in the training set.

[0209] The model training module is used to connect the first feature extraction module and the second feature extraction module to the image fusion layer respectively. The first feature extraction module includes a guided filter layer; the image fusion layer is connected to the generator; and the generator is connected to the discriminator.

[0210] The model training module is used to sequentially connect the first downsampling layer, the first convolutional layer, and the guided filter layer; connect the residual image transformation layer to the first convolutional layer and the guided filter layer respectively to obtain the first feature extraction module; connect the second convolutional layer to the second downsampling layer to obtain the second feature extraction module; and connect the second downsampling layer and the guided filter layer to the image fusion layer respectively.

[0211] The model training module is used to obtain sample image pairs from the training set; through the first feature extraction module, the second feature extraction module, and the image fusion layer, a fusion feature map corresponding to the fused sample image in the sample image pair is generated; the fusion feature map corresponding to the fused sample image and random noise are input into the generator to generate the target fused image corresponding to the fused sample image; the original sample image, the fused sample image, and the target fused image corresponding to the fused sample image in the sample image pair are input into the discriminator to obtain the discrimination result; based on the original sample image, the fused sample image, the target fused image corresponding to the fused sample image, and the discrimination result, the overall loss value of the current training cycle is calculated.

[0212] The model training module is used to calculate the discriminator's loss value based on the discrimination results; calculate the content loss value based on the target fused image and the fused sample image; calculate the smoothing loss value and the style loss value based on the target fused image and the original sample image; calculate the total variation regularization loss value based on the target fused image; and calculate the overall loss value for the current training cycle based on the discriminator's loss value, the content loss value, the smoothing loss value, the style loss value, and the total variation regularization loss value.

[0213] The model training module is used to determine whether the current training period meets the preset convergence condition based on the current number of training periods or the overall loss value of the current training period; if the preset convergence condition is met, it obtains the model parameters corresponding to the training period with the minimum overall loss value; and combines the part of the model structure other than the discriminator with the model parameters to form a trained image synthesis model.

[0214] The image processing apparatus and the image processing method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0215] This application also provides an electronic device for performing the above-described image processing method. Please refer to... Figure 6 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802, and a communication interface 803. The processor 800, the communication interface 803, and the memory 801 are connected via the bus 802. The memory 801 stores a computer program that can run on the processor 800. When the processor 800 runs the computer program, it executes the image processing method provided in any of the foregoing embodiments of this application.

[0216] The memory 801 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 803 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0217] Bus 802 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs. After receiving an execution instruction, the processor 800 executes the program. The image processing method disclosed in any of the foregoing embodiments of this application can be applied to the processor 800, or implemented by the processor 800.

[0218] The processor 800 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 800 or by instructions in software form. The processor 800 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 801. Processor 800 reads the information in memory 801 and, in conjunction with its hardware, completes the steps of the above method.

[0219] The electronic device provided in this application embodiment and the image processing method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0220] This application also provides a computer-readable storage medium corresponding to the image processing method provided in the foregoing embodiments. Please refer to... Figure 7 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the image processing method provided in any of the foregoing embodiments.

[0221] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0222] The computer-readable storage medium provided in the above embodiments of this application and the image processing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0223] It should be noted that:

[0224] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0225] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0226] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0227] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Obtain a first image and a second image to be fused, wherein the first image contains the target object and the second image does not contain the target object; The image of the target object in the first image is fused into the second image to obtain an initial fused image; The initial fused image is input into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guided filtering layer, comprising: The high-frequency feature vector of the initial fused image is extracted by a first feature extraction module included in the pre-trained image synthesis model. The first feature extraction module includes a guided filtering layer and a residual image transformation layer. The image synthesis model extracts the conventional feature vector of the initial fused image through its second feature extraction module. The image synthesis model fuses the conventional feature vector and the high-frequency feature vector through an image fusion layer to obtain a fused feature map. The fused feature map and random noise information are input into the generator included in the image synthesis model to generate the final target fused image; The high-frequency feature vector of the initial fused image is extracted by the first feature extraction module of the pre-trained image synthesis model, including: Based on the initial fused image and the low-frequency feature vector of the initial fused image, a residual image is generated through the residual image transformation layer; Based on the initial fused image, the low-frequency feature vector, and the residual image, the high-frequency feature vector corresponding to the initial fused image is obtained through the guided filtering layer.

2. The method according to claim 1, characterized in that, The step of fusing the image of the target object in the first image into the second image to obtain an initial fused image includes: A preset fusion algorithm is used to extract the first low-frequency component and the first high-frequency component of the first image, and to extract the second low-frequency component and the second high-frequency component of the second image. The first low-frequency component and the second low-frequency component are fused to obtain a low-frequency fused image; The first high-frequency component and the second high-frequency component are fused to obtain a high-frequency fused image; The low-frequency fused image and the high-frequency fused image are added together to obtain the initial fused image.

3. The method according to claim 1, characterized in that, The first feature extraction module includes a first downsampling layer and a first convolutional layer, comprising: The low-frequency feature vector of the initial fused image is extracted through the first downsampling layer and the first convolutional layer.

4. The method according to claim 3, characterized in that, The step of obtaining the high-frequency feature vector corresponding to the initial fused image through the guided filtering layer based on the initial fused image, the low-frequency feature vector, and the residual image includes: The low-frequency weighting coefficients are determined based on the low-frequency feature vector and the residual image. The low-frequency weighting coefficients are upsampled to obtain the high-frequency weighting coefficients; Based on the initial fused image and the high-frequency weight coefficients, a high-frequency feature vector corresponding to the initial fused image is generated.

5. The method according to claim 4, characterized in that, The step of determining the low-frequency weight coefficients based on the low-frequency feature vector and the residual image includes: Based on the low-frequency feature vector and the residual image, the low-frequency weight coefficient is determined by formula (1); …（1） In the above formula (1), i For pixel position, For filtering window, k For the label of the filter window, and The low-frequency weighting coefficient is... The low-frequency feature vector, The residual image is referred to here.

6. The method according to claim 4, characterized in that, The step of generating a high-frequency feature vector corresponding to the initial fused image based on the initial fused image and the high-frequency weight coefficients includes: Based on the initial fused image and the high-frequency weight coefficients, the low-frequency feature vector corresponding to the initial fused image is generated by formula (2); Subtract the low-frequency feature vector from the initial fused image to obtain the high-frequency feature vector corresponding to the initial fused image; …（2） In the above formula (2), The low-frequency feature vector, For the initial fused image, and The high-frequency weighting coefficient is denoted as .

7. The method according to any one of claims 1-6, characterized in that, Before inputting the initial fused image into the pre-trained image synthesis model, the method further includes: Obtain a training set, which includes multiple pairs of sample images, each pair of sample images including an original sample image and its corresponding fused sample image, wherein the original sample image contains the target object; Construct the model structure corresponding to the network model used for image synthesis; The constructed model structure is trained based on the training set to obtain a trained image synthesis model.

8. The method according to claim 7, characterized in that, The acquisition of the training set includes: Obtain the original sample image containing the target object; The original sample image is segmented into a first sub-image containing only the target object and a second sub-image not containing the target object by using a preset semantic segmentation model; Noise is added to the first sub-image and the second sub-image; A preset fusion algorithm is used to fuse the image of the target object in the first sub-image after adding noise into the second sub-image to obtain a fused sample image; The original sample image and the fused sample image are combined to form a set of sample image pairs for the training set.

9. The method according to claim 7, characterized in that, The model structure corresponding to the network model used for image synthesis includes: The first feature extraction module and the second feature extraction module are respectively connected to the image fusion layer, and the first feature extraction module includes a guided filter layer; Connect the image fusion layer to the generator; Connect the generator to the discriminator.

10. The method according to claim 9, characterized in that, The step of connecting the first feature extraction module and the second feature extraction module to the image fusion layer respectively includes: The first downsampling layer, the first convolutional layer, and the guided filter layer are connected in sequence, and the residual image conversion layer is connected to the first convolutional layer and the guided filter layer respectively to obtain the first feature extraction module; The second convolutional layer is connected to the second downsampling layer to obtain the second feature extraction module; The second downsampling layer and the guided filter layer are respectively connected to the image fusion layer.

11. The method according to claim 9, characterized in that, The step of training the constructed model structure based on the training set to obtain the trained image synthesis model includes: Obtain sample image pairs from the training set; The first feature extraction module, the second feature extraction module, and the image fusion layer are used to generate a fusion feature map corresponding to the fusion sample image in the sample image pair. The fusion feature map and random noise corresponding to the fusion sample image are input into the generator to generate the target fusion image corresponding to the fusion sample image; The original sample image, the fused sample image, and the target fused image corresponding to the fused sample image in the sample image pair are input into the discriminator to obtain the discrimination result; The overall loss value for the current training cycle is calculated based on the original sample image, the fused sample image, the target fused image corresponding to the fused sample image, and the discrimination result.

12. The method according to claim 11, characterized in that, The step of calculating the overall loss value for the current training cycle based on the original sample image, the fused sample image, the target fused image corresponding to the fused sample image, and the discrimination result includes: Based on the discrimination result, the loss value of the discriminator is calculated; Calculate the content loss value based on the target fused image and the fused sample image; Based on the target fused image and the original sample image, calculate the smoothing loss value and the style loss value; Calculate the total variation regularization loss value based on the target fused image; The overall loss value for the current training cycle is calculated based on the discriminator's loss value, the content loss value, the smoothing loss value, the style loss value, and the total variation regularization loss value.

13. The method according to claim 11, characterized in that, The method further includes: Based on the current number of training epochs or the overall loss value of the current training epoch, determine whether the preset convergence condition is met. Under the condition that the preset convergence condition is met, obtain the model parameters corresponding to the training cycle with the minimum overall loss value. The parts of the model structure other than the discriminator, together with the model parameters, form a trained image synthesis model.

14. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire a first image and a second image to be fused, wherein the first image contains an image of the target object and the second image does not contain an image of the target object; The fusion module is used to fuse the image of the target object in the first image into the second image to obtain an initial fused image; An image synthesis module is used to input the initial fused image into a pre-trained image synthesis model to obtain the final target fused image. The image synthesis model includes a guided filtering layer, comprising: The high-frequency feature vector of the initial fused image is extracted by a first feature extraction module included in the pre-trained image synthesis model. The first feature extraction module includes a guided filtering layer and a residual image transformation layer. The image synthesis model extracts the conventional feature vector of the initial fused image through its second feature extraction module. The image synthesis model fuses the conventional feature vector and the high-frequency feature vector through an image fusion layer to obtain a fused feature map. The fused feature map and random noise information are input into the generator included in the image synthesis model to generate the final target fused image; The high-frequency feature vector of the initial fused image is extracted by the first feature extraction module of the pre-trained image synthesis model, including: Based on the initial fused image and the low-frequency feature vector of the initial fused image, a residual image is generated through the residual image transformation layer; Based on the initial fused image, the low-frequency feature vector, and the residual image, the high-frequency feature vector corresponding to the initial fused image is obtained through the guided filtering layer.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-13.

Images