Image content filling method and device, readable storage medium and program product

Abstract

The invention relates to an image content filling method and device, a readable storage medium and a program product. The method comprises the following steps: acquiring a to-be-processed image, wherein the to-be-processed image comprises a content keeping area and a fillable area containing a to-be-eliminated target object; the fillable area is used for eliminating and filling the target object; performing down-sampling processing on the to-be-processed image to obtain a down-sampled image; image generation is carried out according to the downsampling image, a filling image is obtained, and in the filling image, the target object is eliminated, and the content of an image area corresponding to the position of the fillable area is generated according to the content of the content keeping area; performing up-sampling on the filling image to the image size of the to-be-processed image to obtain an up-sampling image; and carrying out image fusion on an image region corresponding to the position of the fillable region in the up-sampling image and the content maintaining region to obtain a target image. By adopting the method, the processing efficiency of image content filling can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image content filling method, device, readable storage medium and program product. Background Art

[0002] With the development of computer technology, the demand for image processing is becoming increasingly diverse. In some scenarios, there is a need to remove objects in the image. For example, when shooting outdoors, there is a messy background or passers-by in the frame, which affects the shooting effect. In order to meet this demand, image content filling technology has emerged. Image content filling technology is a technology that removes a specified area in an image and fills it in, so that the filled image looks natural and coherent visually.

[0003] However, existing image content filling technologies have the problem of low processing efficiency. Summary of the invention

[0004] Based on this, the present application provides an image content filling method, device, readable storage medium and program product, which can improve the processing efficiency of image content filling.

[0005] On the one hand, the present application provides an image content filling method, comprising:

[0006] Acquire an image to be processed, wherein the image to be processed includes a content retention area and a fillable area containing a target object to be eliminated; the fillable area is used to eliminate the target object and fill it;

[0007] Downsampling the image to be processed to obtain a downsampled image;

[0008] Performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area;

[0009] Upsampling the padded image to the image size of the image to be processed to obtain an upsampled image;

[0010] An image area corresponding to the position of the fillable area in the up-sampled image is fused with the content retention area to obtain a target image.

[0011] In one embodiment, obtaining the image to be processed includes:

[0012] Acquire an initial image for region recognition;

[0013] Identifying an object region in the initial image that contains a target object to be eliminated;

[0014] Determining, according to the object region, a shadow region in the initial image that contains a shadow of the target object;

[0015] The object area and the shadow area in the initial image are determined as fillable areas, and the image area other than the object area and the shadow area in the initial image is determined as a content retention area to obtain an image to be processed.

[0016] In one embodiment, determining, according to the object area, a shadow area in the initial image that contains the shadow of the target object includes:

[0017] Acquire an object mask of the target object that matches the image size of the initial image, the object mask comprising an area corresponding to the position of the object area in the initial image;

[0018] Inputting the initial image and the object mask into a trained shadow segmentation model to perform image segmentation, and obtaining a segmentation result output by the shadow segmentation model;

[0019] Based on the segmentation result, a shadow area in the initial image including the shadow of the target object is determined.

[0020] In one embodiment, obtaining an initial image for region recognition includes:

[0021] Get the input original image;

[0022] When the original image meets the preset quality constraint condition and the original image does not belong to the preset type that produces negative impact, obtaining the direction information of the original image;

[0023] According to the direction information, the original image is rotationally corrected to obtain an initial image for region recognition.

[0024] In one embodiment, generating an image according to the downsampled image to obtain a filled image comprises:

[0025] Setting the content in the image area corresponding to the position of the fillable area in the downsampled image to empty to obtain a covered image;

[0026] Extracting features according to the cover image through the trained control network to obtain control features; the control features are used to indicate that image generation is performed in the image area corresponding to the position of the fillable area in the cover image during the image generation process;

[0027] According to the mask image and the control feature, an image is generated by using a trained diffusion model to obtain a filled image.

[0028] In one embodiment, the step of generating an image by using a trained diffusion model according to the mask image and the control feature to obtain a filled image comprises:

[0029] Acquire a mask matching the image size of the downsampled image, wherein the mask is used to indicate the position of an image area in the downsampled image corresponding to the position of the fillable area;

[0030] adding noise to the masked image to obtain a noisy feature;

[0031] A feature obtained by performing matrix operation on the cover mask and the noisy feature is subjected to feature fusion processing with the noisy feature and the cover mask to obtain a fused feature;

[0032] According to the fusion feature and the control feature, an image is generated through a trained diffusion model to obtain a filled image.

[0033] In one embodiment, the step of fusing the image area corresponding to the position of the fillable area in the upsampled image with the content retention area to obtain the target image includes:

[0034] In each color channel of the plurality of color channels, obtaining a first channel image included in the upsampled image, and obtaining a second channel image included in the image to be processed;

[0035] The image area corresponding to the position of the fillable area in the first channel image under the color channel and the image area corresponding to the content retention area in the second channel image under the color channel are fused by using a Poisson fusion algorithm to obtain a fused image under the color channel;

[0036] A target image is generated according to the fused images under the multiple color channels.

[0037] On the one hand, the present application also provides an image content filling device, comprising:

[0038] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a content holding area and a fillable area including a target object to be eliminated; the fillable area is used for eliminating the target object and filling it;

[0039] A downsampling module, used for downsampling the image to be processed to obtain a downsampled image;

[0040] A generating module, performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area;

[0041] An upsampling module, upsampling the padded image to the image size of the image to be processed to obtain an upsampled image;

[0042] The fusion module is used to fuse the image area corresponding to the position of the fillable area in the upsampled image with the content retention area to obtain a target image.

[0043] In one aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Acquire an image to be processed, wherein the image to be processed includes a content retention area and a fillable area containing a target object to be eliminated; the fillable area is used to eliminate the target object and fill it;

[0045] Downsampling the image to be processed to obtain a downsampled image;

[0046] Performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area;

[0047] Upsampling the padded image to the image size of the image to be processed to obtain an upsampled image;

[0048] An image area corresponding to the position of the fillable area in the up-sampled image is fused with the content retention area to obtain a target image.

[0049] In one aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0050] Acquire an image to be processed, wherein the image to be processed includes a content retention area and a fillable area containing a target object to be eliminated; the fillable area is used to eliminate the target object and fill it;

[0051] Downsampling the image to be processed to obtain a downsampled image;

[0052] Performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area;

[0053] Upsampling the padded image to the image size of the image to be processed to obtain an upsampled image;

[0054] An image area corresponding to the position of the fillable area in the up-sampled image is fused with the content retention area to obtain a target image.

[0055] In one aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the following steps:

[0056] Acquire an image to be processed, wherein the image to be processed includes a content retention area and a fillable area containing a target object to be eliminated; the fillable area is used to eliminate the target object and fill it;

[0057] Downsampling the image to be processed to obtain a downsampled image;

[0058] Performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area;

[0059] Upsampling the padded image to the image size of the image to be processed to obtain an upsampled image;

[0060] An image area corresponding to the position of the fillable area in the up-sampled image is fused with the content retention area to obtain a target image.

[0061] In the above-mentioned image content filling method, device, readable storage medium and program product, the image to be processed includes a content retention area and a fillable area including a target object to be eliminated. Image generation is performed based on the downsampled image obtained after downsampling the image to be processed to obtain a filled image, which can eliminate the target object. Moreover, since the content of the image area corresponding to the position of the fillable area is generated according to the content of the content retention area, the obtained filled image is more natural. Moreover, since the image generation is performed based on the downsampled image, the amount of processed data is smaller than that directly generated based on the image to be processed, which can improve the image processing efficiency. The filled image is then upsampled, and the image area corresponding to the position of the fillable area in the upsampled image is fused with the content retention area to obtain the target image. In this way, the influence of the downsampling process on the image resolution can be reduced to a large extent, so that the processing efficiency of the image content filling is improved while the target image retains the high-frequency details of the image to be processed to a large extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 is a schematic flow chart of a method for filling image content in one embodiment;

[0064] Figure 2 A schematic diagram of a frame filled with image content in one embodiment;

[0065] Figure 3 A schematic diagram of a pre-processing step in an embodiment;

[0066] Figure 4 A schematic diagram of a shadow recognition algorithm framework in one embodiment;

[0067] Figure 5 A schematic diagram of an image generation framework in one embodiment;

[0068] Figure 6 A schematic diagram of the image fusion process in one embodiment;

[0069] Figure 7 is a structural block diagram of an image content filling device in one embodiment;

[0070] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] In one embodiment, Figure 1 As shown, a method for filling image content is provided. This embodiment takes the method applied to a computer device as an example. The computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be a personal computer, a laptop, a smart phone, a tablet computer or others. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the method includes the following steps:

[0073] Step 102, obtaining an image to be processed, wherein the image to be processed includes a content retention area and a fillable area including a target object to be eliminated; the fillable area is used to eliminate the target object and fill it.

[0074] The target object is an unnecessary object in the image to be processed that has a visual impact on the subject. The subject is an object that needs to be visually highlighted in the image to be processed. The object may include objects that exist in the real world, such as people, animals, plants, man-made objects, or others, such as electronic products, furniture, etc. The object may also include an intangible object, which is an object without a physical body. For example, an intangible object may be a shadow caused by an object blocking a light source, or a reflection caused by an object reflecting a light source, or others.

[0075] For example, in a tourist photography scene, other tourists may appear in the location of the person being photographed. The image may include the person being photographed as the subject and other tourists as the target objects to be eliminated. By eliminating the target objects and filling them, the image can be made cleaner and more professional. In a commercial product photography scene, the product display image taken of the product may have background objects and reflections that affect the visual effect of the product. The product in the product display image can be used as the subject, and the background objects and reflections can be used as the target objects to be eliminated. By eliminating the target objects and filling them, a pure background is created, so that the product display image can highlight the product itself. In a portrait photography scene, there may be interference elements in the background of the portrait image, such as mottled areas on the background wall. The person in the portrait image can be used as the subject and the interference elements can be used as the target objects to be eliminated. By eliminating the target objects and filling them, the portrait image can highlight the person itself. In a real estate photography scene, in the room space image taken, neatly arranged objects can be used as the subject, such as various furniture, and the messy objects in the subject can be used as the target objects to be eliminated. By eliminating the target objects and filling them, the room space image can highlight the neat space.

[0076] Exemplarily, a computer device may acquire an initial image for region recognition, identify an object region in the initial image that contains a target object to be eliminated, determine a fillable region of the image to be processed from the initial image based on the object region, and determine a content retention region of the image to be processed based on the fillable region.

[0077] The initial image is an image whose fillable area and content retention area need to be identified. The initial image can be an original image input by a user, or a preprocessed image after preprocessing the original image. Preprocessing can be, for example, rotation correction of the original image. The target object is an unnecessary object in the image to be processed that has a visual impact on the subject.

[0078] The image to be processed may be an image obtained after marking the fillable area and the content retention area in the initial image. The image to be processed may also be an initial image, and the positions of the fillable area and the content retention area in the image to be processed are characterized by an object mask. The object mask is a binary image with the same image size as the image to be processed. The object mask may include a first area with a pixel value of a first preset value and a second area with a second preset pixel value. The area corresponding to the position of the first area in the image to be processed is the fillable area, and the area corresponding to the position of the second area is the content retention area. The position correspondence between areas means that the relative positions of different areas in their respective images are the same. For example, the relative position of the fillable area relative to the image to be processed is the same as the relative position of the first area relative to the object mask. The first preset value may be 0, and the second preset value may be 1.

[0079] In one embodiment, the computer device may perform image segmentation on the initial image through a trained general image segmentation model to obtain multiple candidate object regions containing candidate objects in the initial image, and based on the sizes of the candidate object regions, screen out the object region in the initial image containing the target object to be eliminated from the multiple candidate object regions containing the candidate objects.

[0080] Among them, the general image segmentation model is an image segmentation model that can segment various objects. The general image segmentation model can be a SAM model (Segment Anything Model, a segmentation model released by Meta), a SAM-2 model (a second-generation SAM model released by Meta), or others. The object region containing the target object to be eliminated can be a candidate object region whose size is smaller than a preset size among multiple candidate object regions containing candidate objects. The size of the region can be the area occupied by the region, or the height or width of the region. The size of the region can be relative or absolute, such as a relative area or an absolute area. The preset size can be 100 pixels, 200 pixels, or others, or 10%, 20%, or others.

[0081] In one embodiment, the computer device may determine the object area in the initial image containing the target object to be eliminated as the fillable area of ​​the image to be processed, and determine the image area outside the fillable area as the content retention area of ​​the image to be processed.

[0082] In one embodiment, a computer device may obtain an initial image to be subjected to region recognition, and in response to a region selection operation, determine a region selected by the region selection operation in the initial image, determine the region selected in the initial image as a fillable region of the image to be processed, and determine other regions outside the region selected in the initial image as content retention regions of the image to be processed. The region selection operation is an operation of interactively specifying the region to be filled. The region selection operation may be a box selection operation, a circle selection operation, a smear operation, or other interactive operations.

[0083] Step 104: downsample the image to be processed to obtain a downsampled image.

[0084] The downsampling process is to reduce the size of the image to be processed so as to reduce the resolution of the image to be processed.

[0085] Exemplarily, the image to be processed is downsampled to an image size supported by the image generation model to obtain a downsampled image. The image generation model is used to generate images in subsequent steps. The image generation model can be a generative adversarial network (GANs), a variational autoencoder (VAE), a diffusion model (Diffusion Models) or others. The downsampling process can reduce the size of the image to be processed in proportion to the image size supported by the image generation model.

[0086] Step 106 , performing image generation based on the downsampled image to obtain a filled image, in which the target object is eliminated, and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content retention area.

[0087] The relative position of the image area corresponding to the position of the fillable area in the filling image relative to the filling image is the same as the relative position of the fillable area relative to the image to be processed.

[0088] Exemplarily, the computer device may generate an image based on the downsampled image by using a trained image generation model to obtain a filled image.

[0089] In one embodiment, a computer device may obtain a cover mask that matches the image size of the downsampled image, where the cover mask is used to indicate the position of an image area in the downsampled image corresponding to the position of a fillable area, and perform image generation through a trained image generation model based on the downsampled image and the cover mask to obtain a filled image.

[0090] When the downsampled image and the cover mask are superimposed, the position of the image area corresponding to the position of the fillable area in the cover mask may be the position of the image area corresponding to the position of the fillable area in the downsampled image. The superposition may be understood as the overlap of the positions of the downsampled image and the cover mask. The pixel value of the image area corresponding to the position of the fillable area in the cover mask may be a first preset value, and the pixel value of the image area corresponding to the position of the content retention area in the cover mask may be a second preset value. It can be understood that the cover mask may be obtained by downsampling the above-mentioned object mask to an image size supported by the image generation model.

[0091] In one embodiment, the computer device may set the content in the image area corresponding to the position of the fillable area in the downsampled image to empty to obtain a covered image, and generate an image based on the covered image by using a trained diffusion model to obtain a filled image. The content in the image area corresponding to the position of the fillable area in the downsampled image may be set to empty by setting the pixel value of each pixel in the image area corresponding to the position of the fillable area in the downsampled image to a preset pixel value. The preset pixel value may be the first preset value mentioned above.

[0092] Step 108 , up-sampling the padded image to the image size of the image to be processed to obtain an up-sampled image.

[0093] Among them, upsampling is the process of expanding the size of the padded image.

[0094] For example, the computer device may use a preconfigured interpolation algorithm to upsample the padded image to the image size of the image to be processed to obtain an upsampled image, wherein the interpolation algorithm may be a bilinear interpolation algorithm, a bicubic interpolation algorithm, a nearest neighbor interpolation algorithm or others.

[0095] Step 110 , the image area corresponding to the position of the fillable area in the upsampled image is fused with the content preservation area to obtain a target image.

[0096] The relative position of the image area corresponding to the position of the fillable area in the upsampled image relative to the upsampled image is the same as the relative position of the fillable area relative to the image to be processed.

[0097] Exemplarily, the computer device may fuse the image area corresponding to the position of the fillable area in the upsampled image with the content retention area using a preset image fusion algorithm to obtain a target image. The preset image fusion algorithm may be a Poisson fusion algorithm, an Alpha fusion algorithm, or others.

[0098] In one embodiment, the computer device may splice the image area corresponding to the position of the fillable area in the upsampled image with the content retention area in the image to be processed to obtain a spliced ​​image; in each color channel of multiple color channels, the Poisson fusion algorithm is used to fuse the image area corresponding to the position of the fillable area in the spliced ​​image and the image area corresponding to the position of the content retention area in the spliced ​​image to obtain a fused image in the color channel; and the target image is generated according to the fused images in the multiple color channels. The multiple color channels may include an R (Red) channel, a G (Green) channel, and a B (Blue) channel.

[0099] In the above-mentioned image content filling method, the image to be processed includes a content retention area and a fillable area including a target object to be eliminated. Image generation is performed based on the downsampled image obtained after downsampling the image to be processed to obtain a filled image, which can eliminate the target object. Moreover, since the content of the image area corresponding to the position of the fillable area is generated according to the content of the content retention area, the obtained filled image is more natural. Moreover, since the image generation is performed based on the downsampled image, the amount of processed data is smaller than that directly generated based on the image to be processed, which can improve the image processing efficiency. The filled image is then upsampled, and the image area corresponding to the position of the fillable area in the upsampled image is fused with the content retention area to obtain the target image. In this way, the impact of the downsampling process on the image resolution can be reduced to a large extent, so that the target image retains the high-frequency details of the image to be processed to a large extent while improving the processing efficiency of image content filling.

[0100] In an exemplary embodiment, step 102 includes: acquiring an initial image for region identification; identifying an object region in the initial image that contains a target object to be eliminated; determining a shadow region in the initial image that contains a shadow of the target object based on the object region; determining the object region and the shadow region in the initial image as fillable regions, and determining the image region other than the object region and the shadow region in the initial image as a content retention region, to obtain an image to be processed.

[0101] The shadow of the target object is the projection of the target object generated when the target object blocks the light source.

[0102] In this embodiment, by identifying the object area in the initial image that contains the target object to be eliminated, and then determining the shadow area in the initial image that contains the shadow of the target object based on the object area, the object area and the shadow area are determined as fillable areas. In combination with subsequent steps, the shadow of the target object can be eliminated while eliminating the target object, avoiding image distortion caused by residual shadows, and improving the quality of the target image.

[0103] In one embodiment, the step of identifying the object area in the initial image that contains the target object to be eliminated includes: the computer device may perform image segmentation on the initial image through a trained object segmentation model to obtain an object mask of the target object output by the object segmentation model to identify the object area in the initial image that contains the target object to be eliminated.

[0104] The object mask of the target object may represent the object region of the initial image that contains the target object to be eliminated. Specifically, the object mask of the target object may include an object representation region that represents the location of the target object and a background representation region that represents the location of the target object. Then, the image region in the initial image corresponding to the location of the object representation region in the object mask of the target object may be the object region of the initial image that contains the target object to be eliminated. The pixel value in the object representation region may be a first preset value, and the pixel value in the background representation region may be a second preset value.

[0105] The object segmentation model is obtained by training the image segmentation model to be trained based on the training data of an object. The object segmentation model can be used to identify an object. The training data of an object may include multiple sample images and the area containing the object marked in each sample image. The object may be a passerby, a trash can or others, and the object segmentation model may be a passerby segmentation model, a trash can segmentation model or others.

[0106] The image segmentation model to be trained can be an FCN network (Fully Convolutional Networks), a U-Net network (an end-to-end image segmentation model based on a CNN convolutional neural network, the main feature of which is a symmetrical encoder-decoder structure), a SegNet network (a semantic segmentation network proposed by Vijay Badrinarayanan et al. of the University of Cambridge in 2015), a DeepLab series of models (a deep learning model for semantic segmentation proposed by Google, including DeepLab V1, DeepLab V2, DeepLab V3, and DeepLab V3+), a PSPNet network (Pyramid Scene Parsing Network), a Mask R-CNN (Mask Region-based Convolutional Neural Network), a HRNet (High-Resolution Network), or others.

[0107] In one embodiment, the step of identifying the object area in the initial image that contains the target object to be eliminated includes: the computer device can perform image segmentation on the initial image through different trained object segmentation models, obtain the object mask output by each object segmentation model, and fuse the object masks output by different object segmentation models to obtain the object mask of the target object, so as to identify the object area in the initial image that contains the target object to be eliminated.

[0108] Among them, different object segmentation models can be understood as image segmentation models that recognize different objects. The object mask output by each object segmentation model may include an object representation area that represents the location of the object recognized by the object segmentation model. The above-mentioned step of fusing the object masks output by different object segmentation models to obtain the object mask of the target object may include: merging the object representation areas in the object masks output by different object segmentation models to obtain the object mask of the target object.

[0109] In an exemplary embodiment, the step of determining the shadow area in the initial image that contains the shadow of the target object based on the object area includes: obtaining an object mask of the target object that matches the image size of the initial image, the object mask including an area corresponding to the position of the object area in the initial image; inputting the initial image and the object mask into a trained shadow segmentation model for image segmentation to obtain a segmentation result output by the shadow segmentation model; and determining the shadow area in the initial image that contains the shadow of the target object based on the segmentation result.

[0110] The object mask of the target object may represent the object region in the initial image that contains the target object to be eliminated. It is understood that the position of the object region in the initial image may be represented by the object mask of the target object without directly marking the object region in the initial image.

[0111] The shadow segmentation model can be obtained by training the image segmentation model to be trained based on the shadow training data. The shadow training data may include multiple training samples and shadow annotations for the target object to be eliminated in each training sample, each training sample may include a sample image and an object mask of the target object in the sample image, the shadow annotation may be a shadow mask, and the shadow mask may characterize a shadow area containing the shadow of the target object in the sample image.

[0112] When training the shadow segmentation model, the image segmentation model to be trained can be an FCN network (Fully Convolutional Networks), which may include an encoder and a decoder. The encoder is a pre-trained backbone network, and the encoder can use the VGG-16 network (a deep convolutional neural network proposed by the Visual Geometry Group of the University of Oxford), ResNet 34 (a version of the ResNet network proposed by Microsoft Labs, including 34 convolutional layers), or others. The decoder can use multiple deconvolution layers, and the deconvolution layer upsamples the feature map downsampled by the decoder through deconvolution processing (Transposed Convolution). A jump connection structure can be used between the encoder and the decoder to fuse feature maps of different sizes.

[0113] In the shadow segmentation model, the last deconvolution layer in the above decoder can be used as an output layer to output a two-channel feature map, which may include a shadow feature map and a non-shadow feature map. The pixel value of each pixel in the shadow feature map can represent the probability that the pixel belongs to the shadow of the target object, and the pixel value of each pixel in the non-shadow feature map can represent the probability that the pixel does not belong to the shadow of the target object.

[0114] After obtaining the feature maps of the two channels, the shadow segmentation model can also use an activation function to process the feature maps of the two channels. Specifically, the Softmax function can be used to normalize the feature maps of the two channels so that the sum of the pixel values ​​at the same pixel position in the feature maps of the two channels is 1, that is, the sum of the probability that the pixel at the same pixel position belongs to the shadow of the target object and the probability that it does not belong to the shadow of the target object is 1.

[0115] After the activation function is used to process the feature maps of the two channels, the shadow segmentation model can also perform thresholding processing, and each pixel can be classified according to the preset threshold to obtain an initial shadow mask. In the initial shadow mask, the pixel value of the pixel belonging to the shadow of the target object can be the first preset value, and the pixel value of the pixel not belonging to the shadow can be the second preset value. The preset threshold can be 0.5, and the pixels at the pixel position with a pixel value greater than 0.5 can be classified as belonging to the shadow of the target object. The preset threshold can be 0.9, and the pixels at the pixel position with a pixel value greater than 0.9 can be classified as belonging to the shadow of the target object.

[0116] The initial shadow mask may be used as the segmentation result output by the shadow segmentation model, and the target shadow mask obtained by post-segmentation processing of the initial shadow mask may be used as the segmentation result output by the shadow segmentation model. The post-segmentation processing may include at least one of conditional random field processing and morphological operation. The conditional random field processing is a processing performed by inputting the initial shadow mask into the conditional random field model. The morphological operation may include an expansion operation, an erosion operation, and the like.

[0117] The segmentation result can be represented by a mask, which can be the above-mentioned initial shadow mask or the target shadow mask. The image size of the segmentation result is the same as the image size of the initial image. The segmentation result can include a shadow representation area, which can represent the position of the shadow area containing the shadow of the target object. In the initial image, the area corresponding to the position of the shadow representation area can be determined as the shadow area containing the shadow of the target object to be eliminated.

[0118] In this embodiment, the object mask of the target object can characterize the position of the object area. The initial image and the object mask are input into the trained shadow segmentation model for image segmentation, so that the shadow area containing the shadow of the target object in the initial image can be determined. Combined with subsequent steps, the shadow of the target object can be eliminated while eliminating the target object, avoiding image distortion caused by residual shadows and improving the quality of the target image.

[0119] In one embodiment, the step of inputting the initial image and the object mask into the trained shadow segmentation model for image segmentation to obtain the segmentation result output by the shadow segmentation model may include: the computer device may input the initial image and the object mask into the trained shadow segmentation model for image segmentation; during the image segmentation process, feature extraction is performed through the encoder in the shadow segmentation model, and deconvolution processing is performed based on the features extracted by the encoder through the decoder to output feature maps of two channels; the feature maps of the two channels are normalized by the Softmax function and then thresholded to obtain an initial shadow mask; the initial shadow mask is post-segmented and processed to obtain a target shadow mask; and the target shadow mask is determined as the segmentation result output by the shadow segmentation model.

[0120] In an exemplary embodiment, the step of obtaining the initial image for region recognition may include: obtaining an input original image; when the original image meets preset quality constraints and the original image does not belong to a preset type that produces negative impacts, obtaining directional information of the original image; and performing rotation correction on the original image according to the directional information to obtain the initial image for region recognition.

[0121] Among them, the preset quality constraint condition is used to constrain the image quality of the original image. The preset quality constraint condition may include a preset resolution constraint condition and a preset noise constraint condition. The preset resolution constraint condition is that the resolution of the original image reaches the preset resolution. The preset resolution is, for example, 1280*1280. The preset noise constraint condition is that the noise level evaluation value of the original image meets the preset low noise condition. The noise level evaluation value may be a noise variance, and the noise level evaluation value meets the preset low noise condition, and the noise level evaluation value may be less than the preset noise variance, and the preset noise variance is, for example, 0.03. The noise level evaluation value may also be a PSNR (Peak Signal to Noise Ratio) value between a preset standard image and the original image, and the noise level evaluation value meets the preset low noise condition, and the noise level evaluation value may be greater than a preset peak signal-to-noise ratio value, and the preset peak signal-to-noise ratio value is, for example, 30dB.

[0122] The preset type that produces negative impact is a category of images containing negative information. Negative information includes bad information, sensitive information, inappropriate content, etc. When determining whether the original image belongs to the preset type that produces negative impact, an image recognition algorithm can be used to determine the category of the original image. Specifically, the image recognition algorithm can extract features from the original image and the image of the preset type, respectively, and analyze the extracted features to determine whether the original image belongs to the preset type that produces negative impact.

[0123] The direction information of the original image may be the direction information in the metadata of the original image. The metadata is used to record various attribute information of the original image. The metadata may include shooting equipment, shooting parameters, time, location, direction information or others. The metadata may specifically be Exif (Exchangeable Image File Format) metadata. Exif metadata may be extracted by an image processing library, such as ImageMagick (an open source command line tool), PIL (Python Image Library) or others. Specifically, the "_getexif ()" function in PIL may be used to extract Exif metadata.

[0124] The orientation information may be the value of the Orientation tag in the Exif metadata. The values ​​of the Orientation tag may include values ​​representing normal orientation, horizontal flip, 180-degree clockwise rotation, vertical flip, 90-degree clockwise rotation and horizontal flip, 90-degree clockwise rotation, 90-degree clockwise rotation and vertical flip, and 90-degree counterclockwise rotation. Corresponding one by one to the values ​​of the above-mentioned Orientation tags, the rotation corrections required for the original image are, respectively, no correction required, horizontal flip, 180-degree clockwise rotation, vertical flip, 90-degree clockwise rotation and horizontal flip, 90-degree clockwise rotation, 90-degree clockwise rotation and vertical flip, and 90-degree counterclockwise rotation. It can be understood that when the orientation information is a value representing the normal orientation, there is no need to perform rotation correction on the original image.

[0125] In this embodiment, when the original image meets the preset quality constraints and the original image does not belong to the preset type that produces negative impacts, it means that the image quality of the original image is good and will not produce negative impacts. At this time, the direction information of the original image is obtained, and the original image is rotated and corrected according to the direction information to obtain the initial image, ensuring that the image is in the normal direction to facilitate subsequent image generation.

[0126] In an exemplary embodiment, step 106 includes: setting the content in the image area corresponding to the position of the fillable area in the downsampled image to empty to obtain a masked image; extracting features based on the masked image through a trained control network to obtain control features; the control features are used to indicate that image generation is performed in the image area corresponding to the position of the fillable area in the masked image during the image generation process; and generating an image through a trained diffusion model based on the masked image and the control features to obtain a filled image.

[0127] The diffusion model generates data by simulating the forward diffusion process and the reverse diffusion process. The forward diffusion process starts from a clear image without adding noise, and gradually adds noise, so that the clear image gradually transforms into a noise distribution. The reverse diffusion process starts from the noise distribution transformed by the forward diffusion process, and gradually restores the original clear image or generates a new image. In the reverse diffusion process, the diffusion model learns how to restore the original clear image or generate a new image from the noisy data, which can be achieved by training the diffusion model. The diffusion model can be a latent diffusion model (Latent Diffusion Models, LDMs, the diffusion model provided in the paper High-Resolution Image Synthesis with Latent Diffusion Models).

[0128] The control network is used to control the image generation process of the diffusion model. The control network can be ControlNet (the network proposed in the paper Adding Conditional Control to Text-to-Image Diffusion Models). When performing feature extraction through ControlNet, the mask image can be used as the Condition information of ControlNet. ControlNet can extract features based on the Condition information and the Input of the diffusion model, output control features, and act on the Unet network of the diffusion model. It can be understood that the features output by ControlNet to the diffusion model are control features. The Input of the diffusion model can be a mask image or a fused feature obtained after feature fusion processing mentioned in the subsequent steps.

[0129] The Unet network of the diffusion model may include multiple Encoder Blocks and Decoder Blocks corresponding to each EncoderBlock. ControlNet may include a convolution layer with a convolution kernel size of 1*1 and multiple trainable Encoder Block copies. In ControlNet, the Condition information is subjected to feature extraction through a 1*1 convolution layer and then concatenated with the Input of the diffusion model. Feature extraction is performed through multiple trainable EncoderBlock copies to obtain the control features output by each Encoder Block copy. The control features can be input into the corresponding Decoder Block, thereby controlling the generation process of the diffusion model.

[0130] In this embodiment, the content in the image area corresponding to the position of the fillable area in the downsampled image is set to empty to obtain a masked image. In this way, the target object can be eliminated first and the fillable area can be marked. Then, the control feature is extracted through the control network. The control feature can indicate that the image area corresponding to the position of the fillable area in the masked image is to be generated. Therefore, according to the masked image and the control feature, the image is generated through the diffusion model. It is possible to generate only the image area corresponding to the position of the fillable area, while retaining the content of the content holding area, thereby improving the accuracy of image generation.

[0131] In one embodiment, a computer device may obtain a mask that matches the image size of the downsampled image, in which the pixel value of the image area corresponding to the position of the fillable area may be 0, and a matrix multiplication operation is performed on the downsampled image and the mask to set the content in the image area corresponding to the position of the fillable area in the downsampled image to empty.

[0132] In one embodiment, according to the fusion features and the control features, the image generation is performed through the diffusion model, and multiple filling images can be obtained. In each filling image, the target object is eliminated, and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content retention area. In this embodiment, multiple filling images are obtained, and in combination with the subsequent steps 108 to 110, multiple target images can be obtained for the user to choose, which avoids the common mode collapse problem of the GAN model and can improve the diversity of the generated results.

[0133] In an exemplary embodiment, the above-mentioned step of generating an image through a trained diffusion model based on the cover image and the control feature to obtain a filled image may include: obtaining a cover mask that matches the image size of the downsampled image, the cover mask is used to represent the position of the image area in the downsampled image corresponding to the position of the fillable area; adding noise to the cover image to obtain a noisy feature; performing feature fusion processing on the features obtained by performing matrix operation on the cover mask and the noisy feature, and the noisy feature and the cover mask to obtain a fused feature; and generating an image through a trained diffusion model based on the fused feature and the control feature to obtain a filled image.

[0134] Among them, the cover mask is used to indicate the position of the image area corresponding to the position of the fillable area in the downsampled image. It can be understood that when the downsampled image and the cover mask are superimposed, the position of the image area corresponding to the position of the fillable area in the cover mask can be the position of the image area corresponding to the position of the fillable area in the downsampled image.

[0135] In the cover image, the pixel value of the image area corresponding to the position of the fillable area may be 0. In the cover mask, the pixel value of the image area corresponding to the position of the fillable area may be 0. The matrix operation may be a matrix multiplication operation.

[0136] Noise addition can be adding Gaussian noise. When adding noise, Gaussian noise can be added in multiple time steps. In each time step, after adding Gaussian noise, the noisy feature z_t of the time step is obtained, z_t=sqrt(1-β_t)*z+sqrt(β_t)*ε, where ε is the standard Gaussian distribution, β_t is the noise variance at time step t, sqrt represents the square root operation, * represents the multiplication operation, and z represents the noisy feature obtained in the previous time step.

[0137] The noisy feature can be in the form of feature map data. The feature fusion process can be a cascade process of the feature obtained by performing matrix multiplication operation on the cover mask and the noisy feature, the noisy feature, and the cover mask. The cascade can be understood as a process of feature splicing in the channel direction. The noisy feature can be a 4-channel feature map, and the cover mask can be understood as a single-channel feature map. Then, the feature obtained by performing matrix multiplication operation on the cover mask and the noisy feature can be a 4-channel feature map, and the fused feature obtained by performing feature fusion processing can be a 9-channel feature map. The fused feature can be used as the input of the diffusion model.

[0138] In this embodiment, since noise is added to the masked image, the obtained noisy features may reduce the information representing the position of the fillable area in the masked image, and the mask is used to represent the position of the image area corresponding to the position of the fillable area in the downsampled image. The features obtained by performing matrix operation on the mask and the noisy features are fused with the noisy features and the mask to obtain fused features. The fused features can introduce information representing the position of the fillable area and provide richer input features for the subsequent diffusion model to generate images, thereby improving the accuracy of image generation.

[0139] In one embodiment, the step of adding noise to the masked image to obtain the noisy features may include: the computer device may input the masked image into an encoder in a trained autoencoder model to obtain four-channel features, and add noise to the four-channel features for multiple time steps to obtain the noisy features. The autoencoder model may be an AutoEncoder, which may include an encoder and a decoder.

[0140] In one embodiment, the above-mentioned step of generating an image based on the fusion features and the control features through a trained diffusion model to obtain a filled image may include: a computer device may input the fusion features and the control features into the trained diffusion model to generate an image, obtain the output features of the diffusion model, input the output features into a decoder in the autoencoder model, and obtain the filled image output by the decoder.

[0141] In an exemplary embodiment, step 110 may include: obtaining a first channel image included in the upsampled image and obtaining a second channel image included in the image to be processed in each color channel of multiple color channels; fusing the image area corresponding to the position of the fillable area in the first channel image under the color channel and the image area corresponding to the content retention area in the second channel image under the color channel using a Poisson fusion algorithm to obtain a fused image under the color channel; and generating a target image based on the fused images under the multiple color channels.

[0142] The upsampled image may include a first channel image in each of the multiple color channels. The image to be processed may include a second channel image in each of the multiple color channels. It can be understood that the first channel image and the second channel image are both single-channel images. The fused images in the multiple color channels may be combined to obtain a target image. The Poisson fusion algorithm may obtain a fused image by solving the Poisson equation. The Poisson equation may refer to the following formula (1).

[0143] Formula (1)

[0144] Among them, formula (1) can express that the fused image under the color channel satisfies And meet . It can represent the fused image under the color channel to be solved. The image area corresponding to the position of the fillable area in the fused image under the color channel can be represented. It can represent the first channel image under this color channel The gradient field of the image area corresponding to the position, Characterizable The boundary conditions, Can represent surrounding the border. The Laplace operator can be characterized as the sum of second-order partial derivatives. In a two-dimensional image, the Laplace operator describes the local curvature of each pixel in the image. The divergence operator can be characterized, which describes the divergence of the gradient field.

[0145] The meaning of can be that the fused image under this color channel exist The result of the Laplacian operator calculation in is equal to the value in the first channel image under this color channel. The divergence of the gradient field of the image area corresponding to the position, that is, the Laplace operator is used to fusion the image under the color channel The calculation result obtained by calculating the image area corresponding to the fillable area position in the first channel image under the color channel is equal to the divergence of the gradient field of the image area corresponding to the fillable area position in the first channel image under the color channel. This makes the image area corresponding to the fillable area position after fusion maintain the same gradient structure as the image area corresponding to the fillable area position in the first channel image under the color channel, thereby maintaining the texture characteristics of the first channel image in the area corresponding to the fillable area position.

[0146] The meaning of can be that the fused image under this color channel Boundary conditions The fused image under this color channel is exist The pixel at the color channel is in the same pixel as the second channel image at the color channel. The pixels at corresponding positions are the same. In other words, the fused image under this color channel The pixels contained in the boundary of the image area corresponding to the position of the fillable area in the image are the same as the pixels contained in the boundary of the image area corresponding to the position of the fillable area in the second channel image under the color channel, so that there is a smooth transition at the boundary after fusion.

[0147] In this embodiment, image fusion is performed through the Poisson fusion algorithm to obtain fused images under each color channel, and then a target image is generated based on the fused images under multiple color channels, so that the portion of the target image corresponding to the position of the content preservation area is the same as the content of the content preservation area in the image to be processed, thereby retaining the original resolution and detail texture of the image to be processed. The image area in the target image corresponding to the position of the fillable area has a smooth transition with the portion corresponding to the position of the content preservation area in the target image while retaining the gradient structure of the image area in the upsampled image corresponding to the position of the fillable area, so that the target image is more realistic and natural.

[0148] In a specific embodiment, see Figure 2 As shown in the schematic diagram of the image content filling framework, the above-mentioned image content filling method may specifically include the following steps.

[0149] See Figure 3 In the schematic flow diagram of the preprocessing steps shown, the terminal can obtain the input original image, and perform image quality assessment on the original image based on the preset quality constraint condition, and the preset quality constraint condition may include the preset resolution constraint condition and the preset noise constraint condition. When the original image does not meet the preset resolution constraint condition, a prompt message indicating low resolution is generated to prompt the user to re-enter. When the original image does not meet the preset noise constraint condition, a prompt message indicating high noise is generated to prompt the user to re-enter. When the original image meets the preset resolution constraint condition and the preset noise constraint condition, the image content of the original image is reviewed, and the image recognition algorithm can be used to identify whether the original image belongs to the preset type that has a negative impact. When the original image belongs to the preset type, a prompt message indicating that it contains negative information is generated to prompt the user to re-enter, or the original image is sent to the review end for manual review by the reviewer. When the review result indicates that the original image belongs to the preset type, a prompt message indicating that it contains negative information is generated to prompt the user to re-enter; when the original image does not belong to the preset type, the direction information of the original image is obtained. According to the direction information, the original image is rotationally corrected to obtain the initial image to be identified for the area.

[0150] The terminal can determine the image to be processed and the fillable area in the image to be processed through interactive operations. Specifically, the terminal can determine the area selected in the initial image by the area selection operation in response to the area selection operation, determine the area selected in the initial image as the fillable area of ​​the image to be processed, and determine other areas outside the area selected in the initial image as the content retention area of ​​the image to be processed.

[0151] The terminal can also automatically determine the image to be processed and the fillable area in the image to be processed, specifically, this can be achieved by sequentially performing the target object recognition step and the shadow recognition step. The target object recognition step may include: the terminal may perform image segmentation on the initial image through different trained object segmentation models, obtain the object mask output by each object segmentation model, fuse the object masks output by different object segmentation models, and obtain the object mask of the target object to identify the object area in the initial image that contains the target object to be eliminated. Among them, the object mask of the target object can represent the object area in the initial image that contains the target object to be eliminated. The object mask of the target object may specifically include an area corresponding to the position of the object area in the initial image.

[0152] Based on Figure 4 The shadow recognition algorithm framework schematic diagram shown in the figure, the shadow recognition step may include: the terminal may input the initial image and the object mask into the trained shadow segmentation model for image segmentation; during the image segmentation process, feature extraction is performed through the encoder in the shadow segmentation model, and deconvolution is performed based on the features extracted by the encoder through the decoder to output feature maps of two channels, the feature maps of the two channels are normalized by the Softmax function, and then thresholded to obtain an initial shadow mask, the initial shadow mask is post-segmented to obtain a target shadow mask, and the target shadow mask is determined as the segmentation result output by the shadow segmentation model; based on the segmentation result, the shadow area in the initial image containing the shadow of the target object is determined.

[0153] The terminal may determine the target object and the shadow of the target object as the target objects to be eliminated, determine the object area and the shadow area in the initial image as the fillable area, and determine the image area other than the object area and the shadow area in the initial image as the content retention area to obtain the image to be processed. The terminal may downsample the image to be processed to an image size supported by the trained diffusion model to obtain the downsampled image.

[0154] See Figure 5In the image generation framework schematic diagram shown in the figure, the terminal can obtain a cover mask that matches the image size of the downsampled image. In the cover mask, the pixel value of the image area corresponding to the position of the fillable area can be 0; the downsampled image and the cover mask are matrix multiplied to set the content in the image area corresponding to the position of the fillable area in the downsampled image to empty, and the cover image is obtained. Through the trained control network, feature extraction is performed according to the cover image to obtain the control feature. The cover image is input into the encoder in the trained autoencoder model, and the encoder maps the cover image to the latent space to obtain a four-channel feature, and the four-channel feature is denoised for multiple time steps to obtain a noisy feature. The feature obtained by matrix multiplication of the cover mask and the noisy feature is fused with the noisy feature and the cover mask to obtain a fused feature. The fusion features and the control features are input into the trained diffusion model for image generation to obtain the output features of the diffusion model. The output features are input into the decoder in the autoencoder model to obtain a filled image output by the decoder. In the filled image, the target object is eliminated, and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content-preserving area.

[0155] The terminal may upsample the padded image to the image size of the image to be processed to obtain an upsampled image.

[0156] The terminal can use the following processing method to perform image fusion to obtain a target image: in each color channel of multiple color channels, a first channel image included in the upsampled image can be obtained, and a second channel image included in the image to be processed can be obtained; an image area corresponding to the position of the fillable area in the first channel image under the color channel and an image area corresponding to the content retention area in the second channel image under the color channel are fused using a Poisson fusion algorithm to obtain a fused image under the color channel; and a target image is generated based on the fused images under multiple color channels.

[0157] The terminal can also use Figure 6 The processing method in the image fusion step flow diagram shown in the figure is used to perform image fusion to obtain a target image. Specifically, the terminal can segment an image area corresponding to the position of the fillable area from the upsampled image, segment a content retention area from the image to be processed, and splice the image area corresponding to the position of the fillable area in the upsampled image with the content retention area to obtain a spliced ​​image; in each color channel of multiple color channels, the Poisson fusion algorithm is used to fuse the image area corresponding to the position of the fillable area in the spliced ​​image and the image area corresponding to the position of the content retention area in the spliced ​​image to obtain a fused image in the color channel; and the target image is generated based on the fused images in the multiple color channels.

[0158] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0159] Based on the same inventive concept, the embodiment of the present application also provides an image content filling device for implementing the image content filling method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image content filling device embodiments provided below can refer to the limitations of the image content filling method above, and will not be repeated here.

[0160] In an exemplary embodiment, Figure 7 As shown, an image content filling device 700 is provided, comprising: an acquisition module 710, a downsampling module 720, a generation module 730, an upsampling module 740 and a fusion module 750, wherein:

[0161] The acquisition module 710 is used to acquire the image to be processed, which includes a content retention area and a fillable area containing a target object to be eliminated; the fillable area is used to eliminate the target object and fill it.

[0162] The downsampling module 720 is used to downsample the image to be processed to obtain a downsampled image.

[0163] The generation module 730 generates an image based on the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content retention area.

[0164] The up-sampling module 740 up-samples the padded image to the image size of the image to be processed to obtain an up-sampled image.

[0165] The fusion module 750 is used to fuse the image area corresponding to the position of the fillable area in the up-sampled image with the content preservation area to obtain a target image.

[0166] In an exemplary embodiment, the acquisition module 710 may include an image acquisition module and a region recognition module, the image acquisition module is used to acquire an initial image to be subjected to region recognition; the region recognition module is used to identify an object region in the initial image that contains a target object to be eliminated; based on the object region, a shadow region in the initial image that contains a shadow of the target object is determined; the object region and the shadow region in the initial image are determined as fillable regions, and the image region other than the object region and the shadow region in the initial image is determined as a content retention region, to obtain an image to be processed.

[0167] In an exemplary embodiment, the region recognition module is also used to obtain an object mask of the target object that matches the image size of the initial image, the object mask including an area corresponding to the position of the object area in the initial image; the initial image and the object mask are input into a trained shadow segmentation model for image segmentation to obtain a segmentation result output by the shadow segmentation model; based on the segmentation result, a shadow area in the initial image containing the shadow of the target object is determined.

[0168] In an exemplary embodiment, the image acquisition module is also used to acquire an input original image; when the original image meets the preset quality constraint conditions and the original image does not belong to the preset type that produces negative impacts, the direction information of the original image is acquired; according to the direction information, the original image is rotationally corrected to obtain an initial image for area recognition.

[0169] In an exemplary embodiment, the generation module 730 is also used to set the content in the image area corresponding to the position of the fillable area in the downsampled image to empty to obtain a masked image; perform feature extraction based on the masked image through a trained control network to obtain control features; the control features are used to indicate that image generation is performed in the image area corresponding to the position of the fillable area in the masked image during the image generation process; and perform image generation through a trained diffusion model based on the masked image and the control features to obtain a filled image.

[0170] In an exemplary embodiment, the generation module 730 is also used to obtain a cover mask that matches the image size of the downsampled image, the cover mask is used to indicate the position of the image area in the downsampled image corresponding to the position of the fillable area; add noise to the cover image to obtain a noisy feature; perform feature fusion processing on the features obtained by performing matrix operations on the cover mask and the noisy feature, and the noisy feature and the cover mask to obtain a fused feature; and generate an image based on the fused feature and the control feature through a trained diffusion model to obtain a filled image.

[0171] In an exemplary embodiment, the fusion module 750 is also used to obtain a first channel image included in the upsampled image and a second channel image included in the image to be processed in each color channel of multiple color channels; the image area corresponding to the position of the fillable area in the first channel image under the color channel and the image area corresponding to the content retention area in the second channel image under the color channel are fused using a Poisson fusion algorithm to obtain a fused image under the color channel; and a target image is generated based on the fused images under multiple color channels.

[0172] Each module in the above-mentioned image content filling device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0173] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image content filling method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0174] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0175] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0177] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0179] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0181] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for filling image content, characterized in that: The method comprises: Acquire an image to be processed, wherein the image to be processed includes a content retention area and a fillable area containing a target object to be eliminated; the fillable area is used to eliminate the target object and fill it; Downsampling the image to be processed to obtain a downsampled image; Performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area; Upsampling the padded image to the image size of the image to be processed to obtain an upsampled image; An image area corresponding to the position of the fillable area in the up-sampled image is fused with the content retention area to obtain a target image.

2. The method according to claim 1, characterized in that The obtaining of the image to be processed comprises: Acquire an initial image for region recognition; Identifying an object region in the initial image that contains a target object to be eliminated; Determining, according to the object region, a shadow region in the initial image that contains a shadow of the target object; The object area and the shadow area in the initial image are determined as fillable areas, and the image area other than the object area and the shadow area in the initial image is determined as a content retention area to obtain an image to be processed.

3. The method according to claim 2, characterized in that The determining, according to the object area, a shadow area in the initial image that contains the shadow of the target object comprises: Acquire an object mask of the target object that matches the image size of the initial image, the object mask comprising an area corresponding to the position of the object area in the initial image; Inputting the initial image and the object mask into a trained shadow segmentation model to perform image segmentation, and obtaining a segmentation result output by the shadow segmentation model; Based on the segmentation result, a shadow area in the initial image including the shadow of the target object is determined.

4. The method according to claim 2, characterized in that: The obtaining of the initial image to be subjected to region recognition comprises: Get the input original image; When the original image meets the preset quality constraint condition and the original image does not belong to the preset type that produces negative impact, obtaining the direction information of the original image; According to the direction information, the original image is rotationally corrected to obtain an initial image for region recognition.

5. The method according to claim 1, characterized in that Generating an image according to the downsampled image to obtain a filled image comprises: Setting the content in the image area corresponding to the position of the fillable area in the downsampled image to empty to obtain a covered image; Extracting features according to the cover image through the trained control network to obtain control features; the control features are used to indicate that image generation is performed in the image area corresponding to the position of the fillable area in the cover image during the image generation process; According to the mask image and the control feature, an image is generated by using a trained diffusion model to obtain a filled image.

6. The method according to claim 5, characterized in that The step of generating an image by using a trained diffusion model according to the mask image and the control feature to obtain a filled image includes: Acquire a mask matching the image size of the downsampled image, wherein the mask is used to indicate the position of an image area in the downsampled image corresponding to the position of the fillable area; adding noise to the masked image to obtain a noisy feature; A feature obtained by performing matrix operation on the cover mask and the noisy feature is subjected to feature fusion processing with the noisy feature and the cover mask to obtain a fused feature; According to the fusion feature and the control feature, an image is generated through a trained diffusion model to obtain a filled image.

7. The method according to any one of claims 1 to 6, characterized in that: The step of fusing the image area corresponding to the position of the fillable area in the upsampled image with the content retention area to obtain a target image includes: In each color channel of the plurality of color channels, obtaining a first channel image included in the upsampled image, and obtaining a second channel image included in the image to be processed; The image area corresponding to the position of the fillable area in the first channel image under the color channel and the image area corresponding to the content retention area in the second channel image under the color channel are fused by using a Poisson fusion algorithm to obtain a fused image under the color channel; A target image is generated according to the fused images under the multiple color channels.

8. An image content filling device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a content holding area and a fillable area including a target object to be eliminated; the fillable area is used for eliminating the target object and filling it; A downsampling module, used for downsampling the image to be processed to obtain a downsampled image; A generating module, performing image generation according to the downsampled image to obtain a filled image, in which the target object is eliminated and the content of the image area corresponding to the position of the fillable area is generated according to the content of the content holding area; An upsampling module, upsampling the padded image to the image size of the image to be processed to obtain an upsampled image; The fusion module is used to fuse the image area corresponding to the position of the fillable area in the upsampled image with the content retention area to obtain a target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images