Image inpainting method, device, computer readable storage medium and computer program product

Abstract

This application relates to an image restoration method, apparatus, computer-readable storage medium, and computer program product. The method includes: acquiring an original image; identifying a background image and a foreground image corresponding to the original image; identifying defect regions in the background image to obtain regions to be restored; generating an image based on image features of other regions in the background image besides the region to be restored, to obtain a restored image corresponding to the region to be restored; and fusing the restored image with the foreground image to obtain a target image with restored background corresponding to the original image. This method can improve the efficiency and visual effect of image background restoration.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of photographic and digital image processing technologies, portrait photography is increasingly widely used in commercial photography, personal portraits, and ID photos. In professional studio shooting, using a backdrop is intended to create a clean, uniform, and undisturbed shooting environment, thereby maximizing the prominence of the subject. However, maintaining the ideal state of the backdrop during actual shooting is quite difficult. Environmental noise and other elements can cause the background to appear mottled, uneven in brightness, or faded in the final image, severely compromising the professionalism and immersive feel of the picture.

[0003] To address the aforementioned issues, related technologies generally rely on manual post-processing corrections. Photographers and retouchers need to perform meticulous manual operations in image editing software: for localized dirt, shoe prints, or tape residue on the backdrop, retouchers repeatedly use the clone stamp or patch tool to repair it; for large areas of wrinkles on the fabric backdrop, retouchers may use the liquefy filter to smooth them out; when background problems are too severe, retouchers need to redraw and replace large areas of the background. These related technologies suffer from low processing efficiency and poor results. Summary of the Invention

[0004] Based on this, this application provides an image restoration method, apparatus, computer-readable storage medium, and computer program product, which can improve the efficiency and visual effect of image background restoration.

[0005] On the one hand, this application provides an image restoration method, including:

[0006] Acquire the original image and identify the background and foreground images corresponding to the original image;

[0007] Defect region identification is performed on the background image to obtain the region to be repaired in the background image;

[0008] Image generation is performed based on the image features of other regions in the background image besides the region to be repaired, to obtain the repaired image corresponding to the region to be repaired.

[0009] The restored image is then fused with the foreground image to obtain the target image with restored background corresponding to the original image.

[0010] On the one hand, this application also provides an image restoration apparatus, the apparatus comprising:

[0011] The acquisition module is used to acquire the original image and identify the background image and foreground image corresponding to the original image;

[0012] The identification module is used to identify defect areas in the background image to obtain the areas to be repaired in the background image.

[0013] The generation module is used to generate an image based on the image features of other regions in the background image besides the region to be repaired, to obtain the repaired image corresponding to the region to be repaired.

[0014] The fusion module is used to fuse the repaired image with the foreground image to obtain the target image with background repair corresponding to the original image.

[0015] On the one hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps included in any of the foregoing method embodiments.

[0016] On the one hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0017] On the one hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0018] The aforementioned image restoration method, apparatus, computer-readable storage medium, and computer program product automatically detect and locate flawed and obtrusive areas by identifying defective areas in the background image. Furthermore, they generate images based on the image features of other areas in the background image besides the area to be restored. They can understand the contextual lighting, texture, and shadow information, thereby avoiding the visual abruptness problem of manual restoration. In summary, this embodiment can improve efficiency while ensuring output quality, achieving automatic, efficient, and high-quality restoration of flawed and obtrusive areas in the background of a photographic backdrop without relying on tedious manual operations. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is an application environment diagram of an image restoration method in one embodiment;

[0021] Figure 2 This is a flowchart illustrating an image restoration method in one embodiment;

[0022] Figure 3 This is a structural block diagram of an image restoration device in one embodiment;

[0023] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] Before describing the embodiments of this application, the relevant technologies and their existing problems will be further explained:

[0026] In professional photography studios, the initial purpose of using a backdrop (whether paper, cloth, or vinyl) is to create a clean, uniform, and undisturbed background environment to maximize the prominence of the subject. However, in practice, maintaining this ideal state is quite difficult, and photographers and post-production staff generally face a series of complex visual problems caused by backdrop defects and environmental limitations.

[0027] Maintaining the physical cleanliness of the backdrop is one of the biggest challenges. During studio operations such as staff movement, model costume changes, and prop placement, the backdrop (especially the floor-mounted portion) inevitably accumulates dirt, shoe prints, dust, and scratches. Fabric backdrops are also prone to developing stubborn wrinkles and creases, which, under studio lighting, create uneven lines of light and shadow, disrupting the smoothness of the background. Furthermore, within the limited studio space, to achieve specific shooting angles, the lens easily captures unwanted elements outside the backdrop, such as the edges and support poles of the backdrop, floor lamp stands and sandbags, exposed power cords, and even corners or floors of the studio itself.

[0028] Traditional manual post-processing methods have several limitations: First, manual retouching is tedious and time-consuming; a photo with many issues may require a senior retoucher to spend tens of minutes or even an hour processing it, severely reducing work efficiency. Second, backgrounds often have subtle variations in light and shadow under different lighting conditions, and using tools like the clone stamp tool can easily create a "patchwork" effect, meaning the brightness or tone of the sampled source doesn't match the target area, resulting in obvious retouching marks. Finally, when using background replacement, the accuracy of subject cutout is difficult to guarantee, especially when dealing with complex edges such as hair and semi-transparent clothing, easily producing harsh "white edges" or "jagged edges," and the replaced background loses the original light and shadow relationships, resulting in a fake and unnatural final product.

[0029] To address the pain points of low efficiency and poor results in manual post-processing, the following image algorithm-based solutions exist: for example, inpainting algorithms can be used to repair local blemishes in images, and outpainting algorithms can be used to handle artifacts at the edges of the image. However, these algorithms still have significant shortcomings in practical applications: on the one hand, most of these algorithms rely on users manually selecting the area to be repaired, and the algorithms themselves cannot automatically identify "where is dirty" or "where is the artifact," requiring users to invest a lot of time in precise masking; on the other hand, these algorithms are mostly single-function "point-like" solutions, lacking a systematic solution for the specific scenario of a photography studio backdrop, and unable to simultaneously handle multiple problems such as physical blemishes, artifacts, and the resulting uneven tones.

[0030] The image restoration method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0031] In one exemplary embodiment, such as Figure 2As shown, an image restoration method is provided, which can be applied to... Figure 1 Taking terminal 102 or server 104 as an example, the following steps are included:

[0032] Step 202: Obtain the original image and identify the background image and foreground image corresponding to the original image.

[0033] The original image can be a photo taken in a studio using a backdrop. The backdrop can be made of paper, cloth, or vinyl, and its initial design purpose is to provide a clean, uniform, and undisturbed background environment. However, in actual shooting, due to factors such as people moving around in the studio, models changing clothes, and prop placement, the backdrop (especially the part laid on the floor) inevitably accumulates dirt, shoe prints, dust, and scratches. Cloth backdrops are also prone to developing stubborn wrinkles and creases, creating uneven lines of light and shadow under lighting. Furthermore, when shooting in the limited space of a studio, the lens can easily capture information about objects outside the backdrop, such as light stands, sandbags, power cords, and even the corners or floor of the studio itself. These physical defects and other visual flaws combine to result in a "blemished," mottled, and unevenly lit background in the final image, severely damaging the professionalism and immersive feel of the image. Therefore, this embodiment separates the portrait foreground from the defective background, so that subsequent processing can independently repair the background without affecting the original details of the foreground.

[0034] The process of identifying the background and foreground images utilizes high-precision image matting technology. Unlike simple binary segmentation, matting can handle complex edges such as hair strands and semi-transparent clothing, generating an alpha value (transparency value) between 0 and 1 for each pixel. This allows for precise differentiation of the foreground, background, and semi-transparent areas where they blend. Specifically, after acquiring the original image, a coarse image segmentation network is used to quickly locate the approximate area of ​​the portrait, generating a coarse mask. It should be noted that while this coarse mask cannot finely process edge details, it effectively defines the main subject area of ​​the portrait. Then, based on this coarse mask, morphological erosion and dilation operations are used to generate a trimap, dividing the image into three regions: an absolute foreground region (pixel value of 255), an absolute background region (pixel value of 0), and an unknown region containing complex edges such as hair strands (pixel value of 128). Finally, the original image and ternary map are input into a preset image matting model (such as PP-Matting or MODNet). This model performs fine-grained transparency prediction in unknown areas and outputs a high-precision foreground mask (Alpha Matte). The foreground mask is a single-channel image with pixel values ​​ranging from 0 to 255, where 0 represents completely transparent (background), 255 represents completely opaque (foreground), and values ​​between 0 and 255 represent semi-transparent states.

[0035] Based on this foreground mask, the foreground and background images can be separated from the original image: Foreground image = Original image × (Foreground mask / 255); Background image = Original image × (1 - Foreground mask / 255). Thus, the portrait area in the background image is cleared, retaining only the information of the backdrop area, providing a clean input for subsequent defect identification and repair.

[0036] Step 204: Perform defect region identification on the background image to obtain the region to be repaired in the background image.

[0037] In this context, "defective areas" refer to regions in the background image that differ significantly from the ideal background due to physical flaws or limitations imposed by shooting conditions. These areas understandably detract from the purity and professionalism of the background, thus requiring restoration. Furthermore, considering the causes, the defective areas in this embodiment can be categorized into flawed areas and "blemishes." Flawed areas refer to physical defects inherent in the backdrop itself, such as dirt and shoe prints (caused by footsteps), tape residue (formed by removing tape after temporarily securing the backdrop or props), wrinkles and creases (caused by folding or pulling the backdrop), and scratches. These flaws visually manifest as localized color anomalies, abrupt texture changes, or uneven brightness. "Bloopers," on the other hand, refer to non-backdrop elements appearing within the lens's field of view, including backdrop edges and support poles, floor lamp stands and sandbags, exposed power cords, and studio corners or floors. These defective elements exhibit significant differences in color, texture, and lighting characteristics from the backdrop background, disrupting the integrity of the image.

[0038] Therefore, in this embodiment, it is necessary to detect and locate these defective areas and generate a mask for the area to be repaired, thereby providing a precise "construction range" for subsequent targeted repairs. Specifically, a parallel processing strategy can be used for defect area identification, separately identifying the areas where the flaws are visible and the areas with defects. For identifying the areas where the flaws are visible, considering that the backdrop is usually characterized by large areas and relatively solid colors, an elimination strategy is used in this embodiment to accurately segment the backdrop area. First, pixel statistical analysis is performed on the background image to determine the main color range of the backdrop, and an initial backdrop mask is generated based on this. Then, this initial mask is used as a prompt and input along with the background image into a preset image segmentation model (such as the SegmentAnything Model, SAM). The powerful segmentation capabilities of SAM are used to output a precise backdrop mask. Finally, the foreground mask and the precise backdrop mask are subtracted from the global mask, and the remaining part is the mask for the areas where the flaws are visible.

[0039] Correspondingly, for the identification of defective areas, this embodiment utilizes the prior knowledge that the background of the curtain should be a smooth gradient. A low-pass filter (such as median filtering) is applied to the curtain area to estimate the "proper" smoothness of the curtain, resulting in a smoothed background image. Then, the original curtain image and the smoothed background image are compared pixel-by-pixel to obtain a high-frequency difference image. In clean, smooth areas of the curtain, the difference value is close to zero; while in areas with dirt, tape, wrinkles, etc., the difference value increases significantly. By performing thresholding and morphological refinement (such as opening operations to remove noise and closing operations to fill holes) on this difference image, high-frequency abnormal information in the curtain area is extracted, thus obtaining an accurate defect mask. Finally, the above-mentioned flaw mask and defect mask are merged to obtain the mask for the area to be repaired. This mask accurately identifies all areas in the background image that need to be repaired.

[0040] Step 206: Generate an image based on the image features of other regions in the background image besides the region to be repaired, to obtain the repaired image corresponding to the region to be repaired.

[0041] In order to intelligently fill in the area to be repaired and make it visually blend naturally with the surrounding clean background, this embodiment uses the contextual information around the area to be repaired as a reference to generate new content that is consistent with the relevant background in terms of texture, lighting, and color, rather than simply copying and pasting or using a single color to fill it. Considering that non-defective areas represent the "ideal state" of the backdrop background—that is, in a photographic backdrop scene, the background should be smooth, uniform, or have a uniform gradient—while defective areas (dirt, wrinkles, flaws) are localized, abnormal damage, this embodiment uses the clean, undamaged background around the defective area as a reference. This ensures that the generated content conforms to the original visual rules of the backdrop, such as the direction of lighting gradients, texture features, and color distribution, thereby fundamentally avoiding the "patchwork" and "distortion" that traditional repair methods (such as clone stamps) easily produce on gradient backgrounds.

[0042] Specifically, image inpainting techniques based on diffusion models can be employed. Diffusion models are a type of generative artificial intelligence model whose core principle is to progressively add noise to "destroy" an image into pure noise, and then learn to reverse the process to gradually "de-noise" and restore the original image. In the inpainting task, the diffusion model can generate reasonable new content within the masked area based on the contextual information surrounding the area to be repaired. Specifically, the background image, the mask of the area to be repaired, and a textual cue describing the visual features of the target background (such as "a smooth, clean, seamless photo studio backdrop") are input into a pre-trained diffusion model. The model iteratively denoises and generates content in the latent space, only filling in the content in the area covered by the mask, while leaving the areas outside the mask unchanged. The final output is the repaired image, where areas that originally contained dirt, tape, wrinkles, or other unwanted elements have been filled in with natural and coherent new content.

[0043] The generative repair method used in this embodiment can not only handle single-point defects, but also cope with large-area complex defects (such as large wrinkles or multiple flaws). At the same time, since the generation process takes into account the contextual information of the surrounding background, the repaired area and the original background can be seamlessly connected in terms of lighting gradation, texture direction and color transition, avoiding the visual abruptness problem common in manual repair.

[0044] Step 208: Perform image fusion between the repaired image and the foreground image to obtain the target image with background repair corresponding to the original image.

[0045] In order to seamlessly synthesize the clean and complete restored background with the high-precision portrait foreground separated in the previous steps and output the final target image, the fusion process in this embodiment can be based on Alpha Compositing. Specifically, it can include: based on the foreground mask, performing a pixel-by-pixel weighted summation of the pixel values ​​of the foreground image and the background image. The summation formula can be as follows: Output_Pixel = (Foreground_Pixel × α) + (Background_Pixel × (1 - α)).

[0046] Here, α is the normalized transparency value of the foreground mask (ranging from 0 to 1). When processing solid areas of a portrait, α can be approximately 1, meaning that the output pixels almost entirely come from the foreground image; when processing pure background areas, α≈0, meaning the output pixels come entirely from the restored background image; when processing semi-transparent edges such as hair strands, α takes an intermediate value, and the output pixels are a mixture of the foreground and background colors, thus achieving a natural transition effect and effectively avoiding the "white edges" or "jagged edges" commonly seen in traditional image cutout.

[0047] Optionally, to further improve the fusion effect, the restored image can be tone-corrected before alpha synthesis. Since the region generated by the generative inpainting model may have subtle color differences or noise texture differences with the original background, this solution can use multi-band fusion technology to smoothly transition the restored region with the original background, and add camera noise (grain) consistent with the original background to the restored region through noise matching technology, so that the restored region is consistent with the original background in microscopic texture, making the restored region "invisible" in the overall background.

[0048] This embodiment achieves automated, high-quality repair of background imperfections and continuity errors in photographic backdrops. The entire process requires no manual intervention, preserving the edge details of the portrait while generating repaired content visually consistent with the original background, significantly improving post-processing efficiency and final image quality.

[0049] In some embodiments, acquiring the original image and identifying the background and foreground images corresponding to the original image includes:

[0050] Perform portrait segmentation on the original image to obtain a coarse portrait mask corresponding to the original image;

[0051] Morphological processing is performed on the portrait mask to obtain the predicted foreground region, predicted background region, and unknown region of the original image;

[0052] Based on the original image, the predicted foreground region, and the background region, a preset image matting model is used to predict the transparency of the unknown region, thereby obtaining the foreground mask corresponding to the original image.

[0053] The foreground image and the background image are identified from the original image based on the foreground mask.

[0054] Portrait segmentation refers to the image processing operation of initially distinguishing pixel regions belonging to the human image from background regions in the original image. A coarse portrait mask is a single-channel probability map with the same resolution as the original image, generated by the portrait segmentation algorithm. Each pixel in this probability map has a value between 0 and 1, representing the confidence that the pixel belongs to the human image region. Specifically, a value closer to 1 indicates a higher probability that the pixel belongs to the human image, while a value closer to 0 indicates a higher probability that the pixel belongs to the background. Because this mask is generated with the priority of processing speed, its prediction results in complex edge areas such as hair and semi-transparent clothing are relatively blurry and cannot be directly used for fine-grained image matting; therefore, it is called a "coarse mask."

[0055] Specifically, after acquiring the original image, it is input into a pre-defined coarse portrait segmentation network. This network employs a lightweight deep learning model with an encoder-decoder architecture, designed to achieve fast processing while maintaining a certain level of accuracy to meet the needs of real-time or batch processing. The encoder part can use a lightweight backbone network (such as MobileNetV3) to downsample the original image layer by layer, extracting multi-level feature maps from low-level edge features to high-level semantic features. The decoder part gradually enlarges the feature map size through deconvolution or upsampling operations, and uses skip connections to concatenate the feature maps extracted from each layer of the encoder with the corresponding feature maps of the decoder layers, so as to preserve spatial detail information while restoring resolution. The last layer of the coarse portrait segmentation network outputs a coarse mask of the portrait through convolution operations and a sigmoid activation function. This embodiment uses a lightweight coarse portrait segmentation network to quickly locate the approximate area of ​​the portrait, providing guiding information for subsequent fine matting, while avoiding the computational overhead of directly using a high-precision matting model to process the entire image.

[0056] Morphological processing refers to operations performed on binary or grayscale images based on image morphological operations (such as erosion and dilation). The predicted foreground region refers to the area obtained after performing an erosion operation on a coarse mask of the portrait. Pixels within this region are considered 100% foreground, and their alpha value is 1. The predicted background region refers to the area obtained after performing a dilation operation on a coarse mask of the portrait and then excluding the predicted foreground region. Pixels within this region are considered 100% background, and their alpha value is 0. The unknown region refers to the transition zone between the predicted foreground and predicted background regions. This region contains complex edge details such as hair strands and semi-transparent clothing, and its pixel alpha values ​​are between 0 and 1, requiring further fine-tuning in subsequent steps.

[0057] Specifically, first, preset erosion and dilation kernel sizes (e.g., both 15×15 pixels) are set, and morphological operations are performed on the portrait coarse mask. The erosion operation shrinks the areas with median values ​​greater than 0 in the portrait coarse mask inwards; the shrunken area is the predicted foreground region. The dilation operation expands the areas with median values ​​greater than 0 in the portrait coarse mask outwards; the portion of the expanded area not covered by the predicted foreground region is the predicted background region. The kernel sizes for erosion and dilation can be adaptively adjusted according to the resolution of the original image; for example, a larger kernel size can be set for high-resolution images, and a smaller kernel size can be set for low-resolution images. The number of pixels for erosion and dilation can be the same, or different values ​​can be set as needed, for example, making the number of pixels for erosion less than the number of pixels for dilation, to reserve a wider unknown region for finer processing. Through the above morphological processing, the portrait coarse mask is converted into a ternary graph structure, where the pixel value of the predicted foreground region is set to 255, the pixel value of the predicted background region is set to 0, and the pixel value of the unknown region is set to 128. This ternary graph provides clear guidance for subsequent matting models, indicating which areas the model should perform fine-grained transparency predictions in, thus concentrating computational resources on the most complex edge regions that require the most processing.

[0058] Transparency prediction refers to the process of calculating the weight value (Alpha value) of each pixel within an unknown region, based on the color information of the original image and prior information provided by the predicted foreground and background regions. The result of transparency prediction is a single-channel image with the same resolution as the original image, called a foreground mask. The value of each pixel in this foreground mask represents the degree to which that pixel belongs to the foreground: a value of 0 indicates that it belongs entirely to the background, a value of 255 indicates that it belongs entirely to the foreground, and values ​​between 0 and 255 represent a semi-transparent state (such as hair strands, the edge of a wedding dress, etc.).

[0059] Specifically, the original image and the ternary image are used as input and fed into a pre-trained image matting model. This model is a pre-trained deep learning network specifically designed to predict detailed alpha masks from images guided by ternary images. Taking the PP-Matting model as an example, its implementation process includes two stages: The first stage is the basic prediction stage, where the original image and the ternary image are concatenated along the channel dimension to form an input tensor. Features are extracted through an encoder-decoder network, and a preliminary alpha prediction map is output. The second stage is the edge refinement stage, where the model identifies edge regions with low confidence in the preliminary prediction map (usually corresponding to unknown regions). High-resolution feature maps and edge attention mechanisms are used to locally optimize these regions, accurately capturing the direction of hair strands and the texture of semi-transparent clothing, ultimately outputting a high-precision foreground mask.

[0060] Alternatively, other matting networks can be used for image matting models, such as MODNet (Mobile Object Detection Network), which can accurately predict the transparency value of each pixel in unknown regions given the original image and a ternary image as guidance. The beneficial effect of transparency prediction is that, compared with simple binarization segmentation, this step can accurately restore the semi-transparent effect of complex edges such as hair strands, enabling subsequent image fusion to achieve a natural transition, thereby effectively suppressing the "white edges" or "jagged edges" commonly found in traditional matting.

[0061] Based on the foreground mask obtained above, the foreground image and background image are separated from the original image. The separation operation uses a pixel-by-pixel weighted calculation, and the calculation process is as follows: First, the foreground mask is normalized. Each pixel value of the foreground mask is divided by 255 to obtain the normalized transparency value α, which ranges from 0.0 to 1.0. Here, α=0.0 represents completely transparent (background), α=1.0 represents completely opaque (foreground), and 0.0<α<1.0 represents semi-transparent. Then, pixel-level weighted calculation is performed based on the α value: The calculation formula for the foreground image is: Foreground image pixel value = Original image pixel value × α. For pixels with α=1.0, the foreground pixel retains its original pixel value completely; for pixels with α=0.0, the foreground pixel value is 0 (i.e., set to empty); for pixels with an intermediate α value (such as hair edges), the foreground pixel retains its original pixel value proportionally. The calculation formula for the background image is: Background image pixel value = Original image pixel value × (1 - α). For a pixel with α=0.0, the background pixel retains its original pixel value completely; for a pixel with α=1.0, the background pixel value is 0; for a pixel with an intermediate value for α, the background pixel retains its original pixel value proportionally.

[0062] In the foreground image obtained through the above calculations, the main subject of the portrait and the semi-transparent edges are completely preserved, while the background area is left blank; in the background image, the backdrop area is completely preserved, while the portrait area is left blank. Thus, the original image is successfully decomposed into two independent parts: a foreground image and a background image, providing clear input data for subsequent defect area identification, background restoration, and image fusion. This embodiment uses a coarse portrait segmentation network to quickly locate the approximate area of ​​the portrait, generates a ternary graph structure of predicted foreground area, predicted background area, and unknown area through morphological processing, performs fine transparency prediction in the unknown area using an image matting model, and finally achieves accurate separation of foreground and background based on a foreground mask. The entire process balances processing efficiency and matting accuracy: the coarse segmentation network ensures processing speed, the ternary graph guides the concentration of computational resources in the edge areas, and the edge refinement mechanism of the matting model ensures accurate restoration of complex details such as hair strands. The resulting foreground mask provides accurate definition of the portrait area for subsequent defect area identification and provides high-quality transparency information for the final image fusion, thereby effectively ensuring the naturalness and professionalism of the final synthesized image.

[0063] In some embodiments, in order to automatically and accurately locate the flawed areas of all non-backdrop elements in the background image and provide an accurate range to be repaired for subsequent image generation, the flawed areas include the flawed areas; the step of identifying the flawed areas in the background image to obtain the areas to be repaired in the background image includes:

[0064] The target background color of the background image is determined based on the pixel value distribution of the background image.

[0065] The background image is binarized according to the target background color to obtain the initial background mask corresponding to the original image;

[0066] The initial background mask is used as a prompt and is input together with the background image into a preset image segmentation model to obtain the target background mask corresponding to the original image;

[0067] The background image is filtered based on the global mask corresponding to the original image, the foreground mask corresponding to the foreground image, and the target background mask to obtain the flaw area.

[0068] The target background color refers to the dominant color tone of the backdrop area in the background image. In studio backdrop shooting scenarios, the backdrop typically presents a large area of ​​relatively pure color, thus its color dominates the entire background image. The purpose of determining the target background color is to provide a color reference for subsequent backdrop area segmentation. Specifically, pixel value statistical analysis is performed on the background image to construct a color histogram of the background image, and the frequency of occurrence of each pixel value interval is counted. The color histogram can be constructed in the RGB color space or the HSV color space. Considering the influence of lighting changes on hue, it is preferable to perform the statistics in the HSV color space because the HSV space separates hue and value, and can more robustly handle gradual changes in lighting on the backdrop. The color histogram is analyzed, and the interval of pixel values ​​with the highest frequency is determined as the target background color. This target background color can be a specific RGB value or a color range. For example, for a gray backdrop, its RGB value can be determined to be between (120,120,120) and (140,140,140). The main color tone of the screen is automatically identified through pixel statistical analysis, providing a color benchmark for subsequent screen area segmentation. This avoids the tedious operation of manually setting color thresholds and can adapt to screens of different colors and materials.

[0069] The initial background mask is a binary image obtained by binarizing the background image based on the target background color. It is used to roughly identify candidate areas for the backdrop. This mask has the same resolution as the background image. Areas with a pixel value of 1 indicate that the pixel color falls within the target background color range (i.e., candidate areas for the backdrop), while areas with a pixel value of 0 indicate that the pixel color falls outside the target background color range (i.e., non-backdrop areas). Specifically, the background image is binarized using the target background color as a threshold. Each pixel in the background image is traversed, and it is determined whether the pixel's color value falls within the target background color range. If the pixel color falls within this range, it is marked as 1 in the initial background mask; otherwise, it is marked as 0. It should be noted that due to uneven lighting, shadows, or local color shifts on the backdrop, binarization based solely on a color threshold often fails to yield ideal backdrop segmentation results. That is, some backdrop areas may be incorrectly marked as 0 due to color deviations from the threshold, and some non-backdrop elements with similar colors (such as lamp stands or walls with similar colors to the backdrop) may be incorrectly marked as 1. Therefore, the initial background mask is a rough mask that needs further refinement. By binarizing the target background color, an initial background mask is quickly generated, providing guiding information for subsequent precise segmentation, while initially excluding most obvious non-background areas.

[0070] The preset image segmentation model refers to a pre-trained deep learning model capable of accurately segmenting specific targets in an image based on prompts. The target background mask refers to a binary mask output by the image segmentation model that precisely identifies the curtain region. This mask has the same resolution as the background image and accurately outlines the complete contour of the curtain, including its edges and the boundaries of parts occluded by the foreground. Specifically, the initial background mask is used as prompts and input along with the background image into the preset image segmentation model, which can employ the Segment Anything Model (SAM). SAM is a large-scale pre-trained segmentation model based on the Transformer architecture, possessing powerful zero-shot segmentation capabilities. It can segment any target in an image based on user-provided prompts (such as points, boxes, or coarse masks) without training for a specific category. In this embodiment, the initial background mask is converted into a prompt format acceptable to SAM.

[0071] Specifically, all pixels in the region with a pixel value of 1 in the initial background mask (i.e., the candidate curtain region) are input into the SAM as a dense set of positive cue points. These dense cue points provide the SAM with strong guiding information about "what is a curtain". The SAM uses its encoder to extract features from the background image, and then, guided by these cue points, performs target segmentation in the feature space, ultimately outputting an accurate target background mask. This target background mask accurately distinguishes between the curtain region and the non-curtain region, effectively handling irregular shapes at the curtain edges and color changes on the curtain caused by uneven lighting. Thus, leveraging the powerful segmentation capabilities of the SAM, the coarse mask based on color thresholding is refined into an accurate curtain mask, effectively solving the problem that relying solely on color thresholding cannot handle uneven lighting and complex boundaries.

[0072] A global mask is a binary mask with the same resolution as the background image, where all pixels have a value of 1, representing the entire area of ​​the image. A foreground mask is used to identify the position of the foreground subject within the image. Screening refers to the process of removing the foreground subject and the background backdrop from the global area using set operations, thereby extracting the areas where elements appear to be out of place. Out-of-place areas are regions in the background image that are neither part of the foreground subject nor the background backdrop; that is, non-backdrop elements appearing within the camera's field of view, including but not limited to the edges and support poles of the backdrop, light stands and sandbags on the ground, exposed power cords, and corners or floors of the studio.

[0073] Specifically, an elimination strategy can be used for screening. For example, the flaw area mask can be calculated using the following formula: Flaw Area Mask = Global Mask - Foreground Mask - Target Background Mask. The subtraction operation refers to a pixel-by-pixel logical subtraction: for each pixel position, if the global mask's pixel value is 1, and the pixel values ​​in both the foreground and target background masks at that position are 0, then the flaw area mask's pixel value at that position is 1; otherwise, it is 0. Since the global mask is 1 at all positions, the positions in the flaw area mask with a pixel value of 1 are areas that belong neither to the foreground portrait nor to the background, which are the flaw areas that need to be repaired.

[0074] It should be noted that before the subtraction operation, the foreground mask and the target background mask need to be binarized to convert them into binary masks (pixel values ​​are only 0 or 1). For the foreground mask, a threshold (such as 128) can be set to treat areas with pixel values ​​greater than the threshold as foreground areas (set to 1), and otherwise as non-foreground areas (set to 0). For the target background mask, since it is already a binary mask, it can be used directly.

[0075] Through the above filtering process, a mask for the flawed areas is obtained. This mask accurately identifies the positions of all flawed elements in the background image that need to be repaired, providing accurate input for subsequent merging of the areas to be repaired and image generation.

[0076] This embodiment automatically determines the target background color of the backdrop through pixel statistical analysis, quickly generates an initial background mask through binarization processing, refines the initial mask into a precise target background mask using the powerful segmentation capabilities of SAM, and finally obtains the flawed areas through elimination processing. The entire process achieves automatic and accurate identification of flawed areas without manual intervention and can adapt to backdrops of different colors and materials as well as complex shooting environments. The flawed area mask generated in this embodiment is merged with the subsequent defect area mask to form the area to be repaired, providing a precise construction range for generative image restoration, thereby ensuring the targeted nature of the restoration process and the naturalness of the restoration result.

[0077] In some embodiments, the defective region includes a flawed region; the step of identifying the defective region in the background image to obtain the region to be repaired in the background image includes:

[0078] The background region to be processed is determined from the background image based on the target background mask, and the background region to be processed is filtered to obtain the smoothed background image corresponding to the original image.

[0079] The background image and the smoothed background image are compared, and the initial defect mask corresponding to the background image is determined based on the comparison result.

[0080] The initial defect mask is subjected to morphological processing to obtain the defect area.

[0081] The background region to be processed refers to the set of pixels belonging to the curtain area in the background image, which is extracted from the background image using a target background mask. Filtering refers to applying a low-pass filter to the image to remove high-frequency details and retain low-frequency gradient information. The smoothed background image is the image obtained after applying a low-pass filter to the curtain area. In this image, local high-frequency imperfections (such as dirt, wrinkled edges, tape marks, etc.) are smoothed out, retaining only the low-frequency background information of the curtain caused by uneven lighting.

[0082] Specifically, firstly, based on the target background mask, the background region to be processed is extracted from the background image. This can be achieved by multiplying the background image and the target background mask pixel-by-pixel to obtain the curtain region image. In this image, the regions in the target background mask with a pixel value of 1 (the curtain region) retain the pixel values ​​of the original background image, while regions with a pixel value of 0 (non-curtain regions) are set to 0. Therefore, all subsequent calculations are performed only on the curtain region, eliminating interference from visible areas in defect detection. Then, the extracted curtain region image is filtered. This embodiment uses a median filter as the low-pass filter. Median filtering is a non-linear filtering method that replaces the original value of a pixel with the median value of all pixels in its neighborhood. Compared to mean filtering, median filtering better preserves edge information while effectively removing isolated noise points. Specifically, the size of the filter kernel can be set according to the resolution of the curtain image and the desired scale of the defect to be smoothed. For example, for an image with a resolution of 1920×1080, the filter kernel size can be set to 15×15 pixels or 21×21 pixels. A larger filter kernel results in a stronger smoothing effect, but also leads to the loss of more detailed information. Median filtering effectively smooths out localized imperfections such as dirt, shoe prints, tape residue, and wrinkled edges, outputting a smoothed background image. In this smoothed background image, the curtain area exhibits an ideal smooth gradient, reflecting the curtain's "proper" appearance. A target background mask precisely defines the area to be processed, ensuring that imperfection detection is performed only within the curtain area, avoiding false detections. Median filtering establishes an "ideal state" model of the curtain, providing a benchmark reference for subsequent imperfection identification.

[0083] Comparison refers to the process of calculating the pixel-by-pixel difference between the original backdrop image and the smoothed background image. The initial defect mask is a binary image generated based on the comparison results, used to roughly identify areas on the backdrop that may have defects. In this mask, areas with a pixel value of 1 indicate that there is an anomaly at that location (i.e., a significant difference from the ideal background), and areas with a pixel value of 0 indicate that the location is normal.

[0084] Specifically, the original screen area image (i.e., the background area to be processed extracted in the previous steps) is compared with the smoothed background image pixel by pixel. Specifically, for each pixel location, the difference in pixel value between the two is calculated. The difference can be calculated using various methods, such as calculating the Euclidean distance between the three RGB channels, or calculating the grayscale difference after converting the image to grayscale space. Since the screen is usually a single color or a simple gradient, grayscale difference is sufficient. It is understandable that a larger difference value indicates a greater deviation between the original screen and the ideal smooth state at that location, and is more likely to be a defective area. In clean, smooth areas of the screen, the original pixel value and the smoothed pixel value are very close, with the difference value approaching zero; while in areas with dirt, shoe prints, tape residue, or wrinkled edges, there is a significant difference between the original pixel value and the smoothed pixel value, with a larger difference value.

[0085] Therefore, in this embodiment, thresholding is applied to the difference image to separate abnormal points from the normal background. Thresholding can use a fixed threshold or an adaptive thresholding method, such as Otsu's Binarization, which automatically selects the optimal threshold based on the distribution of image pixel values ​​to maximize inter-class variance. Through thresholding, pixels with a difference value greater than the threshold are marked as 1 (candidate defect region), and pixels with a difference value less than or equal to the threshold are marked as 0 (normal region), generating an initial defect mask. By comparing the difference between the original backdrop image and the smoothed background image, and utilizing the prior knowledge that the backdrop should have a smooth gradient, local abnormal regions are effectively extracted, providing a foundation for accurate defect localization. Morphological processing refers to the post-processing of binary images based on mathematical morphological operations, used to optimize the quality of the defect mask. The defect region refers to the final mask obtained after morphological refinement, accurately identifying physical defects in the backdrop; each connected component in this mask corresponds to a defect that needs to be repaired. Specifically, a series of morphological operations are performed on the initial defect mask to remove noise, fill holes, and connect neighboring regions. First, an opening operation is performed, i.e., erosion followed by dilation. The opening operation removes isolated small noise points (false positives) in the image, which may be false positives caused by image noise or slight color fluctuations. The erosion operation shrinks the foreground area, eliminating isolated small areas; the dilation operation restores the normal size of the eroded area. Through the opening operation, scattered noise points in the initial defect mask are effectively removed. Then, a closing operation is performed, i.e., dilation followed by erosion. The closing operation fills small holes (false negatives) inside the defect area, connecting broken parts of the same defect into complete connected regions. The dilation operation expands the foreground area, filling internal holes; the erosion operation restores the boundaries of over-dilated regions. Through the closing operation, internal holes that might have been caused by thresholding are filled, and the defect area forms a complete and continuous connected region. After morphological processing, the final defect mask is obtained. This mask precisely identifies all physical defects on the screen that need repair, including dirt, shoe prints, tape residue, wrinkled edges, scratches, etc. This mask is then merged with the previously generated mask for the areas where defects are visible, together forming the mask for the areas to be repaired, providing an accurate repair area for subsequent image generation.

[0086] This embodiment precisely defines the background area to be processed using a target background mask, establishes an ideal smooth state model of the backdrop through median filtering, extracts high-frequency anomaly information through difference comparison and thresholding, and optimizes the mask quality through morphological refinement, thus achieving automatic and accurate identification of defective areas. The entire process utilizes the inherent characteristics of the backdrop background (large area, smooth gradient) and employs traditional image processing algorithms to achieve efficient and robust defect detection without the need for complex deep learning model training. It can also adapt to backdrops of different materials and colors, thereby ensuring the targeted nature of the repair process and the naturalness of the repair results.

[0087] In some embodiments, image generation is performed based on image features of other regions in the background image besides the region to be repaired, to obtain a repaired image corresponding to the region to be repaired, including:

[0088] The defect areas corresponding to the defect areas of different defect types are merged to obtain a mask of the area to be repaired; the defect types include flaws and defects that expose defects.

[0089] Obtain the background repair prompt text corresponding to the original image; wherein, the background repair prompt text is used to describe the visual features of the target background;

[0090] The background image, the mask of the area to be repaired, and the background repair prompt text are input into a preset diffusion generation model to compress the background image into the latent space through the diffusion generation model. In the latent space, the area to be repaired is iteratively denoised according to the mask of the area to be repaired and the background repair prompt text to generate the repaired latent space features.

[0091] The repaired latent space features are decoded into pixel space to obtain the repaired image corresponding to the region to be repaired.

[0092] The defect types can include continuity errors and minor flaws. Continuity errors refer to non-curtain elements appearing within the camera's field of view, such as light stands, wires, and corners. Minor flaws refer to physical defects inherent in the curtain itself, such as dirt, shoe prints, tape residue, and wrinkles. The merging process involves performing a pixel-by-pixel logical OR operation on the continuity error mask and the minor flaw mask to generate a unified and complete mask for the area to be repaired. The mask for the area to be repaired is a binary image with the same resolution as the background image, where areas with a pixel value of 1 represent locations that need repair, and areas with a pixel value of 0 represent locations that remain unchanged. Specifically, the continuity error mask and the minor flaw mask generated in the aforementioned embodiment are obtained. Both masks are binary images, where a pixel value of 1 indicates the location of the continuity error element and the minor flaw, respectively. A pixel-by-pixel logical OR operation is performed on these two masks: for each pixel location, if at least one of the continuity error mask or the minor flaw mask has a pixel value of 1, then the pixel value of the mask for the area to be repaired at that location is set to 1; otherwise, it is set to 0. By merging the images, a unified mask of the areas to be repaired is obtained. This mask accurately identifies all areas in the background image that need to be repaired, including flaws and physical defects. Merging unifies defective areas from different sources and of different types into a single mask, providing a complete guide to the repair scope for subsequent image generation and avoiding potential overlap or omissions that might occur with separate processing.

[0093] Background restoration prompt text refers to the textual description used to guide the diffusion generation model in content generation. This textual description should accurately reflect the visual characteristics of the target background, including the material, color, lighting effects, and overall style of the backdrop. The role of the background restoration prompt text is to provide semantic-level generation guidance to the diffusion model, ensuring that the generated content meets the professional requirements of a studio backdrop. Specifically, the background restoration prompt text can be preset or dynamically generated based on the analysis results of the original image. It can be understood that for the specific scenario of studio backdrop restoration, a concise and effective prompt text is crucial. An exemplary prompt text could be: "A photo of a smooth, clean, seamless studio backdrop, professional studio lighting, high quality." This prompt text contains key information such as the physical characteristics of the backdrop (smooth, clean, seamless), the application scenario (studio backdrop), lighting conditions (professional studio lighting), and quality requirements (high quality), effectively guiding the diffusion model to generate background content that meets expectations. In practical applications, the prompt text can be adaptively adjusted based on the actual color and material of the backdrop. For example, for a gray fabric backdrop, the prompt text can be adjusted to: "A smooth, clean, seamless photo of a gray fabric studio backdrop, professional studio lighting, high quality." By introducing background restoration prompt text, semantic-level generation guidance is provided to the diffusion model, enabling the model to understand "what kind of background should be generated," thereby generating restoration content that better meets expectations.

[0094] A pre-trained diffusion generative model refers to a diffusion model specifically designed for image restoration tasks. Diffusion models are a type of generative artificial intelligence model whose core principle is to progressively add noise to "destroy" an image into pure noise, and then learn the reverse process—progressively "de-noiseing" the noise to restore the original image. In restoration tasks, diffusion models can "fill in" reasonable new content within a masked area based on the contextual information surrounding the area to be restored.

[0095] Specifically, the Inpainting Pipeline of Stable Diffusion 1.5 can be used as the diffusion generation model, such as the StableDiffusionInpaintPipeline in the Hugging Face diffusers library. This model receives three key inputs: image (the input image, i.e., the aforementioned background image), mask_image (the mask image, i.e., the aforementioned mask of the area to be repaired), and prompt (the text prompt, i.e., the aforementioned background repair prompt text). After the inputs are prepared, these three inputs are fed together into the diffusion generation model to start the generation process.

[0096] The latent space refers to the low-dimensional feature space obtained by compressing the high-dimensional pixel space using a variational autoencoder (VAE). In the latent space, the semantic information and structural features of the image are encoded into compact vector representations, significantly reducing the computational cost of the diffusion process. Iterative denoising refers to the process of gradually removing noise and restoring image content, starting from random noise. Each denoising step is based on predictions made using the current latent space features, mask information, and text guidance. The restored latent space features refer to the latent space feature maps obtained after the complete denoising process, where the corresponding restored regions have been filled in with content.

[0097] Specifically, the execution flow of the diffusion generation model is as follows: First, the background image is compressed into the latent space using the VAE encoder to obtain a latent space feature map. Simultaneously, the mask for the region to be repaired is downsampled to the same resolution as the latent space feature map to obtain a latent space mask. Then, in the latent space, the model performs an iterative denoising process using a U-Net structure. The U-Net takes the current latent space feature map, the latent space mask, and the text embedding vector of the background repair prompt text as conditional inputs. During denoising, the U-Net generates features within the area covered by the latent space mask, while the areas outside the mask retain their original latent space features. After a preset number of denoising steps (e.g., 30 to 50 steps; more steps result in higher quality but also longer processing time), the U-Net outputs the repaired latent space feature map. In this feature map, the latent space features corresponding to the region to be repaired have been replaced with new features generated based on the surrounding context and text guidance, while the originally clean areas remain unchanged. By performing iterative denoising in the latent space and leveraging the powerful generative capabilities of the diffusion model, new content with consistent texture, lighting, and color with the surrounding background is generated within the mask area, while preserving the original information of the area outside the mask.

[0098] Decoding refers to the process of converting the latent space feature map back into pixel space using a VAE decoder. The restored image is the complete background image obtained after decoding, where the area to be repaired has been intelligently filled in. The decoder can use deconvolution and upsampling operations to gradually restore the low-dimensional latent space features into a pixel image with the same resolution as the original background image. In the final output restored image, areas that originally contained dirt, tape, wrinkles, or unwanted objects have been naturally and coherently filled with new content. The repaired content is highly consistent with the surrounding backdrop in terms of texture, lighting, and color, with no obvious stitching marks.

[0099] This embodiment obtains a unified mask for the area to be repaired through merging processing, provides semantic guidance through background repair prompt text, iteratively denoises and generates the image in the latent space using a diffusion generation model, and finally obtains the repaired image through decoding. The entire process achieves unified and intelligent repair of both flawed and obtrusive areas. Compared with traditional image repair methods (such as clone stamp), the diffusion model used in this embodiment can "understand" the contextual information of the image, generating content that seamlessly connects with the surrounding background in terms of lighting gradation, texture direction, and color transition, fundamentally avoiding the "patchwork" and "distortion" feel. At the same time, the diffusion model can handle large-area complex defects (such as large wrinkles or multiple obtrusive objects), and the generated results are visually more natural and realistic.

[0100] In some embodiments, the step of image fusion between the restored image and the foreground image to obtain the background-restored target image corresponding to the original image includes:

[0101] Obtain the background image and the mask of the area to be repaired;

[0102] Multi-band fusion processing is performed on the background image and the repaired image to obtain a corrected background image with color correction.

[0103] Obtain the noise features of the non-repaired area in the background image, and add noise to the area to be repaired in the corrected background image according to the noise features to obtain a target background image that is consistent with the noise features of the background image.

[0104] The target image is obtained by weighted summing of the pixel values ​​of the foreground image and the pixel values ​​of the target background image based on the foreground mask corresponding to the foreground image.

[0105] Multi-band fusion processing refers to a technique that decomposes an image into multiple layers of different frequencies, fuses them separately, and then reconstructs the image to eliminate visual differences between the repaired area and the original background. The corrected background image refers to the background image obtained after multi-band fusion processing, where the repaired area and the original background are unified and coordinated in tone and lighting.

[0106] Specifically, multi-band fusion processing can employ the Laplacian Pyramid Blending technique. First, a Laplacian pyramid is constructed for both the background image and the image to be repaired, essentially decomposing each image into multiple layers of different frequencies. The bottom layer (lowest frequency layer) of the Laplacian pyramid contains global color and lighting information, while the higher layers (higher frequency layers) contain texture, edge, and other detailed information. The number of pyramid layers can be set according to the image resolution and fusion accuracy requirements; for example, for a 1920×1080 image, six pyramid layers can be used. Simultaneously, a Gaussian pyramid is constructed for the mask of the area to be repaired, involving multiple Gaussian blurring and downsampling processes to obtain a series of edge-smoothing weight maps. The number of layers in the Gaussian pyramid is the same as that in the Laplacian pyramid, with each layer corresponding to the fusion weights of a specific frequency layer.

[0107] Then, for each layer of the Laplacian pyramid, the corresponding Gaussian pyramid weight map is used as a guide to perform a weighted average fusion of the features of the background image and the restored image at that layer. For low-frequency layers (reflecting color and lighting gradations), the weight map transitions smoothly over a large range, ensuring a uniform tonal transition in the fused area; for high-frequency layers (reflecting texture and edge details), the weight map remains sharp to preserve clear texture boundaries and avoid blurring. The fusion formula is: Fuded features = Background features × (1 - Weight) + Restored features × Weight, where the weight is taken from the weight value of the corresponding layer of the Gaussian pyramid. Through this multi-band fusion method, low-frequency information (color, lighting) transitions smoothly over a large range, while high-frequency information (texture) remains clear locally, fundamentally eliminating the "patchwork" between the restored area and the original background. Finally, all the fused frequency layers are reconstructed into a complete image, i.e., the corrected background image. In this image, the restored area and the original background achieve a smooth transition in color and lighting gradations, with no obvious stitching marks. By using multi-band fusion processing, the problem of slight color difference or discontinuity in lighting gradient that may exist between the repair area generated by the diffusion model and the original background is solved, so that the repair area can be visually completely integrated with the original background.

[0108] The non-repaired area refers to the region in the background image where the mask marking of the area to be repaired is 0, i.e., the originally clean background area that does not require repair. Noise characteristics refer to the statistical properties of high-frequency noise generated by the camera sensor (ISO) in the image, including parameters such as noise type (e.g., Gaussian noise or Poisson noise) and standard deviation. Adding noise refers to artificially synthesizing and superimposing noise with the same statistical characteristics as the original background into the repaired area. The target background image refers to the background image obtained after noise matching processing, where the repaired area and the original background are completely identical in microscopic texture.

[0109] Specifically, sampling can be performed from the non-repaired area of ​​the background image to estimate the noise distribution characteristics of the original image. Specifically, a flat, uniform region in the background image (such as the center of a backdrop) is selected, and statistical information of the pixel values ​​in this region is extracted. By calculating the pixel values ​​in this region, the standard deviation of the noise can be estimated; by analyzing the power spectral density of the noise, the type of noise (such as Gaussian noise or Poisson noise) can be determined. Then, based on the estimated noise characteristics, a synthetic noise map with the same statistical characteristics is generated. This noise map has the same size as the repaired area, and its pixel values ​​conform to the estimated noise distribution. Finally, this synthetic noise map is added only to the repaired area of ​​the corrected background image (i.e., the area covered by the mask of the repaired area) to obtain the target background image. For the non-repaired areas, the original pixel values ​​remain unchanged. Through this operation, the repaired area acquires a microscopic noise texture consistent with the original background, making the repaired area no longer appear "too clean" or "blurry" visually. Noise matching processing solves the problem that the areas generated by diffusion models are often too clean and lack camera noise texture, ensuring that the repaired area maintains consistency with the original background in microscopic texture, further enhancing the realism of the repair result.

[0110] The foreground mask, also known as the high-precision foreground mask (Alpha Matte), represents the weight of each pixel in relation to its foreground status. Weighted summation refers to blending the foreground image and the target background image pixel-by-pixel according to the Alpha compositing formula, using the normalized value of the foreground mask as the weight. The target image refers to the final output image, seamlessly composited with the restored background. Specifically, the foreground mask is first normalized. Each pixel value of the foreground mask is divided by 255 to obtain the normalized transparency value α, which ranges from 0.0 to 1.0. Here, α = 0.0 indicates that the pixel is completely transparent (belonging to the background), α = 1.0 indicates that the pixel is completely opaque (belonging to the foreground), and 0.0 < α < 1.0 indicates a semi-transparent state (such as hair strands or the edge of a wedding dress). Then, for each pixel position, the pixel value of the target image is calculated using the following formula: Output_Pixel = (Foreground_Pixel × α) + (TargetBackground_Pixel × (1 - α));

[0111] Wherein, Foreground_Pixel represents the pixel value of the foreground image at the current pixel point, and TargetBackground_Pixel represents the pixel value of the target background image at the current pixel point.

[0112] When processing solid areas of a person's image (such as cheeks or body), α ≈ 1.0, and the formula becomes Output_Pixel ≈ Foreground_Pixel, meaning the output pixels come almost entirely from the foreground image. When processing pure background areas, α ≈ 0.0, and the formula becomes Output_Pixel ≈ TargetBackground_Pixel, meaning the output pixels come entirely from the restored background image. When processing semi-transparent edges such as hair, α takes a middle value (e.g., 0.3), and the formula becomes Output_Pixel = Foreground_Pixel × 0.3 + TargetBackground_Pixel × 0.7, meaning the output pixels are a mixture of foreground and background colors, simulating the optical appearance of a semi-transparent object against a new background. This pixel-by-pixel weighted summation achieves a natural blending of the foreground and background, effectively avoiding the "white edges" or "jagged edges" common in traditional image cutout methods.

[0113] Compared with traditional direct background replacement or simple blending methods, this embodiment makes the repaired area "invisible" in the overall background through fine tone correction and noise matching; by using weighted summation based on alpha mask, complex edges such as hair can achieve natural transition, fundamentally ensuring the professionalism and realism of the final image.

[0114] In some embodiments, the step of performing multi-band fusion processing on the background image and the restored image to obtain a tone-corrected background image includes:

[0115] The background image and the repaired image are respectively decomposed into multiple image layers of different frequencies;

[0116] Using a Gaussian blurred mask of the region to be repaired as a guide, weighted average fusion is performed on each frequency layer;

[0117] The fused frequency layers are reconstructed into the corrected background image.

[0118] In this context, image layers of different frequencies refer to a series of image layers containing information at different scales, decomposed from the original image using image pyramid technology. Low-frequency layers (the top layer of the pyramid) contain macroscopic information such as global color and lighting variations; high-frequency layers (the bottom layer of the pyramid) contain microscopic details such as texture and edges. The purpose of decomposing the image into multiple frequency layers is to achieve a "divide and conquer" fusion strategy: different fusion weights are applied to information of different frequencies, resulting in a smooth transition in the fusion result on a macroscopic scale while preserving clear details on a microscopic scale.

[0119] Specifically, the Laplacian Pyramid is used for image decomposition. The construction process of the Laplacian Pyramid is as follows: First, a Gaussian Pyramid is constructed. The original image is used as the 0th layer of the Gaussian Pyramid. .right Perform Gaussian blur (using a 5×5 or 7×7 Gaussian kernel) and downsampling (interlaced sampling) to obtain the first layer. ;right Repeat the above steps to obtain the second layer. This process continues until the preset number of pyramid levels N is reached. Each level of the Gaussian pyramid is the result of low-pass filtering and downsampling of the previous level's image. Therefore, as the number of levels increases, the image resolution decreases layer by layer, and high-frequency information is lost layer by layer. Then, the Laplacian pyramid is constructed. Each level of the Laplacian pyramid... Defined as the difference between the upsampled and interpolated values ​​of the k-th and (k+1)-th layers of the Gaussian pyramid: The upsample() operation upsamples a low-resolution image to a higher resolution. Same resolution. The Pyramid of Laplace. The layer contains the highest frequency detail information of the image. The layer contains the next highest frequency information, ... The layers contain lower-frequency information, while the top layer of the Gaussian pyramid... This contains the lowest frequency information of the image (global color and illumination). Through this decomposition method, the original image can be accurately reconstructed from all layers of the Laplacian pyramid plus the top layer of the Gaussian pyramid.

[0120] In this embodiment, the above decomposition operation is performed on both the background image and the restored image to obtain their respective Laplacian pyramids. The number of pyramid layers can be set according to the image resolution and fusion accuracy requirements. For example, for a 1920×1080 image, the number of pyramid layers can be set to 6. The more layers, the finer the fusion control, but the computational load also increases accordingly. Through Laplacian pyramid decomposition, the background image and the restored image are separated into multiple independent layers from high frequency to low frequency, providing a foundation for subsequent frequency division fusion.

[0121] Gaussian blurring refers to applying a Gaussian filter to the mask of the area to be repaired, smoothing its edges. The Gaussian-blurred mask serves as a guide, controlling the fusion weights of the background and repaired images at each frequency level. Weighted average fusion refers to summing the corresponding features of the background and repaired images at the same frequency level according to the guide weights. Specifically, a Gaussian pyramid is constructed on the mask of the area to be repaired, generating a series of mask layers that have undergone Gaussian blurring and downsampling. For example, the mask of the area to be repaired can be used as the 0th layer of the Gaussian pyramid. .right Gaussian blur and downsampling are performed to obtain the first layer. ;right Repeat the above steps to obtain the second layer. And so on, until the same number of levels N as the Pyramid of Laplace is reached. Each level of the Pyramid of Gauss These are all results of blurring and downsampling the original mask to varying degrees. The higher the number of layers, the greater the blurring and the smoother the edge transitions. These mask layers will serve as guiding weights for the fusion of each frequency layer.

[0122] Then, for each layer of the Laplacian pyramid, a weighted average fusion is performed. Let the Laplacian pyramid layer of the background image be... The Laplacian pyramid layers of the restored image are The Gaussian pyramid mask for the corresponding layer is (Values ​​range from 0 to 1), then the merged Laplace pyramid layers Calculate using the following formula: ;

[0123] For high-frequency layers (where k is relatively small, corresponding to...) , wait), Due to Gaussian blurring and downsampling, the edges are relatively smooth, yet the boundaries of the fusion region are still effectively defined. This smooth weight transition allows high-frequency information (texture, edges) to blend naturally at the boundaries of the repaired region, avoiding harsh texture breaks. For low-frequency layers (where k is relatively large, corresponding to...) , wait), After multiple Gaussian blurring and downsampling processes, the image has become extremely smooth, with weights transitioning uniformly across a large area of ​​the fusion region. This allows for a smooth gradient of low-frequency information (color, lighting) across a wide range between the repaired area and the original background, fundamentally eliminating color difference and lighting discontinuities. By using masks with varying degrees of smoothness as guiding weights on different frequency layers, a multi-band fusion effect of "sharp high-frequency fusion and smooth low-frequency transition" is achieved.

[0124] Reconstruction refers to performing an inverse transformation on each layer of the merged Laplacian pyramid and the top layer of the Gaussian pyramid to recover the complete image. The corrected background image refers to the background image obtained after multi-band fusion processing, in which the repaired area and the original background are unified and coordinated in tone and lighting.

[0125] Specifically, the reconstruction process includes: starting from the top layer of the Gaussian pyramid... We begin by restoring the image layer by layer upwards. For the k-th layer (from k=n-1 to 0), we first upsample and interpolate the image of the (k+1)-th layer to make its resolution the same as that of the k-th layer, and then add the fused Laplacian pyramid image of the k-th layer. : Repeat the above steps until level 0 is recovered. This refers to the reconstructed complete image. This image is the corrected background image. In the corrected background image, the repaired area and the original background are seamlessly integrated: in terms of color and lighting, the hue of the repaired area transitions smoothly with the surrounding background without obvious color difference; in terms of texture and detail, the texture of the repaired area blends naturally with the surrounding background without any harsh stitching marks.

[0126] This embodiment decomposes the image into multiple frequency layers using the Laplacian pyramid, employs a Gaussian pyramid-processed mask as guiding weights, and selectively fuses different frequency layers. Finally, the fused layers are reconstructed into the corrected background image. This multi-band fusion method fully utilizes the characteristics of the human visual system: the human eye is more sensitive to changes in low-frequency information (color, illumination), thus requiring a smooth transition across a wide range in the low-frequency layers; while it has a higher tolerance for changes in high-frequency information (texture, detail), maintaining sharp fusion boundaries in the high-frequency layers.

[0127] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0128] Based on the same inventive concept, this application also provides an image restoration apparatus for implementing the image restoration method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image restoration apparatus embodiments provided below can be found in the limitations of the image restoration method described above, and will not be repeated here.

[0129] In one exemplary embodiment, such as Figure 3 As shown, an image restoration apparatus is provided, comprising:

[0130] The acquisition module is used to acquire the original image and identify the background image and foreground image corresponding to the original image;

[0131] The identification module is used to identify defect areas in the background image to obtain the areas to be repaired in the background image.

[0132] The generation module is used to generate an image based on the image features of other regions in the background image besides the region to be repaired, to obtain the repaired image corresponding to the region to be repaired.

[0133] The fusion module is used to fuse the repaired image with the foreground image to obtain the target image with background repair corresponding to the original image.

[0134] Each module in the aforementioned image restoration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0135] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image restoration method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0136] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps included in any of the foregoing method embodiments.

[0138] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image restoration method, characterized in that, The method includes: Acquire the original image and identify the background and foreground images corresponding to the original image; Defect region identification is performed on the background image to obtain the region to be repaired in the background image; Image generation is performed based on the image features of other regions in the background image besides the region to be repaired, to obtain the repaired image corresponding to the region to be repaired. The restored image is then fused with the foreground image to obtain the target image with restored background corresponding to the original image.

2. The method according to claim 1, characterized in that, The step of acquiring the original image and identifying the background and foreground images corresponding to the original image includes: Perform portrait segmentation on the original image to obtain a coarse portrait mask corresponding to the original image; Morphological processing is performed on the portrait mask to obtain the predicted foreground region, predicted background region, and unknown region of the original image; Based on the original image, the predicted foreground region, and the background region, a preset image matting model is used to predict the transparency of the unknown region, thereby obtaining the foreground mask corresponding to the original image. The foreground image and the background image are identified from the original image based on the foreground mask.

3. The method according to claim 1, characterized in that, The defective region includes a visible flaw area; the process of identifying the defective region in the background image to obtain the region to be repaired in the background image includes: The target background color of the background image is determined based on the pixel value distribution of the background image. The background image is binarized according to the target background color to obtain the initial background mask corresponding to the original image; The initial background mask is used as a prompt and is input together with the background image into a preset image segmentation model to obtain the target background mask corresponding to the original image; The background image is filtered based on the global mask corresponding to the original image, the foreground mask corresponding to the foreground image, and the target background mask to obtain the flaw area.

4. The method according to claim 3, characterized in that, The defective region includes a flawed region; the step of identifying the defective region in the background image to obtain the region to be repaired in the background image includes: The background region to be processed is determined from the background image based on the target background mask, and the background region to be processed is filtered to obtain the smoothed background image corresponding to the original image. The background image and the smoothed background image are compared, and the initial defect mask corresponding to the background image is determined based on the comparison result. The initial defect mask is subjected to morphological processing to obtain the defect area.

5. The method according to claim 1, characterized in that, Image generation is performed based on image features of regions other than the region to be repaired in the background image to obtain a repaired image corresponding to the region to be repaired, including: The defect areas corresponding to the defect areas of different defect types are merged to obtain a mask of the area to be repaired; the defect types include flaws and defects that expose defects. Obtain the background repair prompt text corresponding to the original image; wherein, the background repair prompt text is used to describe the visual features of the target background; The background image, the mask of the area to be repaired, and the background repair prompt text are input into a preset diffusion generation model to compress the background image into the latent space through the diffusion generation model. In the latent space, the area to be repaired is iteratively denoised according to the mask of the area to be repaired and the background repair prompt text to generate the repaired latent space features. The repaired latent space features are decoded into pixel space to obtain the repaired image corresponding to the region to be repaired.

6. The method according to claim 5, characterized in that, The step of fusing the repaired image with the foreground image to obtain the target image with background repair corresponding to the original image includes: Obtain the background image and the mask of the area to be repaired; Multi-band fusion processing is performed on the background image and the repaired image to obtain a corrected background image with color correction. Obtain the noise features of the non-repaired area in the background image, and add noise to the area to be repaired in the corrected background image according to the noise features to obtain a target background image that is consistent with the noise features of the background image. The target image is obtained by weighted summing of the pixel values ​​of the foreground image and the pixel values ​​of the target background image based on the foreground mask corresponding to the foreground image.

7. The method according to claim 6, characterized in that, The step of performing multi-band fusion processing on the background image and the repaired image to obtain a tone-corrected background image includes: The background image and the repaired image are respectively decomposed into multiple image layers of different frequencies; Using a Gaussian blurred mask of the region to be repaired as a guide, weighted average fusion is performed on each frequency layer; The fused frequency layers are reconstructed into the corrected background image.

8. An image restoration device, characterized in that, The device includes: The acquisition module is used to acquire the original image and identify the background image and foreground image corresponding to the original image; The identification module is used to identify defect areas in the background image to obtain the areas to be repaired in the background image. The generation module is used to generate an image based on the image features of other regions in the background image besides the region to be repaired, to obtain the repaired image corresponding to the region to be repaired. The fusion module is used to fuse the repaired image with the foreground image to obtain the target image with background repair corresponding to the original image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images