Prop intelligent repairing method for a portrait photography scene

Abstract

The application discloses a kind of props intelligent repair method of portrait photography scene, wherein the method includes: obtaining the pixel-level binary mask of target prop in original image, calculating minimum circumscribed rectangle based on the mask, and cutting original image and prop mask according to this, to obtain cutting image and cutting mask;The cutting image and cutting mask are input into a repair network based on diffusion model, the repair network combines the structure control network for providing original prop information and the keyword embedding trained by text inversion technology, and the generation result is controlled to be towards flawless state by classifier unguided technology, and low-resolution repair image is output, etc.The application aims to provide a kind of high-precision intelligent prop repair method, realize pixel-level accurate positioning of any shape prop, high-precision prop repair, and high-resolution detail recovery referring to original image, to ensure that the repaired prop has high degree of realism and texture fidelity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for intelligent restoration of props in portrait photography scenes. Background Technology

[0002] In many fields such as film and television photography and post-production, prop restoration is a crucial task in images containing props or objects. These props may exhibit various imperfections due to physical wear and tear, reflections from the shooting environment, stains, or age, such as scratches, cracks, spots, and fading. Existing technologies mainly include the following categories: Manual repair methods, represented by professional image editing software such as Adobe Photoshop, rely on tools like the healing brush, clone stamp, and content-aware fill. Their disadvantages are: extremely time-consuming and labor-intensive, high cost, and difficulty in standardization and scaling.

[0003] Traditional algorithmic restoration methods, such as PatchMatch, are based on image statistics or partial differential equations. These methods fill in damaged areas by finding similar intact texture blocks within the image. Their drawbacks include poor performance for non-local, large-area, or structurally complex defects, and difficulty in understanding the semantic information of the object or prop.

[0004] General Inpainting Methods Based on Deep Learning: In recent years, deep learning models, represented by Generative Adversarial Networks (GANs) and Diffusion Models, have achieved great success in the field of image inpainting. However, when these general models are directly applied to high-precision prop restoration, significant shortcomings remain, such as insufficient control precision, insufficient semantic understanding, and loss of texture realism. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose an intelligent prop restoration method for portrait photography scenes. Addressing the problems of low restoration accuracy, limited semantic control, and poor texture realism in existing technologies, this invention aims to provide a high-precision intelligent prop restoration method. It achieves pixel-level precise positioning of props of arbitrary shapes, high-precision prop restoration, and high-resolution detail restoration based on the original image, ensuring that the restored props possess high realism and texture fidelity.

[0006] According to one aspect of the present invention, a method for intelligent prop repair in portrait photography scenes is provided, the method comprising: Obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropped mask. The cropped image and cropping mask are input into a diffusion-based restoration network, which combines a structural control network to provide information about props in the original image with keyword embedding trained by text inversion technology. The network also uses a classifier-guided technique to control the generation result toward a flawless state and outputs a low-resolution restored image. The low-resolution restored image is upsampled to obtain an interpolated enlarged image with the same resolution as the cropped image; the interpolated enlarged image and the cropped image are input together into a Transformer-based super-resolution network to generate a high-resolution restored image. The high-resolution restored image is fused back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image.

[0007] In the above technical solution, the method not only effectively solves the problems of low repair accuracy, single semantic control and poor texture realism in the existing technology, but also achieves precise processing and high detail restoration of the prop area through systematic process design.

[0008] Specifically, the target prop is first located and cropped using pixel-level binarization masks and minimum bounding rectangle calculations. This step ensures the repair operation is confined to the prop's outline, avoiding background contamination and significantly reducing subsequent computational resource consumption. Subsequently, a diffusion-based repair network combined with a structure control network and text inversion techniques enhances the model's understanding and control of the prop's semantics. Unguided classifier techniques guide the generated results towards a flawless state, achieving high-precision semantic repair and texture generation at low resolution. Further, upsampling and a Transformer-based super-resolution network fuse the low-resolution repair image with high-frequency details of the original cropped image, effectively restoring the prop's original material texture and avoiding the smoothing distortion common in traditional super-resolution methods, ensuring the realism and fidelity of the repair result. Finally, seamlessly fusing the high-resolution repair image back into the original image guarantees overall visual consistency and naturalness.

[0009] In summary, this technical solution comprehensively improves the positioning accuracy, semantic control capability, texture fidelity, and processing efficiency of prop repair through a coherent process from positioning and repair to super-resolution.

[0010] In some embodiments, a pixel-level binary mask of the target prop in the original image is obtained, the minimum bounding rectangle is calculated based on the mask, and the original image and the prop mask are cropped accordingly to obtain a cropped image and a cropping mask. Specifically: The input original image is used to identify and crop the prop region: the instance segmentation network is used to obtain the pixel-level binary mask of the target prop, the minimum bounding rectangle is calculated based on the mask, and the original image and the prop mask are cropped accordingly to obtain the cropped image and the cropped mask. The instance segmentation network is used to identify one or more prop instances in an image and obtain a mask for a specific target prop through category filtering.

[0011] The above technical solution effectively solves the common problems of positioning ambiguity and background interference in general restoration methods. Specifically, firstly, by using an instance segmentation network to generate pixel-level binary masks, it can accurately identify and separate specific prop instances in the image, and even accurately lock the target in multi-instance scenes through category filtering. This pixel-level operational boundary ensures that the initial input of the restoration process has extremely high spatial accuracy, providing a fundamental guarantee that the subsequent restoration network will strictly constrain the generated content within the prop outline, thereby fundamentally avoiding the pollution and damage to complex backgrounds during the restoration process. Secondly, by calculating the minimum bounding rectangle based on the mask and performing collaborative cropping of the original image and the mask, the area that would otherwise require computationally intensive diffusion models and super-resolution networks is focused from the entire high-resolution image to the minimum effective area containing the target prop, significantly reducing the model's memory usage and computational complexity, lowering hardware resource requirements, and thus improving the versatility of the entire technical solution.

[0012] In some embodiments, the cropped image and cropping mask are input into a diffusion-based inpainting network. This inpainting network combines a structured control network for providing prop information from the original image with keyword embeddings trained using text inversion techniques. It also uses an unguided classifier technique to guide the generated result towards a flawless state, outputting a low-resolution inpainted image. Specifically: An architecture is adopted with a pre-trained SDXL-Inpainting model as the backbone and a parallel ControlNet structure control network; the ControlNet structure control network is used to process the cropped image to obtain the structural condition information of the prop. Use embedding vector pairs trained with text inversion techniques that contain semantics of perfect and flawed states; The cropped image, its corresponding cropping mask, the structural condition information, and the semantic embedding vector pair are all input into an SDXL-Inpainting model; During the inference process of the SDXL-Inpainting model, the classifier-unguided technique is used to generate a guiding signal that moves towards the perfect state and away from the flawed state based on the semantic embedding vector pair, and under the constraint of the structural condition information, the repaired cropped image is output.

[0013] The above technical solution constructs a specialized and highly controllable generation system for prop repair tasks. Specifically, the innovation of this solution lies first in its parallel architecture of the SDXL-Inpainting model and the ControlNet structural control network. ControlNet processes the original cropped image, extracting and injecting essential structural information (such as contour and shape) into the prop, serving as a strong constraint on the diffusion model's generation process. This effectively ensures that the repaired prop maintains geometric consistency with the original target, avoiding common structural distortions or semantic inconsistencies in general repair models. Secondly, it introduces semantic embedding vector pairs trained using text inversion technology. This technique compresses abstract semantics such as "perfect state" and "flawed state" into efficient embedding representations that the model can understand, providing the model with task semantic understanding beyond the pixel level. Finally, during inference, the unguided classifier technique utilizes the aforementioned embedding vector pairs to dynamically calculate and apply a guiding signal that points towards the perfect state and away from the flawed state. This signal, along with the structural conditions, guides the generation process to output the expected content. This strategy enables adjustable two-way semantic control over the generated results, greatly enhancing the directionality and clarity of the repair task and ensuring that flaws are effectively removed while ideal textures are accurately generated.

[0014] In summary, this repair network scheme deeply integrates three core technologies: structural control network, text inversion, and unguided classifier. It combines prior generation capability (SDXL), structural constraints, and fine-grained semantic control to achieve high-precision and highly semantically consistent repair of props in complex portrait photography scenarios.

[0015] In some embodiments, the upsampling method includes bicubic interpolation.

[0016] In the above technical solution, bicubic interpolation is used as the upsampling method for the low-resolution inpainted image in the super-resolution process, aiming to provide a structurally stable and highly compatible input foundation for the subsequent reference super-resolution network. Specifically, bicubic interpolation, as a classic linear interpolation algorithm, has the core advantage of high computational efficiency and reliability. After receiving the low-resolution inpainted image generated by the preceding diffusion model, its primary task is to quickly upscale the image to match the original cropped image, so as to perform subsequent pixel-level fusion and detail enhancement. Compared with complex learning-based upsampling methods, bicubic interpolation has a deterministic mathematical form and extremely low computational overhead, enabling smooth size expansion without introducing any additional model parameters or uncontrollable generation noise. This determinism ensures that it can maintain the overall composition and basic structural information generated by the preceding inpainting network, while avoiding texture distortion or artifacts that may result from upsampling using another unconstrained generative model. Although its output interpolated magnified image has inherent limitations in smoothing high-frequency details, it provides a "clean" underlying structural framework for subsequent Transformer-based super-resolution networks that only lacks high-frequency textures. This allows the super-resolution network to focus its full capacity on transferring and fusing high-frequency details from the original cropped image, rather than correcting underlying structural errors.

[0017] In some embodiments, the super-resolution network is a network based on the Swing Transformer architecture.

[0018] In the above technical solution, a super-resolution network based on the Swing Transformer architecture is adopted, which aims to overcome the inherent limitations of existing super-resolution methods in long-distance dependency modeling and high-frequency texture reproduction.

[0019] Specifically, the advantage of the Swin Transformer architecture lies in its hierarchical design and shift-window self-attention mechanism. This design enables it to efficiently model long-range dependencies between features at different scales in an image, thus globally understanding the texture structure of the prop surface (such as the continuity of wood rings and the directionality of metal wire drawing), rather than being limited to a local neighborhood as in traditional convolutional networks. This global perspective is crucial for generating structurally coherent and semantically reasonable high-frequency textures. Simultaneously, the shift-window mechanism restricts the computation of self-attention to non-overlapping local windows and achieves cross-window information interaction at deeper levels through window shifting. This maintains powerful modeling capabilities while keeping computational complexity within the linear range of the image size, allowing it to efficiently process upsampled high-resolution images output from previous steps. Therefore, the network can fully utilize the complete high-frequency information retained in the original cropped image as a reference, transferring and fusing realistic and natural high-frequency details to the corresponding structural positions in the interpolated and enlarged image, effectively avoiding the "plastic" distortion caused by over-smoothing or hallucination in traditional methods.

[0020] In summary, the Swin Transformer-based super-resolution network ensures that the final restoration result can perfectly reproduce the original complex texture of the object while removing imperfections, thus achieving a high degree of visual realism at the pixel level.

[0021] In some embodiments, the super-resolution network receives both the interpolated magnified image and the cropped image as input simultaneously via channel stitching.

[0022] In the above technical solution, the super-resolution network adopts a channel stitching method, taking the previously generated interpolated enlarged image and the original cropped image as input together, aiming to maximize information utilization and guide the generation of details.

[0023] Specifically, this channel-stitching architecture constructs a clear "guided-generation" collaborative workflow. The interpolated enlarged image carries low-frequency information that has been repaired by the pre-diffusion model, resulting in semantically and structurally complete but less detailed information; it defines the "target structure" that needs to be enhanced. The original cropped image provides "reference information," rich in realistic high-frequency textures but without degradation, although it contains imperfections. Through channel stitching, the network achieves pixel-level alignment and early fusion of these two information streams at the input. This allows the Swing Transformer-based network, within its self-attention mechanism, to simultaneously capture global semantics from the "target structure" and local texture details from the "reference image," thereby learning a precise mapping from imperfect textures to intact textures and directly transferring effective, realistic high-frequency details from the original image to the repaired area. Compared to strategies that use a single input or later fusion, this early and direct fusion approach provides the network with the richest and most aligned contextual information, greatly reducing the uncertainty of the network "fabricating" details and ensuring that the final high-resolution repair image removes blemishes while maintaining a high degree of consistency between its texture and the surrounding undamaged areas and the physical materiality of the object itself.

[0024] In summary, by using a channel-stitched input method, the semantic correctness of the generated content is combined with the realism of the reference image. This allows the super-resolution network to restore details under strong information guidance, effectively avoiding the texture distortion problem common in traditional methods and outputting restoration results with high visual credibility.

[0025] In some embodiments, the high-resolution restored image is fused back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image, specifically: The cropping mask is used as the definition of the foreground object, the original image is used as the target image, and the high-resolution repair image is used as the source image; By solving the Poisson equation, under the constraint of minimizing the change in the gradient field inside the restored image, the boundary pixel values ​​are matched with the background of the original image to generate the final restored image.

[0026] In the above technical solution, in the final stage, an image fusion method guided by the Poisson equation is adopted. While preserving the integrity of the repaired content, it ensures that the lighting and color transitions at the boundary achieve a visually natural continuity.

[0027] Specifically, the core advantage of Poisson fusion lies in its gradient field-driven working mechanism. This method treats the high-resolution restoration image as the source image and represents its internal texture and detail information through its gradient field; simultaneously, it uses the original image background as the boundary condition for fusion. By solving a Poisson equation that preserves the gradient field (i.e., detail texture) of the source image as much as possible within the restoration region, while smoothly transitioning pixel values ​​to the target background at the region boundary, this method achieves two goals: First, it maximizes the preservation of high-frequency textures and materiality within the restoration image generated by the preceding super-resolution network, avoiding secondary smoothing or information loss in the restoration result during the fusion process; second, by harmonizing pixel values ​​at the boundary, it eliminates "seams" or "halos" artifacts caused by minor differences in brightness and color, allowing the restored prop to visually "embed" into the original background, with its lighting and color consistent with the surrounding environment. This gradient domain-based fusion strategy is particularly effective for restoration tasks under complex lighting conditions or textured backgrounds.

[0028] In summary, the application of Poisson image fusion technology ensures that all the high-precision restoration and super-resolution results in the early stages can be visually seamlessly fed back into the original image, thereby outputting a final restored image with no flaws in overall appearance.

[0029] According to another aspect of the present invention, a smart prop repair device for portrait photography scenes is provided, the device comprising, based on the above-described method: The cropping module is used to obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropping mask. The low-resolution restoration module is used to input the cropped image and cropping mask into a restoration network based on a diffusion model. This restoration network combines a structured control network that provides information about props in the original image with keyword embedding trained by text inversion technology, and uses classifier unguided technology to control the generation result toward a flawless state, outputting a low-resolution restored image. A high-resolution restoration module is used to upsample the low-resolution restoration image to obtain an interpolated magnified image with the same resolution as the cropped image; the interpolated magnified image and the cropped image are input together into a super-resolution network to generate a high-resolution restoration image; The fusion module is used to fuse the high-resolution restored image back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image.

[0030] In order to better utilize the above method, this application proposes an intelligent prop repair device for portrait photography scenes. Each module corresponds to a step of the above method, and its specific principle has been described above and will not be repeated here.

[0031] According to another aspect of the present invention, a smart prop restoration device for portrait photography scenes is provided, comprising: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0032] In the above technical solution, to better operate and process the method, the method is stored in memory, and the processor executes the stored method. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here.

[0033] According to another aspect of the present invention, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method.

[0034] In the above technical solution, to better operate and use the method, the method is stored in a computer-readable storage medium and implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating an embodiment of an intelligent prop repair method for portrait photography scenes according to the present invention. Figure 2 This is a schematic diagram of the workflow of an embodiment of the intelligent prop repair method for portrait photography scenes according to the present invention; Figure 3 This is a schematic diagram of an embodiment of an intelligent prop repair device for portrait photography scenes according to the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] This invention proposes an intelligent prop restoration method for portrait photography scenes. This method divides the clothing area into a main body region and an edge zone, modeling the overall structure and local details separately. By combining consistency constraints in the image domain and frequency domain, it achieves a balance between edge smoothness and natural details. Its aim is to overcome the instability and unnaturalness caused by traditional methods relying solely on post-processing or single supervision, ultimately achieving high-precision, stable, and reliable clothing edge restoration results.

[0039] Example 1 Please see Figure 1 A method for intelligently repairing props in portrait photography scenes, the method comprising: S1. Obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropped mask. In this embodiment, a pixel-level binary mask of the target prop in the original image is obtained. The minimum bounding rectangle is calculated based on this mask, and the original image and prop mask are cropped accordingly to obtain a cropped image and a cropped mask. Specifically: The input original image is used to identify and crop the prop region: the instance segmentation network is used to obtain the pixel-level binary mask of the target prop, the minimum bounding rectangle is calculated based on the mask, and the original image and the prop mask are cropped accordingly to obtain the cropped image and the cropped mask. The instance segmentation network is used to identify one or more prop instances in an image and obtain a mask for a specific target prop through category filtering.

[0040] It is important to note several potential issues that may arise when relying on instance segmentation networks to obtain pixel-level masks. These include: segmentation errors or mask blurring due to similar colors and textures between the prop and background, or the presence of occlusion or shadows, which can affect the accuracy of subsequent restoration areas; instance merging or incorrect association when multiple similar props exist in an image, affecting the accurate differentiation and cropping of individual instances when obtaining masks for specific targets through "category filtering"; and the potential for instance merging or incorrect association when using minimum bounding rectangles for irregularly shaped props (such as long, thin, or curved objects) during the cropping stage, which may introduce a large amount of background area and interfere with the restoration network's correct understanding of the context. This application adopts the following solution: First, the instance segmentation network itself has high accuracy. By expanding the acquired prop mask outward and appropriately expanding the circumscribed rectangle used for cropping, it ensures that the target prop is completely contained within the cropping area, reducing incomplete repairs caused by boundary errors. Second, a random gain strategy for the mask is introduced during model training, enhancing the model's generalization ability to situations such as inaccurate masks and instance merging, thereby improving the robustness of segmentation and subsequent processing. Finally, regarding the issue that irregular objects may contain a lot of background after cropping, experiments show that this design does not negatively affect the contextual understanding and generation capabilities of the repair network. The repair network can effectively utilize information within the provided area for accurate restoration.

[0041] In summary, through accuracy optimization, expansion of the mask and cropping regions, random gain during training, and strong fault-tolerant design of the repair network, the system can maintain stable segmentation and repair performance in complex scenarios.

[0042] S2. The cropped image and cropping mask are input into a diffusion model-based repair network. This repair network combines a structural control network that provides information about the props in the original image with keyword embedding trained by text inversion technology. The network also controls the generation result to be in a flawless state through classifier unguided technology and outputs a low-resolution repaired image. In this embodiment, the cropped image and cropping mask are input into a diffusion-based restoration network. This restoration network combines a structured control network for providing prop information from the original image with keyword embeddings trained using text inversion techniques. It also uses an unguided classifier to guide the generation of the result towards a flawless state, outputting a low-resolution restored image. Specifically: An architecture is adopted with a pre-trained SDXL-Inpainting model as the backbone and a parallel ControlNet structure control network; the ControlNet structure control network is used to process the cropped image to obtain the structural condition information of the prop. Use embedding vector pairs trained with text inversion techniques that contain semantics of perfect and flawed states; The cropped image, its corresponding cropping mask, the structural condition information, and the semantic embedding vector pair are all input into an SDXL-Inpainting model; During the inference process of the SDXL-Inpainting model, the classifier-unguided technique is used to generate a guiding signal that moves towards the perfect state and away from the flawed state based on the semantic embedding vector pair, and under the constraint of the structural condition information, the repaired cropped image is output.

[0043] S3. Upsample the low-resolution restored image to obtain an interpolated enlarged image with the same resolution as the cropped image; input the interpolated enlarged image and the cropped image into a super-resolution network to generate a high-resolution restored image; In this embodiment, the upsampling method includes bicubic interpolation.

[0044] In this embodiment, the super-resolution network is a network based on the Swing Transformer architecture.

[0045] In this embodiment, the super-resolution network simultaneously receives the interpolated magnified image and the cropped image as input via channel stitching.

[0046] S4. The high-resolution repaired image is fused back into the original image based on the position information of the minimum bounding rectangle to obtain the final repaired image.

[0047] In this embodiment, the high-resolution restored image is fused back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image. Specifically: The cropping mask is used as the definition of the foreground object, the original image is used as the target image, and the high-resolution repair image is used as the source image; By solving the Poisson equation, under the constraint of minimizing the change in the gradient field inside the restored image, the boundary pixel values ​​are matched with the background of the original image to generate the final restored image.

[0048] For example, please refer to Figure 2 The specific steps are as follows: a. Use a segmentation network to obtain the mask region of the prop and cut out the bounding rectangle of the prop.

[0049] The purpose of this step is to accurately locate the target object and improve computational efficiency.

[0050] The original image is input into an instance segmentation network within the internal algorithm library. This network identifies and outputs the category and mask for each instance in the image. Through category filtering and logical processing, a pixel-level binary mask of the target prop is obtained, which we call the prop mask. The minimum bounding box is calculated based on the prop mask. Using the coordinates of this bounding box, both the original image and the prop mask are cropped to obtain smaller cropped images and cropped masks. The advantage of this approach is that subsequent complex calculations are concentrated within this smallest region containing the prop, significantly reducing computational load and GPU memory consumption, thus improving overall processing speed.

[0051] b. Achieving intelligent repair and effect control at low resolution based on a diffusion model.

[0052] This step is the core of the invention, enabling precise spatial control and flexible semantic control of the repair task.

[0053] Model building: Based on stable-diffusion-xl-1.0-inpainting-0.1, a ControlNet specifically designed to provide prop information from the original image is trained. The input to SDXL-inpainting is the masked prop image and the mask, while the input to ControlNet is the prop image.

[0054] Meanwhile, using Textual Inversion technology, two specific token embeddings are trained in the SDXL text encoder.

[0055] Keyword 1: Semantically, it represents a state of "perfection" or "flawlessness." The training objective is a flawless prop image.

[0056] Keyword 2: Semantically, it represents a state of "flaw" or "damage". A large number of prop images with scratches, wear, and other defects are used as training target images.

[0057] During the inference phase, the Classifier-Free Guidance (CFG) technique is used to make the generated results closer to "perfect" and farther from "flawed".

[0058] c. Use Transformer-based reference super-resolution to restore details.

[0059] This step aims to address the texture loss caused by low-resolution scaling and the jagged edges and blurring issues resulting from bicubic sampling scaling, in order to generate a high-fidelity, high-resolution final result.

[0060] The low-resolution restored image is enlarged using traditional interpolation algorithms (such as bicubic interpolation) to make its resolution completely consistent with the cropped image, resulting in an interpolated enlarged image. The content of this image is restored, but the texture is blurred. The cropped image is used as a reference image for resolution and texture. Although this image has flaws, it retains the most original and realistic high-frequency texture details of the prop.

[0061] Process: Design and train a Transformer-based super-resolution network (using and modifying the SwinTransformer architecture). This network receives two inputs: an interpolated upscaled image and the original cropped image, which are then stitched together across channels. The final output is a high-resolution restored image with resolution and detail levels equal to or exceeding that of the original cropped image.

[0062] d. Based on the position information of the circumscribed rectangle, accurately paste the high-resolution restored image back to the corresponding position of the original image to obtain the final restored image.

[0063] Based on the above embodiments, the present invention has the following advantages: 1. Extremely high localization capability and background integrity: Through precise segmentation and strong constraints from SDXL-Inpainting + ControlNet, this invention ensures that all repair operations are strictly confined within the prop outline, completely avoiding any contamination or damage to the background. This is especially important for scenes with complex backgrounds or blurred boundaries between props and background, and its accuracy far exceeds that of general repair models.

[0064] 2. Unprecedented repair accuracy: This invention utilizes high-quality datasets and combines a strategy of using TextualInversion dual keywords with CFG guidance to enable the model to understand the task to a great extent and achieve bidirectional, adjustable semantic control, thereby improving the clarity and specificity of the task.

[0065] 3. Superior Texture Fidelity and Realism: Traditional super-resolution methods often produce smooth, distorted "plastic-like" textures. The Transformer-based reference super-resolution scheme proposed in this invention cleverly utilizes the valuable high-frequency texture information still present in the original flawed image as a reference. This allows the final generated image to perfectly reproduce the original material texture of the object (such as the annual rings of wood, the brushed texture of metal, and the fibers of fabric) while removing flaws, resulting in extremely realistic restoration results.

[0066] 4. Higher processing efficiency and resource utilization: By using prop region clipping operations, this invention focuses the computationally intensive diffusion model and Transformer model on the smallest effective area, significantly reducing memory usage and computation time, making it possible to process high-resolution images. The overall efficiency is better than methods that directly operate on the entire image.

[0067] Example 2 Please see Figure 3 A prop intelligent repair device for portrait photography scenes, based on the method described in one embodiment, the device comprising: The cropping module is used to obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropping mask. The low-resolution restoration module is used to input the cropped image and cropping mask into a restoration network based on a diffusion model. This restoration network combines a structured control network that provides information about props in the original image with keyword embedding trained by text inversion technology, and uses classifier unguided technology to control the generation result toward a flawless state, outputting a low-resolution restored image. A high-resolution restoration module is used to upsample the low-resolution restoration image to obtain an interpolated magnified image with the same resolution as the cropped image; the interpolated magnified image and the cropped image are input together into a super-resolution network to generate a high-resolution restoration image; The fusion module is used to fuse the high-resolution restored image back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image.

[0068] In this embodiment, in order to better utilize the method described in one of the embodiments, this application proposes an intelligent prop repair device for portrait photography scenes. Each module corresponds to each step of the above method, and its specific principle has been described above and will not be repeated here.

[0069] Example 3 A smart prop restoration device for portrait photography scenes includes: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in one of the embodiments.

[0070] In this embodiment, to better run and process the method described in one of the embodiments, the above method is stored in a memory, and the stored method is executed using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated further here.

[0071] Example 4 A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in one of the embodiments.

[0072] In this embodiment, to better operate and use the method described in one of the embodiments, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated further here.

[0073] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for intelligently restoring props in portrait photography scenes, characterized in that, The method includes: Obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropped mask. The cropped image and cropping mask are input into a diffusion-based restoration network. This network combines a structured control network to provide prop information from the original image with keyword embeddings trained using text inversion techniques. It also uses an unguided classifier to guide the generated result towards a flawless state, outputting a low-resolution restored image. Specifically: An architecture is adopted with a pre-trained SDXL-Inpainting model as the backbone and a parallel ControlNet structure control network; the ControlNet structure control network is used to process the cropped image to obtain the structural condition information of the prop. Use embedding vector pairs trained with text inversion techniques that contain semantics of perfect and flawed states; The cropped image, its corresponding cropping mask, the structural condition information, and the semantic embedding vector pair are all input into an SDXL-Inpainting model; During the inference process of the SDXL-Inpainting model, the classifier unguided technique is used to generate a guiding signal that moves towards the perfect state and away from the flawed state based on the semantic embedding vector pair, and under the constraint of the structural condition information, the repaired cropped image is output. The low-resolution restored image is upsampled to obtain an interpolated enlarged image with the same resolution as the cropped image; the interpolated enlarged image and the cropped image are input together into a Transformer-based super-resolution network to generate a high-resolution restored image. The high-resolution restored image is fused back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image.

2. The intelligent prop restoration method for portrait photography scenes as described in claim 1, characterized in that, Obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropped mask. Specifically: The input original image is used to identify and crop the prop region: the instance segmentation network is used to obtain the pixel-level binary mask of the target prop, the minimum bounding rectangle is calculated based on the mask, and the original image and the prop mask are cropped accordingly to obtain the cropped image and the cropped mask. The instance segmentation network is used to identify one or more prop instances in an image and obtain a mask for a specific target prop through category filtering.

3. The method for intelligent prop restoration in portrait photography scenes as described in claim 1, characterized in that, The upsampling method includes bicubic interpolation.

4. The method for intelligent prop restoration in portrait photography scenes as described in claim 1, characterized in that, The super-resolution network is based on the Swing Transformer architecture.

5. The method for intelligent prop restoration in portrait photography scenes as described in claim 4, characterized in that, The super-resolution network simultaneously receives the interpolated magnified image and the cropped image as input via channel stitching.

6. The intelligent prop restoration method for portrait photography scenes as described in claim 1, characterized in that, The high-resolution restored image is fused back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image. Specifically: The cropping mask is used as the definition of the foreground object, the original image is used as the target image, and the high-resolution repair image is used as the source image; By solving the Poisson equation, under the constraint of minimizing the change in the gradient field inside the restored image, the boundary pixel values ​​are matched with the background of the original image to generate the final restored image.

7. A smart prop restoration device for portrait photography scenes, characterized in that, Based on the method according to any one of claims 1-6, the apparatus comprises: The cropping module is used to obtain the pixel-level binary mask of the target prop in the original image, calculate the minimum bounding rectangle based on the mask, and crop the original image and prop mask accordingly to obtain the cropped image and cropping mask. A low-resolution restoration module is used to input the cropped image and cropping mask into a diffusion-based restoration network. This restoration network combines a structured control network to provide prop information from the original image with keyword embeddings trained using text inversion techniques. It also uses an unguided classifier technique to guide the generated result towards a flawless state, outputting a low-resolution restored image. Specifically: An architecture is adopted with a pre-trained SDXL-Inpainting model as the backbone and a parallel ControlNet structure control network; the ControlNet structure control network is used to process the cropped image to obtain the structural condition information of the prop. Use embedding vector pairs trained with text inversion techniques that contain semantics of perfect and flawed states; The cropped image, its corresponding cropping mask, the structural condition information, and the semantic embedding vector pair are all input into an SDXL-Inpainting model; During the inference process of the SDXL-Inpainting model, the classifier unguided technique is used to generate a guiding signal that moves towards the perfect state and away from the flawed state based on the semantic embedding vector pair, and under the constraint of the structural condition information, the repaired cropped image is output. A high-resolution restoration module is used to upsample the low-resolution restoration image to obtain an interpolated magnified image with the same resolution as the cropped image; the interpolated magnified image and the cropped image are input together into a Transformer-based super-resolution network to generate a high-resolution restoration image; The fusion module is used to fuse the high-resolution restored image back into the original image based on the position information of the minimum bounding rectangle to obtain the final restored image.

8. A smart prop restoration device for portrait photography scenes, characterized in that, include: At least one processor and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images