Training method for removing wearing object model

Abstract

The invention discloses a training method for removing a wearing object model. The method comprises the steps that a data set is constructed, the data set comprises a plurality of generated perfect images, a plurality of penetrating images generated based on the perfect images and real tags GT corresponding to the penetrating images, and the real tags GT comprise penetrating Mask tags and main Mask tags; each image in the data set is input into a segmentation network, and the segmentation network outputs and predicts corresponding penetration Mask and main body Mask; calculating a first loss value based on the predicted side-through Mask, the main body Mask and a side-through Mask label and a main body Mask label in the real label GT; and training the model based on the first loss value until convergence to obtain a segmentation model M1. According to the method, a solid data basis can be provided for training a deep learning model for removing the penetrating object in the image. Through the data pair generated by the method, the performance of the model can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of removing blooper objects, and in particular to a training method for removing blooper object models. Background Art

[0002] Use image editing software such as Adobe Photoshop to manually remove unnecessary objects from the background, but manual operations are very time-consuming, especially in large-scale image processing. At the same time, skilled image editing skills are required, otherwise obvious traces of photo editing may be left. Users are required to perform certain annotations and interactions, and it cannot be fully automated. For automatic methods based on traditional image processing, filtering, edge detection and other traditional image processing techniques are used to automatically remove background objects. Traditional methods are unstable when processing complex backgrounds or high-resolution images, and different images require different parameter settings, and the tuning process is cumbersome. Currently, the common automatic method based on deep learning is to use deep learning technologies such as convolutional neural networks (CNN) and generative adversarial networks (GAN) to automatically remove background objects. First, a large amount of labeled data is required for training, and the cost of data collection and annotation is high. Secondly, training deep learning models requires a lot of computing resources, which may require high-performance GPUs, and the performance of the model may be inconsistent on different types of images, and the generalization ability needs to be improved. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a training method for removing the model of the out-of-the-box objects, which can provide a solid data foundation for training the deep learning model for removing the out-of-the-box objects in the image. The data pairs generated by this method can significantly improve the performance of the model.

[0004] According to one aspect of the present invention, a training method for removing a model of a goof object is provided, wherein the model includes a segmentation network and a generation network. The method includes the following steps: Construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and the real labels GT corresponding to the goof images. The real labels GT include goof Mask labels and subject Mask labels. Input each image in the data set into the segmentation network, and the segmentation network outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the segmentation model M1; Each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, are input into the generation network, and the generation network outputs a repaired image; the repaired image is fused with the original image using the goof Mask and the main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X, and the model is trained based on the second loss value until convergence to obtain a model M2.

[0005] In the above technical solution, an image A containing glitches and the corresponding true label GT are obtained. The true label GT contains a glitches mask label and a main body mask label. This step is to provide the target output required for the training model. Image A is input into the segmentation network, the task of which is to identify and segment the glitches area and the main body area in the image, and output the predicted glitches mask C and the main body mask D respectively. The first loss value is calculated based on the predicted mask and the corresponding mask in the true label GT. This loss value measures the difference between the mask output by the segmentation network and the true mask. The segmentation network is trained using the first loss value until the model converges to obtain a model M1. This step is to optimize the segmentation network so that it can more accurately identify and segment the glitches and the main body areas. Image A containing glitches, as well as the glitches mask B and the main body mask C predicted by the segmentation network are input into the generation network together. The task of the generation network is to use this information to generate a repaired image D, that is, an image after removing the glitches. The generated image D is fused with the original image A using the glitches mask B and the main body mask C to obtain the final result image R. The second loss value is calculated based on the result image R and the perfect image X. This loss value measures the difference between the restored image and the perfect image. The generative network is trained with the second loss value until the model converges to obtain the model M. This step is to optimize the generative network so that it can generate higher quality restored images. This method trains the segmentation network and the generative network in an end-to-end manner, so that the two networks can work together to improve the quality of the restored image. The segmentation network focuses on identifying and segmenting the out-of-gag areas, and can more accurately locate the areas that need to be restored. The generative network uses the segmentation results to generate the restored image, which can produce a more natural and realistic restoration effect. The entire process is automated, reducing manual intervention and improving efficiency.

[0006] In some embodiments, the image A containing the glitch and the true label GT corresponding to the image A are constructed as follows: 3D data modeling is performed through the blender rendering engine and HumanGen to obtain perfect image data of the background and subject under various light sources, as well as perfect image data collected through public data sets; Use Photoshop software to combine the goof objects into the perfect image generated above; The perfect images generated above are used through Stable Diffusion and StyleGAN to batch generate goof pictures, and the generated pictures are labeled. The labels include: goof Mask label and subject Mask label.

[0007] In the above technical solution, a method combining 3D modeling, image synthesis and deep learning technology is used to construct an image containing goofs and its corresponding real label. This method can be used to train a deep learning model to identify and repair goofs in an image. Specifically: 3D data modeling is performed using the Blender rendering engine and HumanGen. Blender is a powerful open source 3D modeling and rendering tool that can create high-quality 3D models and scenes. HumanGen is a tool for generating 3D character models. Through these tools, perfect image data of the background and subject can be generated under a variety of light source conditions. The goof objects are synthesized into the generated perfect image using Photoshop software. Photoshop is a professional image editing software that can precisely control each detail of image synthesis to ensure that the goof objects blend naturally and realistically with the background and subject. The generated perfect images are batch processed by Stable Diffusion and StyleGAN to generate goof pictures. Stable Diffusion and StyleGAN are both deep learning models that can generate high-quality images. These models can be used to simulate different goof situations to generate a large number of goof pictures. The generated goof images are labeled to create goof mask labels and subject mask labels. These labels will be used as training data to train the deep learning model. The goof mask label is used to indicate the location of the goof object in the image, the subject mask label is used to indicate the location of the subject in the image, and the perfect image X is the original image without goofs. The advantages of this method include: (1) High-quality data generation: Through 3D modeling and image synthesis technology, high-quality image data can be generated, which is closer to the complexity of the real world.

[0008] (2) Diversity: By generating blooper images in batches, we can create a large amount of diverse training data, which helps improve the generalization ability of the model.

[0009] (3) Automation: Using deep learning models for image generation and labeling can automate the entire data preparation process and improve efficiency.

[0010] In some embodiments, the repaired image is fused with the original image using the goof mask and the main body mask, and the fusion is performed using the following formula:

[0011]

[0012] In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents image D, In the above technical solution, the two images and Perform weighted combination according to mask M. is the restored image D, is the original image A, and M is the combination of the goof mask B and the main mask C. The mask M is a binary image with a value range of [0, 1]. Each pixel value in the mask determines whether the pixel at the corresponding position is still If a pixel value in the mask M is 1, the result image The corresponding pixels in will come entirely from (The restored image). If a pixel value in the mask M is 0, the result image The corresponding pixels in will come entirely from (original image). The operation ⊙ represents an element-by-element multiplication, also known as the Hadamard product. This approach allows for flexible control of the degree of fusion between the two images by adjusting the value of the mask M, which can accurately locate the image that needs to be restored. The selected area and the original image need to be retained Since the value range of the mask M is continuous, this method can produce a natural and smooth transition effect, especially in the mask edge area.

[0013] In some embodiments, the generating network is a Transformer-like network, which includes a background restoration module and a color adjustment and fusion module; wherein, A background restoration module, which is configured to use the goof MASK and the subject MASK to infer and generate background content that matches the surrounding environment; A color adjustment and fusion module is configured to adjust the color and lighting of the repaired out-of-band area.

[0014] In the above technical solutions, CNN: is good at extracting local features, especially in image processing tasks. It uses convolutional layers and pooling layers to identify local features such as edges, textures, and shapes, while maintaining a certain degree of translation invariance in the spatial dimension. The self-attention mechanism of Transformer can capture long-range dependencies and global information in the image, which helps to ensure that the repaired image is visually consistent with the entire scene. This integration of global and local information helps to generate more natural and coordinated background content. The background repair module uses the goof mask and subject mask provided by the segmentation model to infer and generate background content that matches the surrounding environment. The global information processing ability of Transformer is particularly important here because it can help the model better understand the context around the goof area and generate a suitable background to fill these areas. After repairing the goof area, it is necessary to ensure that the repaired content matches the tone and lighting conditions of the original image. The color adjustment and fusion module is responsible for adjusting the color and lighting of the repaired area to reduce the visual difference between the repaired part and the original image and achieve seamless fusion. It can reduce the sense of boundary between the repaired area and the original image and avoid obvious "sense of splicing", which is very important for improving the realism of the repaired image. The Transformer model architecture has good scalability, and the complexity and performance of the model can be easily improved by increasing the number of layers or adjusting the number of heads. Although Transformer models usually require a lot of data and computing resources to train, once trained, they usually perform better than traditional CNNs on image restoration tasks. The design idea of ​​using the Transformer model for image restoration can improve the quality and realism of the restored image and reduce the visual difference between the restored area and the original image by effectively combining global and local information processing.

[0015] In some embodiments, the first loss value is calculated using CrossEntropy Loss.

[0016] In the above technical solution, the goal of the segmentation network is to identify and segment the glitches and main areas in the image. Using the cross entropy loss function to calculate the first loss value can help the model learn to distinguish the glitches mask from the main area more accurately. The cross entropy loss function can quantify the difference between the probability distribution predicted by the model and the true label, thereby guiding the model to more accurately segment the glitches and main areas. Since the gradient of the cross entropy loss function is relatively smooth, it helps the optimization algorithm (such as gradient descent) to converge more stably, which is particularly important for the training of the segmentation network. Even in the face of complex image backgrounds and various glitches, the model can maintain good performance. The cross entropy loss provides an intuitive loss metric that allows developers and researchers to easily understand the performance of the model on the segmentation task and make adjustments accordingly. By applying the cross entropy loss function to the training of the segmentation network, the accuracy of the model in identifying and segmenting the glitches can be ensured, which is the basis for the subsequent image restoration steps and directly affects the quality of the final restored image.

[0017] In some embodiments, the second loss value is calculated using Gan Loss and VGG Loss.

[0018] In the above technical solution, GAN Loss enables the generator to learn to generate more realistic images through adversarial training, while the discriminator learns to better distinguish between real images and generated images. This adversarial training can improve the performance of the model. VGG Loss can capture higher-level perceptual differences by comparing the feature maps of the generated image and the target image at different layers of the VGG network, which helps to generate more visually accurate images. Since the VGG network has multiple convolutional layers, VGG Loss can evaluate the differences in images at multiple scales, which helps to generate more detailed and clear images. Combining the advantages of GAN Loss and VGG Loss, using these two loss functions to calculate the second loss value can improve the performance of image restoration tasks. GAN Loss helps to generate high-quality images, while VGG Loss provides powerful feature extraction capabilities and perceptual losses, making the generated images more visually accurate and realistic. This combination allows the model to better learn how to repair the goofs in the image during training, thereby achieving better restoration effects.

[0019] According to another aspect of the present invention, there is provided a model for removing blooper objects, which is trained by the above-mentioned training method for removing blooper object models.

[0020] In the above technical solution, the model relies on the training method, and a model for removing the bloopers is obtained based on the training method, so that ordinary users can complete high-quality image removal operations without having to master complex editing software and techniques. Through the automated processing of the deep learning model, the technical threshold for image processing has been significantly lowered, providing an efficient and convenient solution for a wide range of user groups. It should be noted that the principles and effects of each step have been described above and will not be elaborated here.

[0021] According to another aspect of the present invention, a training device for removing a goof object model is provided, comprising: A data acquisition module is used to construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and the real labels GT corresponding to the images, where the real labels GT include goof Mask labels and subject Mask labels; A segmentation module is used to input each image in the data set into a segmentation network, and the segmentation network outputs a predicted corresponding gaffe Mask and subject Mask; a first loss value is calculated based on the predicted gaffe Mask, subject Mask and the gaffe Mask label and subject Mask label in the true label GT; the model is trained based on the first loss value until convergence to obtain a model M1; A generation module is used to input each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, into the generation network, and the generation network outputs a repaired image; the repaired image is fused with the original image using the goof Mask and the main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X in the true label GT, and the model is trained based on the second loss value until convergence to obtain the generation model M2.

[0022] In the above technical solution, in order to better use the above method, the present application proposes a training device for removing the model of the object with the gaffe, and each module corresponds to each step of the above method. The specific principle has been described above and will not be repeated here.

[0023] According to another aspect of the present invention, there is provided a training device for removing a goof object model, comprising: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0024] In the above technical solution, in order to better run and process the method, the above method is stored in a memory, and the stored method is executed by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0026] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of an embodiment of a training method for removing a goof object model according to the present invention; Figure 2 It is a schematic diagram of a segmentation network training process of an embodiment of a training method for removing a glitch object model of the present invention; Figure 3 It is a schematic diagram of a generation network training process of an embodiment of a training method for removing a goof object model of the present invention; Figure 4 It is a structural schematic diagram of an embodiment of a training device for removing a goof object model according to the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0030] The present invention provides a training method for removing a model of out-of-focus objects, which can provide a solid data foundation for training a deep learning model for removing out-of-focus objects in images. The data pairs generated by this method can significantly improve the performance of the model.

[0031] Embodiment 1 See also Figure 1 A training method for removing a model of a goof object, the model comprising a segmentation network and a generation network, the method comprising the following steps S1. Construct a data set, which includes several generated perfect images, several goof images generated based on the perfect images, and the real labels GT corresponding to the goof images, and the real labels GT include goof Mask labels and subject Mask labels; S2. Input each image in the data set into the segmentation network, and the segmentation network outputs the predicted corresponding gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the segmentation model M1; S3, input each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, into the generation network, and the generation network outputs a repaired image; the repaired image is fused with the original image using the goof Mask and the main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X, and the model is trained based on the second loss value until convergence to obtain the generation model M2.

[0032] In this embodiment, an image A containing a glitch and a corresponding real label GT are obtained. The real label GT includes a glitch Mask label and a main body Mask label. This step is to provide the target output required for the training model. Image A is input into the segmentation network, the task of which is to identify and segment the glitch area and the main body area in the image, and output the predicted glitch Mask C and the main body Mask D respectively. The first loss value is calculated based on the predicted Mask and the corresponding Mask in the real label GT. This loss value measures the difference between the Mask output by the segmentation network and the real Mask. The segmentation network is trained with the first loss value until the model converges to obtain the model M1. This step is to optimize the segmentation network so that it can more accurately identify and segment the glitch and main body areas. The image A containing the glitch, as well as the glitch Mask B and the main body Mask C predicted by the segmentation network are input into the generation network together. The task of the generation network is to use this information to generate a repaired image D, that is, an image after removing the glitch. The generated image D is fused with the original image A using the glitch Mask B and the main body Mask C to obtain the final result image R. The second loss value is calculated based on the result image R and the perfect image X. This loss value measures the difference between the restored image and the perfect image. The generative network is trained with the second loss value until the model converges to obtain the model M. This step is to optimize the generative network so that it can generate higher quality restored images. This method trains the segmentation network and the generative network in an end-to-end manner, so that the two networks can work together to improve the quality of the restored image. The segmentation network focuses on identifying and segmenting the out-of-gag areas, and can more accurately locate the areas that need to be restored. The generative network uses the segmentation results to generate the restored image, which can produce a more natural and realistic restoration effect. The entire process is automated, reducing manual intervention and improving efficiency.

[0033] In this embodiment, the image A containing the goof and the real label GT corresponding to the image A are constructed as follows: S11, 3D data modeling is performed through blender rendering engine and HumanGen to obtain perfect image data of background and subject under various light sources, as well as perfect image data collected through public data sets; S12, using Photoshop software, synthesizing the goof object into the perfect image generated above; S13. Batch generate goof images for the perfect images generated above by means of Stable Diffusion and StyleGAN, and label the generated images. The labels include: goof Mask label and subject Mask label.

[0034] In this embodiment, a method combining 3D modeling, image synthesis and deep learning technology is used to construct an image containing goofs and its corresponding real label. This method can be used to train a deep learning model to identify and repair goofs in an image. Specifically: 3D data modeling is performed using the Blender rendering engine and HumanGen. Blender is a powerful open source 3D modeling and rendering tool that can create high-quality 3D models and scenes. HumanGen is a tool for generating 3D character models. Through these tools, perfect image data of the background and subject can be generated under a variety of light source conditions. The goof objects are synthesized into the generated perfect image using Photoshop software. Photoshop is a professional image editing software that can accurately control each detail of image synthesis to ensure that the fusion of the goof objects with the background and subject is natural and real. The generated perfect images are batch processed by means of Stable Diffusion and StyleGAN to generate goof pictures. Stable Diffusion and StyleGAN are both deep learning models that can generate high-quality images. These models can be used to simulate different goof situations, thereby generating a large number of goof pictures. The generated goof images are labeled to create real labels including goof mask labels and subject mask labels. These labels will be used as training data to train the deep learning model. The goof mask label is used to indicate the location of the goof object in the image, the subject mask label is used to indicate the location of the subject in the image, and the perfect image X is the original image without goofs. The advantages of this method include: (1) High-quality data generation: Through 3D modeling and image synthesis technology, high-quality image data can be generated, which is closer to the complexity of the real world.

[0035] (2) Diversity: By generating blooper images in batches, we can create a large amount of diverse training data, which helps improve the generalization ability of the model.

[0036] (3) Automation: Using deep learning models for image generation and labeling can automate the entire data preparation process and improve efficiency.

[0037] In this embodiment, the image D is fused with the original image A using the goof Mask B and the main body Mask C, and the fusion is performed using the following formula:

[0038]

[0039] In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents image D, In this embodiment, the two images are and Perform weighted combination according to mask M. is the restored image D, is the original image A, and M is the combination of the goof mask B and the main mask C. The mask M is a binary image with a value range of [0, 1]. Each pixel value in the mask determines whether the pixel at the corresponding position is still If a pixel value in the mask M is 1, the result image The corresponding pixels in will come entirely from (The restored image). If a pixel value in the mask M is 0, the result image The corresponding pixels in will come entirely from (original image). The operation ⊙ represents an element-by-element multiplication, also known as the Hadamard product. This approach allows for flexible control of the degree of fusion between the two images by adjusting the value of the mask M, which can accurately locate the image that needs to be restored. The selected area and the original image need to be retained Since the value range of the mask M is continuous, this method can produce a natural and smooth transition effect, especially in the mask edge area.

[0040] In this embodiment, the generation network is a Transformer-like network, which includes a background restoration module and a color adjustment and fusion module; wherein, A background restoration module, which is configured to use the goof MASK and the subject MASK to infer and generate background content that matches the surrounding environment; A color adjustment and fusion module is configured to adjust the color and lighting of the repaired out-of-band area.

[0041] In this embodiment, CNN is good at extracting local features, especially in image processing tasks. It uses convolutional layers and pooling layers to identify local features such as edges, textures, and shapes, while maintaining a certain degree of translation invariance in the spatial dimension. The self-attention mechanism of Transformer can capture long-range dependencies and global information in the image, which helps to ensure that the repaired image is visually consistent with the entire scene. This integration of global and local information helps to generate more natural and coordinated background content. The background repair module uses the goof mask and subject mask provided by the segmentation model to infer and generate background content that matches the surrounding environment. The global information processing ability of Transformer is particularly important here because it can help the model better understand the context around the goof area and generate a suitable background to fill these areas. After repairing the goof area, it is necessary to ensure that the repaired content matches the tone and lighting conditions of the original image. The color adjustment and fusion module is responsible for adjusting the color and lighting of the repaired area to reduce the visual difference between the repaired part and the original image and achieve seamless fusion. It can reduce the sense of boundary between the repaired area and the original image and avoid obvious "sense of splicing", which is very important for improving the realism of the repaired image. The Transformer model architecture has good scalability, and the complexity and performance of the model can be easily improved by increasing the number of layers or adjusting the number of heads. Although Transformer models usually require a lot of data and computing resources to train, once trained, they usually perform better than traditional CNNs on image restoration tasks. The design idea of ​​using the Transformer model for image restoration can improve the quality and realism of the restored image and reduce the visual difference between the restored area and the original image by effectively combining global and local information processing.

[0042] In this embodiment, the first loss value adopts CrossEntropy Loss.

[0043]

[0044] in, Represents the one-hot encoding value of the true label, where only the value of the correct category is 1 and the others are 0. The model is a category The predicted probability.

[0045] The goal of the segmentation network is to identify and segment the glitches and main areas in the image. Using the cross-entropy loss function to calculate the first loss value can help the model learn to distinguish the glitches and main areas more accurately. The cross-entropy loss function can quantify the difference between the probability distribution predicted by the model and the true label, thereby guiding the model to more accurately segment the glitches and main areas. Since the gradient of the cross-entropy loss function is relatively smooth, it helps optimization algorithms (such as gradient descent) converge more stably, which is particularly important for the training of the segmentation network. The model can maintain good performance even when faced with complex image backgrounds and diverse glitches. The cross-entropy loss provides an intuitive loss metric that allows developers and researchers to easily understand the performance of the model on the segmentation task and make adjustments accordingly. By applying the cross-entropy loss function to the training of the segmentation network, the accuracy of the model in identifying and segmenting the glitches can be ensured, which is the basis for subsequent image restoration steps and directly affects the quality of the final restored image.

[0046] In this embodiment, the second loss value uses Gan Loss and VGG Loss (the specific function formula refers to the prior art). GAN Loss uses adversarial training to enable the generator to learn to generate more realistic images, while the discriminator learns to better distinguish between real images and generated images. This adversarial training can improve the performance of the model. VGG Loss can capture higher-level perceptual differences by comparing the feature maps of the generated image and the target image at different layers of the VGG network, which helps to generate more visually accurate images. Since the VGG network has multiple convolutional layers, VGG Loss can evaluate the differences in images at multiple scales, which helps to generate more detailed and clear images. Combining the advantages of GAN Loss and VGG Loss, using these two loss functions to calculate the second loss value can improve the performance of image restoration tasks. GAN Loss helps to generate high-quality images, while VGG Loss provides powerful feature extraction capabilities and perceptual losses, making the generated images more visually accurate and realistic. This combination allows the model to better learn how to repair the goofs in the image during training, thereby achieving better restoration effects.

[0047] In order to better explain the present invention, the main parts will be further described as follows: 1) Data Generation Perfect image generation uses blender rendering engine and HumanGen (3D character library) to perform 3D data modeling to obtain perfect image data of various light sources, backgrounds and subjects.

[0048] Photoshop (PS) synthesis of fake goof data: Use Photoshop software to synthesize goof objects (such as photographic equipment, reflectors, etc.) into the perfect image generated above. Keep the natural fusion of goof objects and image background to avoid obvious cutting marks. Use Stable Diffusion and StyleGAN to batch generate more realistic goof images for the generated goof images. 2) Model design The model consists of two parts: 1. Segmentation model: Use segmentation technology to efficiently identify the goof area and the main area. Please refer to the training process Figure 2 The advantages are as follows: (1) Achieve multi-task output through a single model. Utilize the dual-head structure of the segmentation model to achieve simultaneous generation of foreground and background masks. Treat foreground protection and background removal as multi-task learning problems, improve the accuracy and efficiency of the model, and enhance the understanding of the foreground and background of the image.

[0049] (2) Reduce computational complexity: One model completes multiple tasks, reducing model call and data flow costs.

[0050] (3) Improving segmentation consistency: Through joint learning, foreground and background mask generation can interact with each other, reducing the possibility of mis-segmentation.

[0051] (4) Detail processing enhancement: This network retains high-resolution and low-resolution information, ensures accurate segmentation edges, and performs additional refinement on the subject protection area.

[0052] 2. Transformer-like generation model: Traditional convolutional neural networks (CNNs) are good at processing local information, while Transformer's self-attention mechanism makes it perform well in processing long-distance dependencies and global information. It can comprehensively consider the visual consistency of the entire scene, ensuring that after removing the bloopers, the overall visual effect of the image will not be unbalanced due to local processing. This model follows this idea for model design. For the training process, please refer to Figure 3 .

[0053] a. Background repair module: Based on the goof mask and subject mask output by the segmentation model, this module is responsible for inferring and generating reasonable background content based on the surrounding environment information to fill in the goof area.

[0054] b. Color adjustment and fusion module: After repairing the out-of-focus area, the repaired content needs to be consistent with the original image's tone and lighting conditions to avoid an obvious "splice sense". The color adjustment and fusion module is responsible for visually blending the repaired part with the original image seamlessly.

[0055] Embodiment 2 A model for removing blooper objects is trained by a training method for removing blooper object models described in one of the embodiments.

[0056] In the above technical solution, the model relies on the training method described in one of the embodiments, and a model for removing goof objects is obtained based on the training method, so that ordinary users can complete high-quality image removal operations without having to master complex editing software and techniques. Through the automated processing of the deep learning model, the technical threshold for image processing has been significantly reduced, providing an efficient and convenient solution for a wide range of user groups. It should be noted that the principles and effects of each step have been described above and will not be elaborated here.

[0057] Embodiment 3 See also Figure 4 , a training device for removing a goof object model, comprising A data acquisition module is used to construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and real labels GT corresponding to the goof images, where the real labels GT include goof Mask labels and subject Mask labels; A segmentation module is used to input each image in the data set into a segmentation network, and the segmentation network outputs a predicted corresponding gaffe Mask and subject Mask; a first loss value is calculated based on the predicted gaffe Mask, subject Mask and the gaffe Mask label and subject Mask label in the true label GT; the model is trained based on the first loss value until convergence to obtain a segmentation model M1; A generation module is used to input each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, into the generation network, and the generation network outputs a repaired image; the repaired image is fused with the original image using the goof Mask and the main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X in the true label GT, and the model is trained based on the second loss value until convergence to obtain the generation model M2.

[0058] In the above technical solution, in order to better use the method described in one of the embodiments, the present application proposes a training device for removing the model of the object that is out of place, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0059] Embodiment 4 A training device for removing a goof object model, comprising: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in one of the embodiments.

[0060] In the above technical solution, in order to better run and process the method described in one of the embodiments, the above method is stored in a memory, and a processor is used to execute the stored method. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0061] Embodiment 5 A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method described in one of the embodiments is implemented.

[0062] In the above technical solution, in order to better run and use the method described in one of the embodiments, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0063] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A training method for removing a goof object model, characterized in that: The model includes a segmentation network and a generation network. The method includes the following steps Construct a dataset, which includes several generated perfect images, several goof images generated based on the perfect images, and the real labels GT corresponding to the goof images. The real labels GT include goof Mask labels and subject Mask labels. Input each image in the data set into the segmentation network, and the segmentation network outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the segmentation model M1; Each image in the dataset, as well as the corresponding predicted goof mask and subject mask, are input into the generative network, and the generative network outputs the repaired image; The repaired image is fused with the original image using the goof Mask and the main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X, and the model is trained based on the second loss value until convergence to obtain a generated model M2.

2. A training method for removing a goof object model as claimed in claim 1, characterized in that: The image A containing the goof and the true label GT corresponding to the image A are constructed as follows: 3D data modeling is performed using the blender rendering engine and HumanGen to obtain perfect image data of the background and subject under various light sources, as well as perfect image data collected through public datasets; Use Photoshop software to combine the goof objects into the perfect image generated above; The perfect images generated above are used through Stable Diffusion and StyleGAN to batch generate goof pictures, and the generated pictures are labeled. The labels include: goof Mask label and subject Mask label.

3. A training method for removing a goof object model as claimed in claim 2, characterized in that: The repaired image is fused with the original image using the goof Mask and the main body Mask, and the fusion is performed using the following formula: In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents image D, Represents the original image A.

4. A training method for removing a goof object model as claimed in claim 1, characterized in that: The generating network is a Transformer-like network.

5. A training method for removing a goof object model as claimed in claim 1, characterized in that: The first loss value is calculated using CrossEntropy Loss.

6. A training method for removing a goof object model as claimed in claim 1, characterized in that: The second loss value is calculated using Gan Loss and VGG Loss.

7. A model for removing goof objects, characterized in that: The method is trained by a training method for removing blooper object models as described in any one of claims 1 to 6.

8. A training device for removing a goof object model, characterized in that: include A data acquisition module is used to construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and real labels GT corresponding to the goof images, where the real labels GT include goof Mask labels and subject Mask labels; A segmentation module is used to input each image in the data set into a segmentation network, and the segmentation network outputs a predicted corresponding gaffe Mask and subject Mask; a first loss value is calculated based on the predicted gaffe Mask, subject Mask and the gaffe Mask label and subject Mask label in the true label GT; the model is trained based on the first loss value until convergence to obtain a segmentation model M1; The generation module is used to input each image in the dataset, as well as the corresponding predicted goof mask and subject mask, into the generation network, and the generation network outputs the repaired image; The repaired image is fused with the original image using the goof Mask and the main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X in the true label GT, and the model is trained based on the second loss value until convergence to obtain a generated model M2.

9. A training device for removing a goof object model, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images