A training method for removing blooper object models

Abstract

The present invention discloses a training method for removing a model of a goof object. The method comprises: constructing a data set, the data set includes several generated perfect images, several goof images generated based on the perfect images, and marking the generated goof images to obtain a corresponding real label GT, the real label GT includes a goof Mask label and a main body Mask label; inputting each image in the data set into a segmentation network model, and the segmentation network model outputs a predicted goof Mask and main body Mask; calculating a first loss value based on the predicted goof Mask, main body Mask and the goof Mask label and main body Mask label in the real label GT; training the model based on the first loss value until convergence to obtain an optimized segmentation network model M1. The present invention can provide a solid data foundation for training a deep learning model for removing goof objects in an image. The data pairs generated by this method can significantly improve the performance of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of removing blooper objects, and in particular to a training method for removing blooper object models. Background Art

[0002] Use image editing software such as Adobe Photoshop to manually remove unnecessary objects from the background, but manual operations are very time-consuming, especially in large-scale image processing. At the same time, skilled image editing skills are required, otherwise obvious traces of photo editing may be left. Users are required to perform certain annotations and interactions, and it cannot be fully automated. For automatic methods based on traditional image processing, filtering, edge detection and other traditional image processing techniques are used to automatically remove background objects. Traditional methods are unstable when processing complex backgrounds or high-resolution images, and different images require different parameter settings, and the tuning process is cumbersome. Currently, the common automatic method based on deep learning is to use deep learning technologies such as convolutional neural networks (CNN) and generative adversarial networks (GAN) to automatically remove background objects. First, a large amount of labeled data is required for training, and the cost of data collection and annotation is high. Secondly, training deep learning models requires a lot of computing resources, which may require high-performance GPUs, and the performance of the model may be inconsistent on different types of images, and the generalization ability needs to be improved. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a training method for removing the model of the out-of-the-box objects, which can provide a solid data foundation for training the deep learning model for removing the out-of-the-box objects in the image. The data pairs generated by this method can significantly improve the performance of the model.

[0004] According to one aspect of the present invention, a training method for removing a model of an object with a glitch is provided, wherein the model includes a segmentation network model and a generation network model. The method includes the following steps:

[0005] Construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and label the generated goof images to obtain the corresponding true label GT, which includes the goof Mask label and the subject Mask label;

[0006] Input each image in the data set into the segmentation network model, and the segmentation network model outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the optimized segmentation network model M1;

[0007] Each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, are input into the generative network model, and the generative network model outputs a repaired image; the repaired image is fused with the original image using the goof Mask and main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X, and the model is trained based on the second loss value until convergence to obtain an optimized model M2.

[0008] In the above technical solution, an image A containing glitches and the corresponding true label GT are obtained. The true label GT contains a glitches Mask label and a main body Mask label. This step is to provide the target output required for the training model. Image A is input into the segmentation network model, the task of which is to identify and segment the glitches area and the main body area in the image, and output the predicted glitches Mask C and the main body Mask D respectively. The first loss value is calculated based on the predicted Mask and the corresponding Mask in the true label GT. This loss value measures the difference between the Mask output by the segmentation network model and the true Mask. The segmentation network model is trained using the first loss value until the model converges to obtain a model M1. This step is to optimize the segmentation network model so that it can more accurately identify and segment the glitches and the main body areas. The image A containing glitches, as well as the glitches Mask B and the main body Mask C predicted by the segmentation network model are input into the generative network model together. The task of the generative network model is to use this information to generate a repaired image D, that is, an image after the glitches are removed. The generated image D is fused with the original image A using the goof mask B and the main mask C to obtain the final result image R. The second loss value is calculated based on the result image R and the perfect image X. This loss value measures the difference between the restored image and the perfect image. The generative network model is trained using the second loss value until the model converges to obtain the model M. This step is to optimize the generative network model so that it can generate higher quality restored images. This method trains the segmentation network model and the generative network model in an end-to-end manner, so that the two networks can work together to improve the quality of the restored image. The segmentation network model focuses on identifying and segmenting the goof area, and can more accurately locate the area that needs to be restored. The generative network model uses the segmentation results to generate the restored image, which can produce a more natural and realistic restoration effect. The entire process is automated, reducing manual intervention and improving efficiency.

[0009] In some embodiments, the image A containing the glitch and the true label GT corresponding to the image A are constructed as follows:

[0010] 3D data modeling is performed using the blender rendering engine and HumanGen to obtain perfect image data of the background and subject under various light sources, as well as perfect image data collected through public datasets;

[0011] Use Photoshop software to combine the goof objects into the perfect image generated above;

[0012] The perfect images generated above are used through Stable Diffusion and StyleGAN to batch generate goof pictures, and the generated pictures are labeled. The labels include: goof Mask label and subject Mask label.

[0013] In the above technical solution, a method combining 3D modeling, image synthesis and deep learning technology is used to construct an image containing goofs and its corresponding real label. This method can be used to train a deep learning model to identify and repair goofs in an image. Specifically: 3D data modeling is performed using the Blender rendering engine and HumanGen. Blender is a powerful open source 3D modeling and rendering tool that can create high-quality 3D models and scenes. HumanGen is a tool for generating 3D character models. Through these tools, perfect image data of the background and subject can be generated under a variety of light source conditions. The goof objects are synthesized into the generated perfect image using Photoshop software. Photoshop is a professional image editing software that can precisely control each detail of image synthesis to ensure that the goof objects blend naturally and realistically with the background and subject. The generated perfect images are batch processed by Stable Diffusion and StyleGAN to generate goof pictures. Stable Diffusion and StyleGAN are both deep learning models that can generate high-quality images. These models can be used to simulate different goof situations to generate a large number of goof pictures. The generated goof images are labeled to create goof mask labels and subject mask labels. These labels will be used as training data to train the deep learning model. The goof mask label is used to indicate the location of the goof object in the image, the subject mask label is used to indicate the location of the subject in the image, and the perfect image X is the original image without goofs. The advantages of this method include:

[0014] (1) High-quality data generation: Through 3D modeling and image synthesis technology, high-quality image data can be generated, which is closer to the complexity of the real world.

[0015] (2) Diversity: By generating blooper images in batches, we can create a large amount of diverse training data, which helps improve the generalization ability of the model.

[0016] (3) Automation: Using deep learning models for image generation and labeling can automate the entire data preparation process and improve efficiency.

[0017] In some embodiments, the repaired image is fused with the original image using the goof mask and the main body mask, and the fusion is performed using the following formula:

[0018]

[0019]

[0020] In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents image D, In the above technical solution, the two images and Perform weighted combination according to mask M. is the restored image D, is the original image A, and M is the combination of the goof mask B and the main mask C. The mask M is a binary image with a value range of [0, 1]. Each pixel value in the mask determines whether the pixel at the corresponding position is still If a pixel value in the mask M is 1, the result image The corresponding pixels in will come entirely from (The restored image). If a pixel value in the mask M is 0, the result image The corresponding pixels in will come entirely from (original image). The operation ⊙ represents an element-by-element multiplication, also known as the Hadamard product. This approach allows for flexible control of the degree of fusion between the two images by adjusting the value of the mask M, which can accurately locate the image that needs to be restored. The selected area and the original image need to be retained Since the value range of the mask M is continuous, this method can produce a natural and smooth transition effect, especially in the mask edge area.

[0021] In some embodiments, the generation network model is a Transformer-like network, which includes a background restoration module and a color adjustment and fusion module; wherein,

[0022] A background restoration module, which is configured to use the goof MASK and the subject MASK to infer and generate background content that matches the surrounding environment;

[0023] A color adjustment and fusion module is configured to adjust the color and lighting of the repaired out-of-band area.

[0024] In the above technical solutions, CNN: is good at extracting local features, especially in image processing tasks. It uses convolutional layers and pooling layers to identify local features such as edges, textures, and shapes, while maintaining a certain degree of translation invariance in the spatial dimension. The self-attention mechanism of Transformer can capture long-range dependencies and global information in the image, which helps to ensure that the repaired image is visually consistent with the entire scene. This integration of global and local information helps to generate more natural and coordinated background content. The background repair module uses the goof mask and subject mask provided by the segmentation network model to infer and generate background content that matches the surrounding environment. The global information processing ability of Transformer is particularly important here because it can help the model better understand the context around the goof area and generate a suitable background to fill these areas. After repairing the goof area, it is necessary to ensure that the repaired content matches the tone and lighting conditions of the original image. The color adjustment and fusion module is responsible for adjusting the color and lighting of the repaired area to reduce the visual difference between the repaired part and the original image and achieve seamless fusion. It can reduce the sense of boundary between the repaired area and the original image and avoid obvious "sense of splicing", which is very important for improving the realism of the repaired image. The Transformer model architecture has good scalability, and the complexity and performance of the model can be easily improved by increasing the number of layers or adjusting the number of heads. Although Transformer models usually require a lot of data and computing resources to train, once trained, they usually perform better than traditional CNNs on image restoration tasks. The design idea of ​​using the Transformer model for image restoration can improve the quality and realism of the restored image and reduce the visual difference between the restored area and the original image by effectively combining global and local information processing.

[0025] In some embodiments, the first loss value is calculated using CrossEntropy Loss.

[0026] In the above technical solution, the goal of the segmentation network model is to identify and segment the out-of-band area and the main area in the image. Using the cross-entropy loss function to calculate the first loss value can help the model learn to distinguish the out-of-band mask from the main area more accurately. The cross-entropy loss function can quantify the difference between the probability distribution predicted by the model and the true label, thereby guiding the model to more accurately segment the out-of-band area and the main area. Since the gradient of the cross-entropy loss function is relatively smooth, it helps the optimization algorithm (such as gradient descent) to converge more stably, which is particularly important for the training of the segmentation network model. Even in the face of complex image backgrounds and various out-of-band situations, the model can maintain good performance. The cross-entropy loss provides an intuitive loss metric that allows developers and researchers to easily understand the performance of the model on the segmentation task and make adjustments accordingly. By applying the cross-entropy loss function to the training of the segmentation network model, the accuracy of the model in identifying and segmenting the out-of-band area can be ensured, which is the basis for the subsequent image restoration steps and directly affects the quality of the final restored image.

[0027] In some embodiments, the second loss value is calculated using Gan Loss and VGG Loss.

[0028] In the above technical solution, GAN Loss enables the generator to learn to generate more realistic images through adversarial training, while the discriminator learns to better distinguish between real images and generated images. This adversarial training can improve the performance of the model. VGG Loss can capture higher-level perceptual differences by comparing the feature maps of the generated image and the target image at different layers of the VGG network, which helps to generate more visually accurate images. Since the VGG network has multiple convolutional layers, VGG Loss can evaluate the differences in images at multiple scales, which helps to generate more detailed and clear images. Combining the advantages of GAN Loss and VGG Loss, using these two loss functions to calculate the second loss value can improve the performance of image restoration tasks. GAN Loss helps to generate high-quality images, while VGG Loss provides powerful feature extraction capabilities and perceptual losses, making the generated images more visually accurate and realistic. This combination allows the model to better learn how to repair the goofs in the image during training, thereby achieving better restoration effects.

[0029] According to another aspect of the present invention, there is provided a model for removing blooper objects, which is trained by the above-mentioned training method for removing blooper object models.

[0030] In the above technical solution, the model relies on the training method, and a model for removing the bloopers is obtained based on the training method, so that ordinary users can complete high-quality image removal operations without having to master complex editing software and techniques. Through the automated processing of the deep learning model, the technical threshold for image processing has been significantly lowered, providing an efficient and convenient solution for a wide range of user groups. It should be noted that the principles and effects of each step have been described above and will not be elaborated here.

[0031] According to another aspect of the present invention, a training device for removing a goof object model is provided, comprising:

[0032] A data acquisition module is used to construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and the real labels GT corresponding to the images, where the real labels GT include goof Mask labels and subject Mask labels;

[0033] A segmentation module is used to input each image in the data set into a segmentation network model, and the segmentation network model outputs a predicted corresponding gaffe Mask and main body Mask; a first loss value is calculated based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; the model is trained based on the first loss value until convergence to obtain an optimized model M1;

[0034] A generation module is used to input each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, into the generation network model, and the generation network model outputs a repaired image; the repaired image is merged with the original image using the goof Mask and main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image, and the model is trained based on the second loss value until convergence to obtain an optimized generation network model M2.

[0035] In the above technical solution, in order to better use the above method, the present application proposes a training device for removing the model of the object with the gaffe, and each module corresponds to each step of the above method. The specific principle has been described above and will not be repeated here.

[0036] According to another aspect of the present invention, there is provided a training device for removing a goof object model, comprising:

[0037] at least one processor and a memory communicatively coupled to the at least one processor;

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0039] In the above technical solution, in order to better run and process the method, the above method is stored in a memory, and the stored method is executed by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0040] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0041] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flow chart of an embodiment of a training method for removing a goof object model according to the present invention;

[0044] Figure 2 It is a schematic diagram of a training process of a segmentation network model according to an embodiment of a training method for removing a model of an object that is out of place in the present invention;

[0045] Figure 3 It is a schematic diagram of a training process of generating a network model according to an embodiment of a training method for removing a goof object model of the present invention;

[0046] Figure 4 It is a structural schematic diagram of an embodiment of a training device for removing a goof object model according to the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0048] The present invention provides a training method for removing a model of out-of-focus objects, which can provide a solid data foundation for training a deep learning model for removing out-of-focus objects in images. The data pairs generated by this method can significantly improve the performance of the model.

[0049] Embodiment 1

[0050] See also Figure 1 A training method for removing a model of a goof object, the model comprising a segmentation network model and a generation network model, the method comprising the following steps

[0051] S1. Construct a data set, which includes several generated perfect images, several goof images generated based on the perfect images, and label the generated goof images to obtain the corresponding true label GT, which includes the goof Mask label and the subject Mask label;

[0052] S2. Input each image in the data set into the segmentation network model, and the segmentation network model outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the optimized segmentation network model M1;

[0053] S3, input each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, into the generative network model, and generate the network model to output the repaired image; use the goof Mask and main body Mask to fuse the repaired image with the original image to obtain the final result image; calculate the second loss value based on the result image and the perfect image X, and train the model based on the second loss value until convergence to obtain the optimized generative network model M2.

[0054] In this embodiment, an image A containing goofs and a corresponding real label GT are obtained. The real label GT contains a goof Mask label and a main body Mask label. This step is to provide the target output required for the training model. Image A is input into the segmentation network model. The task of the network is to identify and segment the goof area and the main body area in the image, and output the predicted goof Mask C and main body Mask D respectively. The first loss value is calculated based on the predicted Mask and the corresponding Mask in the real label GT. This loss value measures the difference between the Mask output by the segmentation network model and the real Mask. The segmentation network model is trained with the first loss value until the model converges to obtain a model M1. This step is to optimize the segmentation network model so that it can more accurately identify and segment the goof and main body areas. The image A containing goofs, as well as the goof Mask B and main body Mask C predicted by the segmentation network model are input into the generation network model. The task of the generation network model is to use this information to generate a repaired image D, that is, an image after the goofs are removed. The generated image D is fused with the original image A using the goof mask B and the main mask C to obtain the final result image R. The second loss value is calculated based on the result image R and the perfect image X. This loss value measures the difference between the restored image and the perfect image. The generative network model is trained using the second loss value until the model converges to obtain the model M. This step is to optimize the generative network model so that it can generate higher quality restored images. This method trains the segmentation network model and the generative network model in an end-to-end manner, so that the two networks can work together to improve the quality of the restored image. The segmentation network model focuses on identifying and segmenting the goof area, and can more accurately locate the area that needs to be restored. The generative network model uses the segmentation results to generate the restored image, which can produce a more natural and realistic restoration effect. The entire process is automated, reducing manual intervention and improving efficiency.

[0055] In this embodiment, the image A containing the goof and the true label GT corresponding to the image A are constructed in the following way:

[0056] S11, 3D data modeling is performed using the blender rendering engine and HumanGen to obtain perfect image data of the background and subject under various light sources, as well as perfect image data collected through public datasets;

[0057] S12. Using Photoshop software, synthesize the goof object into the perfect image generated above;

[0058] S13. Batch generate goof images for the perfect images generated above by means of Stable Diffusion and StyleGAN, and label the generated images. The labels include: goof Mask label and subject Mask label.

[0059] In this embodiment, a method combining 3D modeling, image synthesis and deep learning technology is used to construct an image containing goofs and its corresponding real label. This method can be used to train a deep learning model to identify and repair goofs in an image. Specifically: 3D data modeling is performed using the Blender rendering engine and HumanGen. Blender is a powerful open source 3D modeling and rendering tool that can create high-quality 3D models and scenes. HumanGen is a tool for generating 3D character models. Through these tools, perfect image data of the background and subject can be generated under a variety of light source conditions. The goof objects are synthesized into the generated perfect image using Photoshop software. Photoshop is a professional image editing software that can accurately control each detail of image synthesis to ensure that the fusion of the goof objects with the background and subject is natural and real. The generated perfect images are batch processed by means of Stable Diffusion and StyleGAN to generate goof pictures. Stable Diffusion and StyleGAN are both deep learning models that can generate high-quality images. These models can be used to simulate different goof situations, thereby generating a large number of goof pictures. The generated goof images are labeled to create real labels including goof mask labels and subject mask labels. These labels will be used as training data to train the deep learning model. The goof mask label is used to indicate the location of the goof object in the image, the subject mask label is used to indicate the location of the subject in the image, and the perfect image X is the original image without goofs. The advantages of this method include:

[0060] (1) High-quality data generation: Through 3D modeling and image synthesis technology, high-quality image data can be generated, which is closer to the complexity of the real world.

[0061] (2) Diversity: By generating blooper images in batches, we can create a large amount of diverse training data, which helps improve the generalization ability of the model.

[0062] (3) Automation: Using deep learning models for image generation and labeling can automate the entire data preparation process and improve efficiency.

[0063] In this embodiment, the image D is fused with the original image A using the goof Mask B and the main body Mask C, and the fusion is performed using the following formula:

[0064]

[0065]

[0066] In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents image D, In this embodiment, the two images are and Perform weighted combination according to mask M. is the restored image D, is the original image A, and M is the combination of the goof mask B and the main mask C. The mask M is a binary image with a value range of [0, 1]. Each pixel value in the mask determines whether the pixel at the corresponding position is still If a pixel value in the mask M is 1, the result image The corresponding pixels in will come entirely from (The restored image). If a pixel value in the mask M is 0, the result image The corresponding pixels in will come entirely from (original image). The operation ⊙ represents an element-by-element multiplication, also known as the Hadamard product. This approach allows for flexible control of the degree of fusion between the two images by adjusting the value of the mask M, which can accurately locate the image that needs to be restored. The selected area and the original image need to be retained Since the value range of the mask M is continuous, this method can produce a natural and smooth transition effect, especially in the mask edge area.

[0067] In this embodiment, the generation network model is a Transformer-like network, which includes a background restoration module and a color adjustment and fusion module; wherein,

[0068] A background restoration module, which is configured to use the goof MASK and the subject MASK to infer and generate background content that matches the surrounding environment;

[0069] A color adjustment and fusion module is configured to adjust the color and lighting of the repaired out-of-band area.

[0070] In this embodiment, CNN is good at extracting local features, especially in image processing tasks. It uses convolutional layers and pooling layers to identify local features such as edges, textures, and shapes, while maintaining a certain degree of translation invariance in the spatial dimension. The self-attention mechanism of Transformer can capture long-range dependencies and global information in the image, which helps to ensure that the repaired image is visually consistent with the entire scene. This integration of global and local information helps to generate more natural and coordinated background content. The background repair module uses the goof mask and subject mask provided by the segmentation network model to infer and generate background content that matches the surrounding environment. The global information processing ability of Transformer is particularly important here because it can help the model better understand the context around the goof area and generate a suitable background to fill these areas. After repairing the goof area, it is necessary to ensure that the repaired content matches the tone and lighting conditions of the original image. The color adjustment and fusion module is responsible for adjusting the color and lighting of the repaired area to reduce the visual difference between the repaired part and the original image and achieve seamless fusion. It can reduce the sense of boundary between the repaired area and the original image and avoid obvious "sense of splicing", which is very important for improving the realism of the repaired image. The Transformer model architecture has good scalability, and the complexity and performance of the model can be easily improved by increasing the number of layers or adjusting the number of heads. Although Transformer models usually require a lot of data and computing resources to train, once trained, they usually perform better than traditional CNNs on image restoration tasks. The design idea of ​​using the Transformer model for image restoration can improve the quality and realism of the restored image and reduce the visual difference between the restored area and the original image by effectively combining global and local information processing.

[0071] In this embodiment, the first loss value adopts CrossEntropy Loss.

[0072]

[0073] in, Represents the one-hot encoding value of the true label, where only the value of the correct category is 1 and the others are 0. The model is a category The predicted probability.

[0074] The goal of the segmentation network model is to identify and segment the glitches and main areas in the image. Using the cross entropy loss function to calculate the first loss value can help the model learn to distinguish the glitches and main areas more accurately. The cross entropy loss function can quantify the difference between the probability distribution predicted by the model and the true label, thereby guiding the model to more accurately segment the glitches and main areas. Since the gradient of the cross entropy loss function is relatively smooth, it helps optimization algorithms (such as gradient descent) converge more stably, which is particularly important for the training of the segmentation network model. Even in the face of complex image backgrounds and various glitches, the model can maintain good performance. The cross entropy loss provides an intuitive loss metric that allows developers and researchers to easily understand the performance of the model on the segmentation task and make adjustments accordingly. By applying the cross entropy loss function to the training of the segmentation network model, the accuracy of the model in identifying and segmenting the glitches can be ensured, which is the basis for the subsequent image restoration steps and directly affects the quality of the final restored image.

[0075] In this embodiment, the second loss value uses Gan Loss and VGG Loss (the specific function formula refers to the prior art). GAN Loss uses adversarial training to enable the generator to learn to generate more realistic images, while the discriminator learns to better distinguish between real images and generated images. This adversarial training can improve the performance of the model. VGG Loss can capture higher-level perceptual differences by comparing the feature maps of the generated image and the target image at different layers of the VGG network, which helps to generate more visually accurate images. Since the VGG network has multiple convolutional layers, VGG Loss can evaluate the differences in images at multiple scales, which helps to generate more detailed and clear images. Combining the advantages of GAN Loss and VGG Loss, using these two loss functions to calculate the second loss value can improve the performance of image restoration tasks. GAN Loss helps to generate high-quality images, while VGG Loss provides powerful feature extraction capabilities and perceptual losses, making the generated images more visually accurate and realistic. This combination allows the model to better learn how to repair the goofs in the image during training, thereby achieving better restoration effects.

[0076] In order to better explain the present invention, the main parts will be further described as follows:

[0077] 1) Data Generation

[0078] Perfect image generation uses blender rendering engine and HumanGen (3D character library) to perform 3D data modeling to obtain perfect image data of various light sources, backgrounds and subjects.

[0079] Photoshop (PS) synthesis of fake goof data: Use Photoshop software to synthesize goof objects (such as photographic equipment, reflectors, etc.) into the perfect image generated above. Keep the natural fusion of goof objects and image background to avoid obvious cutting marks. Use Stable Diffusion and StyleGAN to batch generate more realistic goof images for the generated goof images.

[0080] 2) Model design

[0081] The model consists of two parts:

[0082] 1. Segmentation network model: Use segmentation technology to efficiently identify the area of ​​​​the goof and the main area. Please refer to the training process Figure 2 The advantages are as follows:

[0083] (1) Achieve multi-task output through a single model. The dual-head structure of the segmentation network model is used to achieve simultaneous generation of foreground and background masks. Foreground protection and background removal are considered as multi-task learning problems, which improves the accuracy and efficiency of the model and enhances the understanding of the foreground and background of the image.

[0084] (2) Reduce computational complexity: One model completes multiple tasks, reducing model call and data flow costs.

[0085] (3) Improving segmentation consistency: Through joint learning, foreground and background mask generation can interact with each other, reducing the possibility of mis-segmentation.

[0086] (4) Detail processing enhancement: This network retains high-resolution and low-resolution information, ensures accurate segmentation edges, and performs additional refinement on the subject protection area.

[0087] 2. Transformer-like generative network model:

[0088] Traditional convolutional neural networks (CNNs) are good at processing local information, while Transformer's self-attention mechanism makes it perform well in processing long-distance dependencies and global information. It can comprehensively consider the visual consistency of the entire scene, ensuring that after removing the bloopers, the overall visual effect of the image will not be unbalanced due to local processing. This model follows this idea for model design. For the training process, please refer to Figure 3 .

[0089] a. Background repair module: Based on the out-of-focus MASK and main body MASK output by the segmentation network model, this module is responsible for inferring and generating reasonable background content based on the surrounding environment information to fill in the out-of-focus area.

[0090] b. Color adjustment and fusion module: After repairing the out-of-focus area, the repaired content needs to be consistent with the original image's tone and lighting conditions to avoid an obvious "splice sense". The color adjustment and fusion module is responsible for visually blending the repaired part with the original image seamlessly.

[0091] Embodiment 2

[0092] A model for removing blooper objects is trained by a training method for removing blooper object models described in one of the embodiments.

[0093] In the above technical solution, the model relies on the training method described in one of the embodiments, and a model for removing goof objects is obtained based on the training method, so that ordinary users can complete high-quality image removal operations without having to master complex editing software and techniques. Through the automated processing of the deep learning model, the technical threshold for image processing has been significantly reduced, providing an efficient and convenient solution for a wide range of user groups. It should be noted that the principles and effects of each step have been described above and will not be elaborated here.

[0094] Embodiment 3

[0095] See also Figure 4 , a training device for removing a goof object model, comprising

[0096] The data acquisition module is used to construct a data set, which includes several generated perfect images X, several goof images generated based on the perfect images, and label the generated goof images to obtain the corresponding real labels GT, which include goof Mask labels and subject Mask labels;

[0097] The segmentation module is used to input each image in the data set into the segmentation network model, and the segmentation network model outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the optimized segmentation network model M1;

[0098] A generation module is used to input each image in the data set, as well as the corresponding predicted goof Mask and main body Mask, into the generation network model, and the generation network model outputs a repaired image; the repaired image is merged with the original image using the goof Mask and main body Mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image, and the model is trained based on the second loss value until convergence to obtain an optimized generation network model M2.

[0099] In the above technical solution, in order to better use the method described in one of the embodiments, the present application proposes a training device for removing the model of the object that is out of place, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0100] Embodiment 4

[0101] A training device for removing a goof object model, comprising:

[0102] at least one processor and a memory communicatively coupled to the at least one processor;

[0103] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in one of the embodiments.

[0104] In the above technical solution, in order to better run and process the method described in one of the embodiments, the above method is stored in a memory, and a processor is used to execute the stored method. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0105] Embodiment 5

[0106] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method described in one of the embodiments is implemented.

[0107] In the above technical solution, in order to better run and use the method described in one of the embodiments, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0108] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A training method for removing a goof object model, characterized in that: The model includes a segmentation network model and a generation network model. The method includes the following steps Construct a data set, which includes several generated perfect images, several goof images generated based on the perfect images, and label the generated goof images to obtain the corresponding real label GT, which includes a goof Mask label and a subject Mask label; the goof Mask label is used to indicate the position of the goof object in the image, and the subject Mask label is used to indicate the position of the subject in the image, and the perfect image is an original image without goofs; the perfect image is generated by rendering, modeling, or, public data collection; Input each image in the data set into the segmentation network model, and the segmentation network model outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the optimized segmentation network model M1; Each image in the dataset, as well as the corresponding predicted goof mask and subject mask, are input into the generative network model, and the generative network model outputs the repaired image; The repaired image is merged with the original image using the goof mask and the main body mask to obtain a final result image; a second loss value is calculated based on the result image and the perfect image X, and the model is trained based on the second loss value until convergence to obtain an optimized generative network model M2; The generation network model is a Transformer-like network, which includes a background restoration module and a color adjustment and fusion module; wherein, A background restoration module, which is configured to use the goof MASK and the subject MASK to infer and generate background content that matches the surrounding environment; A color adjustment and fusion module, which is configured to adjust the color and lighting of the repaired out-of-gag area; The repaired image is fused with the original image using the goof Mask and the main body Mask, and the fusion is performed using the following formula: In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents the restored image D, represents the perfect image A.

2. A training method for removing a goof object model as claimed in claim 1, characterized in that: The image A containing the goof and the true label GT corresponding to the image A are constructed as follows: 3D data modeling is performed using the blender rendering engine and HumanGen to obtain perfect image data of the background and subject under various light sources, as well as perfect image data collected through public datasets; Use Photoshop software to combine the goof objects into the perfect image generated above; The perfect images generated above are used through Stable Diffusion and StyleGAN to batch generate goof pictures, and the generated pictures are labeled. The labels include: goof Mask label and subject Mask label.

3. A training method for removing a goof object model as claimed in claim 1, characterized in that: The first loss value is calculated using CrossEntropy Loss.

4. A training method for removing a goof object model as claimed in claim 1, characterized in that: The second loss value is calculated using Gan Loss and VGG Loss.

5. A model for removing goof objects, characterized in that: The method is trained by the training method for removing blooper object models as described in any one of claims 1 to 4.

6. A training device for removing a model of a goof object, characterized in that: include A data acquisition module is used to construct a data set, which includes several generated perfect images, several goof images generated based on the perfect images, and the generated goof images are labeled to obtain the corresponding real label GT, which includes a goof Mask label and a subject Mask label; the goof Mask label is used to indicate the position of the goof object in the image, and the subject Mask label is used to indicate the position of the subject in the image, and the perfect image is an original image without goofs; the perfect image is generated by rendering, modeling, or public data collection; The segmentation module is used to input each image in the data set into the segmentation network model, and the segmentation network model outputs the corresponding predicted gaffe Mask and main body Mask; calculate the first loss value based on the predicted gaffe Mask, main body Mask and the gaffe Mask label and main body Mask label in the true label GT; train the model based on the first loss value until convergence to obtain the optimized segmentation network model M1; The generation module is used to input each image in the data set, as well as the corresponding predicted goof mask and subject mask, into the generation network model, and the generation network model outputs the repaired image; Use the goof mask and main body mask to merge the repaired image with the original image to get the final result image; Calculating a second loss value based on the result image and the perfect image, and training the model based on the second loss value until convergence to obtain an optimized generative network model M2; The generation network model is a Transformer-like network, which includes a background restoration module and a color adjustment and fusion module; wherein, A background restoration module, which is configured to use the goof MASK and the subject MASK to infer and generate background content that matches the surrounding environment; A color adjustment and fusion module, which is configured to adjust the color and lighting of the repaired out-of-gag area; The repaired image is fused with the original image using the goof Mask and the main body Mask, and the fusion is performed using the following formula: In the formula, The result graph is shown as follows: Indicates the corrected mask, M1 indicates the goof mask, M2 indicates the main body mask, and the value range is [0, 1]. represents the restored image D, represents the perfect image A.

7. A training device for removing a goof object model, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images