Image matting precision expert discrimination method and system, storage medium and program product

Abstract

The invention provides an image matting precision expert discrimination method and system, a storage medium and a program product. The method comprises the following steps: acquiring a true value mask map; using a boundary detection technology to identify a boundary region of the truth value mask pattern, and randomly determining to add interference items to obtain a processed training data set; training the processed training data set by using a ViT model as a backbone network to obtain an expert discriminator; evaluating a boundary region generated by the image matting model through a scoring mechanism by using an expert discriminator; and a scoring result of the expert discriminator is fed back to the image matting model as a part of a loss function, so that the image matting model is helped to improve the accuracy of boundary processing. According to the method, the segmentation precision of the fine boundary can be remarkably improved, especially when a fine and complex scene is processed, the finally output foreground image or video boundary is clearer and more accurate, and more accurate boundary segmentation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a matting accuracy expert judgment method, system, storage medium and program product. Background Art

[0002] Image segmentation and matting technology is an important research direction in the field of computer vision and is widely used in many fields such as film production, virtual reality, image processing and video editing. The core task of matting technology is to accurately extract the foreground object from the image or video, which usually requires that the segmented foreground and background have a clear boundary. However, in practical applications, the problem of inaccurate boundaries has always been one of the major challenges facing matting technology.

[0003] With the rapid development of deep learning, matting models based on convolutional neural networks (CNN) have gradually become mainstream. These models are trained with large-scale labeled data and have achieved certain success in processing matting tasks in complex scenes. However, due to the limited ability of deep learning models to depict boundary details, especially when dealing with fine structures such as hair and grass leaves, the models often have problems such as blurred boundaries, missed segments or over-segmentation, resulting in unsatisfactory visual quality of matting results.

[0004] Existing matting methods usually rely on the color, texture and other information of the foreground and background for segmentation, and have weak processing capabilities for fine boundaries. Although some models have introduced attention mechanisms or multi-scale feature fusion techniques to try to improve the accuracy of boundary processing, the effect is still limited. To solve this problem, in recent years, researchers have proposed the use of auxiliary networks or discriminators to specifically optimize the boundary areas of the model output. However, these methods still have room for improvement in computational complexity and universality in practical application scenarios. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a matting accuracy expert discriminant method, system, storage medium and program product, which aims to optimize the accuracy of the matting model when processing boundaries, especially in small and complex structures, to achieve more accurate boundary segmentation. The matting accuracy expert discriminator training process does not rely on the generation process, but focuses on improving the boundary accuracy of the matting model through unsupervised learning methods.

[0006] In a first aspect, the present invention provides a method for expert determination of matting accuracy, comprising the following steps:

[0007] Get the true value mask map;

[0008] Use boundary detection technology to identify the boundary area of ​​the true value mask image, randomly decide to add interference items, and obtain a processed training data set;

[0009] The processed training data set is trained using the ViT model as the backbone network to obtain an expert discriminator;

[0010] Use an expert discriminator to evaluate the boundary area generated by the matting model through a scoring mechanism;

[0011] The scoring results of the expert discriminator are fed back to the matting model as part of the loss function to help the matting model improve the accuracy of boundary processing.

[0012] As a further improvement of the present invention, the step of using a boundary detection technology to identify a boundary area of ​​a true value mask image includes:

[0013] Use the Sobel edge detection algorithm to identify the boundary area of ​​the true value mask image;

[0014] The boundary extraction formula of the Sobel edge detection algorithm is:

[0015]

[0016] Among them, G x and G y Represent the gradients in the horizontal and vertical directions respectively, I(x,y) represents the pixel value of the true mask image, and the symbol * represents the convolution operation;

[0017] Calculate the edge strength of each pixel in the true mask image using the formula:

[0018]

[0019] G(x,y) represents the edge strength of each pixel in the image.

[0020] As a further improvement of the present invention, the randomly deciding to add interference items includes: first randomly selecting whether to add interference items, and if adding interference items is selected, then randomly selecting whether to add black and white blocks or Gaussian noise.

[0021] As a further improvement of the present invention, the method for adding black and white blocks includes:

[0022] Randomly select pixels in the entire image area;

[0023] The randomly selected pixels are assigned a value of 0 or 255 according to probability to simulate the blur or over-segmentation phenomenon at the boundary between the foreground and the background. The formula is:

[0024]

[0025] Among them, I block Represents the processed true value mask image pixel value.

[0026] As a further improvement of the present invention, the Gaussian noise addition formula is:

[0027] I noise (x,y)=I(x,y)+N(0,σ 2 ),

[0028] Among them, I(x,y) represents the original value of the pixel in the true mask image, N(0,σ 2 ) means the mean is 0 and the variance is σ 2 Gaussian random noise.

[0029] As a further improvement of the present invention, the processed training data set is trained using the ViT model as a backbone network to obtain an expert discriminator, including:

[0030] Divide the true value mask image into several image blocks;

[0031] After embedding the image block into a high-dimensional feature space, it is input into the multi-layer Transformer encoder of the ViT model for processing to obtain the features processed by the ViT model;

[0032] The ViT model outputs a predicted image with the same resolution as the input true mask image through the output layer;

[0033] The output layer mapping formula of the ViT model is:

[0034] P(x,y)=Upsample(F ViT (x,y)),

[0035] Among them, F ViT is the feature processed by the ViT model, and P(x,y) represents the predicted image with the same resolution as the input true mask image;

[0036] The predicted image and the true value mask image are compared using a pixel-by-pixel binary cross entropy loss function to calculate the loss, update the model parameters, and obtain the expert discriminator.

[0037] As a further improvement of the present invention, the scoring mechanism includes:

[0038] The score is calculated based on the error between the predicted value of the expert discriminator and the actual boundary. The formula is:

[0039]

[0040] Among them, y i is the true value label of the i-th pixel, is the predicted label of the i-th pixel.

[0041] In a second aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0042] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0043] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] First, the invention adds interference items to help the training model learn how to deal with incorrect segmentation areas and noisy boundary areas, and reduce errors in practical applications, thereby improving the accuracy of the matting model. In this way, the model can better handle extreme situations that may occur in boundary areas, especially when the foreground and background colors or textures are close, the model may have incorrect segmentation, thereby enhancing the robustness and accuracy of the model.

[0046] Second, the output of the ViT model in the present invention is redesigned to have the same resolution as the original input image, which facilitates the expert discriminator to score pixel by pixel and improves the accuracy of the matting model.

[0047] Third, the expert discriminator provided by the present invention can accurately identify whether the boundary area in the image has been disturbed. In actual use, the discriminator can be seamlessly integrated into a variety of matting models, providing reliable boundary accuracy feedback and significantly improving the boundary processing performance in matting tasks.

[0048] In summary, the present invention provides a method, system, storage medium and program product for expert discrimination of matting accuracy. The matting model optimized by the expert discriminator can significantly improve the segmentation accuracy of subtle boundaries, especially when processing small and complex scenes. The final output foreground image or video boundary is clearer and more accurate, achieving more precise boundary segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of a method for training an expert discriminator for matting accuracy provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of adding and labeling interference items provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the input and output of the ViT model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S100, S200... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models described in the present invention are not the only limiting methods of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0054] The purpose of the present invention is to address the shortcomings of existing matting technologies in boundary processing, and to propose a boundary accuracy expert discrimination method based on deep learning, which aims to optimize the accuracy of the matting model when processing boundaries, especially in small and complex structures such as hair, plants, etc., to achieve more accurate boundary segmentation. Its training process does not rely on the generation process, but focuses on improving the boundary accuracy of the matting model through unsupervised learning methods. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict.

[0055] In a first aspect, an embodiment of the present invention provides a method for expert determination of matting accuracy, and the specific steps may be as follows:

[0056] like Figure 1 As shown, a matting accuracy expert discriminator is trained. The expert discriminator is used as an independent module in the matting model training process, and its main function is to evaluate the accuracy of the boundary and optimize the feedback. The training method of the expert discriminator is shown in steps S100 to S300.

[0057] S100: Obtain a true value mask image;

[0058] In one embodiment of the present invention, a ground-truth mask (i.e., a true value mask) is obtained as training data in the expert discriminator training process. The ground-truth mask can be derived from a labeled matting dataset containing an image and a corresponding ground-truth foreground mask. The present invention does not limit the data source of the ground-truth mask. Figure 2 As shown in Figure 1, the ground-truth mask corresponds to the foreground area annotation of each image, accurately describing the true boundary between the foreground and background.

[0059] S200: Use boundary detection technology to identify the boundary area of ​​the true value mask image, randomly decide to add interference items, and obtain a processed training data set;

[0060] The boundary region refers to the boundary between the foreground object and the background. In the ground-truth mask image, the foreground and the background are clearly separated, and the boundary region is the transition area between the foreground pixels and the background pixels.

[0061] Preferably, the step of using a boundary detection technique to identify a boundary area of ​​a true value mask image comprises:

[0062] S211: Use the Sobel edge detection algorithm to identify the boundary area of ​​the true value mask image. The Sobel algorithm is an edge detection method based on the first-order derivative. It detects the boundary by calculating the gradient value of each pixel in the mask image in the horizontal and vertical directions.

[0063] The boundary extraction formula of the Sobel edge detection algorithm is:

[0064]

[0065] Among them, G x and G y Represent the gradients in the horizontal and vertical directions respectively, I(x,y) represents the pixel value of the true mask image, and the symbol * represents the convolution operation;

[0066] S212: Calculate the edge strength of each pixel in the true value mask image. The formula is:

[0067]

[0068] G(x,y) represents the edge strength of each pixel in the image.

[0069] Since only the ground-truth mask is used in the training process of the present invention, the expert discriminator needs to be able to identify the accuracy of the boundary. To this end, interference items can be added to the ground-truth mask to simulate the boundary error in the actual matting task. The addition of interference items includes but is not limited to adding black and white blocks and adding Gaussian noise. Based on the interference items in the specific implementation of the present invention, all other interference item embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0070] Preferably, the randomly deciding to add interference items comprises: firstly randomly selecting whether to add interference items, and if adding interference items is selected, then randomly selecting whether to add black and white blocks or Gaussian noise.

[0071] Because the overall structure of the matting model is a generative adversarial network, which consists of a generator and a discriminator. Among them, the generator process is simulated by randomly adding interference and adding labels. Preferably, if the true value mask image is interfered, the interfered pixels are labeled 1 and the uninterrupted pixels are labeled 0; if the true value mask image is not interfered, all pixel labels are 0.

[0072] like Figure 2 As shown, the adding of black and white blocks refers to randomly inserting black and white blocks in the whole image area of ​​the true value mask image to simulate the situation where the blurring of the boundary area causes the misclassification of the matting model;

[0073] Preferably, the method of adding black and white blocks comprises:

[0074] S221: randomly selecting pixel points in the entire image area;

[0075] S222: randomly selected pixels are assigned values ​​of 0 or 255 according to probability to simulate blur or over-segmentation of the foreground and background at the boundary.

[0076] In one embodiment of the present invention, in order to simulate the problem of under-segmentation or over-segmentation of the matting model in the actual scene, the present invention interferes with the boundary information of the mask image by randomly inserting black and white blocks in the whole image area of ​​the true value mask image. This processing method is intended to guide the model to recognize and process the matting error phenomenon caused by unclear distinction between foreground and background or noise. The insertion method of black and white blocks can be performed by randomly selecting pixel points in the whole image area and assigning them a value of 0 (black) or 255 (white) to simulate the blur or over-segmentation phenomenon at the boundary between the foreground and the background. The formula is:

[0077]

[0078] Among them, I blockRepresents the processed true value mask image pixel value. The randomly selected pixels in the entire image area are assigned 0 (black) or 255 (white) according to the probability.

[0079] This method of inserting obvious black and white blocks in the whole image area is intended to simulate the phenomenon of regional discontinuity and omission that may occur when the model mis-segments. This interference helps the training model learn how to identify mis-segmented areas and reduce errors in practical applications, improving the accuracy of the matting model. In this way, the model can better handle extreme situations that may occur in boundary areas, especially when the foreground and background colors or textures are close, the model may mis-segment. The random insertion of black and white blocks makes the training data more diverse and enhances the robustness and adaptability of the model.

[0080] like Figure 2 As shown, the adding of Gaussian noise refers to adding random noise in the boundary area of ​​the true value mask image to simulate the situation where the blurring of the boundary area causes the matting model to make mistakes when processing fine boundaries.

[0081] Gaussian noise is random. By adding noise, we can simulate the unclear boundary caused by various uncertain factors, and can simulate the boundary blur problem that may be encountered in the matting task.

[0082] Preferably, the Gaussian noise addition formula is:

[0083] I noise (x,y)=I(x,y)+N(0,σ 2 ),

[0084] Among them, I(x,y) represents the original value of the pixel in the true mask image, N(0,σ 2 ) means the mean is 0 and the variance is σ 2 Gaussian random noise. The noise can be added with different variances σ according to different scenarios. 2 , in order to flexibly control the intensity of the noise, so as to better simulate various boundary blur situations. Through this processing, the model can learn to identify and process the boundary areas with noise, and improve the robustness and accuracy of the model when dealing with actual boundaries.

[0085] S300: training the processed training data set using the ViT model as a backbone network to obtain an expert discriminator;

[0086] The ViT model divides the input ground-truth mask into several fixed-size image blocks and models the features of these image blocks through the Transformer encoder. Compared with traditional convolutional neural networks (CNNs), ViT has stronger global feature capture capabilities and can better handle boundary detail issues in matting tasks.

[0087] like Figure 3 As shown in the figure, the present invention uses the ViT model (i.e., Vision Transformer model) as the backbone (i.e., backbone network), and modifies its output layer so that the model can generate a prediction output that is consistent with the original image size, so as to calculate the loss of the label and the prediction value. During the training process, the model scores the ground-truth mask image with added interference and optimizes the boundary accuracy.

[0088] Preferably, the processed training data set is trained using the ViT model as a backbone network to obtain an expert discriminator, including:

[0089] S310: dividing the true value mask image into a plurality of image blocks;

[0090] S320: embedding the image block into a high-dimensional feature space and inputting it into a multi-layer Transformer encoder of the ViT model for processing to obtain features processed by the ViT model;

[0091] After multi-layer Transformer encoding, the ViT model is able to capture the global relationship and contextual information between image patches, thereby extracting features of boundary areas.

[0092] S330: The ViT model outputs a predicted image with the same resolution as the input true value mask image through the output layer;

[0093] Preferably, in order to enable the ViT model to process the details in the matting task and generate a prediction result consistent with the size of the original image, the present invention improves the output layer of the ViT model. The traditional ViT model usually outputs a global feature vector for classification tasks, while in the present invention, the output of the ViT model is redesigned to have the same resolution as the original input image so that it can be scored pixel by pixel.

[0094] The output layer mapping formula of the ViT model is:

[0095] P(x,y)=Upsample(F ViT (x,y)),

[0096] Among them, F ViTis the feature processed by the ViT model, and P(x,y) represents the predicted image with the same resolution as the input true mask image;

[0097] Through this improvement, the model can generate output pixel by pixel, and the output value of each pixel can be compared with the corresponding pixel in the label image (the labeled true value mask image). Preferably, in the labeled true value mask image, if the true value mask image is interfered, the interfered pixels are labeled 1 and the uninterrupted pixels are labeled 0; if the true value mask image is not interfered, all pixel labels are 0.

[0098] S340: The predicted image and the true value mask image are subjected to loss calculation using a pixel-by-pixel binary cross entropy loss function, the model parameters are updated, and an expert discriminator is obtained.

[0099] The loss function used is a pixel-by-pixel binary cross entropy loss, which aims to optimize the gap between the predicted value and the true label value of each pixel.

[0100] During the training process, the ground-truth mask image and the perturbed mask image are used as input, ViTbackbone extracts global features, and generates prediction results that are consistent with the original image size through the modified output layer. The predicted value of each pixel is compared with the true label through the loss function, and the model parameters are updated through back propagation. The goal of training is to enable the model to learn to recognize the difference between the real boundary and the noise interference boundary, so as to improve the boundary accuracy of the model when processing the matting task. After the training is completed, the parameters of the expert discriminator are fixed and no longer updated. The expert discriminator will be inserted into the training process of other matting models as an independent scorer module, specifically used to score the boundary area output by the matting model and evaluate its accuracy.

[0101] S400: Using an expert discriminator to evaluate the boundary area generated by the matting model through a scoring mechanism;

[0102] The expert discriminator does not participate in the generation of the matting model. Preferably, the scoring mechanism includes:

[0103] The score is calculated based on the error between the predicted value of the expert discriminator and the actual boundary. The formula is:

[0104]

[0105] Among them, y i is the true value label of the i-th pixel, is the predicted label of the i-th pixel.

[0106] S500: Feeding back the scoring result of the expert discriminator to the matting model as part of the loss function to help the matting model improve the accuracy of boundary processing.

[0107] The expert discriminator's scoring results are fed back to the matting model as part of the loss function, helping the model to gradually improve the accuracy of boundary processing. The expert discriminator can accurately identify whether the boundary area in the image has been disturbed. In actual use, the discriminator can be seamlessly integrated into a variety of matting models, providing reliable boundary accuracy feedback and significantly improving the boundary processing performance in matting tasks.

[0108] The matting model optimized by the expert discriminator can significantly improve the segmentation accuracy of subtle boundaries, especially when processing complex scenes. The final output foreground image or video boundary is clearer and more accurate.

[0109] In a second aspect, an embodiment of the present invention provides a computer system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0110] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0111] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0112] The embodiments of the present invention provide a method, system, storage medium and program product for expert discrimination of matting accuracy. The matting model optimized by the expert discriminator can significantly improve the segmentation accuracy of subtle boundaries, especially when processing small and complex scenes. The final output foreground image or video boundary is clearer and more accurate, achieving more precise boundary segmentation.

Claims

1. A method for expert determination of matting accuracy, characterized in that: The following steps are involved: Get the true value mask map; Use boundary detection technology to identify the boundary area of ​​the true value mask image, randomly decide to add interference items, and obtain a processed training data set; The processed training data set is trained using the ViT model as the backbone network to obtain an expert discriminator; Use an expert discriminator to evaluate the boundary area generated by the matting model through a scoring mechanism; The scoring results of the expert discriminator are fed back to the matting model as part of the loss function to help the matting model improve the accuracy of boundary processing.

2. The method for expert determination of matting accuracy according to claim 1, characterized in that: The method of using a boundary detection technology to identify the boundary area of ​​the true value mask image includes: Use the Sobel edge detection algorithm to identify the boundary area of ​​the true value mask image; The boundary extraction formula of the Sobel edge detection algorithm is: Among them, G x and G y Represent the gradients in the horizontal and vertical directions respectively, I(x, y) represents the pixel value of the true mask image, and the symbol * represents the convolution operation; Calculate the edge strength of each pixel in the true mask image using the formula: G(x, y) represents the edge strength of each pixel in the image.

3. The method for expert determination of matting accuracy according to claim 1, characterized in that: The randomly deciding to add interference items includes: firstly randomly selecting whether to add interference items, and if adding interference items is selected, then randomly selecting whether to add black and white blocks or Gaussian noise.

4. The method for expert determination of matting accuracy according to claim 3, characterized in that: The method for adding the black and white blocks comprises: Randomly select pixels in the entire image area; The randomly selected pixels are assigned a value of 0 or 255 according to probability to simulate the blur or over-segmentation phenomenon at the boundary between the foreground and the background. The formula is: Among them, I block Represents the processed true value mask image pixel value.

5. The method for expert determination of matting accuracy according to claim 3, characterized in that: The Gaussian noise addition formula is: I noise (x,y)=I(x,y)+N(0,σ 2 ), Among them, I(x, y) represents the original value of the pixel in the true mask image, N(0, σ 2 ) means the mean is 0 and the variance is σ 2 Gaussian random noise.

6. The method for expert determination of matting accuracy according to claim 1, characterized in that: The processed training data set is trained using the ViT model as a backbone network to obtain an expert discriminator, including: Divide the true value mask image into several image blocks; After embedding the image block into a high-dimensional feature space, it is input into the multi-layer Transformer encoder of the ViT model for processing to obtain the features processed by the ViT model; The ViT model outputs a predicted image with the same resolution as the input true mask image through the output layer; The output layer mapping formula of the ViT model is: P(x, y) = Upsample(F) ViT (x, y)), Among them, F ViT is the feature processed by the ViT model, and P(x, y) represents the predicted image with the same resolution as the input true value mask image; The predicted image and the true value mask image are compared using a pixel-by-pixel binary cross entropy loss function to calculate the loss, update the model parameters, and obtain the expert discriminator.

7. The method for expert determination of matting accuracy according to claim 1, characterized in that: The scoring mechanism includes: The score is calculated based on the error between the predicted value of the expert discriminator and the actual boundary. The formula is: Among them, yi is the true value label of the i-th pixel, is the predicted label of the i-th pixel.

8. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images