Loss-based image matting model training methods, devices, equipment, and media

Abstract

This invention discloses a method, apparatus, device, and medium for training a matting model based on loss values. This application applies to the technical field of video matting, and includes: inputting an RGB three-channel image obtained from a preset image library into a matting model to obtain a first feature map, a second feature map, and a third feature map; calculating the loss between the first feature map and the labeled mask image using a first composite loss function based on the preset three-part image to obtain a first loss value; calculating the loss between the second feature map and the labeled mask image using a second composite loss function to obtain a second loss value; calculating the loss between the third feature map and the labeled mask image using a third composite loss function based on the preset three-part image to obtain a third loss value; optimizing the matting parameters in the matting model based on the first, second, and third loss values, and returning to execution until preset training conditions are met. This application embodiment can improve the matting effect.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video matting technology, and in particular to a method, apparatus, device and medium for training a matting model based on loss values. Background Technology

[0002] Existing matting models typically perform matting on videos based on image segmentation loss functions and video matting loss functions. Specifically, image segmentation loss functions treat the video as a series of frames, segmenting each frame to obtain the background and foreground, and modeling each feature map separately. The fitted loss functions include CrossEntropy, Weighted CrossEntropy, Focal loss, and SSIM loss. Video matting loss functions treat matting as a regression process, then model it to output a single feature map. The fitted loss functions include L1 loss, L2 loss, MSE loss, SSIM loss, and Laplacian pyramid loss. However, using a single loss function results in poor matting performance. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and medium for training a matting model based on loss values, aiming to solve the problem of poor matting results in existing methods.

[0004] In a first aspect, embodiments of the present invention provide a method for training a matting model based on a loss value, comprising:

[0005] If a matting model training instruction is received, an RGB three-channel image is obtained from a preset image library according to the matting model training instruction, and the RGB three-channel image is input into the matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions;

[0006] Based on the preset three-part image, the first loss value is obtained by calculating the loss between the first feature map and the labeled mask map using the first loss value calculation method and the first composite loss function.

[0007] The second loss value is obtained by calculating the loss between the second feature map and the labeled mask map using the second loss value calculation method and the second composite loss function.

[0008] Based on the preset three-part image, the loss between the third feature image and the labeled mask image is calculated using the third loss value calculation method and the third composite loss function to obtain the third loss value;

[0009] The matting parameters in the matting model are optimized based on the first loss value, the second loss value, and the third loss value, and the process returns to the step of obtaining the RGB three-channel image from the preset image library according to the matting model training instructions until the preset training conditions are met.

[0010] Further, using an edge masking method, the labeled edge mask corresponding to the preset triad image is set to Boolean data containing only edge labels; the preset triad image and the first feature image are merged to obtain a predicted mask image, and the predicted mask image is extracted based on the labeled edge mask image to obtain a predicted edge mask image; a preset loss value of the predicted edge mask image and the first feature image is calculated using a first composite loss function based on the labeled mask image; the predicted edge mask image, the first feature image, and the labeled mask image are respectively calculated with the RGB three-channel image to obtain a predicted edge foreground image, a predicted foreground image, and a true foreground image; the preset loss value of the predicted edge foreground image and the predicted foreground image is calculated using the first composite loss function; the average loss value of the preset loss values ​​of the predicted edge mask image, the first feature image, the predicted edge foreground image, and the predicted foreground image is calculated, and the average loss value is used as the first loss value.

[0011] Furthermore, pixels in the labeled edge mask image corresponding to the preset three-part image whose pixel value is equal to the preset pixel value are set to the boolean data true; pixels whose pixel value is not equal to the preset pixel value are set to the boolean data false.

[0012] Further, the loss values ​​between the predicted edge mask and the labeled mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted edge mask; the loss values ​​between the first feature map and the labeled mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted mask.

[0013] Further, the loss values ​​between the predicted edge foreground map and the true foreground map are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted edge foreground map; the loss values ​​between the predicted foreground map and the true foreground map are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted foreground map.

[0014] Further, the labeled mask image is downsampled by a preset factor to obtain a low-resolution labeled mask image; the loss values ​​between the low-resolution labeled mask image and the second feature map are calculated using the MSE loss function and the SSIM loss function, respectively, to obtain the MSE loss value and the SSIM loss value; a second loss value is calculated based on the MSE loss value and the SSIM loss value.

[0015] Further, the preset triangulation image and the labeled mask image are downsampled by a preset factor to obtain a low-resolution preset triangulation image and the labeled mask image; the third feature image is merged with the low-resolution preset triangulation image to obtain a predicted mask image, and the predicted mask image is extracted based on the labeled edge mask image to obtain a predicted edge mask image; the low-resolution preset triangulation image and the labeled mask image are merged to obtain a ground truth mask image, and the ground truth mask image is extracted based on the labeled edge mask image to obtain a ground truth edge mask image; the loss values ​​between the predicted edge mask image and the ground truth edge mask image are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value, respectively; a third loss value is calculated based on the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value.

[0016] Secondly, embodiments of the present invention also provide a loss-based image matting model training device, comprising:

[0017] The acquisition unit is configured to, upon receiving a matting model training instruction, acquire an RGB three-channel image from a preset image library according to the matting model training instruction, and input the RGB three-channel image into the matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions;

[0018] The first calculation unit is used to calculate the loss between the first feature map and the labeled mask map according to the preset three-part map and the first loss value calculation method using the first composite loss function to obtain the first loss value.

[0019] The second calculation unit is used to calculate the loss between the second feature map and the labeled mask map by using the second loss value calculation method and the second composite loss function to obtain the second loss value;

[0020] The third calculation unit is used to calculate the loss between the third feature map and the labeled mask map according to the preset three-part map and the third loss value calculation method using the third composite loss function to obtain the third loss value;

[0021] The optimization unit is used to optimize the matting parameters in the matting model according to the first loss value, the second loss value and the third loss value, and return to execute the step of obtaining the RGB three-channel image from the preset image library according to the matting model training instruction until the preset training conditions are met.

[0022] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0023] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0024] This invention provides a method, apparatus, device, and medium for training an image matting model based on loss values. The method includes: upon receiving an image matting model training instruction, obtaining an RGB three-channel image from a preset image library according to the instruction, and inputting the RGB three-channel image into an image matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions; calculating the loss between the first feature map and the labeled mask using a first composite loss function based on the preset three-channel image to obtain a first loss value; calculating the loss between the second feature map and the labeled mask using a second composite loss function based on a second loss value calculation method to obtain a second loss value; calculating the loss between the third feature map and the labeled mask using a third composite loss function based on the preset three-channel image to obtain a third loss value; optimizing the matting parameters in the image matting model based on the first loss value, the second loss value, and the third loss value, and then returning to the step of obtaining the RGB three-channel image from the preset image library according to the image matting model training instruction until preset training conditions are met. The technical solution of this invention uses a first composite loss function, a second composite loss function, and a third composite function to calculate a first loss value, a second loss value, and a third loss value, respectively. Based on the first loss value, the second loss value, and the third loss value, the matting parameters in the matting model are optimized, which enables the matting model to quickly fit the real scene, thereby improving the matting effect. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a loss-based image matting model training method provided in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a sub-process of a loss-based image matting model training method provided in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of a sub-process of a loss-based image matting model training method provided in an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of a sub-process of a loss-based image matting model training method provided in an embodiment of the present invention;

[0030] Figure 5 A schematic block diagram of a loss-based image matting model training device provided in an embodiment of the present invention; and

[0031] Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0034] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0037] Please see Figure 1 , Figure 1This is a flowchart illustrating the loss-value-based image matting model training method provided in this embodiment of the invention. The loss-value-based image matting model training method of this invention can be applied to a server, and implemented through an application program installed on the server to improve the image matting effect. The loss-value-based image matting model training method will be described in detail below. Figure 1 As shown, the method includes the following steps S100-S140.

[0038] S100. If a matting model training instruction is received, an RGB three-channel image is obtained from a preset image library according to the matting model training instruction, and the RGB three-channel image is input into the matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions.

[0039] In this embodiment of the invention, if the server receives a matting model training instruction, indicating that the matting model needs to be trained, it retrieves an RGB three-channel image from a preset image library according to the instruction and inputs the RGB three-channel image into the matting model to obtain a first feature image, a second feature image, and a third feature image. The first, second, and third feature images are feature images with different resolutions. Understandably, the RGB three-channel image is a 3-channel color image, and the first, second, and third feature images are single-channel images. It should be noted that in this embodiment, the RGB three-channel image is a batch × 3 × height × width image, where batch represents a batch size, 3 represents a channel size, and height and width represent height and width; the single-channel image is a batch × 1 × height × width image, where 1 represents a channel size, and height and width can be arbitrarily set according to actual conditions. In practical applications, the RGB three-channel image is 1×3×320×320, the first feature image is 1×1×320×320, the second feature image is 1×1×40×40, and the third feature image is 1×1×160×160.

[0040] S110. Based on the preset three-part image, the first loss value is obtained by using the first composite loss function to calculate the loss between the first feature image and the labeled mask image using the first loss value calculation method.

[0041] In this embodiment of the invention, after inputting the RGB three-channel image into the matting model to obtain a first feature map, a second feature map, and a third feature map, a first loss value is obtained by calculating the loss between the first feature map and the labeled mask image using a first composite loss function based on a preset tripartite image. The labeled mask image is a single-channel image that extracts the foreground; the preset tripartite image is a single-channel image containing edge regions derived from the labeled mask image. It should be noted that in this embodiment of the invention, the first loss value calculation method is a method for calculating the loss between the first feature map and the labeled mask image using a first composite loss function on the first feature map from the matting model's fusion layer. The first composite function is an L1 loss function, a SSIM loss function, and a Laplacian pyramid loss function.

[0042] Please see Figure 2 In one embodiment, for example, in an embodiment of the present invention, step S110 includes the following steps S111-S116.

[0043] S111. Using the edge masking method, set the labeled edge mask image corresponding to the preset triangulation image to Boolean data with only edge labels;

[0044] S112. The preset three-part image and the first feature image are merged to obtain a prediction mask image, and the prediction mask image is extracted based on the labeled edge mask image to obtain a prediction edge mask image.

[0045] S113. Calculate the preset loss value of the predicted edge mask and the first feature map based on the labeled mask map using the first composite loss function;

[0046] S114. The predicted edge mask, the first feature map, and the labeled mask are respectively calculated with the RGB three-channel image to obtain the predicted edge foreground image, the predicted foreground image, and the real foreground image.

[0047] S115. Calculate the predicted edge foreground map and the preset loss value of the predicted foreground map using the first composite loss function;

[0048] S116. Calculate the average loss value of the predicted edge mask map, the first feature map, the predicted edge foreground map, and the preset loss value of the predicted foreground map, and use the average loss value as the first loss value.

[0049] In this embodiment of the invention, an edge masking method is used to set the labeled edge mask corresponding to the preset triangulation image as Boolean data containing only edge labels. Specifically, pixels in the labeled edge mask corresponding to the preset triangulation image whose pixel value is equal to a preset pixel value are set to Boolean data true, where the preset pixel value is 0.5; pixels whose pixel value is not equal to the preset pixel value are set to Boolean data false. The preset triangulation image and the first feature map are merged to obtain a prediction mask, and the prediction mask is extracted based on the labeled edge mask to obtain a prediction edge mask. In practical applications, extraction involves... Pixel values ​​with a value of "true" at the boundary of the predicted mask are retained; understandably, pixel values ​​with a value of "false" are set to 0. Based on the labeled mask, a preset loss value is calculated for the predicted edge mask and the first feature map using a first composite loss function. Specifically, the loss values ​​between the predicted edge mask and the labeled mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted edge mask. The function calculates the loss value between the first feature map and the labeled mask to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted mask. The predicted edge mask, the first feature map, and the labeled mask are multiplied by the RGB three-channel image to obtain the predicted edge foreground image, the predicted foreground image, and the ground truth foreground image. The first composite loss function calculates the preset loss values ​​of the predicted edge foreground image and the predicted foreground image. Specifically, the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function are used to calculate the loss values ​​between the predicted edge foreground image and the ground truth foreground image. The loss values ​​between the foreground images are used to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted edge foreground image. The loss values ​​between the predicted foreground image and the ground truth foreground image are calculated using the L1 loss function, SSIM loss function, and Laplacian pyramid loss function, respectively, to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted foreground image. The predicted edge mask image, the first feature map, the predicted edge foreground image, and the preset loss value of the predicted foreground image are added together to obtain the average loss value, which is then used as the first loss value. Understandably, the labeled mask image is a batch×1×height×width image, which in practical applications is 1×1×320×320.It should be noted that, in this embodiment of the invention, the smaller the L1 loss value, SSIM loss value, and Laplacian pyramid loss value, the smaller the first loss value, indicating that the matting is more accurate.

[0050] S120. The second loss value is obtained by calculating the loss between the second feature map and the labeled mask map using the second loss value calculation method and the second composite loss function.

[0051] In this embodiment of the invention, after calculating the loss between the first feature map and the labeled mask using a first composite loss function to obtain a first loss value, the loss between the second feature map and the labeled mask is calculated using a second composite loss function through a second loss value calculation method to obtain a second loss value. The second composite loss function includes the MSE loss function and the SSIM loss function. The second loss value calculation method is a method for calculating the loss between the second feature map and the labeled mask using the second composite loss function on the second feature map output from the high-level semantic layer of the matting model.

[0052] Please see Figure 3 In one embodiment, for example, in an embodiment of the present invention, step S120 includes the following steps S121-S123.

[0053] S121. The labeled mask image is downsampled by a preset factor to obtain a low-resolution labeled mask image;

[0054] S122. Calculate the loss values ​​between the low-resolution labeled mask and the second feature map using the MSE loss function and the SSIM loss function respectively to obtain the MSE loss value and the SSIM loss value.

[0055] S123. Calculate the second loss value based on the MSE loss value and the SSIM loss value.

[0056] In this embodiment of the invention, the labeled mask is first downsampled by a preset factor to obtain a low-resolution labeled mask, wherein the preset factor is 8 times. For example, assuming the labeled mask is 1×1×320×320, downsampling by 8 times results in 1×1×40×40. Then, the loss values ​​between the low-resolution labeled mask and the second feature map are calculated using the MSE loss function and the SSIM loss function, respectively, to obtain the MSE loss value and the SSIM loss value. After obtaining the MSE loss value and the SSIM loss value, a second loss value is calculated based on the MSE loss value and the SSIM loss value. Specifically, the average loss value is obtained by calculating the MSE loss value and the SSIM loss value, and then multiplied by a preset weight to obtain the second loss value, wherein the preset weight is 10. It is understood that the preset factor and the preset weight value can be flexibly set according to actual needs without any restrictions. It should be noted that, in this embodiment of the invention, the smaller the MSE loss value and the SSIM loss value, the smaller the second loss value, indicating that the matting is more accurate.

[0057] S130. Based on the preset three-part image, the loss between the third feature image and the labeled mask image is calculated using the third composite loss function through the third loss value calculation method to obtain the third loss value.

[0058] In this embodiment of the invention, after calculating the loss between the second feature map and the labeled mask to obtain a second loss value, the loss between the third feature map and the labeled mask is calculated using a third composite loss function based on the preset three-part image to obtain a third loss value. The third composite loss function includes an L1 loss function, a SSIM loss function, and a Laplacian pyramid loss function. The third loss value calculation method is a method for calculating the loss between the third feature map and the labeled mask using the third composite loss function on the third feature map output from the shallow neural network layer of the matting model. It should be noted that in this embodiment of the invention, the image data input to the first composite loss function, the second composite loss function, and the third composite loss function are all normalized first. That is, the image data such as the preset three-part image, the labeled mask, the first feature map, the second feature map, and the second feature map are first normalized before being input to the corresponding composite loss function.

[0059] Please see Figure 4 In one embodiment, for example, in an embodiment of the present invention, step S130 includes the following steps S131-S135.

[0060] S131. Downsample the preset three-part image and the labeled mask image by a preset factor to obtain the preset three-part image and the labeled mask image with a low resolution;

[0061] S132. The third feature map is merged with the preset three-part map with low resolution to obtain a prediction mask map, and the prediction mask map is extracted based on the labeled edge mask map to obtain a prediction edge mask map.

[0062] S133. Merge the low-resolution preset three-part image with the labeled mask image to obtain the real mask image, and extract the real mask image based on the labeled edge mask image to obtain the real edge mask image.

[0063] S134. Calculate the loss values ​​between the predicted edge mask and the real edge mask using the L1 loss function, SSIM loss function, and Laplacian pyramid loss function respectively to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value.

[0064] S135. Calculate the third loss value based on the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value.

[0065] In this embodiment of the invention, the preset triangulation image and the labeled mask image are first downsampled by a preset factor to obtain a low-resolution version of the preset triangulation image and the labeled mask image. The preset factor is 2 times. For example, assuming both the preset triangulation image and the labeled mask image are 1×1×320×320, the preset triangulation image and the labeled mask image after downsampling by 2 times are both 1×1×160×160. The third feature image is merged with the low-resolution preset triangulation image to obtain a predicted mask image, and the predicted mask image is extracted based on the labeled edge mask image to obtain a predicted edge mask image. The low-resolution preset triangulation image is merged with the labeled mask image to obtain a true mask image, and the true mask image is extracted based on the labeled edge mask image to obtain a true edge mask image. Understandably, the two extractions are for extracting the predicted mask image and the true mask image, respectively. Pixel values ​​with a true value at the boundary are retained in practice, while those with a false value are set to 0. The loss values ​​between the predicted edge mask and the ground truth edge mask are calculated using the L1 loss function, SSIM loss function, and Laplacian pyramid loss function, respectively, to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value. After calculating these values, a third loss value is calculated. Specifically, the average loss value obtained from the L1 loss value, SSIM loss value, and Laplacian pyramid loss value is multiplied by a preset weight of 10 to obtain the third loss value. Understandably, the preset multiplier and preset weight values ​​can be flexibly set according to actual needs without any restrictions. It should be noted that, in this embodiment of the invention, the smaller the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value, the smaller the third loss value, indicating that the matting is more accurate.

[0066] S140. Optimize the matting parameters in the matting model based on the first loss value, the second loss value, and the third loss value, and return to the step of obtaining RGB three-channel images from the preset image library according to the matting model training instructions until the preset training conditions are met.

[0067] In this embodiment of the invention, the matting parameters in the matting model are optimized based on the first loss value, the second loss value, and the third loss value. Understandably, the matting parameters are multi-dimensional parameters of the matting model, and the specific parameters can be determined according to the actual application. The process then returns to the step of obtaining an RGB three-channel image from a preset image library according to the matting model training instructions until preset training conditions are met. The preset training conditions are reaching a preset number of training iterations or the first loss value, the second loss value, and the third loss value reaching preset values. It should be noted that in this embodiment of the invention, the smaller the values ​​of the first loss value, the second loss value, and the third loss value, the better the matting model fits and the more accurate the matting.

[0068] Figure 5 This is a schematic block diagram of a loss-based image matting model training device 200 provided in an embodiment of the present invention. Figure 5 As shown, corresponding to the above-described loss-based matting model training method, the present invention also provides a loss-based matting model training apparatus 200. This loss-based matting model training apparatus 200 includes a unit for executing the above-described loss-based matting model training method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 5 The loss-based matting model training device 200 includes an acquisition unit 201, a first calculation unit 202, a second calculation unit 203, a third calculation unit 204, and an optimization unit 205.

[0069] The acquisition unit 201 is configured to, upon receiving a matting model training instruction, acquire an RGB three-channel image from a preset image library according to the instruction, and input the RGB three-channel image into a matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions; the first calculation unit 202 is configured to, based on the preset three-part image, calculate the loss between the first feature map and the labeled mask image using a first loss value calculation method and a first composite loss function to obtain a first loss value; the second calculation unit 203 is configured to, based on a second loss value calculation method and a second composite loss function, calculate the loss between the first feature map and the labeled mask image to obtain a first loss value; The loss calculation method uses a second composite loss function to calculate the loss between the second feature map and the labeled mask to obtain a second loss value; the third calculation unit 204 is used to calculate the loss between the third feature map and the labeled mask using a third composite loss function based on the preset three-part image to obtain a third loss value; the optimization unit 205 is used to optimize the matting parameters in the matting model based on the first loss value, the second loss value and the third loss value, and return to execute the step of obtaining the RGB three-channel image from the preset image library according to the matting model training instructions until the preset training conditions are met.

[0070] In some embodiments, such as this embodiment, the first calculation unit 202 includes a setting unit 2021, a first extraction unit 2022, a first calculation subunit 2023, a second calculation subunit 2024, a third calculation subunit 2025, and a fourth calculation subunit 2026.

[0071] The setting unit 2021 is used to set the labeled edge mask image corresponding to the preset triangulation image to Boolean data with only edge labels using an edge masking method. Specifically, pixels in the labeled edge mask image corresponding to the preset triangulation image whose pixel value is equal to a preset pixel value are set to Boolean data true; pixels whose pixel value is not equal to the preset pixel value are set to Boolean data false. The first extraction unit 2022 is used to merge the preset triangulation image and the first feature map to obtain a prediction mask image, and extract the prediction mask image based on the labeled edge mask image to obtain a prediction edge mask image. The first calculation subunit... 2023 is used to calculate preset loss values ​​of the predicted edge mask and the first feature map based on the labeled mask using a first composite loss function. Specifically, the loss values ​​between the predicted edge mask and the labeled mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted edge mask; the loss values ​​between the first feature map and the labeled mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value of the predicted mask. The first calculation subunit 2024 is used to calculate the predicted edge foreground image, the predicted foreground image, and the real foreground image by combining the predicted edge mask, the first feature map, and the labeled mask with the RGB three-channel image respectively; the second calculation subunit 2025 is used to calculate the preset loss values ​​of the predicted edge foreground image and the predicted foreground image through the first composite loss function, specifically, to calculate the loss value between the predicted edge foreground image and the real foreground image through the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function respectively. The L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted edge foreground map are obtained; the loss values ​​between the predicted foreground map and the real foreground map are calculated using the L1 loss function, SSIM loss function, and Laplacian pyramid loss function, respectively, to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted foreground map; the fourth calculation subunit 2026 is used to calculate the average loss value of the predicted edge mask map, the first feature map, the predicted edge foreground map, and the preset loss value of the predicted foreground map, and use the average loss value as the first loss value.

[0072] In some embodiments, such as this one, the second calculation unit 203 includes a first sampling unit 2031, a fifth calculation subunit 2032, and a sixth calculation subunit 2033.

[0073] Wherein, the first sampling unit 2031 is used to downsample the labeled mask image by a preset factor to obtain the labeled mask image with a small resolution; the fifth calculation subunit 2032 is used to calculate the loss value between the small resolution labeled mask image and the second feature map through the MSE loss function and the SSIM loss function respectively to obtain the MSE loss value and the SSIM loss value; the sixth calculation subunit 2033 is used to calculate the second loss value based on the MSE loss value and the SSIM loss value.

[0074] In some embodiments, such as this one, the third calculation unit 204 includes a second sampling unit 2041, a second extraction unit 2042, a third extraction unit 2043, a seventh calculation subunit 2044, and an eighth calculation subunit 2045.

[0075] Wherein, the second sampling unit 2041 is used to downsample the preset triangulation image and the labeled mask image by a preset factor to obtain the preset triangulation image and the labeled mask image at a low resolution; the second extraction unit 2042 is used to merge the third feature image and the preset triangulation image at a low resolution to obtain a prediction mask image, and extract the prediction mask image based on the labeled edge mask image to obtain a prediction edge mask image; the third extraction unit 2043 is used to merge the preset triangulation image at a low resolution with the labeled mask image to obtain a true mask image, and extract the predicted edge mask image based on the labeled edge mask image. The seventh calculation subunit 2044 is used to extract the real edge mask from the labeled edge mask to obtain the real edge mask; the seventh calculation subunit 2044 is used to calculate the loss value between the predicted edge mask and the real edge mask through the L1 loss function, the SSIM loss function and the Laplacian pyramid loss function respectively to obtain the L1 loss value, the SSIM loss value and the Laplacian pyramid loss value; the eighth calculation subunit 2045 is used to calculate the third loss value based on the L1 loss value, the SSIM loss value and the Laplacian pyramid loss value.

[0076] The specific implementation of the image matting model training device 200 based on loss value in this embodiment of the invention corresponds to the above-described process verification method, and will not be repeated here.

[0077] The aforementioned loss-based matting model training device can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0078] Please see Figure 6 , Figure 6This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 300 is a server, which can be a standalone server or a server cluster.

[0079] See Figure 6 The computer device 300 includes a processor 302, a memory, and a network interface 305 connected via a system bus 301. The memory may include a storage medium 303 and internal memory 304.

[0080] The storage medium 303 may store an operating system 3031 and a computer program 3032. When the computer program 3032 is executed, it causes the processor 302 to execute a loss-based matting model training method.

[0081] The processor 302 provides computing and control capabilities to support the operation of the entire computer device 300.

[0082] The internal memory 304 provides an environment for the computer program 3032 in the storage medium 303 to run. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a loss-value-based matting model training method.

[0083] This network interface 305 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 300 to which the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0084] The processor 302 is used to run a computer program 3032 stored in the memory to perform the following steps: if a matting model training instruction is received, an RGB three-channel image is obtained from a preset image library according to the matting model training instruction, and the RGB three-channel image is input into the matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions; the loss between the first feature map and the labeled mask is calculated using a first composite loss function based on the preset three-part image to obtain a first loss value; the loss between the second feature map and the labeled mask is calculated using a second composite loss function based on a second loss value calculation method to obtain a second loss value; the loss between the third feature map and the labeled mask is calculated using a third composite loss function based on the preset three-part image to obtain a third loss value; the matting parameters in the matting model are optimized based on the first loss value, the second loss value, and the third loss value, and the process returns to the step of obtaining the RGB three-channel image from the preset image library according to the matting model training instruction until the preset training conditions are met.

[0085] In some embodiments, such as this embodiment, when the processor 302 implements the step of calculating the loss between the first feature map and the labeled mask map using a first composite loss function based on a preset three-part image to obtain a first loss value, the specific implementation includes the following steps: setting the labeled edge mask map corresponding to the preset three-part image to Boolean data with only edge labels using an edge masking method; merging the preset three-part image and the first feature map to obtain a prediction mask map, and extracting the prediction mask map based on the labeled edge mask map to obtain a prediction edge mask map; and then, based on the labeled mask map... The preset loss values ​​of the predicted edge mask and the first feature map are calculated using the first composite loss function; the predicted edge mask, the first feature map, and the labeled mask are respectively calculated with the RGB three-channel image to obtain the predicted edge foreground image, the predicted foreground image, and the true foreground image; the preset loss values ​​of the predicted edge foreground image and the predicted foreground image are calculated using the first composite loss function; the average loss value of the preset loss values ​​of the predicted edge mask, the first feature map, the predicted edge foreground image, and the predicted foreground image is calculated, and the average loss value is used as the first loss value.

[0086] In some embodiments, such as this embodiment, when the processor 302 implements the step of setting the labeled edge mask map corresponding to the preset triangulation map to Boolean data with only edge labels using the edge masking method, the specific steps are as follows: setting the pixel value in the labeled edge mask map corresponding to the preset triangulation map that is equal to the preset pixel value to Boolean data true; setting the pixel value that is not equal to the preset pixel value to Boolean data false.

[0087] In some embodiments, such as this embodiment, when the processor 302 implements the step of calculating the preset loss values ​​of the predicted edge mask and the first feature map based on the labeled mask using the first composite loss function, it specifically implements the following steps: calculating the loss values ​​between the predicted edge mask and the labeled mask using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function respectively to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted edge mask; calculating the loss values ​​between the first feature map and the labeled mask using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function respectively to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted mask.

[0088] In some embodiments, such as this embodiment, when the processor 302 implements the step of calculating the predicted edge foreground map and the preset loss value of the predicted foreground map using the first composite loss function, it specifically implements the following steps: calculating the loss value between the predicted edge foreground map and the real foreground map using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function respectively to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted edge foreground map; calculating the loss value between the predicted foreground map and the real foreground map using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function respectively to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted foreground map.

[0089] In some embodiments, such as this embodiment, when the processor 302 implements the step of calculating the loss between the second feature map and the labeled mask map using the second composite loss function through the second loss value calculation method to obtain the second loss value, the specific steps are as follows: downsampling the labeled mask map by a preset factor to obtain the labeled mask map with a small resolution; calculating the loss value between the labeled mask map and the second feature map with the small resolution loss function and the SSIM loss function respectively to obtain the MSE loss value and the SSIM loss value; and calculating the second loss value based on the MSE loss value and the SSIM loss value.

[0090] In some embodiments, such as this embodiment, when the processor 302 implements the step of calculating the loss between the third feature map and the labeled mask map using the third composite loss function based on the preset three-part image to obtain the third loss value, the specific steps are as follows: downsampling the preset three-part image and the labeled mask map by a preset factor to obtain the preset three-part image and the labeled mask map at a lower resolution; merging the third feature map and the lower resolution preset three-part image to obtain the prediction mask map, and extracting the prediction mask map based on the labeled edge mask map to obtain the prediction edge. The mask image is obtained by merging the low-resolution preset triangulation image with the labeled mask image to obtain the true mask image, and by extracting the true mask image based on the labeled edge mask image to obtain the true edge mask image; the loss values ​​between the predicted edge mask image and the true edge mask image are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value; a third loss value is calculated based on the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value.

[0091] It should be understood that in the embodiments of this application, the processor 302 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0092] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0093] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the loss-value-based matting model training method described above.

[0094] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0095] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0096] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0099] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0100] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for training an image matting model based on loss values, characterized in that, include: If a matting model training instruction is received, an RGB three-channel image is obtained from a preset image library according to the matting model training instruction, and the RGB three-channel image is input into the matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions; The first loss value is obtained by calculating the loss between the first feature map and the labeled mask map using a first composite loss function based on the preset three-part image and a first loss value calculation method. The first composite loss function includes the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function. The labeled mask map is a single-channel image with the foreground extracted. The preset three-part image is a single-channel image containing edge regions derived from the labeled mask map. The second loss value is obtained by calculating the loss between the second feature map and the labeled mask map using the second composite loss function through the second loss value calculation method. The second composite loss function includes the MSE loss function and the SSIM loss function. The third loss value is obtained by calculating the loss between the third feature map and the labeled mask map using the third loss value calculation method and the third composite loss function based on the preset three-part map. The third composite loss function includes the L1 loss function, the SSIM loss function and the Laplacian pyramid loss function. The matting parameters in the matting model are optimized based on the first loss value, the second loss value, and the third loss value, and the process returns to the step of obtaining the RGB three-channel image from the preset image library according to the matting model training instructions until the preset training conditions are met. The step of calculating the loss between the first feature map and the labeled mask map using a first composite loss function based on a preset three-part image to obtain a first loss value includes: The edge masking method is used to set the labeled edge mask image corresponding to the preset triangulation image to Boolean data with only edge labels. The preset three-part image and the first feature image are merged to obtain a prediction mask image, and the prediction mask image is extracted based on the labeled edge mask image to obtain a prediction edge mask image. The preset loss values ​​of the predicted edge mask and the first feature map are calculated based on the labeled mask map using the first composite loss function; The predicted edge mask, the first feature map, and the labeled mask are respectively calculated with the RGB three-channel image to obtain the predicted edge foreground image, the predicted foreground image, and the real foreground image; The predicted edge foreground map and the preset loss value of the predicted foreground map are calculated using the first composite loss function; Calculate the average loss value of the predicted edge mask map, the first feature map, the predicted edge foreground map, and the preset loss value of the predicted foreground map, and use the average loss value as the first loss value.

2. The image matting model training method based on loss value according to claim 1, characterized in that, The step of setting the labeled edge mask image corresponding to the preset triangulation image to Boolean data containing only edge labels using the edge masking method includes: Set the pixel value in the labeled edge mask image corresponding to the preset three-part image to the boolean data true; Pixels whose pixel value is not equal to the preset pixel value are set to the boolean value false.

3. The image matting model training method based on loss value according to claim 1, characterized in that, The step of calculating the preset loss values ​​of the predicted edge mask and the first feature map based on the labeled mask map using the first composite loss function includes: The loss values ​​between the predicted edge mask and the labeled mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted edge mask. The loss values ​​between the first feature map and the labeled mask map are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted mask map.

4. The image matting model training method based on loss value according to claim 1, characterized in that, The step of calculating the predicted edge foreground map and the preset loss value of the predicted foreground map using the first composite loss function includes: The loss values ​​between the predicted edge foreground map and the true foreground map are calculated using the L1 loss function, SSIM loss function, and Laplacian pyramid loss function, respectively, to obtain the L1 loss value, SSIM loss value, and Laplacian pyramid loss value of the predicted edge foreground map. The loss values ​​between the predicted foreground map and the true foreground map are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value of the predicted foreground map.

5. The image matting model training method based on loss value according to claim 1, characterized in that, The step of calculating the loss between the second feature map and the labeled mask map using the second composite loss function through the second loss value calculation method to obtain the second loss value includes: The labeled mask image is downsampled by a preset factor to obtain a low-resolution labeled mask image; The loss values ​​between the low-resolution labeled mask and the second feature map are calculated using the MSE loss function and the SSIM loss function, respectively, to obtain the MSE loss value and the SSIM loss value; The second loss value is calculated based on the MSE loss value and the SSIM loss value.

6. The image matting model training method based on loss value according to claim 1, characterized in that, The step of calculating the loss between the third feature map and the labeled mask map using the third loss value calculation method based on the preset three-part map to obtain the third loss value includes: The preset triangulation image and the labeled mask image are downsampled by a preset factor to obtain the preset triangulation image and the labeled mask image at a low resolution; The third feature map is merged with the low-resolution preset three-part map to obtain a prediction mask map, and the prediction mask map is extracted based on the labeled edge mask map to obtain a prediction edge mask map. The preset three-part image with low resolution is merged with the labeled mask image to obtain the real mask image, and the real mask image is extracted based on the labeled edge mask image to obtain the real edge mask image. The loss values ​​between the predicted edge mask and the real edge mask are calculated using the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function, respectively, to obtain the L1 loss value, the SSIM loss function, and the Laplacian pyramid loss value. The third loss value is calculated based on the L1 loss value, the SSIM loss value, and the Laplacian pyramid loss value.

7. A training device for a matting model based on loss values, characterized in that, include: The acquisition unit is configured to, upon receiving a matting model training instruction, acquire an RGB three-channel image from a preset image library according to the matting model training instruction, and input the RGB three-channel image into the matting model to obtain a first feature map, a second feature map, and a third feature map, wherein the first feature map, the second feature map, and the third feature map are feature maps with different resolutions; The first calculation unit is used to calculate the loss between the first feature map and the labeled mask map using a first loss value calculation method and a first composite loss function based on the preset three-part image to obtain a first loss value. The first composite loss function includes the L1 loss function, the SSIM loss function, and the Laplacian pyramid loss function. The labeled mask map is a single-channel image with the foreground extracted. The preset three-part image is a single-channel image containing edge regions derived based on the labeled mask map. The second calculation unit is used to calculate the loss between the second feature map and the labeled mask map by using a second composite loss function through a second loss value calculation method to obtain a second loss value, wherein the second composite loss function includes the MSE loss function and the SSIM loss function; The third calculation unit is used to calculate the loss between the third feature map and the labeled mask map according to the preset three-part map and the third loss value calculation method using the third composite loss function to obtain the third loss value. The third composite loss function includes the L1 loss function, the SSIM loss function and the Laplacian pyramid loss function. An optimization unit is used to optimize the matting parameters in the matting model based on the first loss value, the second loss value, and the third loss value, and return to execute the step of obtaining the RGB three-channel image from the preset image library according to the matting model training instruction until the preset training conditions are met. The first computing unit includes: The setting unit is used to set the labeled edge mask map corresponding to the preset three-part map to Boolean data with only edge labels by using the edge masking method; The first extraction unit is used to merge the preset three-part image and the first feature image to obtain a prediction mask image, and extract the prediction mask image based on the labeled edge mask image to obtain a prediction edge mask image. The first calculation subunit is used to calculate the preset loss value of the predicted edge mask and the first feature map based on the labeled mask map and the first composite loss function; The second calculation subunit is used to calculate the predicted edge mask, the first feature map and the labeled mask with the RGB three-channel image to obtain the predicted edge foreground image, the predicted foreground image and the real foreground image. The third calculation subunit is used to calculate the predicted edge foreground map and the preset loss value of the predicted foreground map through the first composite loss function; The fourth calculation subunit is used to calculate the average loss value of the predicted edge mask map, the first feature map, the predicted edge foreground map, and the preset loss value of the predicted foreground map, and use the average loss value as the first loss value.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the image matting model training method based on loss value as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the loss-value-based matting model training method as described in any one of claims 1-6.

Images