Method for determining matting parameters, image processing method, equipment and storage medium

Abstract

The embodiment of the invention provides a method for determining matting parameters, an image processing method, equipment and a storage medium. The method comprises the steps of obtaining a to-be-matted image and a value range of a background color of the to-be-matted image in each channel under a specified color model; based on the to-be-matted image and the value range, a pre-constructed target optimization function is optimized to obtain an optimization value of a matting parameter of the to-be-matted image, a variable of the target optimization function is the matting parameter, and a variable of the target optimization function is the matting parameter; the optimization target of the target optimization function comprises one or more of the following items: when the pixel points in the to-be-matted image are divided into a foreground region and a background region based on the matting parameters, the number of misclassified pixel points is minimized; minimizing the number of pixel points belonging to a transition region between the foreground region and the background region in the to-be-matted image; and the accumulated value of the probabilities of the pixel points in the transition area is minimized. In this way, the matting parameters can be automatically and accurately determined.

Description

Method for determining cutout parameters, image processing method, device and storage medium Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for determining cutout parameters, an image processing method, a device, and a storage medium. Background Art

[0002] Image cutout technology is widely used in various fields. By identifying the foreground and background areas of the image to be cutout, and then replacing the background area with the user's desired background, special effects images or images in specific scenes can be obtained without the need for additional dedicated scenes.

[0003] At present, some cutout technologies can pre-determine the difference between the color of each pixel in the image to be cutout and the background color of the image to be cutout, and then determine the probability (i.e., Alpha value) that each pixel is the foreground of the image to be cutout based on the difference and the pre-set cutout parameters, so as to obtain the Alpha image corresponding to the image to be cutout, and perform cutout processing on the image to be cutout based on the Alpha image.

[0004] Typically, when setting cutout parameters, users typically do so manually. For example, users can continually adjust the cutout parameter values, then visually observe the cutout effect. The values ​​that produce the best cutout results are then used as the final cutout parameters for the image to be cutout. This method is cumbersome, time-consuming, and labor-intensive. Furthermore, when the cutout parameter values ​​are close to the optimal values, the changes in the cutout effect are subtle and difficult to detect directly with the naked eye. Therefore, the cutout parameters set in this way are not necessarily the optimal cutout parameters.

[0005] Therefore, it is necessary to provide a solution that is more convenient and quick and can accurately determine the optimal or better clipping parameters.

[0006] Summary of the Invention

[0007] Based on this, this specification provides a method for determining cutout parameters, an image processing method, a device, and a storage medium.

[0008] According to a first aspect of an embodiment of this specification, a method for determining a cutout parameter is provided, the method comprising:

[0009] Obtaining the value range of each channel of the image to be cut out and the background color of the image to be cut out under a specified color model;

[0010] A pre-constructed target optimization function is optimized based on the image to be cutout and the value range to obtain an optimized value of a cutout parameter of the image to be cutout, wherein the cutout parameter is used to determine a probability that a pixel point in the image to be cutout is a foreground area of ​​the image to be cutout, and the probability is determined based on the cutout parameter and a difference between the color of the pixel point and the background color; the variable of the target optimization function is the cutout parameter, and the optimization target of the target optimization function includes one or more of the following:

[0011] When dividing the pixels in the image to be cut out into the foreground area and the background area of ​​the image to be cut out based on the cutout parameters, the number of misclassified pixels is minimized;

[0012] Minimizing the number of pixel points in the image to be cutout that belong to the transition area between the foreground area and the background area determined based on the cutout parameters;

[0013] The accumulated value of the probabilities of the pixels in the transition area determined based on the cutout parameters is minimized.

[0014] According to a second aspect of the embodiments of this specification, there is provided an image processing method, the method comprising:

[0015] Get the image to be cut out;

[0016] Determine the difference between the color of each pixel in the image to be cut out and the background color of the image to be cut out;

[0017] Based on the difference and the optimized value of the predetermined cutout parameter, the probability that each pixel in the image to be cutout belongs to the foreground of the image to be cutout is determined, so as to perform cutout processing on the image to be cutout based on the probability, wherein the optimized value is obtained based on the method for determining the cutout parameter mentioned in the first aspect above.

[0018] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a processor, a memory, and computer instructions stored on the memory, wherein the processor implements the method mentioned in the first and / or second aspects above when executing the computer instructions.

[0019] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the method mentioned in the first and / or second aspects above is implemented.

[0020] By applying the embodiments of this specification, in order to automatically determine the optimal or better cutout parameters, the applicant has designed a target optimization function. This target optimization function uses the cutout parameters as variables and one or more error functions used to measure the cutout errors caused by the cutout parameters as optimization targets. The target optimization function is then optimized based on the image to be cutout, thereby automatically determining the optimized values ​​of the cutout parameters for subsequent cutout processing. In this way, better or optimal cutout parameters can be automatically determined without the need for manual setting by the user, making the determination of the cutout parameters more convenient and quick, and the determined cutout parameters more accurate, and the cutout image obtained using the cutout parameters has a better effect.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0023] FIG1 is a schematic diagram of manually setting cutout parameters in the prior art.

[0024] FIG2 is a flow chart of a method for determining cutout parameters according to an embodiment of the present disclosure.

[0025] FIG. 3 is a schematic diagram of defining a background color distribution range using a reference color and a similarity threshold according to an embodiment of the present disclosure.

[0026] FIG. 4 is a schematic diagram of defining a background color distribution range using a reference color and a similarity threshold according to another embodiment of the present disclosure.

[0027] FIG5 is a schematic diagram of the logical structure of an electronic device according to an embodiment of this specification. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0029] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0031] Image cutout technology is widely used in various fields. By identifying the foreground and background areas of the image to be cutout, and then replacing the background area with the user's desired background, special effects images or images in specific scenes can be obtained without the need for additional dedicated scenes.

[0032] The key to the cutout technology is to accurately identify the foreground and background areas from the image to be cutout. For any image to be cutout I, the subject part can be called the foreground F, and the rest is the background B. Then the image to be cutout I can be regarded as a weighted fusion of F and B: I = Alpha*F+(1-Alpha)*B, where Alpha is a continuous value between [0, 1], which can be understood as the probability that each pixel belongs to the foreground, or the opacity of the foreground image (that is, the higher the probability that the pixel belongs to the foreground, the more opaque it is). The task of cutout is to find the appropriate weight Alpha for each pixel, that is, to obtain an Alpha image corresponding to the image to be cutout. The pixel value of each pixel in the Alpha image represents the probability that the pixel at the corresponding pixel position of the image to be cutout is the foreground. Then, the foreground area can be cut out from the image to be cutout based on the Alpha image.

[0033] At present, some cutout technologies can pre-determine the difference between the color of each pixel in the image to be cutout and the background color of the image to be cutout, and then determine the probability (i.e., Alpha value) that each pixel is the foreground of the image to be cutout based on the difference and the pre-set cutout parameters, so as to obtain the Alpha image corresponding to the image to be cutout, and perform cutout processing on the image to be cutout based on the Alpha image.

[0034] For example, in some scenarios, the difference between the color of each pixel in the image to be cut out and the background color of the image to be cut out can be represented by the distance between the color of each pixel and a preset reference color. The user can set a reference color representing the background color in a certain color space (i.e., a single color coordinate point), and then calculate the distance between the color of each pixel in the image to be cut out and the reference color. Based on the distance and the preset cutout parameters, the probability that each pixel in the image to be cut out is the foreground area (i.e., the Alpha value) is determined, and the Alpha image corresponding to the image to be cut out can be obtained, and then the cutout can be realized based on the Alpha image.

[0035] For example, the cutout parameters may include one or more of a similarity threshold (thres) and a smoothing coefficient (ratio) for measuring the similarity between the color of each pixel and the background color. The Alpha value of each pixel can be calculated using formula (1):

[0036] Wherein, d is the distance between the color of each pixel and the reference color, thres is the preset similarity threshold, and ratio is the preset smoothing coefficient.

[0037] Currently, as shown in FIG1, when setting the cutout parameters, it is generally set manually by the user. For example, the user can continuously adjust the values ​​of the cutout parameters (such as the similarity threshold and the smoothing coefficient). The cutout software can perform cutout processing on the cutout image based on the cutout parameters currently set by the user and display a preview image so that the user can observe the cutout effect with the naked eye and use the value when the cutout effect is better as the final cutout parameter. This method is cumbersome, time-consuming and labor-intensive. Moreover, when the value of the cutout parameter is close to the optimal value, the change in the cutout effect is subtle and difficult to be directly perceived by the naked eye. Therefore, the cutout parameters set in this way are often not the optimal cutout parameters.

[0038] Therefore, it is necessary to provide a solution that is more convenient and quick and can accurately determine the optimal or better clipping parameters.

[0039] Based on this, the embodiment of the present application provides a solution for automatically determining the cutout parameters. In order to automatically determine the optimal or better cutout parameters, the applicant designed a target optimization function. The target optimization function uses the cutout parameters as variables and one or more error functions used to measure the cutout errors caused by the cutout parameters as optimization targets. The target optimization function is then optimized based on the image to be cutout, thereby automatically determining the optimized value of the cutout parameters for subsequent cutout processing. Among them, considering that for a better cutout parameter, it should meet the following conditions as much as possible:

[0040] (1) Based on the cutout parameters, the foreground and background areas of the image to be cutout should be distinguished as accurately as possible, that is, the number of misidentified pixels should be as small as possible;

[0041] (2) In order to avoid the transition area between the foreground and background areas being too large, which would affect the overall effect of the image, the area of ​​the transition area determined based on the cutout parameters should also be as small as possible, that is, the number of pixels belonging to the transition area should be as small as possible;

[0042] (3) In addition, in order to avoid incompleteness or missing foreground areas, the pixels in the transition area should be as close to the background area as possible, that is, the cumulative value of the alpha value of all pixels in the transition area (that is, the probability that the pixel belongs to the foreground area) should be as small as possible.

[0043] When setting the optimization target, in an embodiment of the present application, one or more error functions for measuring the cutout errors caused by the cutout parameters can be set based on the above three consideration dimensions, and then one or more optimization targets can be designed based on the error function to optimize the cutout parameters.

[0044] Through the method provided in the embodiment of the present application, better or optimal cutout parameters can be automatically determined without the need for manual setting by the user, making the determination of the cutout parameters more convenient and quick, and the determined cutout parameters are more accurate, and the effect of obtaining the cutout image using the cutout parameters is better.

[0045] The method for determining the cutout parameters provided in the embodiments of the present application can be executed by various electronic devices, such as mobile phones, computers, various live broadcast devices, or cloud servers, etc. For example, the method for determining the cutout parameters can be integrated into a cutout APP or a cutout SDK. As long as the cutout APP or cutout SDK is installed on the device, the method for determining the cutout parameters can be implemented, and then the image can be automatically cutout based on the determined cutout parameters.

[0046] As shown in FIG2 , the method for determining the cutout parameters may include the following steps:

[0047] S202, obtaining the value range of each channel of the image to be cut out and the background color of the image to be cut out under a specified color model;

[0048] In step S202, the image to be cut out can be obtained, wherein the image to be cut out can be an image in various formats, for example, it can be an image in various formats such as RGB format, YUV format, LAB format, HSV format, etc., and the embodiment of the present application does not limit this.

[0049] Considering that the background color of the image to be cut out usually falls within a color range, the value ranges of each channel of the background color under a specified color model can be predetermined. The specified color model can be an RGB model, a YUV model, an HSV model, a LAB model, etc. For example, assuming the specified color model is the HSV model and the background color is green, the value ranges corresponding to the three channels H, S, and V can be set to: H∈[35°, 77°], [0.168, 1], [0.180, 1].

[0050] Among them, the value range of the background color in each channel is determined. Its purpose is to verify the division result of the cutout parameters based on the value range of the background color when the pixels in the image to be cutout are divided into the foreground area and the background area based on the cutout parameters, so as to determine the cutout error caused by the cutout parameters.

[0051] S204: Optimizing a pre-constructed target optimization function based on the image to be cutout and the value range to obtain an optimized value of a cutout parameter of the image to be cutout, wherein the cutout parameter is used to determine a probability that a pixel in the image to be cutout is a foreground area of ​​the image to be cutout, and the probability is determined based on the cutout parameter and a difference between a color of the pixel and a background color; the variable of the target optimization function is the cutout parameter, and the optimization objective of the target optimization function includes one or more of the following:

[0052] When dividing the pixels in the image to be cut out into the foreground area and the background area of ​​the image to be cut out based on the cutout parameters, the number of misclassified pixels is minimized;

[0053] Minimizing the number of pixel points in the image to be cutout that belong to the transition area between the foreground area and the background area determined based on the cutout parameters;

[0054] Minimizing the cumulative value of the probabilities of the pixels in the transition area determined based on the cutout parameters;

[0055] In step S204, after obtaining the value ranges of each channel of the image to be cut out and the background color under the specified color model, a pre-constructed target optimization function can be optimized based on the two to obtain optimized values ​​of the cutout parameters of the image to be cut out. The cutout parameters can be used to determine the probability that a pixel in the image to be cut out is in the foreground area of ​​the image to be cut out, combined with the difference between the color of the pixel in the image to be cut out and the background color. For example, the cutout parameters can be one or more of a similarity threshold and a smoothing coefficient.

[0056] The pre-built target optimization function can use the cutout parameter as a variable and one or more of the following as optimization objectives:

[0057] (1) When dividing the pixels in the image to be cut out into foreground and background regions based on the cutout parameters, the number of misclassified pixels is minimized. This optimization objective is used to measure the accuracy of the foreground and background determined based on the cutout parameters. By constraining the cutout parameters through this optimization objective, the foreground and background regions determined based on the cutout parameters can be made more accurate.

[0058] (2) Minimize the number of pixels in the transition area determined based on the cutout parameters. The transition area is the area between the foreground area and the background area in the image to be cutout. Usually, in order to ensure that the edge of the foreground area obtained by cutout can transition smoothly and more naturally, a transition area is generally set between the foreground area and the background area. Obviously, the area of ​​the transition area cannot be too large, otherwise it will affect the display effect of the entire image. Therefore, by setting this optimization goal to constrain the cutout parameters, the area of ​​the transition area can be prevented from being too large.

[0059] (3) The cumulative value of the probability that each pixel in the transition area belongs to the foreground area, determined based on the cutout parameters, is minimized. As previously mentioned, in order to ensure the integrity and non-missing nature of the foreground area, the transition area should be as close to the background area as possible. Therefore, by constraining the cumulative value of the above-mentioned probabilities of the pixels in the transition area, we can ensure that the foreground area obtained by cutout is relatively complete and free of serious missing parts.

[0060] Among them, when setting the optimization target of the target optimization function, one or more of the above optimization targets can be selected, and then the target optimization function is optimized based on the set optimization target, that is, better or optimal cutout parameters can be obtained.

[0061] In addition, in some embodiments, multiple optimization goals can also be set to constrain the cutout parameters from multiple dimensions, so that the cutout parameters ultimately automatically determined are more accurate, and cutout processing based on the cutout parameters can achieve better cutout effects.

[0062] In some embodiments, the cutout parameter may be one or more of a similarity threshold and a smoothing coefficient. For example, the smoothing coefficient may be set to a certain empirical value, and then only the similarity threshold is optimized, or both the similarity threshold and the smoothing coefficient may be optimized simultaneously.

[0063] Among them, when the difference between the color of the pixel point of the image to be cut out and the background color is less than the similarity threshold, the probability that the pixel point is the foreground area is 0. When the difference between the color of the pixel point of the image to be cut out and the background color and the similarity threshold is greater than 0 and less than or equal to 1, the probability that the pixel point is the foreground area is positively correlated with the difference between the difference and the similarity threshold. For example, the larger the difference, the greater the probability; when the difference between the color of the pixel point of the image to be cut out and the background color and the similarity threshold is greater than 1, the probability that the pixel point is the foreground area is the smoothing coefficient.

[0064] In some embodiments, when optimizing a pre-constructed target optimization function based on the image to be cut out and the above-mentioned value range to obtain the optimized value of the cutout parameter, the optimal cutout parameter can be determined through multiple iterations. For example, the initial value of the cutout parameter can be set, and then the difference between each pixel in the image to be cut out and the background color can be determined. Then, based on the difference and the initial value of the cutout parameter, the probability that each pixel in the image to be cut out belongs to the foreground area is determined, and based on the probability and the value range of the background color in each channel, the number of misclassified pixels is determined, the number of pixels belonging to the transition area is determined based on the probability, and the cumulative value of the above-mentioned probabilities of each pixel in the transition area is determined based on the probability, i.e., the values ​​of the above-mentioned three optimization objectives under the current cutout parameters. Then, the initial value of the cutout parameter can be updated based on the value of one or more of the above-mentioned three optimization objectives under the current cutout parameters, and the updated value is used as the current value of the cutout parameter, and the numerical values ​​corresponding to the above-mentioned three optimization objectives are determined again based on the current value.

[0065] Repeat the above iterative process, continuously update the current value of the cutout parameter, and replace the current value with the updated value until the preset condition for stopping the iteration is reached. The current value of the cutout parameter obtained at this time is used as the optimized value for subsequent cutout processing.

[0066] In some embodiments, if the target optimization function includes multiple optimization objectives, the target optimization function can be optimized by adopting a multi-objective gradient descent algorithm to find a Pareto optimal solution as the optimized value of the cutout parameter. Taking into account that the values ​​of different optimization objectives in the above three optimization objectives are quite different, if multiple optimization objectives are directly superimposed and the cutout parameters are optimized based on the superposition results, there may be some optimization objectives whose values ​​are much smaller than the values ​​of other optimization objectives, and the constraint effect of the optimization objective on the cutout parameters may be masked. Based on this, in the embodiment of the present application, a multi-objective gradient descent algorithm can be used to solve the target optimization function, so that in the optimization process, the influence of each optimization objective on the cutout parameters can be taken into account, so that the optimized value of the cutout parameter finally determined is more accurate, and the cutout effect based on the cutout parameter is also better.

[0067] In some embodiments, the misclassified pixels may include the following: one is that the pixel points that originally belong to the background are classified into the foreground area, and the other is that the pixel points that originally belong to the foreground are classified into the background area. Since the value range of the background color in each channel of the specified color model has been predetermined, it is possible to roughly determine whether each pixel point in the image to be cut out is the foreground or the background based on the value range. Therefore, the misclassified pixel points can be determined in combination with the value range of the background color in each channel. For example, the misclassified pixel points may include one or more of the following: (1) The pixel values ​​of each channel are all within the value range of the background color in the corresponding channel, and the pixel points are classified into the foreground area. If the pixel values ​​of a certain pixel point in each channel are all within the value range of the background color corresponding channel, then the pixel point can be considered to be the background. If at this time, the pixel point is determined to be the foreground based on the current cutout parameters, then the pixel point is considered to be a misclassified pixel point.

[0068] For example, assuming that the background color is green and is represented by the HSV model, the value ranges corresponding to its H, S, and V channels can be set to: H∈[35°, 77°], [0.168, 1], [0.180, 1]. Suppose a pixel point P1 is (H=40°, S=0.2, V=0.2), and the channel values ​​of its H, S, and V channels are all within the value range of the channels corresponding to the background color. Therefore, P1 is considered to be the background. If P1 is determined to be the foreground based on the cutout parameters at this time, it means that P1 is a misclassified pixel point.

[0069] (2) There are pixels whose pixel values ​​in any channel are outside the range of the background color in the corresponding channel and are classified as background. If a pixel has a pixel value in any channel outside the range of the background color in the corresponding channel, the pixel is considered to be foreground. If the pixel is determined to be background based on the current cutout parameters, the pixel is considered to be misclassified.

[0070] Similarly, assuming that the background color is green and is represented by the HSV model, the value ranges corresponding to its H, S, and V channels can be set to: H∈[35°, 77°], [0.168, 1], [0.180, 1]. A certain pixel point P2 is (H=10°, S=0.2, V=0.2). The channel value of the H channel is not within the value range of the H channel of the background color, so P2 is determined to be the foreground. If P2 is determined to be the background based on the cutout parameters at this time, it means that P2 is a misclassified pixel point.

[0071] In some embodiments, the background color can be set by the user. For example, when the user sets the value range of each channel of the background color in a specified color model, the user can directly enter the value range of each channel based on the background to be cut out, or the user can directly specify the background color, such as red, green, or blue, and then the device executing the image processing method can automatically generate the value range of each channel based on the color selected by the user. Alternatively, in some scenarios, in order to facilitate user settings, the user can also directly select an area in the background of the image to be cut out, and then the device executing the method can automatically determine the value range of each channel based on the color corresponding to the area selected by the user.

[0072] Considering that in the HSV color space or the LAB color space, colors are more intuitive and easier to distinguish, while the distribution range corresponding to the background color usually needs to be set manually by the user, in some embodiments, the background color can be set in the HSV color space or the LAB color space, that is, the specified color model can be the HSV model or the LAB model.

[0073] In some embodiments, the difference between the color of each pixel of the image to be cut out and the background color of the image to be cut out can be represented by the distance between the color of each pixel and a reference color. For example, taking the background color as green, the reference color can be set to (R=0, G=255, B=0), and then the distance between the color of each pixel in the image to be cut out (for example, R=20, G=30, B=40) and the reference color can be calculated. In essence, the distance between the three-dimensional point (0, 255, 0) and the three-dimensional point (20, 30, 40) can be determined.

[0074] Considering that the distribution of background color is usually irregular, for example, if the color is represented in two-dimensional space, the distribution range of the background color is usually polygonal, and if the color is represented in three-dimensional space, the distribution range of the background color is usually a polyhedron. If the alpha value of each pixel is determined based on the distance between each pixel and the reference color and a pre-set threshold, it is equivalent to defining the distribution range of the background color by the reference color (equivalent to the center of the circle or the center of the sphere) and the threshold (equivalent to the radius). This distribution range is circular or spherical, while the distribution range of the background color is usually an irregular polygon or a polygon. The distribution range defined in the above manner cannot accurately cover the distribution range of the background color. Therefore, if the threshold is set too low, the background cannot be completely removed (as shown in Figure 3, where the color is represented in two-dimensional space as an example), and if the threshold is set too high, some foreground areas that are close to the background color will also be removed (as shown in Figure 4, where the color is represented in two-dimensional space as an example). In order to more accurately define the distribution range of the background color, the present application can pre-determine a high-dimensional hyperplane, use the hyperplane to define the distribution range of the background color (that is, as the classification boundary between background color and non-background color), and then determine the distance between the pixel points in the image to be cutout and the hyperplane, and determine the probability of each pixel point being the foreground area based on the distance (that is, the Alpha value).

[0075] Compared to defining the range of background color by using a circular or spherical area determined by a reference color and a preset single threshold, the embodiment of the present application can use a high-dimensional hyperplane and threshold to define the range of background color, so that the defined background color range is more accurate and can fully cover the distribution range of the background color, and thus the foreground area and background area extracted are also more accurate, resulting in a cutout image with a natural transition.

[0076] Therefore, in some embodiments, the difference between the color of each pixel in the image to be cut out and the background color is represented by the distance between the color of each pixel and a preset hyperplane, where the hyperplane is the classification boundary between the background color and the non-background color of the image to be cut out.

[0077] In some embodiments, when determining the hyperplane used to represent the classification boundary of the background color and non-background color of the image to be cutout, a plurality of sample pixels carrying labels (hereinafter referred to as first sample pixels for ease of distinction) can be obtained, wherein the label of each first sample pixel is used to indicate whether the color of the first sample pixel is the background color. The plurality of first sample pixels can then be used as training samples to train a preset support vector machine model, and the first sample pixel is classified by the support vector machine model to determine whether its color is the background color. Then, based on the difference between the predicted result output by the support vector machine model and the actual result indicated by the label, the model parameters of the support vector machine model are adjusted until the accuracy of the color classification result of the pixel by the support vector machine is higher than the preset accuracy (for example, 95%), and then the trained support vector machine model is used as the hyperplane. In order to ensure the accuracy of the trained hyperplane, the first sample pixel can include positive samples and negative samples to ensure the diversity of the samples.

[0078] Among them, the support vector machine model is the expression of the hyperplane (for example, Y=w*X+b), and the training process is the process of determining the parameters w and b in the expression, so that the final hyperplane can accurately classify background colors and non-background colors.

[0079] In some embodiments, the first sample pixel point can be represented by a first color model. When obtaining the first sample pixel point, sampling can be performed at intervals within the value range of the pixel value of each channel of the first color model to obtain multiple sample values ​​of the channel. The sample values ​​of each channel are then combined to obtain multiple first sample pixel points. For each first sample pixel point, it can be determined whether the color of the first sample pixel point is the background color, and the determination result is used as the label of the first sample pixel point.

[0080] For example, taking the RGB model as the first color model, since each pixel in the RGB model needs to be represented by the pixel values ​​of the three channels R, G, and B, and the pixel values ​​of the three channels range from 0 to 255, the R channel can be sampled at intervals between 0 and 255. For example, a sample value is obtained every 5 samples, that is, 0, 5, 10, 15... and so on. Similarly, for the G channel and the B channel, the same method is used to sample at intervals between 0 and 255 to obtain their respective sample values. Then, the sample values ​​of the three channels R, G, and B are combined. For example, R=5, G=5, and B=5 are combined to obtain a first sample pixel, that is, (R=5, G=5, B=5). Then, it can be determined whether the color of the sample pixel is the background color to obtain the label of the first sample pixel.

[0081] Of course, if the first color model is an HSV model, a LAB model or a YUV model, a similar sampling method can also be used to obtain a plurality of first sample pixel points.

[0082] In order to ensure that the trained hyperplane has high accuracy, the interval step size can be smaller during interval sampling to cover as many pixels as possible.

[0083] Typically, the background color is a color distribution range. For example, when a certain color model (such as the RGB or HSV model) is used to represent the background color, the values ​​of each channel of the color model are generally not fixed values, but rather a range of values. Therefore, in some embodiments, when determining whether the color of a first sample pixel is the background color, it can be determined whether the pixel values ​​of each channel of the first sample pixel are all within the value range of the background color within the corresponding channel. If so, the color of the first sample pixel is determined to be the background color.

[0084] For example, assuming that the background color and the first sample pixel are both represented by the HSV model, and the background color is green, then the value ranges corresponding to its H, S, and V channels can be set to: H∈[35°, 77°], [0.168, 1], [0.180, 1], respectively. Assuming that a first sample pixel P1 is (H°=40,S=0.2,V=0.2), the pixel values ​​of its H, S, and V channels are all within the value ranges of the channels corresponding to the background color, so the color of P1 is determined to be the background color. However, a first sample pixel P2 is (H=10°,S=0.2,V=0.2), and the pixel value of its H channel is not within the value range of the H channel of the background color, so the color of P2 is determined to be a non-background color.

[0085] In some embodiments, the background color can be represented by a second color model, wherein the second color model can be the same as the first color model, for example, both are RGB models, or different, for example, the background color is represented by the HSV model and the first sample pixel is represented by the RGB model. For scenes where the first color model and the second color model are different, before determining whether the color of the first sample pixel is the background color, the color model representing the sample pixel and the color model representing the background color can be unified before making a determination. For example, taking the case where the background color is represented by the HSV model and the first sample pixel is represented by the RGB model, the background color can be converted from the HSV format to the RGB format, or the first sample pixel can be converted from the RGB format to the HSV format, to facilitate subsequent pixel value comparison.

[0086] Considering that colors in the HSV color space or LAB color space are more intuitive and easier to distinguish, while the distribution range corresponding to the background color usually requires manual setting by the user, in some embodiments, the background color can be set in the HSV color space or LAB color space. When sampling to obtain sample pixels, sampling can be performed in the RGB color space. Therefore, the first color model mentioned above can be an RGB model, and the second color model mentioned above can be an HSV model or a LAB model.

[0087] In some embodiments, assuming that the background color is represented by a second color model, when the user sets the value range of each channel of the background color in the second color model, the user can directly enter the value range of each channel based on the background to be cut out, or the user can directly specify the background color, such as red, green, or blue, and then the device executing the image processing method can automatically generate the value range of each channel based on the color selected by the user. Alternatively, in some scenarios, to facilitate user settings, the user can also directly select an area in the background of the image to be cut out, and then the device executing the image processing method can automatically determine the value range of each channel based on the color corresponding to the area selected by the user.

[0088] In some embodiments, when determining the distance between the color coordinates of each pixel in the image to be cutout and the hyperplane, the color coordinates of each pixel can be substituted into the expression of the hyperplane to obtain the distance between the color coordinates of the pixel and the hyperplane. This method is computationally intensive and time-consuming, making it unsuitable for scenarios such as live broadcasts where real-time cutout performance is critical.

[0089] In order to reduce the amount of calculation and improve the efficiency of the cutout, in some embodiments, a distance map can be first determined based on the hyperplane and a plurality of preset sample pixel points (hereinafter referred to as second sample pixel points), and the distance map is used to indicate the distance between the color coordinates of each of the plurality of sample pixel points and the hyperplane. When determining the distance between the color coordinates of each pixel point in the image to be cutout and the hyperplane, it is not necessary to substitute the color coordinates of each pixel point into the expression of the hyperplane for calculation. Instead, for the pixel point whose color coordinates in the image to be cutout are the same as the color coordinates of the second sample image pixel point, the distance corresponding to the pixel point can be directly found from the distance map. For the pixel point whose color coordinates in the image to be cutout are different from the color coordinates of the second sample image pixel point, a plurality of second sample pixel points whose color coordinates are relatively close to the color coordinates of the pixel point can be selected, and the distances of these second sample pixel points are interpolated to obtain the distance of the pixel point.

[0090] By constructing a distance map based on the hyperplane in advance, the distance from the color coordinates of each pixel in the image to be cut out to the hyperplane can be determined by table interpolation, which can reduce the amount of calculation, improve processing efficiency, and ensure the real-time performance of the cutout processing.

[0091] Among them, for the same background color (that is, the value ranges of each channel corresponding to the background color are the same, that is, they can be considered to be the same background color), a distance map can be determined. Later, during the cutout process, for all images with the same background color, this distance map can be used to determine the distance between the color coordinates of the pixel points in the image and the hyperplane. For example, for a green background, a distance map can be determined, and all subsequent images to be cutout with a green background can use this distance map to calculate the distance.

[0092] In some embodiments, the first sample pixels and the second image pixels may be the same batch of pixels. For example, when obtaining sample pixels for training a hyperplane, these sample pixels and the trained hyperplane may be used to generate a distance map. In some embodiments, the first sample pixels and the second image pixels may also be different pixels.

[0093] In some embodiments, in order to ensure that the second sample pixel points can be evenly distributed in the color space and facilitate subsequent interpolation, when determining the second sample pixel points, they can also be determined by an interval sampling method. Taking the representation of the multiple second sample pixel points by the first color model (for example, the RGB model) as an example, when determining the multiple second sample pixel points, for each channel of the first color model, interval sampling can be performed within the value range of the pixel value of the channel to obtain multiple sampling values ​​of the channel, and then the sampling values ​​of each channel are combined to obtain multiple second sample pixel points. Among them, the specific interval sampling method can be referred to the description in the above embodiment and will not be repeated here.

[0094] In some embodiments, when calculating the distance between the color coordinates of a pixel point of the image to be cut out and the hyperplane, the pixel values ​​of all channels of the pixel point can be directly used to form the color coordinates, and the distance between the color coordinates and the hyperplane can be calculated. For example, if the image to be cut out is an image in RGB format, assuming that the pixel values ​​of the three channels R, G, and B of a certain pixel point are respectively: R=10, G=20, B=30, then the distance from the three-dimensional point (10, 20, 30) to the hyperplane can be calculated. If the image to be cut out is an image in HSV format, assuming that the pixel values ​​of the three channels H, S, and V of a certain pixel point are respectively: H=10°, S=0.2, V=0.3, then the distance from the three-dimensional point (10, 20, 10) to the hyperplane can be calculated.

[0095] Taking into account that for the image to be cut out represented by certain color models, some channels in the color model are channels that are not related to color, and when calculating the distance, the main goal is to determine the degree of proximity between the color of the pixel point and the background color. Therefore, only the pixel values ​​of the channels related to color will affect the degree of proximity, while the pixel values ​​of the channels that are not related to color will not affect the degree of proximity. Therefore, in some embodiments, in order to reduce the amount of calculation and improve processing efficiency, for the pixel points in the image to be cut out, the color coordinates of the pixel points can be the coordinates composed of the pixel values ​​of the target channel, and the distance between the pixel point and the hyperplane can be the distance between the coordinates composed of the pixel values ​​of the target channel of the pixel point and the hyperplane, wherein the target channel is a channel related to color.

[0096] For example, if the image to be cut out is in YUV format, the target channels are U and V channels. Assuming that the pixel values ​​of the Y, U, and V channels of a certain pixel point are: Y=10, U=20, V=30, then the distance from the two-dimensional point (20, 30) to the hyperplane can be calculated as the distance from the pixel point to the hyperplane.

[0097] If the image to be cut out is an image in HSV format, the target channels are the H and S channels; assuming that the pixel values ​​of the H, S, and V channels of a certain pixel point are: H = 10°, S = 0.2, V = 0.3, then the distance from the two-dimensional point (10, 0.2) to the hyperplane can be calculated as the distance from the pixel point to the hyperplane.

[0098] If the image to be cut out is an image in LAB format, the target channels are A and B channels. Assuming that the pixel values ​​of the L, A, and B channels of a certain pixel point are: L=10, A=20, B=30, then the distance from the two-dimensional point (20, 30) to the hyperplane can be calculated as the distance from the pixel point to the hyperplane.

[0099] If the image to be cut out is an RGB format image, the target channels are R, G, and B channels. Assuming that the pixel values ​​of the R, G, and B channels of a certain pixel point are: R=10, G=20, B=30, the distance from the three-dimensional point (10, 20, 10) to the hyperplane can be calculated.

[0100] By adopting the above method, for scenes where the image to be cutout is in HSV, YUV, or LAB format, when calculating the distance, it is only necessary to calculate the distance from the two-dimensional point to the hyperplane. Compared with taking all pixel values ​​into account, that is, calculating the distance from the three-dimensional point to the hyperplane, this calculation method can greatly reduce the amount of calculation without affecting the final cutout effect.

[0101] Furthermore, an embodiment of the present application also provides an image processing method, the method comprising:

[0102] Get the image to be cut out;

[0103] Determine the difference between each pixel in the image to be cutout and the background color of the image to be cutout;

[0104] Based on the difference and the optimized value of the predetermined cutout parameter, the probability that each pixel point in the image to be cutout belongs to the foreground of the image to be cutout is determined, so as to perform cutout processing on the image to be cutout based on the probability, wherein the optimized value can be obtained based on the method for determining the cutout parameter introduced in any of the above embodiments.

[0105] The specific details of the image processing method can be referred to the introduction in the above embodiment and will not be repeated here.

[0106] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0107] Corresponding to the method embodiment for determining the cutout parameters provided in the embodiment of the present application, the embodiment of the present application further provides an apparatus for determining the cutout parameters, the apparatus comprising:

[0108] An acquisition module is used to acquire the value range of each channel of the image to be cut out and the background color of the image to be cut out under a specified color model;

[0109] An optimization module is configured to optimize a pre-constructed target optimization function based on the image to be cutout and the value range to obtain an optimized value of a cutout parameter of the image to be cutout, wherein the cutout parameter is used to determine a probability that a pixel in the image to be cutout is a foreground area of ​​the image to be cutout, and the probability is determined based on the cutout parameter and a difference between the color of the pixel and the background color; the variable of the target optimization function is the cutout parameter, and the optimization objective of the target optimization function includes one or more of the following:

[0110] When dividing the pixels in the image to be cut out into the foreground area and the background area of ​​the image to be cut out based on the cutout parameters, the number of misclassified pixels is minimized;

[0111] Minimizing the number of pixel points in the image to be cutout that belong to the transition area between the foreground area and the background area determined based on the cutout parameters;

[0112] The accumulated value of the probabilities of the pixels in the transition area determined based on the cutout parameters is minimized.

[0113] Among them, the specific implementation of the processing performed when determining the cutout parameters in the above method of determining the cutout parameters can be referred to the description in the above method embodiment, and will not be repeated here.

[0114] The present application also provides an electronic device, as shown in FIG5 , which is a hardware structure diagram of an electronic device according to an embodiment of the present specification. In addition to the processor 52 and memory 54 shown in FIG5 , the device may also include other hardware, such as a forwarding chip responsible for processing messages. In terms of hardware structure, the device may also be a distributed device, possibly including multiple interface cards, to expand message processing at the hardware level. The memory 54 stores computer instructions, and when the processor 52 executes the computer instructions, it implements the method for determining the cutout parameters and / or the image processing method mentioned in any of the above embodiments.

[0115] Accordingly, an embodiment of the present application further provides a computer storage medium, in which a program is stored. When the program is executed by a processor, the method for determining cutout parameters and / or the image processing method in any of the above embodiments is implemented.

[0116] The embodiments of the present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0117] Those skilled in the art will readily recognize other implementations of the embodiments of the present invention after considering the specification and practicing the instructions disclosed herein. The embodiments of the present invention are intended to cover any variations, uses, or adaptations of the embodiments of the present invention that follow the general principles of the embodiments of the present invention and include common knowledge or customary techniques in the art not disclosed in the embodiments of the present invention. The description and examples are to be considered as exemplary only, and the true scope and spirit of the embodiments of the present invention are indicated by the following claims.

[0118] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

[0119] The above description is only a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the scope of protection of the embodiments of this specification.

Claims

1. A method for determining matte parameters, characterized in that, the method includes: obtaining the image to be matted and the value ranges of each channel of the background color of the image to be matted in a specified color model; optimizing a pre-constructed target optimization function based on the image to be matted and the value ranges to obtain an optimized value of the matte parameters of the image to be matted, wherein the matte parameters are used to determine the probability that a pixel point in the image to be matted is the foreground region of the image to be matted, and the probability is determined based on the matte parameters and the difference between the color of the pixel point and the background color; the variable of the target optimization function is the matte parameters, and the optimization objectives of the target optimization function include one or more of the following: when dividing the pixel points in the image to be matted into the foreground region and the background region of the image to be matted based on the matte parameters, minimizing the number of misclassified pixel points; minimizing the number of pixel points in the image to be matted that belong to the transition region between the foreground region and the background region determined based on the matte parameters; minimizing the cumulative value of the probabilities of the pixel points in the transition region determined based on the matte parameters.

2. The method according to claim 1, characterized in that, the misclassified pixel points include: pixel points whose pixel values of each channel are all within the value range of the background color in the corresponding channel and are classified into the foreground region; and / or pixel points whose pixel value of any one channel is outside the value range of the background color in the corresponding channel and are classified into the background region.

3. The method according to claim 2, characterized in that, the value ranges of each channel of the background color in the specified color model are determined based on the following methods: determined based on the numerical range input by the user; or determined based on the color selected by the user; or determined based on the color corresponding to the area boxed by the user in the image to be matted.

4. The method according to claim 1, characterized in that, the optimizing a pre-constructed target optimization function based on the image to be matted and the value ranges to obtain an optimized value of the matte parameters of the image to be matted includes: repeating the following iterative process until a preset condition for stopping iteration is reached, and taking the determined current value of the matte parameters as the optimized value: determining the difference between the color of each pixel point in the image to be matted and the background color; determining the probability that each pixel point belongs to the foreground region based on the difference and the current value of the matte parameters; determining the number of misclassified pixel points, the number of pixel points belonging to the transition region, and / or the cumulative value of the probabilities of the pixel points in the transition region based on the probability and the value ranges; updating the current value based on the number of misclassified pixel points, the number of pixel points belonging to the transition region, and / or the cumulative value to obtain an updated value; using the updated value to replace the current value.

5. The method according to claim 1, characterized in that, The matte extraction parameters include a similarity threshold and / or a smoothing coefficient; wherein, when the difference is less than the similarity threshold, the probability is 0; when the difference between the difference and the similarity threshold is greater than 0 and less than or equal to 1, the probability is positively correlated with the difference between the difference and the similarity threshold; when the difference between the difference and the similarity threshold is greater than 1, the probability is the smoothing coefficient.

6. The method according to claim 1, wherein, the optimized value is obtained by solving the target optimization function by using a multi-objective gradient descent algorithm.

7. The method according to claim 1, wherein, the difference between the color of each pixel point and the background color is characterized by the distance between the color of each pixel point and a reference color; or the difference between the color of each pixel point and the background color is characterized by the distance between the color of each pixel point and a preset hyperplane, where the hyperplane is the classification boundary between the background color and the non-background color of the image to be matte-extracted.

8. The method according to claim 7, wherein, the hyperplane is determined based on the following method: obtain a plurality of first sample pixel points with labels, where the label is used to indicate whether the color of the first sample pixel point is the background color; train a preset support vector machine model by using the first sample pixel points, and use the trained model as the hyperplane.

9. The method according to claim 1, wherein, the specified color model includes an HSV model or an LAB model.

10. An image processing method, wherein, the method includes: obtain an image to be matte-extracted; determine the difference between the color of each pixel point in the image to be matte-extracted and the background color of the image to be matte-extracted; determine the probability that each pixel point in the image to be matte-extracted belongs to the foreground of the image to be matte-extracted based on the difference and the optimized value of the matte extraction parameters determined in advance, so as to perform matte extraction processing on the image to be matte-extracted based on the probability, where the optimized value is obtained by the method for determining matte extraction parameters according to any one of claims 1-9.

11. An electronic device, wherein, the electronic device includes a processor, a memory, and computer instructions stored on the memory, and when the processor executes the computer instructions, the method according to any one of claims 1-10 is implemented.

12. A computer-readable storage medium, wherein, a computer program is stored on the computer-readable storage medium, and when the computer program is executed, the method according to any one of claims 1-10 is implemented.

Images