Complex background image matting method based on deep learning semantic segmentation

Abstract

The present invention belongs to the field of image processing technology, and particularly to a complex background image matting method based on deep learning semantic segmentation. The method uses the Tensorflow deep learning framework to perform semantic segmentation on the input image, accurately identify all pixel points representing the portrait in the image, and thus obtain the edge contour of the portrait. The Canny operator edge detection is performed on the original image to find the portrait contour. Based on the minimum distance from the contour point to the portrait edge contour and the distance from the facial feature point, it is determined whether the contour point belongs to an unknown area point. Finally, the foreground background point α is sampled by the neighborhood of the unknown area. p Estimation algorithm to calculate the transparency α of the unknown area p .

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a complex background image matting method based on deep learning semantic segmentation. Background Art

[0002] Image matting is to calculate the transparency of each pixel in the image. In the traditional visual field, image threshold segmentation, cluster segmentation, region segmentation and other algorithms are generally used. After the portrait is segmented, the image matting algorithm is combined to complete the image matting operation.

[0003] Traditional visual segmentation and matting are very strict about the background color of the image, requiring the background to be pure color. It is not possible to accurately segment the portrait for uneven lighting and mottled backgrounds. Especially when users use mobile phones to shoot, traditional visual segmentation cannot accurately segment the portrait area. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a complex background image matting method based on deep learning semantic segmentation.

[0005] The present invention is achieved through the following technical solutions:

[0006] A complex background image matting method based on deep learning semantic segmentation, comprising the steps of:

[0007] A. Based on the Tensorflow deep learning framework, the input image is semantically segmented to accurately identify all the pixels representing the portrait in the image, thereby obtaining the edge contour of the portrait;

[0008] B. Performing Canny operator edge detection on the input image to find the portrait contour, and determining whether the contour point belongs to an unknown area point based on the minimum distance from the contour point to the portrait edge contour and the distance from the facial feature point;

[0009] C. Sampling the background point α in the unknown area neighborhood p Estimation algorithm to calculate the transparency α of the unknown area p ; Based on transparency α p Replace the original background and output the final image.

[0010] Furthermore, in step A, the steps are specifically included:

[0011] S1. Perform semantic segmentation on the input image based on Tensorflow, obtain the known areas of the image, namely the portrait area and the background area, and output a binary image;

[0012] S2, performing an image corrosion morphological operation on the binary image, and outputting a corrosion map;

[0013] S3, performing contour search on the erosion map, obtaining all the contour points of the portrait edge, saving all the contour points to a linked list, marking all the contour points as unknown areas, filling the contour points, and outputting a first image.

[0014] Furthermore, in step S1, based on the Tensorflow framework, the deeplabv3 model is used for segmentation, which can accurately segment the portrait pixels.

[0015] Furthermore, in step S2, the erosion morphological operation uses a 3x3 convolution kernel to scan the binary image row by row, and performs an "or" operation on all pixel points in the convolution kernel. If the calculation result is 0, the point value is 0, otherwise it is 1.

[0016] Furthermore, in step B, the steps are specifically included:

[0017] S4, using the Canny operator to perform edge detection on the input image, and setting its low threshold and high threshold to 50 and 80 respectively, and outputting a second image;

[0018] S5, performing face detection on the input image, determining a face rectangular area, and extracting face feature points;

[0019] S6. Find all the contour points of the second image, traverse the contour points, calculate the minimum Euclidean distance from each point to the contour points in the linked list, and when the value of the distance is less than the set threshold, the point is determined to be an unknown area point and the transparency needs to be recalculated, otherwise it is a portrait point; at the same time, calculate the minimum Euclidean distance from each point to the left and right cheeks of the facial feature points. The smaller the value, the closer it is to the portrait, and the smaller the unknown area calibration range is, and the third image is output.

[0020] Further, in step C, the steps are included:

[0021] S7, performing unknown region alpha estimation on the third image, and outputting a fourth image;

[0022] S8, performing alpha correction on the fourth image;

[0023] S9, using the alpha blending formula P(dst) = (1-α p )*P(d)+P(s)*α p , replace the original background and output the final image, where P(dst) represents the output value, P(d1) represents the pixel value to be replaced, and P(s) represents the original pixel value.

[0024] Furthermore, the background point α before sampling in the unknown area neighborhood p Estimation can be divided into four steps:

[0025] S7.1, expanding the known area to reduce the number of pixels in the unknown area;

[0026] S7.2, for each point in the unknown area, sample according to a predetermined rule and select the best pair of foreground and background points; estimate the α of the current point based on the obtained color values ​​of the foreground point, background point and current point p ;

[0027] α p The calculation formula is as follows:

[0028]

[0029] Where C is the current point, F is the foreground point, and B is the background point;

[0030] S7.3, recalculate the foreground, background, transparency and credibility, and the optimal sampling point pair can minimize color distortion;

[0031] S7.4. Locally smooth the foreground and background pairs and the transparent values ​​to reduce noise and determine the final α of each point p value.

[0032] Furthermore, in step S7.2, the sampling rule is to set a parameter k, which represents the maximum number of foreground points and background points that can be sampled by an unknown pixel point; the maximum number of point pairs that can be formed by the sampling points is k*k. For each unknown point, starting from the point, k paths are derived, and the angle between each path is 2π / k. The first foreground or background point encountered in the route passed by each path is recorded until it exceeds the edge of the image.

[0033] Further, in step S7.3, the color distortion can be expressed by the expression M p (Fi,Bj)=||C p -(α p .Fi+(1-α p ).Bj)|| represents; where C p is the color value of point p, α p is the transparency estimate of point p; calculate the optimal sampling point pair for all points in the neighborhood of pixel point p, and each pixel point p can get a set of (F, B, α p ) so that its corresponding M p Minimum, then choose to make M p The smallest k groups (F, B, α p ), use the average value of these k groups of data to calculate the new foreground, background, transparency and credibility.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a complex background image matting method based on deep learning semantic segmentation.

[0035] Compared with the prior art, the present invention has the following advantages: 1. It does not rely on input parameters and has no requirements on background color, brightness, uniformity, lighting, etc., and is suitable for complex backgrounds. 2. It retains the edge area of ​​the portrait as much as possible, and the edge loss is small after the background is changed, and details such as hair are retained. 3. The background can be changed to any color. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described in detail below with reference to the accompanying drawings;

[0037] Figure 1 (a)-(h) are the cutout effect diagrams of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Image matting is to calculate the transparency of each pixel in the image and identify the foreground and background pixels of the image. The image-to-background calculation can be done using equation C p =α p .F p +(1-α p ).B p Indicates. Where C p is the observed image color, F p , B p , α p are all unknown quantities, representing foreground, background, and transparency. To solve this equation, some additional constraints must be added, such as creating an initial image of the same size as the image to be segmented, with only three pixel values, namely 0, 128, and 255, where 0 represents the background, 128 represents the pixel to be calculated, and 255 represents the foreground. The main purpose of the algorithm is to use the similarity, spatial distance, and color distance between adjacent pixels in the image to calculate the transparency α of the unknown area. p .

[0040] The Tensorflow deep learning framework is used to perform semantic segmentation on the input image and accurately identify all the pixels representing the portrait in the image, thereby obtaining the edge contour of the portrait. The Canny operator is used to detect the edge of the original image to find the portrait contour. Based on the minimum distance from the contour point to the edge contour of the portrait and the distance from the facial feature point, it is determined whether the contour point belongs to the unknown area point. Finally, the foreground and background points α are sampled in the neighborhood of the unknown area. p Estimation algorithm to calculate the transparency α of the unknown area p . Different from traditional algorithms, it does not rely on input parameters, does not limit the background color, and has no requirements for image lighting and background color. The algorithm involves semantic segmentation, image morphological operations, edge detection, contour extraction, portrait edge positioning, skin color judgment, and face detection.

[0041] The algorithm processing flow of the present invention includes the following steps:

[0042] S1. Use Tensorflow to perform semantic segmentation on the input image (a), obtain the known areas of the image, namely the portrait area (255) and the background area (0), and output a binary image (b). This algorithm is based on the Tensorflow framework and uses the deeplabv3 model for segmentation, which can accurately segment the portrait pixels.

[0043] S2. Perform image erosion morphological operation on (b), retain the edge details of the image as much as possible, and output the erosion map (c). Erosion uses a 3x3 convolution kernel to scan the image line by line, and performs an "or" operation on all pixel points in the convolution kernel. If the calculation result is 0, the point value is 0, otherwise it is 1.

[0044] S3, perform contour search on image (c), obtain all the edge contour points of the portrait, save all the contour points to the linked list (contours_a), and mark all the contour points as unknown areas, that is, fill the contour points with a gray brush with a radius of 8, and output image (d).

[0045] S4. Use the Canny algorithm to perform edge detection on image (a), and set its low threshold and high threshold to 50 and 80 respectively, and output image (e).

[0046] S5. Perform face detection on image (a), determine the face rectangular area, and extract face feature points.

[0047] S6. Find all contour points (contours_b) of image (e), traverse the contour points, and calculate the minimum Euclidean distance from each point to the contour point of contours_a. When the value is less than the set threshold (max_distance), the point is determined to be an unknown area point and its transparency needs to be recalculated, otherwise it is a portrait point. At the same time, the minimum Euclidean distance from each point to the left and right cheeks of the facial feature points is calculated. The smaller the value, the closer it is to the portrait, and the smaller the unknown area calibration range is, and the output image (f).

[0048] S7, estimate the unknown region alpha of image (f) and output image (g). p Estimation can be divided into four steps:

[0049] S7.1. Expand the known area and reduce the number of pixels in the unknown area. For an unknown point p, if there is a known foreground point or background point q within a certain radius of its domain, calculate the Euclidean distance between p and q in RGB space. If its value is less than 5 / 256 and is closest to the point, point p is attributed to the area where q is located, reducing the number of pixels in the unknown area and the amount of calculation.

[0050] S7.2. For each point in the unknown area, sample according to certain rules and select the best pair of foreground and background points. The sampling rule is to set the parameter k, which represents the maximum number of foreground and background points that can be sampled for an unknown pixel. The maximum number of point pairs that can be formed by the sampling points is k*k. For each unknown point, starting from this point, k paths are derived, and the angle between each path is 2π / k. The first foreground or background point encountered in the route of each path is recorded until it exceeds the edge of the image. According to the obtained foreground point, background point and the color value of the current point, the α of the current point is estimated. p .

[0051] α p Calculation formula:

[0052]

[0053] Where C is the current point, F is the foreground point, B is the background point, and the subscripts r, g, and b represent the three primary colors. p If the value is reasonable, use α p The color recalculated by combining F / B should be very close to the original color. At the same time, within a certain neighborhood, since pixels generally do not have mutations, the average value of the difference should also be small. Finally, the best foreground and background point pair and the α corresponding to the unknown point are selected from the point pair set. p .

[0054] S7.3. Recalculate the foreground, background, transparency and credibility. The optimal sampling point pair should minimize the color distortion. The color distortion can be expressed as p (Fi,Bj)=||C p -(α p .Fi+(1-α p ).Bj)|| represents. Among them, C p is the color value of point p, α p is the estimated transparency value of point p. Calculate the optimal sampling point pair for all points in the neighborhood of pixel point p. Each pixel point p can get a set of (F, B, α p ) so that its corresponding M p Minimum, then choose to make M p The smallest k groups (F, B, α p ), use the average value of these k groups of data to calculate the new foreground, background, transparency and credibility.

[0055] S7.4. Locally smooth the foreground, background, and transparency values ​​to reduce noise and determine the final α of each point. p value.

[0056] S8. Perform alpha correction on the image (g). Traverse the original image, and correct the alpha according to the RGB pixel value of the point, the alpha value of the point, the minimum Euclidean distance and color distance from the point to the edge of the face, and output the final Alpha. The skin color calculation method can estimate the skin color range through the HSB color space and Ycbcr color respectively, and convert the RGB color to HSB and Ycbcr.

[0057] HSB skin color judgment formula: (h>7)&&(h<30)&&(s>=15)&&(s<256)&&(b>50)&&(b<256)

[0058] Ycbcr skin color judgment formula: (cr>=134.5)&&(cr<=173)&&(cb>=77)&&(cb<=127)

[0059] S9, using the alpha blending formula P(dst) = (1-α p )*P(d)+P(s)*α p , replace the original background, and output the final image (h). Where P(dst) represents the output value, P(d) represents the pixel value to be replaced, and P(s) represents the original pixel value. Figure 1 In the image, use white (R: 255, G: 255, B: 255) to replace the original background color.

[0060] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of a complex background image matting method based on deep learning semantic segmentation are implemented.

[0061] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a complex background image matting method based on deep learning semantic segmentation are implemented.

[0062] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention also fall within the scope of protection of the present invention.

Claims

1. A complex background image matting method based on deep learning semantic segmentation, characterized in that: Includes steps: A. Based on the Tensorflow deep learning framework, the input image is semantically segmented to accurately identify all the pixels representing the portrait in the image, thereby obtaining the edge contour of the portrait; B. Performing Canny operator edge detection on the input image to find the portrait contour, and determining whether the contour point belongs to an unknown area point based on the minimum distance from the contour point to the portrait edge contour and the distance from the facial feature point; C. Sampling the background point α in the unknown area neighborhood p Estimation algorithm to calculate the transparency α of the unknown area p ; Based on transparency α p Replace the original background and output the final image; In step A, the steps are specifically included: S1. Perform semantic segmentation on the input image based on Tensorflow, obtain the known areas of the image, namely the portrait area and the background area, and output a binary image; S2, performing an image corrosion morphological operation on the binary image, and outputting a corrosion map; S3, performing contour search on the erosion image, obtaining all the contour points of the portrait edge, saving all the contour points to a linked list, marking all the contour points as unknown areas, filling the contour points, and outputting a first image; In step B, the steps are specifically included: S4, using the Canny operator to perform edge detection on the input image, and setting its low threshold and high threshold to 50 and 80 respectively, and outputting a second image; S5, performing face detection on the input image, determining a face rectangular area, and extracting face feature points; S6. Find all the contour points of the second image, traverse the contour points, calculate the minimum Euclidean distance from each point to the contour points in the linked list, and when the value of the distance is less than the set threshold, the contour point is determined to be an unknown area point and the transparency needs to be recalculated, otherwise it is a portrait point; at the same time, calculate the minimum Euclidean distance from each point to the left and right cheeks of the facial feature points. The smaller the value, the closer it is to the portrait, and the smaller the unknown area calibration range is, and output the third image.

2. The complex background image matting method based on deep learning semantic segmentation according to claim 1, characterized in that: In step S1, based on the Tensorflow framework, the deeplabv3 model is used for segmentation, which can accurately segment the portrait pixels.

3. The complex background image matting method based on deep learning semantic segmentation according to claim 1, characterized in that: In step S2, the erosion morphological operation uses a 3x3 convolution kernel to scan the binary image line by line, and performs an "or" operation on all pixel points in the convolution kernel. If the calculation result is 0, the pixel value is 0, otherwise it is 1.

4. The complex background image matting method based on deep learning semantic segmentation according to claim 1, characterized in that: In step C, the steps include: S7, performing unknown region alpha estimation on the third image, and outputting a fourth image; S8, performing alpha correction on the fourth image; S9. Use alpha blending formula , replace the original background and output the final image, where P(dst) represents the output value, P(d) represents the pixel value to be replaced, and P(s) represents the original pixel value.

5. The complex background image matting method based on deep learning semantic segmentation according to claim 4, characterized in that: The background point α before sampling in the unknown area neighborhood p Estimation can be divided into four steps: S7.1, expanding the known area to reduce the number of pixels in the unknown area; S7.2, for each point in the unknown area, sample according to a predetermined rule and select the best pair of foreground and background points; estimate the α of the current point based on the obtained color values ​​of the foreground point, background point and current point p ; α p The calculation formula is as follows: ； Where C is the current point, F is the foreground point, B is the background point, and the subscripts r, g, and b refer to the three primary colors; S7.3, recalculate the foreground, background, transparency and credibility, and the optimal sampling point pair can minimize color distortion; S7.

4. Locally smooth the foreground, background and transparent values ​​to reduce noise and determine the final α of each point p value.

6. The complex background image matting method based on deep learning semantic segmentation according to claim 5, characterized in that: In step S7.2, the sampling rule is to set the parameter k, which represents the maximum number of foreground points and background points that can be sampled for an unknown pixel point; the maximum number of point pairs that can be formed by the sampling points is k * k; for each pixel point in the unknown area, starting from the pixel point, k paths are derived, and the angle between each path is 2π / k. The first foreground or background point encountered in the route of each path is recorded until it exceeds the edge of the image.

7. The complex background image matting method based on deep learning semantic segmentation according to claim 6, characterized in that: In step S7.3, the color distortion can be expressed as Indicates; where C p is the color value of point p, α p is the estimated transparency value at point p; Calculate the optimal sampling point pairs for all points in the neighborhood of pixel point p. Each pixel point p can obtain a set of (F, B, α p ) so that its corresponding M p Minimum, then choose to make M p The smallest k groups (F, B, α p ), use the average value of these k groups of data to calculate the new foreground, background, transparency and credibility.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the complex background image matting method based on deep learning semantic segmentation described in any one of claims 1-7 are implemented.

Images