Lipprint Removal Method, System, Device and Storage Medium Based on Image Processing

Abstract

The present invention belongs to the technical field of image processing, and particularly relates to a lip print removal method, system, device and storage medium based on image processing. The method includes: obtaining an original image to be retouched, and cropping a lip image based on an outer bounding box of lip key points; performing lip segmentation on the obtained original image to obtain a mask image marking the lip area, and performing lip print detection within the range of the mask image to obtain a single-channel lip print mask; dividing the lip image into a plurality of gradient blocks of the same size and performing lip print marking; replacing the gradient blocks of the repaired area marked with lip prints with gradient blocks of the area without lip prints, and obtaining a lip print removal image after Poisson reconstruction; performing median filtering on the lip image and the lip print removal image to obtain a result image retaining the color feature information of the original image, and performing local high-light recovery on the result image to obtain a final image after high-light recovery. The present invention realizes the retention of the texture of lip gloss while removing lip prints.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a lip wrinkle removal method, system, device and storage medium based on image processing. Background Art

[0002] In the field of photographic editing, when manually retouching images to deal with wrinkles, the high-frequency and low-frequency and neutral gray retouching methods are mostly used, and it is necessary to manually smear each wrinkle one by one. Different from other facial wrinkles, lip wrinkles are very complex and cumbersome to operate because the mouth area is small and the wrinkles are thin. At the same time, in many scenarios, the lips are coated with lipsticks of different materials (for example: matte, moisturizing), and it is very difficult to maintain the texture of the lip color while removing lip wrinkles.

[0003] At present, more and more end-to-end intelligent retouching solutions are replacing traditional manual retouching methods. When retouching images, a wrinkle removal method is adopted. This method mostly targets wrinkles in the skin area and does not deal with lip wrinkles, resulting in a lack of retouching solutions in the market that can automatically detect and remove lip wrinkles. Summary of the Invention

[0004] To solve the problem that the current retouching methods cannot remove lip wrinkles while maintaining the texture of the lip color, the present invention provides a lip wrinkle removal method, system, device and storage medium based on image processing, which processes the detected lip wrinkle area to remove lip wrinkles, and can retain the color, texture, and light and shadow features of the lips, and the removal effect is natural.

[0005] The present invention is realized by the following technical solutions:

[0006] A lip wrinkle removal method based on image processing, comprising the following steps:

[0007] Obtain the original image to be retouched, and crop the lip image based on the outer bounding box of the lip key points;

[0008] Perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip wrinkle detection within the range of the mask image to obtain a single-channel lip wrinkle mask;

[0009] Calculate the gradient of the lip image, divide the lip image into a number of gradient blocks of the same size, and mark the lip wrinkles for the divided gradient blocks according to the lip wrinkle mask; replace the gradient blocks of the marked lip wrinkle repair areas with gradient blocks in the area without lip wrinkles, and obtain a lip wrinkle removal image after Poisson reconstruction;

[0010] Perform median filtering on the lip image and the lip wrinkle removal image to obtain a result image retaining the color feature information of the original image, and perform local high-light restoration on the result image to remove the local light and shadow changes caused by lip wrinkles and retain the high-light area to obtain the final image after high-light restoration.

[0011] As a further solution of the present invention, before the outer bounding box based on the lip key points is cropped to obtain a lip image, it further includes calculating the outer bounding box of the lip key points. The method for calculating the outer bounding box of the lip key points is as follows:

[0012] Obtain the face key points in the original image according to face detection;

[0013] Traverse the face key points and find the key point set of the lips through indexing;

[0014] The rectangle formed by the vertices in the key point set of the lips is the outer bounding box of the lip key points.

[0015] Among them, the vertices include the maximum and minimum values of the x coordinate and y coordinate in the key point set of the lips, including: point (x min , y min ), point (x max , y min ), point (x min , y max ), point (x max , y max ).

[0016] As a further solution of the present invention, the size of each divided gradient block is proportionally divided according to the size of the lip image. When calculating the gradient of the lip image, the gradient is used to measure the change in pixel values between adjacent pixels. The magnitude of the gradient value G(x, y) at the pixel point (x, y) is:

[0017] G(x, y) = sqrt(G x (x, y) 2 +G y (x, y) 2 )

[0018] Among them, G x and G y respectively represent the gradient values of the pixel point (x, y) in the x direction and y direction.

[0019] As a further solution of the present invention, the method for lip pattern marking on the divided gradient blocks according to the lip pattern mask includes:

[0020] Obtain a number of gradient blocks of the same size after division. Each gradient block corresponds to a pixel block at the same position in the lip image and the lip pattern mask image;

[0021] Judge each gradient block one by one. If there is a pixel value greater than 0 in the pixel block corresponding to the gradient block in the lip pattern mask image, that is, there is a lip pattern, this gradient block is marked as a repair area;

[0022] If there are no pixels detected as lip prints in the gradient block, then the gradient block is a wrinkle-free area.

[0023] As a further aspect of the present invention, when replacing the gradient block of the repaired area marked with lip prints with a gradient block of a lip-print-free area, a random selection method is adopted from the unmarked gradient blocks of the lip-print-free area that meet the screening conditions. This can ensure to a certain extent that the same unmarked gradient block is not frequently used, avoiding the problem of repeated textures after repair and causing visual unreality.

[0024] As a further aspect of the present invention, after all the gradient blocks of the area to be repaired are replaced, Poisson reconstruction is performed on the lip image to obtain a lip-print-removed image;

[0025] Perform median filtering with the same radius on the lip-print-removed image and the lip image respectively to obtain the low-frequency feature maps of the filtered lip-print-removed image and the lip image, and calculate the high-frequency feature maps of the lip-print-removed image and the lip image;

[0026] Fuse the high-frequency feature map of the lip-print-removed image with the low-frequency feature map of the lip image to obtain a result map that retains the color feature information of the original image.

[0027] As a further aspect of the present invention, the method for local specular highlight restoration of the result map includes:

[0028] Perform Gaussian filtering on the lip image, set the pixel values of the filtered image that are darker than the lip image to 255, and the pixel values of other areas to 0 to obtain a specular highlight area mask image;

[0029] Perform connected component detection and classification on the specular highlight area mask image, mark the connected components as lip print areas and specular highlight areas according to the aspect ratio, and calculate a specular highlight feature mixed map based on the lip image within the specular highlight area;

[0030] Mix the specular highlight feature mixed map with the result map that retains the color feature information of the original image in a soft light blending mode to obtain an image after specular highlight restoration as the final image.

[0031] The present invention also includes a lip-print removal system based on image processing. The lip-print removal system based on image processing uses the aforementioned lip-print removal method based on image processing to remove lip prints and maintain the texture of lip gloss. The lip-print removal system based on image processing includes a lip print detection module, a lip print removal module, an original image feature retention module, and a local specular highlight restoration module.

[0032] The lip print detection module is used to crop the original image of the image to be retouched to obtain a lip image, perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip print detection within the range of the mask image to obtain a single-channel lip print mask;

[0033] The lip print removal module is used to calculate the gradient of the lip image, divide the lip image into a number of gradient blocks of the same size, and perform lip print marking on the divided gradient blocks according to the lip print mask; use the gradient blocks in the area without lip prints to replace the gradient blocks in the repaired area marked with lip prints, and obtain a lip print removal image after Poisson reconstruction;

[0034] The original image feature retention module is used to perform median filtering on the lip image and the lip print removal image to obtain a result image that retains the color feature information of the original image; and

[0035] The local highlight restoration module is used to perform local highlight restoration on the result image, remove the local light and shadow changes caused by lip prints, and retain the highlight area to obtain a final image after highlight restoration.

[0036] The present invention further includes a lip print removal device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the lip print removal method based on image processing are implemented.

[0037] The present invention further includes a storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the lip print removal method based on image processing are implemented.

[0038] The technical solution provided by the present invention has the following beneficial effects:

[0039] The lip print removal method, system, device and storage medium based on image processing of the present invention use randomly selected gradient blocks that meet the screening conditions and are not marked to replace the gradient blocks corresponding to the repaired area, and obtain a lip print removal image after Poisson reconstruction. After median filtering, not only lip prints are removed, but also the color feature information of the original image is retained. Moreover, in order to better maintain the texture of the original lip, local highlight restoration is also performed to remove the slender highlights distributed around the lip prints and retain the highlights generated by the lip gloss texture and ambient light, obtaining an image after highlight restoration, realizing the maintenance of the texture of the lip gloss while removing lip prints, and greatly improving the working efficiency of image processing and the quality of lip print retouching compared with the traditional manual retouching method. Description of the Drawings

[0040] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0041] Figure 1 Flowchart of a lip print removal method based on image processing according to an embodiment of the present invention.

[0042] Figure 2 Schematic diagram of the effect comparison between the original image and the final result image in a lip print removal method based on image processing according to an embodiment of the present invention.

[0043] Figure 3 Flowchart of performing local highlight restoration in a lip print removal method based on image processing according to an embodiment of the present invention.

[0044] Figure 4 Schematic diagram of the effect between the original image and after local highlight restoration in a lip print removal method based on image processing according to an embodiment of the present invention.

[0045] Figure 5 System block diagram of a lip print removal system based on image processing according to an embodiment of the present invention. Detailed implementation manners

[0046] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] The lip print removal method, system, device and storage medium provided by the present invention use an innovative lip print processing method to process images, maintain the texture of lip color while removing lip prints, and greatly improve the work efficiency of image processing and the quality of lip print refinement compared with traditional manual retouching methods.

[0048] The following will be described in conjunction with specific embodiments.

[0049] As Figure 1 shown, an embodiment of the present invention provides a lip print removal method based on image processing. This method removes lip prints and maintains the texture of lip color. This method mainly includes three parts: lip print removal, original image feature retention, and local highlight restoration. The specific steps are as follows:

[0050] S1. Obtain the original image to be retouched, and crop the lip image based on the outer bounding box of the lip key points.

[0051] In this embodiment, before cropping the lip image based on the outer bounding box of the lip key points, it further includes calculating the outer bounding box of the lip key points. The method for calculating the outer bounding box of the lip key points is:

[0052] Obtain the face key points in the original image according to face detection;

[0053] Traverse the facial key points, and find the key point set of the lips through indexing;

[0054] The vertices in the key point set of the lips form a rectangular box, which is the outer bounding box of the key points of the lips.

[0055] Among them, when processing the lip print area, to optimize the processing time of the algorithm, first calculate the outer bounding box of the key points of the lips according to the facial key points, and crop the lip image based on the outer bounding box Io .

[0056] Among them, the outer bounding box traverses the facial key points, and respectively takes the maximum and minimum values of their x coordinates and y coordinates: xmax, xmin, ymax, ymin; that is: the vertices include the maximum and minimum values of the x coordinates and y coordinates of the key point set of the lips.

[0057] The rectangular box composed of the points (x min , y min ), the points (x max , y min ), the points (x min , y max ), the points (x max , y max ) is the outer bounding box of the key points of the lips.

[0058] S2. Perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip print detection within the range of the mask image to obtain a single-channel lip print mask.

[0059] In this embodiment, lip segmentation adopts an image segmentation scheme based on deep learning, which will not be elaborated here. The single channel in the single-channel lip print mask is used to mark whether it is a lip print. For a single pixel, the pixel value of this single-channel image is between 0 and 255. If it is greater than 0, it means it is a lip print, and the larger the value, the deeper the lip print.

[0060] S3. Calculate the gradient of the lip image, divide the lip image into several gradient blocks of the same size, and perform lip print marking on the divided gradient blocks according to the lip print mask; replace the gradient blocks of the marked lip print repair area with gradient blocks of the non-lip print area, and obtain a lip print removal image after Poisson reconstruction.

[0061] In this embodiment, calculate the lip image Io gradient, and divide it into several gradient blocks of the same size. To better represent the texture information, the size of each gradient block needs to be calculated according to the image size. In this embodiment, the block size = min(image width, image height) / 15.0 is used for calculation.

[0062] Specifically, the size of each divided gradient block is proportionally divided according to the size of the lip image. When calculating the gradient of the lip image, the gradient is used to measure the change in pixel values of adjacent pixels. The magnitude of the gradient value G(x, y) at the pixel point (x, y) is:

[0063] G(x,y) = sqrt(G x (x,y) 2 +G y (x,y) 2 )

[0064] where G x and G y respectively represent the gradient values of the pixel point (x, y) in the x - direction and y - direction.

[0065] For a discrete two - dimensional image, the gradient is approximated by differences:

[0066] G x (x,y) = I(x + 1,y)-(x - 1,y)

[0067] G y (x,y) = I(x,y + 1)-I(x,y - 1)

[0068] where I(x, y) is the pixel value of the image at the (x, y) coordinates.

[0069] In this embodiment, for the gradient block corresponding to the marked repair area, search for the gradient block to replace the repair area in other unmarked gradient blocks. That is, use the gradient of the wrinkle - free area of the lip to replace the gradient block of the repair area. After all the gradient blocks of the areas to be repaired are replaced, perform Poisson reconstruction on the image to obtain the lip - wrinkle - removed image Ir1 .

[0070] In this embodiment, the method for marking lip wrinkles on the divided gradient blocks according to the lip - wrinkle mask includes:

[0071] Obtain a number of gradient blocks of the same size after division, and each gradient block corresponds to a pixel block at the same position in the lip image and the lip - wrinkle mask image;

[0072] Judging each gradient block one by one. If there is a pixel value greater than 0 in the pixel block of the gradient block corresponding to the lip - wrinkle mask image, that is, there are lip wrinkles, the gradient block is marked as a repair area;

[0073] If there are no pixel points detected as lip wrinkles in the gradient block, then the gradient block is a wrinkle - free area.

[0074] When performing lip - wrinkle marking, after obtaining the lip - wrinkle mask and completing the division of the gradient blocks, start marking the gradient blocks. If the lip image IoThe width and height are w and h respectively, the gradient block size is s*s, m = w / s, n = h / s, and the gradient image is divided into m*n blocks of s*s. Each s*s block corresponds to the pixel block at the same position in the lip image Io and the pixel block in the lip print mask image. If there are pixels with values greater than 0 in the pixel block corresponding to the gradient block in the lip print mask image, that is, there is a lip print, then this gradient block is marked as the repair area. Correspondingly, if there are no pixels detected as lip prints in the gradient block, then this gradient block is a wrinkle-free area.

[0075] In this embodiment, when replacing the gradient block of the marked repair area with the gradient block of the wrinkle-free area, a random selection method is adopted from the unmarked gradient blocks of the wrinkle-free area that meet the screening conditions. In this way, to a certain extent, it can ensure that the same unmarked gradient block is not frequently used, avoiding the problem of repeated textures after repair and causing visual unreality.

[0076] In this embodiment, Poisson reconstruction, as a common method of image fusion / image repair, aims to insert the source image region g into the ROI region of the target image S. To achieve this, actually, within the ROI region, the gradient of the target image should be consistent with that of the source image, while at the region edge, the target image maintains its original gradient unchanged. After solving the Poisson equation, the obtained image can visually fuse the source image and the target image naturally and seamlessly. In the present invention, the gradient of the area to be repaired is replaced with the gradient of the wrinkle-free area. After Poisson reconstruction, a lip print removal image can be obtained Ir1 。

[0077] S4. Perform median filtering on the lip image and the lip print removal image to obtain a result image that retains the color feature information of the original image, and perform local highlight restoration on the result image to remove the local light and shadow changes caused by lip prints and retain the highlight area to obtain the final image after highlight restoration.

[0078] In this embodiment, for the lip print removal image after Poisson reconstruction Ir1 , the lip prints are removed, but at this time, the lip color and light and shadow features often differ greatly from the original image. Therefore, this application introduces median filtering processing, which can better separate the low-frequency information and high-frequency information of the image.

[0079] After all the gradient blocks of the areas to be repaired are replaced, perform Poisson reconstruction on the lip image to obtain a lip print removal image;

[0080] Perform median filtering with the same radius on the lip print removal image and the lip image respectively to obtain the low-frequency feature maps of the filtered lip print removal image and the lip image, and calculate the high-frequency feature maps of the lip print removal image and the lip image;

[0081] Fuse the high-frequency feature map of the lip print-removed image with the low-frequency feature map of the lip image to obtain a result map that retains the color feature information of the original image.

[0082] In this embodiment, for the lip image Io and the lip print-removed image Ir1 perform median filtering with the same radius once respectively. The filtered results are the low-frequency feature maps of the lip image Io and the lip print-removed image Io_low and the lip print-removed image Ir1 of the low-frequency feature map Ir1_low , calculate the high-frequency feature maps of the lip image Io and the lip print-removed image Ir1 : Io_high = Io - Io_low, Ir1_high = Ir - Ir1_low . Then add the low-frequency feature map of the lip image Io to the high-frequency feature map of the lip print-removed image Ir1 . The resulting map Ir2 = Io_low + Ir1_high removes the lip prints and retains the color feature information of the original image.

[0083] In order to better maintain the texture of the original lip, perform local specular highlight restoration on the result map Ir2 . In this embodiment, lip specular highlights are generally divided into two categories: one is the local light and shadow changes caused by lip prints. The shape of this kind of specular highlight is generally slender and distributed around the lip prints, which needs to be removed; the other is the specular highlight generated by the texture of lip gloss and environmental light. The shape of this kind of specular highlight is mostly blocky and needs to be retained. As Figure 2 shown, the comparison between the original image and the final image result after local specular highlight restoration. After local specular highlight restoration, the final image removes the lip prints while maintaining the texture of the lip gloss.

[0084] As Figure 3 shown, the method for performing local specular highlight restoration on the result map includes:

[0085] S401. Perform Gaussian filtering on the lip image, set the pixel values of the pixels in the filtered image that are darker than the lip image to 255, and set the pixel values of other areas to 0 to obtain a specular highlight area mask image;

[0086] S402. Perform connected component detection and classification on the specular highlight area mask image. Mark the connected components as lip print areas and specular highlight areas according to the aspect ratio, and calculate a specular highlight feature mixed map based on the lip image within the specular highlight area;

[0087] S403. Mix the specular highlight feature mixed map with the result map that retains the color feature information of the original image in a soft light blending mode to obtain an image after specular highlight restoration as the final image.

[0088] In this embodiment, the specular highlight restoration process is as follows:

[0089] A. First, perform Gaussian filtering on the original image, and set the pixel values darker than the lip image after filtering to 255, and the pixel values in other areas to 0, to obtain the high-light region mask image H Io 。 mask 。

[0090] B. Then, perform connected component detection on the masked area image, classify the connected components, mark the areas with a large aspect ratio of length to width as lip lines, and mark the connected components with an aspect ratio close to or less than 1:1 as high-light regions. In the high-light region, calculate the high-light feature mixed map based on the lip image Io 。

[0091] C. Mix the high-light feature mixed map with the result image that retains the color feature information of the original image in a soft light blending mode Ir2 to obtain the image after high-light restoration Ir3 which is the final result image after removing lip lines.

[0092] See Figure 4 shown in the comparison between the original image and the final image result after local high-light restoration. After local high-light restoration, the final image removes lip lines while maintaining the texture of lip gloss.

[0093] Among them, local high-light restoration is mainly judged by morphology and area: the local light and shadow changes caused by lip lines are generally relatively slender and have a small area: if the aspect ratio is greater than a certain value or the area is less than a certain value, it is determined as the high light caused by lip lines.

[0094] When performing connected component detection, first binarize the image, set the pixels greater than 0 to 255, and keep the pixels equal to 0 unchanged. Then mark the pixel points with the same value and adjacent to each other as a connected component. Among them, the definition of adjacent: generally divided into 4-neighborhood and 8-neighborhood. The 4-neighborhood is the points above, below, left, and right of the pixel point; the 8-neighborhood is the points above, below, left, right, upper left, upper right, lower left, and lower right of the pixel. In the present invention, the points in the 8-neighborhood of the pixel are defined as adjacent points.

[0095] In this embodiment, for the high-light feature mixed map H: for the pixel point located at (x, y), if H mask (x, y)>0, then calculate according to the following formula:

[0096]

[0097] If H mask (x, y)<=0, H(x, y) = 127.5.

[0098] The present invention can automatically and quickly remove lip wrinkles, and can retain the color, texture, and light and shadow features of the lips, with a natural removal effect. It is possible to replace the gradient block corresponding to the repair area with a randomly selected gradient block that meets the screening conditions and is unmarked, and after Poisson reconstruction, a lip wrinkle removal image is obtained. After median filtering, not only are the lip wrinkles removed, but also the color feature information of the original image is retained. Moreover, in order to better maintain the material of the original lip, local specular highlight restoration is performed to remove the slender specular highlights distributed around the lip wrinkles and retain the specular highlights generated by the lip gloss texture and ambient light, obtaining an image after specular highlight restoration, realizing the maintenance of the texture of the lip gloss while removing lip wrinkles, greatly improving the work efficiency of image processing and the quality of lip wrinkle refinement compared to traditional manual retouching methods.

[0099] As Figure 5 shown, in another embodiment of the present invention, a lip wrinkle removal system based on image processing is provided to remove lip wrinkles and maintain the texture of the lip gloss. The system includes a lip wrinkle detection module 100, a lip wrinkle removal module 200, an original image feature retention module 300, and a local specular highlight restoration module 400.

[0100] The lip wrinkle detection module 100 is used to crop the original image of the image to be retouched to obtain a lip image, perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip wrinkle detection within the range of the mask image to obtain a single-channel lip wrinkle mask.

[0101] In this embodiment, when the lip wrinkle detection module 100 processes the lip wrinkle area, to optimize the processing time of the algorithm, first, according to the key points of the face, the outer bounding box of the key points of the lips is calculated, and the lip image is cropped based on the outer bounding box. Io .

[0102] Among them, the outer bounding box traverses the key points of the face and respectively takes the maximum and minimum values of their x coordinates and y coordinates: xmax, xmin, ymax, ymin. The rectangular box composed of the points (x min , y min ), the point (x max , y min ), the point (x min , y max ), and the point (x max , y max ) is the outer bounding box of the key points of the lips.

[0103] The lip wrinkle removal module 200 is used to calculate the gradient of the lip image, divide the lip image into a number of gradient blocks of the same size, and perform lip wrinkle marking on the divided gradient blocks according to the lip wrinkle mask; use the gradient blocks in the non-lip wrinkle area to replace the gradient blocks in the repair area marked with lip wrinkles, and after Poisson reconstruction, a lip wrinkle removal image is obtained.

[0104] In this embodiment, when the lip wrinkle removal module 200 calculates the gradient of the lip image and divides it into a number of gradient blocks of the same size, the size of each gradient block needs to be calculated according to the image size. In this embodiment, the block size = min(image width, image height) / 15.0 is used for calculation. Io For the gradient blocks corresponding to the marked repair areas, search for the gradient blocks in other unmarked gradient blocks to replace the repair areas. That is, use the gradient of the lip area without wrinkles to replace the gradient blocks of the repair areas. After all the gradient blocks of the areas to be repaired are replaced, perform Poisson reconstruction on the image to obtain the lip wrinkle removal image.

[0105] For the gradient blocks corresponding to the marked repair areas, search for the gradient blocks in other unmarked gradient blocks to replace the repair areas. That is, use the gradient of the lip area without wrinkles to replace the gradient blocks of the repair areas. After all the gradient blocks of the areas to be repaired are replaced, perform Poisson reconstruction on the image to obtain the lip wrinkle removal image. Ir1 .

[0106] In this embodiment, when the lip wrinkle removal module 200 marks lip wrinkles for the divided gradient blocks according to the lip wrinkle mask, a number of gradient blocks of the same size are obtained. Each gradient block corresponds to a pixel block at the same position in the lip image and the lip wrinkle mask image; each gradient block is judged one by one. If there is a pixel value greater than 0 in the pixel block of the gradient block corresponding to the lip wrinkle mask image, that is, there are lip wrinkles, the gradient block is marked as a repair area; if there are no pixel points detected as lip wrinkles in the gradient block, then the gradient block is an area without wrinkles.

[0107] In this embodiment, the replacement is performed by randomly selecting from the unmarked gradient blocks of the lip - wrinkle - free areas that meet the screening conditions. In this way, to a certain extent, it can ensure that the same unmarked gradient block is not frequently used, avoiding the problem of repeated textures after repair and causing visual unreality.

[0108] The original image feature retention module 300 is used to perform median filtering on the lip image and the lip wrinkle removal image to obtain a result image that retains the color feature information of the original image.

[0109] Perform median filtering on the lip image Io and the lip wrinkle removal image Ir1 respectively with the same radius. The filtered results are the low - frequency feature maps of the lip image Io and the lip wrinkle removal image Io_low . Calculate the high - frequency feature maps of the lip image Ir1 and the lip wrinkle removal image Ir1_low : Io Ir1 . Then, add the low - frequency feature map of the lip image Io_high = Io - Io_low, Ir1_high = Ir - Ir1_ low to the high - frequency feature map of the lip wrinkle removal image . The resulting image Io Ir1 Ir2 = Io_low + Ir1_high ​​It not only removes lip wrinkles but also retains the color feature information of the original image.

[0110] The local specular highlight restoration module 400 is used to perform local specular highlight restoration on the result image, remove the local light and shadow changes caused by lip wrinkles, retain the specular highlight area, and obtain the final image after specular highlight restoration.

[0111] In this embodiment, the method for the local specular highlight restoration module 400 to perform local specular highlight restoration on the result image is as follows:

[0112] Perform Gaussian filtering on the lip image, set the pixel values of the filtered image that are darker than the lip image to 255, and set the pixel values of other areas to 0 to obtain a specular highlight area mask image;

[0113] Perform connected component detection and classification on the specular highlight area mask image, label the connected components as lip wrinkle areas and specular highlight areas according to the aspect ratio, and calculate a specular highlight feature mixed image based on the lip image within the specular highlight area;

[0114] Mix the specular highlight feature mixed image with the result image that retains the color feature information of the original image in a soft light mixing mode to obtain the image after specular highlight restoration as the final image.

[0115] Among them, the lip wrinkle removal system based on image processing adopts the steps of a lip wrinkle removal method based on image processing as described in the foregoing embodiment when executing. Therefore, the operation process of the lip wrinkle removal system based on image processing in this embodiment will not be introduced in detail.

[0116] In an embodiment of the present invention, a lip wrinkle removal device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the foregoing method embodiment are implemented:

[0117] Obtain the original image of the image to be repaired, and crop the lip image based on the outer bounding box of the lip key points;

[0118] Perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip wrinkle detection within the range of the mask image to obtain a single-channel lip wrinkle mask;

[0119] Calculate the gradient of the lip image, divide the lip image into a number of gradient blocks of the same size, and perform lip wrinkle marking on the divided gradient blocks according to the lip wrinkle mask; replace the gradient blocks of the repaired area marked with lip wrinkles with gradient blocks in the area without lip wrinkles, and obtain a lip wrinkle removal image after Poisson reconstruction;

[0120] Perform median filtering on the lip image and the lip wrinkle removal image to obtain a result image that retains the color feature information of the original image, and perform local specular highlight restoration on the result image to remove the local light and shadow changes caused by lip wrinkles and retain the specular highlight area, thereby obtaining the final image after specular highlight restoration.

[0121] In another embodiment of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented:

[0122] Obtain the original image of the image to be retouched, and crop the lip image based on the outer bounding box of the lip key points;

[0123] Perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip wrinkle detection within the range of the mask image to obtain a single-channel lip wrinkle mask;

[0124] Calculate the gradient of the lip image, divide the lip image into a number of gradient blocks of the same size, and mark the gradient blocks according to the lip wrinkle mask; replace the gradient blocks of the repaired area marked with lip wrinkles with the gradient blocks of the area without lip wrinkles, and obtain the lip wrinkle removal image after Poisson reconstruction;

[0125] Perform median filtering on the lip image and the lip wrinkle removal image to obtain a result image that retains the color feature information of the original image, and perform local specular highlight restoration on the result image to remove the local light and shadow changes caused by lip wrinkles and retain the specular highlight area, thereby obtaining the final image after specular highlight restoration.

[0126] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories.

[0127] In summary, for the lip print removal method, system, device, and storage medium based on image processing of the present invention, randomly selected gradient blocks that meet the screening conditions and are unmarked are used to replace the gradient blocks corresponding to the repair area. After Poisson reconstruction, a lip print removal image is obtained. After median filtering, not only are the lip prints removed, but the color feature information of the original image is also retained. Moreover, in order to better maintain the texture of the lips in the original image, local specular highlight recovery is also performed to remove the slender specular highlights distributed around the lip prints and retain the specular highlights generated by the lip gloss texture and ambient light, resulting in an image after specular highlight recovery, achieving the retention of the texture of the lip gloss while removing the lip prints. Compared with the traditional manual image retouching method, the working efficiency of image processing and the quality of lip print refinement are greatly improved.

[0128] The foregoing are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A lip print removal method based on image processing, characterized in that, It includes the following steps: Obtain the original image to be retouched, and crop the lip image based on the outer bounding box of the lip key points; Perform lip segmentation on the obtained original image to obtain a mask image marking the lip area, and perform lip print detection within the range of the mask image to obtain a single-channel lip print mask; Calculate the gradient of the lip image, divide the lip image into several gradient blocks of the same size, and mark the gradient blocks according to the lip print mask; Replace the gradient blocks of the retouched area marked with lip prints with gradient blocks of the area without lip prints, and obtain an image with lip prints removed after Poisson reconstruction; Perform median filtering on the lip image and the image with lip prints removed to obtain a result image that retains the color feature information of the original image, and perform local highlight restoration on the result image to remove the local light and shadow changes caused by lip prints and retain the highlight area to obtain the final image after highlight restoration; 2. The lip print removal method based on image processing according to claim 1, characterized in that: Before cropping the lip image based on the outer bounding box of the lip key points, it also includes calculating the outer bounding box of the lip key points, and the method for calculating the outer bounding box of the lip key points is: Obtain the face key points in the original image according to face detection; Traverse the face key points and find the key point set of the lips through indexing; The rectangle formed by the vertices in the key point set of the lips is the outer bounding box of the lip key points; 3. The lip print removal method based on image processing according to claim 2, wherein: The size of each divided gradient block is proportionally divided according to the size of the lip image. When calculating the gradient of the lip image, the gradient is used to measure the change in pixel values of adjacent pixels, and the magnitude of the gradient value G(x, y) at the pixel point (x, y) is: G(x,y) = sqrt(G x (x,y) 2 +G y (x,y) 2 ) Among them, G x and G y respectively represent the gradient values of the pixel point (x, y) in the x direction and the y direction.

4. The lip print removal method based on image processing according to claim 3, characterized in that: The method for marking lip prints on the divided gradient blocks according to the lip print mask includes: Obtain several gradient blocks of the same size after division, and each gradient block corresponds to the pixel blocks at the same position in the lip image and the lip print mask image; Judge each gradient block one by one. If there are pixel values greater than 0 in the pixel block corresponding to the gradient block in the lip print mask image, that is, there are lip prints, the gradient block is marked as the retouched area; If there are no pixel points detected as lip prints in the gradient block, then the gradient block is an area without wrinkles; 5. The lip print removal method based on image processing according to claim 4, wherein: When replacing the gradient blocks of the retouched area marked with lip prints with gradient blocks of the area without lip prints, randomly select from the unmarked gradient blocks of the area without lip prints that meet the screening conditions for replacement; 6. The lip print removal method based on image processing according to claim 5, wherein: After all the gradient blocks of the areas to be retouched are replaced, perform Poisson reconstruction on the lip image to obtain an image with lip prints removed; Perform median filtering with the same radius on the lip image with lip prints removed and the lip image respectively to obtain the low-frequency feature maps of the filtered lip image with lip prints removed and the lip image, and calculate the high-frequency feature maps of the lip image with lip prints removed and the lip image; Fuse the high-frequency feature map of the lip image with lip prints removed and the low-frequency feature map of the lip image to obtain a result image that retains the color feature information of the original image; 7. The lip print removal method based on image processing according to claim 6, wherein: The method for performing local highlight restoration on the result image includes: Perform Gaussian filtering on the lip image, set the pixel values of the filtered image that are darker than the lip image to 255, and set the pixel values of other areas to 0 to obtain a highlight area mask image; Perform connected component detection and classification on the masked image of the high - light area, mark the connected components as lip print areas and high - light areas according to the aspect ratio, and calculate the high - light feature mixed map within the high - light area based on the lip image; Mix the high - light feature mixed map with the result map that retains the color feature information of the original image in a soft - light blending mode to obtain the image after high - light restoration, which is used as the final image.

8. A lip print removal system based on image processing, characterized in that: The lip print removal system based on image processing uses the lip print removal method according to any one of claims 1 - 7 to remove lip prints and maintain the texture of lip gloss; The lip print removal system based on image processing includes: A lip print detection module, which is used to crop the original image of the image to be retouched to obtain a lip image, perform lip segmentation on the obtained original image to obtain a masked image marking the lip area, and perform lip print detection within the range of the masked image to obtain a single - channel lip print mask; A lip print removal module, which is used to calculate the gradient of the lip image, divide the lip image into a number of gradient blocks of the same size, and mark the lip prints for the divided gradient blocks according to the lip print mask; replace the gradient blocks of the marked lip print repair areas with the gradient blocks of the area without lip prints, and obtain the lip print - removed image after Poisson reconstruction; An original - image feature retention module, which is used to perform median filtering on the lip image and the lip print - removed image to obtain a result map that retains the color feature information of the original image; A local high - light restoration module, which is used to perform local high - light restoration on the result map, remove the local light and shadow changes caused by lip prints, and retain the high - light area to obtain the final image after high - light restoration.

9. A lip print removal device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A storage medium stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images