Image processing method, device, equipment, storage medium and program product

Abstract

The embodiments of the present application disclose an image processing method, device, equipment, storage medium and program product, which belong to the field of computer technology. The method comprises: obtaining an original image to be processed; performing skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image; performing a first stage processing on the original image based on the first detection result image and the second detection result image to obtain a first stage result image; performing a second stage processing on the first stage result image to obtain a second stage result image; and generating a final result image based on the original image and the second stage result image. The present application provides an image beautification solution that takes into account the ability to remove blemishes, the ability to retain skin texture, and high real-time performance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device, storage medium and program product. Background Art

[0002] In applications such as photo taking and live video streaming, there is a need to process facial images, which is what we usually call the beauty requirement.

[0003] Traditional image processing methods based on edge-preserving filtering algorithms focus on removing blemishes such as pimples and spots in facial images, but they also lose a lot of facial details, resulting in poor skin texture preservation. In related technologies, a neural network model is trained using a machine learning algorithm, and the original facial image is processed through the neural network model to output a beautified facial image. This end-to-end processing method based on a neural network model requires the design of a more complex neural network structure in order to enable the neural network model to have better blemish removal capabilities and skin texture preservation capabilities, which in turn results in a longer processing time through the neural network model and cannot meet the needs of some application scenarios with high real-time requirements.

[0004] Therefore, there is currently no image beauty processing solution that takes into account the ability to remove blemishes, retain skin texture, and achieve high real-time performance. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, storage medium and program product. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, there is provided an image processing method, the method comprising:

[0007] Get the original image to be processed;

[0008] Perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image; wherein the first detection result image is used to represent the skin color area in the original image, and the second detection result image is used to represent the face area in the original image;

[0009] Based on the first detection result map and the second detection result map, the original image is subjected to first-stage processing to obtain a first-stage result map; wherein the first-stage processing is used to remove the area in the facial area that is different from the overall skin color;

[0010] Performing a second-stage processing on the first-stage result image to obtain a second-stage result image; wherein the second-stage processing is used to improve the uniformity of skin color at different positions in the facial area;

[0011] Based on the original image and the second-stage result image, a final result image is generated.

[0012] According to one aspect of an embodiment of the present application, there is provided an image processing device, the device comprising:

[0013] An original image acquisition module, used to acquire the original image to be processed;

[0014] An original image detection module, used to perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image; wherein the first detection result image is used to characterize the skin color area in the original image, and the second detection result image is used to characterize the face area in the original image;

[0015] A first processing module, configured to perform a first-stage processing on the original image based on the first detection result image and the second detection result image to obtain a first-stage result image; wherein the first-stage processing is used to remove an area in the facial area that is different from the overall skin color;

[0016] A second processing module, configured to perform a second-stage processing on the first-stage result image to obtain a second-stage result image; wherein the second-stage processing is configured to improve the uniformity of skin color at different positions in the facial region;

[0017] The result generation module is used to generate a final result image based on the original image and the second-stage result image.

[0018] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned image processing method.

[0019] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image processing method.

[0020] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions stored in a computer-readable storage medium, and a processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned image processing method.

[0021] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0022] The skin color area and face area in the original image are selected by performing skin color detection and face detection on the original image. The area in the face area that is significantly different from the overall skin color is weakened according to the first stage processing, and the difference between the skin color in this area and the overall skin color of the face area is reduced. The second stage processing improves the uniformity of the skin color in the face area to obtain the final image processing result. On the one hand, the first stage of processing weakens the area in the facial area that is significantly different from the overall skin color, and reduces the difference between the skin color of the area and the overall skin color of the facial area. The second stage of processing improves the uniformity of the skin color in the facial area, strengthens the blemish removal capability, and better preserves the skin texture. On the other hand, based on the first detection result map and the second detection result map obtained by skin color detection and face detection, the facial skin color area in the original image is determined, and the image beauty processing is performed on the above-mentioned facial skin color area, and the background area and the edges of the facial features are not subjected to image beauty processing, thereby reducing the wear of the background area and the edges of the facial features during beauty processing. On the other hand, this solution can complete the image beauty processing only through simple calculations, and does not need to perform a large amount of calculations by using neural networks. The calculation process of beauty processing is optimized, thereby shortening the time required, and can meet the needs of some application scenarios with high real-time requirements. For example, the above-mentioned beauty optimization process can be performed while shooting, with high real-time performance. Therefore, the present application provides an image beauty solution that takes into account the blemish removal capability, the skin texture retention capability, and high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of an image before and after processing provided by an embodiment of the present application;

[0026] Figure 4 is a flowchart of an image processing method provided by another embodiment of the present application;

[0027] Figure 5 is a schematic diagram of an image processing method provided by an embodiment of the present application;

[0028] Figure 6 is a block diagram of an image processing device provided by an embodiment of the present application;

[0029] Figure 7 It is a block diagram of an image processing device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0031] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include: a terminal 10 and a server 20.

[0032] The terminal 10 may be an electronic device such as a mobile phone, a tablet computer, a PC (Personal Computer), a wearable device, a vehicle-mounted terminal device, a VR (Virtual Reality) device, and an AR (Augmented Reality) device, and this application does not limit this. A client running a target application may be installed in the terminal 10. For example, the target application may be an image processing application or other application with an image processing function. Optionally, the target application is an application with an image beautification function, such as a shooting application, a live video application, a social application, a video editing application, a short video application, etc., and this application does not limit this.

[0033] The following is an introduction using the target application as an application with an image beautification function. The client of the target application has the function of beautifying facial images. By performing acne removal, scar removal, whitening and other processing on facial images, the facial images can meet the needs of users after processing.

[0034] The server 20 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 20 may be a background server of the target application, used to provide background services for the client of the target application.

[0035] The terminal 10 and the server 20 may communicate with each other via a network, for example, the network may be a wired network or a wireless network.

[0036] The image processing method provided in the embodiment of the present application, the execution subject of each step can be Figure 1 The server 20 in the implementation environment of the scheme shown, that is, the server 20 executes all the steps of the embodiment of the method of the present application; it can also be the terminal 10 (such as the client of the target application), that is, the terminal 10 executes all the steps of the embodiment of the method of the present application; or the server 20 and the terminal 10 interact and cooperate to execute, that is, the server 20 executes part of the steps of the embodiment of the method of the present application, and the terminal 10 executes another part of the steps of the embodiment of the method of the present application.

[0037] In a possible application scenario, the target application is used to process already created pictures and perform beautification processing on faces in the pictures.

[0038] In another possible application scenario, the above-mentioned target application is used in live broadcast and short video applications. While live broadcasting or shooting videos, real-time beautification processing is performed on the faces in the screen. This scenario has high requirements for real-time processing, and the terminal needs to be able to beautify the faces in the shot while shooting.

[0039] In the following method embodiments, for the sake of convenience, the example in which the execution subject of each step is the terminal 10 is taken, but this is not a limitation.

[0040] Please refer to Figure 2 , which shows a flow chart of an image processing method provided by an embodiment of the present application. The method may include at least one of the following steps (210-250):

[0041] Step 210: Obtain the original image to be processed.

[0042] Optionally, the original image contains a facial area of ​​the target object. For example, the original image may be a face image, or an image containing a face. Exemplarily, the original image may be a frontal face image, a side face image, or an image containing multiple facial areas, etc., which is not limited in this application. Through the embodiments of the present application, the original image to be processed is subjected to facial beautification processing to obtain an image after facial beautification processing. The intensity of the facial beautification processing can be adjusted according to the needs of the user.

[0043] Step 220, perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image.

[0044] The first detection result image is used to represent the skin color area in the original image, and the second detection result image is used to represent the face area in the original image.

[0045] The skin color area refers to the area composed of pixels whose color values ​​match the skin color in the original image. The skin color area and the non-skin color area in the original image can be distinguished based on the first detection result image. For example, the skin color area may include the face area, and may also include skin areas such as the neck and arms, or the area where the objects or background whose color values ​​match the skin color in the original image are located.

[0046] In some embodiments, a skin color probability is set for pixels in the original image, and the first detection result map contains the skin color probabilities corresponding to each pixel in the original image. Optionally, the skin color probability corresponding to a certain pixel is used to indicate whether the pixel belongs to a skin color area. Optionally, the skin color detection is performed on the original image using the RGB color value of the pixel, the RGB color value of the pixel in the original image is extracted, and then the RGB color value interval corresponding to the skin color area is set, and the area composed of all pixels whose RGB color values ​​of the pixels in the original image are in the RGB color value interval corresponding to the skin color area is determined as the skin color area. For example, the R value interval in the RGB color value interval corresponding to the skin color area is set to (206-254), the G value interval is (123-234), and the B value interval is (100-230). Each pixel in the original image is detected. When the RGB color value of a pixel is (240, 180, 150), the pixel belongs to the skin color area; when the RGB color value of another pixel is (200, 180, 150), the pixel does not belong to the skin color area.

[0047] In other embodiments, the skin color probability corresponding to a certain pixel is used to indicate the association (or proximity) between the pixel and the skin color area. For example, the skin color probability is set based on the sum of the minimum difference between the RGB color value of the pixel and the upper or lower limit of the RGB color value interval corresponding to the skin color area. The smaller the sum of the minimum difference between the RGB color value of the pixel and the upper or lower limit of the RGB color interval corresponding to the skin color area, the greater the skin color probability. The maximum skin color probability is 1, that is, the pixel is located in the skin color area; the minimum skin color probability is 0, that is, no RGB color value in the RGB color value of the pixel is located in the RGB color value interval corresponding to the skin color area; when there is any RGB color value in the RGB color value of the pixel that is located in the RGB color value interval corresponding to the skin color area, the skin color probability of the point is between 0 and 1.

[0048] In some embodiments, the R value interval in the RGB color value interval corresponding to the skin color area is set to (206-254), the G value interval is set to (123-234), and the B value interval is set to (100-230). Each pixel in the original image is detected. When the RGB color value of pixel E is (240, 180, 150), pixel E belongs to the skin color area, and the skin color probability corresponding to the pixel E is 1; when the RGB color value of pixel F is (200, 60, 255), pixel F does not belong to the skin color area, and no RGB color value of the RGB color value of pixel F is in the RGB color value interval corresponding to the skin color area, then the skin color probability corresponding to the pixel F is 0; when the RGB color value of pixel G is (200, 180, 150 ), the pixel G does not belong to the skin color area, and at the same time, the sum of the minimum difference between the RGB color value of the pixel G and the upper limit or lower limit of the RGB color value interval corresponding to the skin color area is calculated to be 6, specifically 206-200=6, then the skin color probability corresponding to the pixel G is between 0 and 1; when the RGB color value of the pixel H is (200,100,150), the pixel H does not belong to the skin color area, and at the same time, the sum of the minimum difference between the RGB color value of the pixel H and the upper limit or lower limit of the RGB color value interval corresponding to the skin color area is calculated to be 29, specifically (206-200)+(123-100)=29, then the skin color probability corresponding to the pixel H is between 0 and 1, and the skin color probability corresponding to the pixel H is less than the skin color probability corresponding to the pixel H.

[0049] Optionally, the facial key points in the original image are located by a neural network positioning method, and then the facial area is determined by the facial key points to obtain the facial area in the original image. The facial key points are used to assist in face detection so as to obtain a more correct facial area. For example, the facial key points include the key points of the corners of the mouth, the corners of the eyes, the corners of the eyebrows and other facial parts. According to the positions of the key points of the corners of the mouth, the corners of the eyes, the corners of the eyebrows and other facial parts in the original image obtained above, the facial area in the original image is determined. Optionally, the facial area only includes the skin color area of ​​the face, and does not include non-skin color areas such as eyes and eyebrows.

[0050] In some embodiments, face probabilities are set for pixels in the original image, and the second detection result image includes face probabilities for each pixel in the original image. The face probability of a certain pixel is used to indicate whether the pixel belongs to a face region. Figure 3 As shown, in the original image 31, the spot 35 is located in the face area 33, and the face probability of the pixel corresponding to the spot 35 is 1. The corner of the eye 34 does not belong to the face area 33, and the face probability of the pixel corresponding to the corner of the eye 34 is 0. A pixel is located in the background of the original image, and the pixel does not belong to the face area 33, and the face probability of the pixel is 0.

[0051] In other embodiments, the face probability of a certain pixel is used to indicate the association degree between the pixel and the face region. Optionally, the association degree is set according to whether the pixel can be used to help determine the face region. The more the pixel can be used to help determine the face region, the greater the face probability of the pixel. The maximum face probability is 1, that is, the pixel is located in the face region; the minimum face probability is 0, that is, the pixel does not belong to the face region and the pixel cannot be used to help determine the face region; when the pixel does not belong to the face region and the pixel can be used to help determine the face region, the face probability of the point is between 0 and 1.

[0052] In some embodiments, Figure 3 As shown, in the original image 31, the spot 35 is located in the face area 33, and the face probability of the pixel corresponding to the spot 35 is 1. The corner of the eye 34 does not belong to the face area 33, but the corner of the eye 34 is used to help determine the face area 33, and the face probability of the pixel corresponding to the corner of the eye 34 is between 0 and 1. A pixel is located in the background, the pixel does not belong to the face area 33, and the pixel cannot be used to help determine the face area 33, and the face probability of the pixel is 0.

[0053] Step 230, based on the first detection result image and the second detection result image, perform the first stage processing on the original image to obtain a first stage result image.

[0054] The first stage of processing is used to remove areas in the face that differ from the overall skin color.

[0055] By performing the first stage processing on the original image, a first stage result image is obtained based on the first detection result image and the second detection result image. The first stage result image weakens the area in the facial area that is significantly different from the overall skin color, and reduces the difference between the skin color of this area and the overall skin color of the facial area. The first stage processing only processes the facial area in the original image, mainly processing the area in the facial area that is significantly different from the overall skin color. For example, there are areas in the facial area where there are large color differences in acne, spots, scars, etc. The first stage processing is used to weaken the areas where there are large color differences in the above-mentioned acne, spots, scars, etc., and reduce the difference between the skin color of this area and the overall skin color of the facial area, so that the overall color of the facial area is as consistent as possible.

[0056] In some embodiments, Figure 3 As shown, the first stage of processing is used to process areas with color differences such as acne, spots 35, scars, etc. in the facial area 33 of the original image 31.

[0057] Step 240, performing second-stage processing on the first-stage result graph to obtain a second-stage result graph.

[0058] The second stage of processing is used to improve the uniformity of skin tone in different areas of the face.

[0059] In the first-stage result image obtained after the first-stage processing, the facial area weakens the area with large color differences and reduces the difference between the skin color of this area and the overall skin color of the facial area. However, there is still a certain difference between the skin color of this area and the overall skin color of the facial area. From the overall visual effect of the entire facial area, there may be some slight color differences between the above-mentioned spot-removed area and other non-spot-removed areas, resulting in discontinuity and unevenness between the above-mentioned spot-removed area and other non-spot-removed areas. Through the second-stage processing, the uniformity of the skin color in the facial area is improved, and the second-stage result image is obtained.

[0060] Step 250, generating a final result image based on the original image and the second stage result image.

[0061] According to the above steps, the second-stage result image is obtained. The second-stage result image is the result image obtained by the user when the beauty intensity value is the maximum. Among them, the beauty intensity value is used to adjust the proportion of the original image and the second-stage result image when mixed. This value can be controlled by the user. According to the beauty intensity value selected by the user, the original image and the second-stage result image are set with corresponding weights and then fused to obtain the final result image.

[0062] In some implementations, when the beautification intensity value set by the user is the minimum threshold value, the weight of the original image is set to 1, and the weight of the second-stage result image is set to 0, and the final result image is the original image; when the beautification intensity value set by the user is between the minimum threshold value and the maximum threshold value, the original image and the second-stage result image are set with corresponding weights according to the beautification intensity value, and then fused to obtain the final result image; when the beautification intensity value set by the user is the maximum threshold value, the weight of the original image is set to 0, and the weight of the second-stage result image is set to 1, and the final result image is the second-stage result image.

[0063] This application selects the skin color area and face area in the original image by performing skin color detection and face detection on the original image, and weakens the area in the face area that is significantly different from the overall skin color according to the first stage processing, thereby reducing the difference between the skin color of this area and the overall skin color of the face area. The second stage processing improves the uniformity of the skin color in the face area to obtain the final image processing result. On the one hand, the first stage of processing weakens the area in the facial area that is significantly different from the overall skin color, and reduces the difference between the skin color of the area and the overall skin color of the facial area. The second stage of processing improves the uniformity of the skin color in the facial area, strengthens the blemish removal capability, and better preserves the skin texture. On the other hand, based on the first detection result map and the second detection result map obtained by skin color detection and face detection, the facial skin color area in the original image is determined, and the image beauty processing is performed on the above-mentioned facial skin color area, and the background area and the edges of the facial features are not subjected to image beauty processing, thereby reducing the wear of the background area and the edges of the facial features during beauty processing. On the other hand, this solution can complete the image beauty processing only through simple calculations, and does not need to perform a large amount of calculations by using neural networks. The calculation process of beauty processing is optimized, thereby shortening the time required, and can meet the needs of some application scenarios with high real-time requirements. For example, the above-mentioned beauty optimization process can be performed while shooting, with high real-time performance. Therefore, the present application provides an image beauty solution that takes into account the blemish removal capability, the skin texture retention capability, and high real-time performance.

[0064] Please refer to Figure 4 , which shows a flowchart of an image processing method provided by another embodiment of the present application, taking the target application as an image beautification application as an example, the method may include at least one of the following steps (410-490):

[0065] Step 410: Obtain the original image to be processed.

[0066] Step 420, perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image.

[0067] The first detection result image is used to represent the skin color area in the original image, and the second detection result image is used to represent the face area in the original image.

[0068] For the description of step 410 and step 420, please refer to the above embodiment, which will not be repeated here.

[0069] Step 430: Based on the first detection result image and the second detection result image, perform a first filtering process on the original image to obtain a first filtered image.

[0070] The first filtering process is used to perform filtering on the face area while retaining the edge of the original image.

[0071] In the first filtered image, only the face region is filtered. Through the filtering process, the pixel values ​​of the pixels in the face region are made closer and the difference between the face region and other regions is made more obvious.

[0072] Optionally, step 430 includes steps 431 to 434:

[0073] Step 431: determine the face area in the original image based on the second detection result image.

[0074] The second detection result image is used to characterize the face area in the original image. Based on the second detection result image, the face area in the original image is determined, wherein the face area in the original image only includes the skin color area, and does not include non-skin color areas such as eyes and eyebrows.

[0075] Step 432, for the target pixel in the face area, determine the first filtering weights corresponding to the surrounding pixels according to the pixel value difference between the target pixel and the surrounding pixels and the skin color probability corresponding to the surrounding pixels; wherein the skin color probability is obtained based on the first detection result image.

[0076] The surrounding pixels are pixels adjacent to the target pixel. Optionally, the surrounding pixels may also be pixels that are 1 pixel away from the target pixel. The present application does not limit the definition of surrounding pixels. For the target pixel in the face area, the first filter weight corresponding to the target pixel is calculated. The first filter weight is obtained by the pixel value difference between the target pixel and each surrounding pixel and the skin color probability corresponding to each pixel of the surrounding pixel. The pixel value may be the RGB color value mentioned in the previous embodiment.

[0077] According to the pixel value of the target pixel and the pixel values ​​of the surrounding pixels in the original image, the pixel value difference between the target pixel and each surrounding pixel is calculated. For example, if the pixel value of the target pixel is A, and the pixel value of one of the surrounding pixels is B, then the pixel value difference between the target pixel and the surrounding pixel is |AB|. Optionally, the pixel value difference between the target pixel and the surrounding pixel can also be (AB) 2 , this application does not limit the method for calculating the pixel value difference between the target pixel and the surrounding pixels.

[0078] The skin color probabilities corresponding to the pixel values ​​of each surrounding pixel are obtained through the first detection result. The specific introduction of the skin color probability can be found in the content of the previous embodiment, which will not be repeated here.

[0079] After multiplying the pixel value difference of each of the surrounding pixels and the skin color probability, the first filtering weights corresponding to each of the surrounding pixels are obtained.

[0080] In some embodiments, the pixel value of the target pixel is A, the pixel value of one of the surrounding pixels is B, and the skin color probability corresponding to the surrounding pixels is X, then the first filter weight corresponding to the surrounding pixels is X|AB|. Optionally, the first filter weight corresponding to the surrounding pixels is X(AB) 2 , this application does not limit the calculation method of the first filtering weight.

[0081] Step 433: Determine the first filtered pixel value corresponding to the target pixel according to the pixel values ​​corresponding to the surrounding pixels and the first filtering weight.

[0082] According to the pixel values ​​of each surrounding pixel and the first filtering weight, the first filtered pixel value corresponding to the target pixel is determined. Among them, the pixel values ​​of each surrounding pixel can be obtained from the original image, and the first filtering weight is obtained according to the above calculation process. Optionally, the target pixel participates in the calculation process of the first filtered pixel value corresponding to the target pixel, and the weight value of the target pixel can be set to 1. Optionally, the target pixel does not participate in the calculation process of the first filtered pixel value corresponding to the target pixel. This application does not limit the calculation process of the first filtered pixel value corresponding to the target pixel.

[0083] With the target pixel as the center, the surrounding pixels corresponding to the target pixel are determined, and the first filtered pixel value corresponding to the target pixel is determined according to the pixel values ​​corresponding to each of the surrounding pixels and the first filtering weight. The number of pixels around the target pixel can be arbitrary. For example, the area composed of the target pixel and its surrounding pixels can be a 3*3 area. Then, the first filtered pixel value corresponding to the target pixel is determined according to the pixel values ​​of each of the surrounding pixels in the 3*3 area and the first filtering weight. Optionally, the area composed of the target pixel and its surrounding pixels can also be a 5*5 area. Then, the first filtered pixel value corresponding to the target pixel is determined according to the pixel values ​​of each of the surrounding pixels in the 5*5 area and the first filtering weight. This application does not limit the size of the area composed of the target pixel and the corresponding surrounding pixels.

[0084] In some embodiments, the target pixel participates in the calculation process of the first filtered pixel value corresponding to the target pixel, and the weight value of the target pixel is set to 1, and the area composed of the target pixel and its surrounding pixels is set to be a 3*3 area. Then, according to the pixel values ​​of each surrounding pixel in the 3*3 area and the first filtering weight, the product results corresponding to each surrounding pixel are obtained by multiplying the pixel values ​​of each surrounding pixel and the first filtering weight, and at the same time, according to the pixel value and weight of the target pixel itself, the product result corresponding to the target pixel is obtained by multiplying. The average value of the above 9 product results is taken as the first filtered pixel value of the target pixel.

[0085] In some embodiments, the target pixel does not participate in the calculation process of the first filtered pixel value corresponding to the target pixel, and the area composed of the target pixel and its surrounding pixels is set to be a 5*5 area. Then, according to the pixel values ​​of each surrounding pixel in the 5*5 area and the first filtering weight, the product results corresponding to each surrounding pixel are obtained by multiplying the pixel values ​​of each surrounding pixel and the first filtering weight. The average value of the above 24 product results is taken as the first filtered pixel value of the target pixel.

[0086] Step 434: Obtain a first filtered image according to the first filtered pixel values ​​corresponding to the pixels in the face area.

[0087] According to step 433, the first filtered pixel values ​​corresponding to all pixels in the face area are calculated to obtain a first filtered image. The first filtered pixel values ​​corresponding to all pixels in the face area are obtained by the same calculation method.

[0088] Through the first stage of processing, a first filtered image is obtained, and the face area in the original image is filtered through the first filtering weight to reconstruct the pixel values ​​of the pixels in the face area in the original image, so that the determination of the defect position in the following step is more accurate.

[0089] Step 440: Generate a defect detection result image based on the first filtered image and the original image.

[0090] The first filtered image and the original image are processed to obtain a defect detection result map, wherein the defect detection result map is used to characterize the location of defects in the facial area, and the defect refers to an area with a large color difference such as the above-mentioned acne, spots, scars, etc.

[0091] Optionally, step 440 includes steps 441 to 443:

[0092] Step 441, obtaining a first difference image based on the difference between the pixel values ​​at corresponding positions in the first filtered image and the original image.

[0093] Optionally, the pixel value of each pixel in the first filtered image is subtracted from the pixel value of the pixel at the corresponding position in the original image to obtain a first difference image. The first difference image shows the difference between the pixel values ​​at the corresponding position in the first filtered image and the original image. Since there is no change in the area outside the face area, the area outside the face area is exactly the same in the first filtered image and the original image, so the difference in the first difference image is 0. After the face area is filtered, since the pixel value of the defective part is generally dark, the brightness of the defective part will be improved after filtering. Then, the part of the first difference image where the difference of the pixel is greater than 0 is the defective part.

[0094] Step 442, setting the pixel value of each first pixel in the first difference image to a first value to obtain a processed first difference image; wherein the first pixel refers to a pixel in the first difference image whose pixel value meets the first condition.

[0095] Taking the above steps as an example, the first difference image contains the difference of pixels, and the difference is the difference between the pixel values ​​of the first filtered image and the pixel values ​​of the corresponding position of the original image, and the above difference represents the brightness difference between the pixel values ​​of the first filtered image and the pixel values ​​of the corresponding position of the original image. For example, when the difference is greater than 0, it means that the brightness of the pixel in the first filtered image is greater than the brightness of the pixel at the corresponding position in the original image; when the difference is equal to 0, it means that the brightness of the pixel in the first filtered image is the same as the brightness of the pixel at the corresponding position in the original image; when the difference is less than 0, it means that the brightness of the pixel in the first filtered image is less than the brightness of the pixel at the corresponding position in the original image.

[0096] Step 443 , performing difference truncation and smooth remapping processing according to the pixel value of each pixel in the processed first difference image, and generating a defect detection result image.

[0097] According to the difference of each pixel in the obtained first difference image, difference truncation and smooth remapping processing are performed to generate a defect detection result image.

[0098] Difference truncation means setting a maximum value and a minimum value (such as the maximum value and the minimum value can be set manually), and according to the set maximum value and minimum value, setting the difference of each pixel in the first difference image that is greater than the maximum value to the maximum value, and setting the difference of each pixel in the first difference image that is less than the minimum value to the minimum value, and the difference of each pixel in the first difference image that is between the maximum value and the minimum value is temporarily unchanged. Finally, the difference of each pixel after difference truncation is made to be within the interval of the above maximum value and the minimum value. Smooth remapping processing is to smoothly (or proportionally) map the difference of each pixel after difference truncation to a set interval (such as the interval can be set manually), for example, the interval can be [0,1].

[0099] For example, there are 3 pixel differences in the first difference image, which are 0, 100, and 300, respectively. At the same time, the minimum value of the difference truncation is 50 and the maximum value is 250. Then, the difference values ​​of the 3 pixels obtained after the difference truncation are 50, 100, and 250. Then, the difference values ​​of the above three pixels are mapped to the interval [0, 1] through smooth remapping, so the pixel with a difference value of 50 is mapped to 0, the pixel with a difference value of 100 is mapped to 0.25, and the pixel with a difference value of 250 is mapped to 1. Therefore, the difference values ​​of the three pixels with a difference value of 0, 100, and 300 in the first difference image are 0, 0.25, and 1 after difference truncation and smooth remapping.

[0100] The image composed of the difference values ​​of each pixel obtained after the difference truncation and smooth remapping is called the defect detection result map. The position of the defect in the defect detection result map is the position of the pixel where the difference value is greater than 0.

[0101] The defect detection result map is generated by the first filtered image and the original image through difference truncation and smooth remapping, which clearly reflects the location of the defects in the face area, making the defect elimination in the following step, that is, the processing of the first stage result map, more accurate.

[0102] Step 450 , based on the defect detection result image and the second detection result image, the original image and the first filtered image are mixed to generate a first stage result image.

[0103] The product of the defect detection result image and the second detection result image is used as a weight to mix the original image and the first filtered image to generate a first stage result image.

[0104] Optionally, step 450 includes steps 451 to 452:

[0105] Step 451, determining a first weight matrix based on the difference between pixels in the defect detection result image and the product of the face probability at the corresponding position in the second detection result image.

[0106] Since the pixel values ​​of the pixels in the defect detection result image and the second detection result image are both between 0 and 1, the value of any weight value in the first weight matrix is ​​also between 0 and 1.

[0107] Step 452: Mix the original image and the first filtered image based on the first weight matrix to generate a first stage result image.

[0108] According to the calculation formula of the first weight matrix and the first stage result image obtained above, the original image and the first filtered image are mixed to generate the first stage result image.

[0109] The calculation formula for the first stage result graph is:

[0110] The first stage result image = the first filtered image * the first weight matrix + the original image * (1-the first weight matrix).

[0111] For example, when the pixel is at a point outside the face area, the weight value of the pixel in the first weight matrix is ​​0, and the pixel value of the pixel in the first stage result image is the pixel value corresponding to the original image. When the pixel is at a point in the face area, the value of the pixel in the first weight matrix is ​​set to M, M is greater than 0 and less than or equal to 1, and the pixel value of the pixel in the first stage result image is:

[0112] The pixel value of the pixel in the first filtered image*M+the pixel value of the pixel in the original image*(1-M).

[0113] When M is 1, the pixel value of the pixel in the first stage result image is the pixel value of the pixel in the first filtered image.

[0114] By mixing the original image with the first filtered image, the difference points in the original image and the color difference of the overall skin color are weakened and reduced, so that the image after preliminary beautification does not have difference points such as acne, spots, scars, etc.

[0115] Step 460: Generate a fuzzy result map and an edge result map based on the first stage result map.

[0116] The blurred result image is an image generated after filtering the first stage result image, and is used to generate an edge result image. The edge result image is used to show the edge information of the image background and the position of the facial features, so that the beauty result can not lose this edge information. Optionally, step 460 includes steps 461-462:

[0117] Step 461, performing a second filtering process on the first stage result image to obtain a blurred result image.

[0118] According to the pixel values ​​and weights of the selected pixel and each surrounding pixel, a second filtering process is performed to obtain a blurred result image. The selected pixel is an arbitrary pixel in the result image of the first stage, and the second filtering process adopts mean filtering, that is, the weights of the selected pixel and each surrounding pixel are set to the same value, for example, all are set to 1. Optionally, the selected pixel participates in the calculation process of the second filtered pixel value corresponding to the selected pixel, and the weight value of the target pixel is set to 1. Optionally, the selected pixel does not participate in the calculation process of the second filtered pixel value corresponding to the selected pixel. The present application does not limit whether the selected pixel participates in the calculation process of the second filtered pixel value of the selected pixel.

[0119] With the selected pixel as the center, determine the surrounding pixels corresponding to the selected pixel, and determine the second filtered pixel value corresponding to the selected pixel according to the pixel values ​​and the same weights corresponding to the surrounding pixels. The number of pixels around the selected pixel can be arbitrary. For example, the area composed of the target pixel and its surrounding pixels can be a 3*3 area. Then, according to the pixel values ​​and the same weights of the surrounding pixels in the 3*3 area, determine the second filtered pixel value corresponding to the selected pixel. Optionally, the area composed of the selected pixel and its surrounding pixels can be a 5*5 area. Then, according to the pixel values ​​and the same weights of the surrounding pixels in the 5*5 area, determine the second filtered pixel value corresponding to the selected pixel. This application does not limit the size of the area composed of the selected pixel and the corresponding surrounding pixels.

[0120] In some embodiments, the selected pixel participates in the calculation process of the first filtered pixel value corresponding to the selected pixel, and the weight value of the selected pixel is set to 1, and the area composed of the selected pixel and its surrounding pixels is set to be a 3*3 area. Then, according to the pixel values ​​of each surrounding pixel in the 3*3 area and the same weight, the product results corresponding to each surrounding pixel are obtained by multiplying the pixel values ​​of each surrounding pixel and the same weight, and at the same time, according to the pixel value and weight of the selected pixel itself, the product result corresponding to the selected pixel is obtained by multiplying. The average value of the above 9 product results is taken as the second filtered pixel value of the selected pixel.

[0121] In some embodiments, the selected pixel does not participate in the calculation process of the first filtered pixel value corresponding to the selected pixel, and the area composed of the selected pixel and its surrounding pixels is set to be a 5*5 area. Then, according to the pixel values ​​of each surrounding pixel in the 5*5 area and the same weight, the product results corresponding to each surrounding pixel are obtained by multiplying the pixel values ​​of each surrounding pixel and the same weight. The average of the above two product results is taken as the second filtered pixel value of the selected pixel.

[0122] Step 462, obtaining an edge result map based on the difference between the pixel values ​​at corresponding positions in the first stage result map and the fuzzy result map.

[0123] The pixel value of each pixel in the first stage result image is subtracted from the pixel value of the pixel at the corresponding position in the blurred result image to obtain the difference of each pixel, and then the difference of each pixel is multiplied by the second value to obtain the edge result image. Optionally, the second value is a preset value.

[0124] Optionally, the edge result image obtained above is subjected to mean filtering to obtain a smoother edge of the facial region.

[0125] Based on the first stage result map, a blur result map and an edge result map are generated. If the original image is used to generate the edge result map, some spot areas will be regarded as edges, resulting in the final result being unable to completely remove spots. By using the first stage result map to generate the edge result map, it is effectively prevented from treating the spot areas as edges. At the same time, the edge result map can well reflect the edge information in the image, so that the beauty result can achieve the effect of removing spots and acne and improving skin tone uniformity without losing the edges of facial features.

[0126] Step 470: Generate a skin color unevenness result map according to the blurred result map and the first stage result map.

[0127] The Uneven Skin Tone result map is used to indicate areas on the face where uneven skin tone exists.

[0128] Optionally, step 470 includes steps 471 to 472:

[0129] Step 471, based on the difference between the pixel values ​​at the corresponding positions in the first stage result image and the blurred result image, an initial uneven skin color result image is obtained.

[0130] In some embodiments, the pixel value of each pixel in the first stage result image is subtracted from the pixel value of the pixel at the corresponding position in the blurred result image to obtain the difference of each pixel, and then the difference of each pixel is added to the third value to obtain the skin color unevenness value to obtain the skin color unevenness result image. Optionally, the third value is a pre-set arbitrary value. In actual use, the third value can be set to 0.5.

[0131] The skin color unevenness value is used to indicate whether the pixel value of a pixel in the first stage result image is dark, bright, or uniform. A uniform pixel value of a pixel indicates that the pixel value of the pixel is neither dark nor bright.

[0132] Step 472: Perform a third filtering process on the initial uneven skin color result map to obtain an uneven skin color result map.

[0133] The pixels in the face area in the initial uneven skin color result map are subjected to a third filtering process to obtain an uneven skin color result map, wherein the third filtering process uses a 3*3 Gaussian filter, and the closer the pixel is to the pixel, the greater the weight, and the farther away from the center point, the lower the weight. Optionally, the above-mentioned Gaussian filter may not be used, and the initial uneven skin color result map obtained may be directly used to perform the following steps.

[0134] By processing the blurred result map and the first stage result map, the initial uneven skin color result map is obtained, which reflects the brightness and darkness of the facial area, preparing for the brightness and darkness processing of the facial area in the following steps. At the same time, the initial uneven skin color result map is further processed by Gaussian filtering to obtain the uneven skin color result map, which can better retain the skin color texture details.

[0135] Step 480, based on the uneven skin color result map, the first stage result map is processed by using an inverse contrast enhancement method to obtain a second stage result map; wherein the inverse contrast enhancement method is used to narrow the brightness between different pixels.

[0136] The inverse contrast enhancement method is the opposite method of the contrast enhancement method. The contrast enhancement method is to move the value away from the middle value and make the value proportionally expand on both sides of the middle value when the middle value is set. Therefore, the inverse contrast enhancement method is to move the value closer to the middle value and make the value proportionally reduce to the middle value. In this application, the inverse contrast enhancement method is used to close the brightness between different pixels, so that the brightness tends to the middle value and the brightness of the face area is similar.

[0137] Optionally, step 480 includes steps 481 to 482:

[0138] Step 481, determining a first pixel set and a second pixel set according to the skin color unevenness result map; wherein the first pixel set includes pixels whose skin color unevenness values ​​in the skin color unevenness result map belong to a first numerical interval, the second pixel set includes pixels whose skin color unevenness values ​​in the skin color unevenness result map belong to a second numerical interval, and the skin color unevenness values ​​of the pixels in the first pixel set are greater than the skin color unevenness values ​​of the pixels in the second pixel set.

[0139] In some embodiments, taking the embodiment of step 471 as an example, when the skin color unevenness value is greater than 0.5, it means that the pixel value of the pixel in the first stage result image is greater than the pixel value of the pixel at the corresponding position in the blurred result image, that is, it is the bright part of the skin color uneven area, and the pixel value of the corresponding pixel in the first stage result image needs to be reduced; when the skin color unevenness value is less than 0.5, it means that the pixel value of the pixel in the first stage result image is less than the pixel value of the pixel at the corresponding position in the blurred result image, that is, it is the dark part of the skin color uneven area, and the pixel value of the corresponding pixel in the first stage result image needs to be increased; when the skin color unevenness value is equal to 0.5, it indicates a uniform skin color area, and the pixel value of the pixel in the first stage result image corresponding to the skin color unevenness value is the middle value when the inverse contrast enhancement method is used, and the uniform area is not processed when the inverse contrast enhancement method is used.

[0140] In some embodiments, taking the above embodiment as an example, the pixel value of the pixel in the first stage result map corresponding to the skin color unevenness value of the pixel in the skin color unevenness result map is equal to 0.5 is the middle value of the inverse contrast enhancement method, wherein the area where the pixel with the middle value is located is also called a uniform area, and the area where other pixels are located is called an uneven area.

[0141] Step 482, reducing the pixel values ​​of the pixels belonging to the first pixel set in the first stage result image, and increasing the pixel values ​​of the pixels belonging to the second pixel set in the first stage result image, to obtain a second stage result image.

[0142] According to the calculation formula of the above uneven skin color results, when the uneven skin color value is less than 0.5, the corresponding pixel in the first stage result image is a local dark area, and its pixel value needs to be increased; when the uneven skin color value is greater than 0.5, the corresponding pixel is a local bright area, and its pixel value needs to be reduced.

[0143] The above-mentioned inverse contrast enhancement method is used to narrow the brightness and darkness between different pixels, that is, the pixel values ​​of the pixels belonging to the first pixel set in the first stage result image are reduced, and the pixel values ​​of the pixels belonging to the second pixel set in the first stage result image are increased. The uniform skin color area is not processed to obtain the second stage result image.

[0144] In some embodiments, the pixel value of the pixel in the first stage result image is increased or decreased to different degrees according to the difference between the skin color unevenness value of the pixel and the middle value, wherein the greater the difference between the skin color unevenness value of the pixel and the middle value, the greater the degree of increase or decrease of the pixel value of the pixel in the first stage result image; the smaller the difference between the skin color unevenness value of the pixel and the middle value, the smaller the degree of increase or decrease of the pixel value of the pixel in the first stage result image.

[0145] For example, the pixel value of the pixel in the first stage result image is set to A, the uneven skin color value of the pixel in the uneven skin color result image is set to B, the pixel value of the pixel in the second stage result image is set to C, and the unevenness value is set to D. The unevenness value is used to indicate the unevenness degree of the facial area in the first stage result image.

[0146] Among them, the skin color unevenness value of the pixel in the skin color unevenness result map is used to describe the skin color uniformity of the first stage result map. According to the skin color unevenness value of the pixel in the skin color unevenness result map, the first stage result map is divided into a uniform area, a bright part of the local uneven area, and a dark part of the local uneven area. Different processing is taken for the pixel values ​​of the three areas in the above-mentioned first stage result map.

[0147] Among them, when B>0.5, that is, the first pixel set mentioned above, that is, the local uneven bright area in the first stage result image, the unevenness degree is calculated by the skin color unevenness value:

[0148]

[0149] Then the pixel value of the pixel in the second stage result image is:

[0150] C=A-(1-A)D

[0151] When B<0.5, that is, the second pixel set mentioned above, that is, the local uneven dark area in the first stage result map, the unevenness degree is calculated through the skin color unevenness result map:

[0152]

[0153] Then the pixel value of the pixel in the second stage result image is:

[0154] C=A+AD

[0155] When B=0.5, the unevenness degree D=0 is calculated by the skin color unevenness value, and the pixel value of the pixel in the second stage result image is the same as the pixel value of the pixel in the first stage result image.

[0156] The first stage result image is subjected to an inverse contrast enhancement method according to the unevenness degree, wherein when the difference between multiple skin color unevenness values ​​and the middle value in the skin color unevenness result image is the same, the unevenness degree of the corresponding region in the first stage result image is the same. For example, when the middle value is 0.5, and two skin color unevenness values ​​in the skin color unevenness result image are 0.4 and 0.6 respectively, the difference from 0.5 is 0.1, and the unevenness degree of the corresponding region in the first stage result image is the same.

[0157] The inverse contrast enhancement method is used to make the pixel values ​​of the pixels in the first stage result image close to each other, making the overall skin color in the second stage result image more harmonious and consistent.

[0158] Step 490, generating a final result image based on the original image and the second stage result image.

[0159] Step 490 has been introduced in the previous embodiment and will not be repeated here.

[0160] Optionally, step 490 includes steps 491 to 494:

[0161] Step 491, obtaining a second difference image based on the difference between the pixel values ​​at corresponding positions in the second stage result image and the original image.

[0162] The pixel value of the pixel at the corresponding position in the original image is subtracted from the pixel value of the pixel in the second stage result image to obtain the pixel difference and the second difference image. The pixel value of the pixel in the second difference image is the difference obtained by the above method.

[0163] Step 492, performing difference truncation and smooth remapping processing according to the pixel value of each pixel in the second difference image to generate an intermediate result image.

[0164] Similarly, the pixel value interval of each pixel in the second difference image is selected and compressed through difference truncation and smooth remapping processing, so that the pixel difference in the obtained intermediate result image is compressed to the interval of 0 to 1.

[0165] Step 493: Generate a second weight matrix based on the first detection result image, the second detection result image, the edge result image corresponding to the original image, the intermediate result image, and the beauty intensity value, wherein the beauty intensity value is used to adjust the proportion of the original image and the second stage result image when they are mixed.

[0166] The second weight matrix is ​​obtained by multiplying the skin color probability of the pixels of the first detection result image, the face probability of the pixels of the second detection result image, the difference of the pixels of the edge result image corresponding to the original image, the difference of the pixels of the intermediate result image, and the beautification intensity value, wherein the skin color probability of the pixels of the first detection result image, the face probability of the pixels of the second detection result image, the difference of the pixels of the edge result image corresponding to the original image, and the difference of the pixels of the intermediate result image are calculated by the above method, and the beautification intensity value is adjusted according to the user, and the size of the second weight matrix is ​​determined according to the beautification intensity value adjusted by the user.

[0167] Step 494, mixing the original image and the second stage result image based on the second weight matrix to generate a final result image.

[0168] According to the calculation formula of the second weight matrix and the final result image obtained above, the original image and the intermediate result image are mixed to generate the final result image.

[0169] The calculation formula for the final result graph is:

[0170] Final result image = second stage result image * second weight matrix + original image * (1-second weight matrix)

[0171] In some implementations, when the user does not use the beautification function, the beautification strength value is set to 1, the weight of the second-stage result image is 0, and the final result image is the original image; when the user uses the beautification function, the corresponding beautification strength value is set, and the second weight matrix is ​​calculated to be N, where N is greater than 0 and less than or equal to 1, and the pixel value of the pixel in the final result image is:

[0172] Final result image = second stage result image * N + original image * (1-N).

[0173] In this embodiment, a fuzzy result image is first generated by the first stage result image, and then an edge result image is generated by processing the first stage result image and the fuzzy result image. The edge result image can well reflect the edge information in the image, so that the beautification result can achieve the effect of removing spots and acne and improving the uniformity of skin color without losing the edges of facial features.

[0174] At the same time, the initial skin color uneven result map is filtered by Gaussian filtering to obtain a skin color uneven result map. Compared with processing the initial skin color uneven result map, processing the skin color uneven result map obtained after Gaussian filtering can better improve the uniformity of the skin color and better retain the skin texture details.

[0175] In addition, a beautification strength value is set so that the user can control the desired degree of beautification by adjusting the beautification strength value.

[0176] Please refer to Figure 5, which shows a schematic diagram of the overall steps of an image processing method provided by an embodiment of the present application.

[0177] First, the original image to be processed is obtained, and the skin color detection result m2 is obtained by performing skin color detection on the original image, which is the first detection result image. The face detection result m1 is obtained by performing face detection on the original image, which is the second detection result image. The original image is subjected to edge-preserving skin color filtering to obtain the first filtered image. The large defect area detection image is obtained by subtracting the first filtered image obtained by edge-preserving skin color filtering from the original image and performing difference truncation and smooth remapping, which is the defect detection result image. Then, the mixed weight is obtained by multiplying the face probability in the face detection result m1 and the difference between the pixels at the corresponding positions in the large defect area detection image, which is the first weight matrix. The original image and the large defect area detection image are calculated and summed according to the first weight matrix to obtain the freckle and acne removal result image, which is the first stage result image.

[0178] Then, the freckle and acne removal result image is processed by mean filtering to obtain a blurred result image, and then the difference of the pixel values ​​of the corresponding position pixels is calculated based on the first stage result image and the blurred result image to obtain the edge detection result m3, which is the edge result image. Then, based on the difference of the pixel values ​​of the corresponding position pixels of the blurred result image obtained after mean filtering and the freckle and acne removal result image, an uneven area detection image is obtained, which is the initial skin color uneven result image. The initial skin color uneven result image can be processed by Gaussian filtering to obtain the skin color uneven result image. According to the pixel value of the freckle and acne removal result image corresponding to the middle value of the pixel determined in the uneven area detection image, the freckle and acne removal result image is inversely contrast enhanced to obtain a uniform skin color result image, which is the second stage result image. The uniform skin color result image and the original image are differentially truncated and smoothly remapped to obtain large facial blemishes and lighter uneven areas m4, which is the intermediate result image. Finally, the above m1, m2, m3, m4 are multiplied by the beautification intensity value set by the user to obtain the mixing weight, that is, the second weight matrix. The original image and the skin color uniformity result image are mixed according to the second weight matrix to obtain the face beautification result, that is, the final result image.

[0179] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0180] Please refer to Figure 6, which shows a block diagram of an image processing device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image processing method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be a computer device, or it can be set in a computer device. The device 600 may include: an original image acquisition module 610, an original image detection module 620, a first processing module 630, a second processing module 640 and a result generation module 650.

[0181] The original image acquisition module 610 is used to acquire the original image to be processed.

[0182] The original image detection module 620 is used to perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image; wherein the first detection result image is used to characterize the skin color area in the original image, and the second detection result image is used to characterize the face area in the original image.

[0183] The first processing module 630 is used to perform a first stage processing on the original image based on the first detection result image and the second detection result image to obtain a first stage result image; wherein the first stage processing is used to remove the area in the facial area that is different from the overall skin color.

[0184] The second processing module 640 is used to perform a second stage processing on the first stage result image to obtain a second stage result image; wherein the second stage processing is used to improve the uniformity of skin color at different positions in the facial area.

[0185] The result generation module 650 is used to generate a final result image based on the original image and the second-stage result image.

[0186] In some embodiments, the first processing module 630 includes: a first filtering unit 631 , a defect result generating unit 632 , and a first result generating unit 633 .

[0187] The first filtering unit 631 is used to perform a first filtering process on the original image based on the first detection result image and the second detection result image to obtain a first filtered image; wherein the first filtering process is used to filter the face area while retaining the edge of the original image.

[0188] The defect result generating unit 632 is used to generate a defect detection result map based on the first filtered image and the original image; wherein the defect detection result map is used to represent the defect position in the facial area.

[0189] The first result generating unit 633 is used to mix the original image and the first filtered image based on the defect detection result image and the second detection result image to generate the first stage result image.

[0190] In some embodiments, the first filtering unit 631 is used to:

[0191] Based on the second detection result image, determining the face area in the original image;

[0192] For a target pixel in the face area, determining first filtering weights corresponding to each of the surrounding pixels according to a pixel value difference between the target pixel and each of the surrounding pixels and skin color probabilities corresponding to each of the surrounding pixels; wherein the skin color probabilities are obtained based on the first detection result map;

[0193] Determine a first filtered pixel value corresponding to the target pixel according to the pixel values ​​corresponding to the surrounding pixels and the first filtering weight;

[0194] The first filtered image is obtained according to the first filtered pixel values ​​respectively corresponding to the pixels in the face area.

[0195] In some embodiments, the defect result generating unit 632 is used to:

[0196] Obtaining a first difference image based on the difference between the pixel values ​​at corresponding positions in the first filtered image and the original image;

[0197] Setting the pixel value of each first pixel in the first difference image to a first value to obtain a processed first difference image; wherein the first pixel refers to a pixel in the first difference image whose pixel value meets a first condition;

[0198] According to the pixel value of each pixel in the processed first difference image, difference truncation and smooth remapping processing are performed to generate the defect detection result image.

[0199] In some embodiments, the first result generating unit 633 is used to:

[0200] Determining a first weight matrix based on the product of the pixel values ​​at corresponding positions in the defect detection result map and the second detection result map;

[0201] The original image and the first filtered image are mixed based on the first weight matrix to generate the first stage result image.

[0202] In some embodiments, the second processing module 640 includes: a first result using unit 641 , a skin color result generating unit 642 and a second result generating unit 643 .

[0203] The first result using unit 641 is used to generate a fuzzy result map and an edge result map based on the first stage result map.

[0204] The skin color result generating unit 642 is used to generate a skin color unevenness result map according to the blur result map and the first stage result map.

[0205] The second result generating unit 643 is used to process the first stage result image according to the uneven skin color result image by using an inverse contrast enhancement method to obtain the second stage result image; wherein the inverse contrast enhancement method is used to narrow the brightness between different pixels.

[0206] In some embodiments, the first result using unit 641 is configured to:

[0207] Performing a second filtering process on the first stage result image to obtain the fuzzy result image;

[0208] The edge result image is obtained based on the difference between the pixel values ​​at corresponding positions in the first stage result image and the fuzzy result image.

[0209] In some embodiments, the skin color result generating unit 642 is used to:

[0210] Based on the difference between the pixel values ​​at the corresponding positions in the first stage result image and the blurred result image, an initial uneven skin color result image is obtained;

[0211] A third filtering process is performed on the initial uneven skin color result map to obtain the uneven skin color result map.

[0212] In some embodiments, the second result generating unit 643 is configured to:

[0213] Determine a first pixel set and a second pixel set according to the uneven skin color result map; wherein the first pixel set includes pixels whose uneven skin color values ​​in the uneven skin color result map belong to a first numerical interval, the second pixel set includes pixels whose uneven skin color values ​​in the uneven skin color result map belong to a second numerical interval, and the uneven skin color values ​​of the pixels in the first pixel set are greater than the uneven skin color values ​​of the pixels in the second pixel set;

[0214] The pixel values ​​of the pixels belonging to the first pixel set in the first stage result image are reduced, and the pixel values ​​of the pixels belonging to the second pixel set in the first stage result image are increased to obtain the second stage result image.

[0215] In some embodiments, the result generation module 650 is used to:

[0216] Based on the difference between the pixel values ​​at the corresponding positions in the second-stage result image and the original image, a second difference image is obtained;

[0217] Perform difference truncation and smooth remapping processing according to the pixel value of each pixel in the second difference image to generate an intermediate result image;

[0218] Generate a second weight matrix based on the first detection result image, the second detection result image, the edge result image corresponding to the original image, the intermediate result image and the beautification strength value; wherein the beautification strength value is used to adjust the respective proportions of the original image and the second stage result image when they are mixed;

[0219] The original image and the second-stage result image are mixed based on the second weight matrix to generate the final result image.

[0220] This application selects the skin color area and face area in the original image by performing skin color detection and face detection on the original image, and weakens the area in the face area that is significantly different from the overall skin color according to the first stage processing, thereby reducing the difference between the skin color of this area and the overall skin color of the face area. The second stage processing improves the uniformity of the skin color in the face area to obtain the final image processing result. On the one hand, the first stage of processing weakens the area in the facial area that is significantly different from the overall skin color, and reduces the difference between the skin color of the area and the overall skin color of the facial area. The second stage of processing improves the uniformity of the skin color in the facial area, strengthens the blemish removal capability, and better preserves the skin texture. On the other hand, based on the first detection result map and the second detection result map obtained by skin color detection and face detection, the facial skin color area in the original image is determined, and the image beauty processing is performed on the above-mentioned facial skin color area, and the background area and the edges of the facial features are not subjected to image beauty processing, thereby reducing the wear of the background area and the edges of the facial features during beauty processing. On the other hand, this solution can complete the image beauty processing only through simple calculations, and does not need to perform a large amount of calculations by using neural networks. The calculation process of beauty processing is optimized, thereby shortening the time required, and can meet the needs of some application scenarios with high real-time requirements. For example, the above-mentioned beauty optimization process can be performed while shooting, with high real-time performance. Therefore, the present application provides an image beauty solution that takes into account the blemish removal capability, the skin texture retention capability, and high real-time performance.

[0221] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0222] In an exemplary embodiment, a computer device is also provided. The computer device includes a processor and a memory, in which a computer program is stored. The computer device may be the terminal 10 and the server 20 described above, and the computer program is loaded and executed by the processor to implement the above-mentioned image processing method.

[0223] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned image processing method.

[0224] In an exemplary embodiment, a computer program product is also provided. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned image processing method.

[0225] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0226] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0227] The above are merely exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Get the original image to be processed; Perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image; wherein the first detection result image is used to represent the skin color area in the original image, and the second detection result image is used to represent the face area in the original image; Based on the first detection result image and the second detection result image, performing a first filtering process on the original image to obtain a first filtered image; wherein the first filtering process is used to perform filtering process on the face area while retaining the edge in the original image; Based on the first filtered image and the original image, generating a defect detection result map; wherein the defect detection result map is used to characterize the defect position in the facial area; Based on the defect detection result image and the second detection result image, the original image and the first filtered image are mixed to generate a first stage result image; Based on the result graph of the first stage, a fuzzy result graph and an edge result graph are generated; Generate an uneven skin color result map according to the blurred result map and the first stage result map; According to the uneven skin color result map, the first stage result map is processed by using an inverse contrast enhancement method to obtain a second stage result map; wherein the inverse contrast enhancement method is used to narrow the brightness and darkness between different pixels; Based on the original image and the second-stage result image, a final result image is generated.

2. The method according to claim 1, characterized in that The step of performing a first filtering process on the original image based on the first detection result image and the second detection result image to obtain a first filtered image includes: Based on the second detection result image, determining the face area in the original image; For a target pixel in the face area, determining first filtering weights corresponding to each of the surrounding pixels according to a pixel value difference between the target pixel and each of the surrounding pixels and skin color probabilities corresponding to each of the surrounding pixels; wherein the skin color probabilities are obtained based on the first detection result map; Determine a first filtered pixel value corresponding to the target pixel according to the pixel values ​​corresponding to the surrounding pixels and the first filtering weight; The first filtered image is obtained according to the first filtered pixel values ​​respectively corresponding to the pixels in the face area.

3. The method according to claim 1, characterized in that: The step of generating a defect detection result image based on the first filtered image and the original image includes: Obtaining a first difference image based on the difference between the pixel values ​​at corresponding positions in the first filtered image and the original image; Setting the pixel value of each first pixel in the first difference image to a first value to obtain a processed first difference image; wherein the first pixel refers to a pixel in the first difference image whose pixel value meets a first condition; According to the pixel value of each pixel in the processed first difference image, difference truncation and smooth remapping processing are performed to generate the defect detection result image.

4. The method according to claim 1, characterized in that The mixing of the original image and the first filtered image based on the defect detection result image and the second detection result image to generate a first stage result image includes: Determining a first weight matrix based on the product of the pixel values ​​at corresponding positions in the defect detection result map and the second detection result map; The original image and the first filtered image are mixed based on the first weight matrix to generate the first stage result image.

5. The method according to claim 1, characterized in that The generating of a fuzzy result map and an edge result map based on the result map of the first stage includes: Performing a second filtering process on the first stage result image to obtain the fuzzy result image; The edge result image is obtained based on the difference between the pixel values ​​at corresponding positions in the first stage result image and the fuzzy result image.

6. The method according to claim 1, characterized in that The step of generating an uneven skin color result map according to the blurred result map and the first stage result map comprises: Based on the difference between the pixel values ​​at the corresponding positions in the first stage result image and the blurred result image, an initial uneven skin color result image is obtained; A third filtering process is performed on the initial uneven skin color result map to obtain the uneven skin color result map.

7. The method according to claim 1, characterized in that The method of processing the first stage result image according to the uneven skin color result image by using an inverse contrast enhancement method to obtain a second stage result image includes: Determine a first pixel set and a second pixel set according to the uneven skin color result map; wherein the first pixel set includes pixels whose uneven skin color values ​​in the uneven skin color result map belong to a first numerical interval, the second pixel set includes pixels whose uneven skin color values ​​in the uneven skin color result map belong to a second numerical interval, and the uneven skin color values ​​of the pixels in the first pixel set are greater than the uneven skin color values ​​of the pixels in the second pixel set; The pixel values ​​of the pixels belonging to the first pixel set in the first stage result image are reduced, and the pixel values ​​of the pixels belonging to the second pixel set in the first stage result image are increased to obtain the second stage result image.

8. The method according to any one of claims 1 to 7, characterized in that: The step of generating a final result image based on the original image and the second-stage result image comprises: Based on the difference between the pixel values ​​at the corresponding positions in the second-stage result image and the original image, a second difference image is obtained; Perform difference truncation and smooth remapping processing according to the pixel value of each pixel in the second difference image to generate an intermediate result image; Generate a second weight matrix based on the first detection result image, the second detection result image, the edge result image corresponding to the original image, the intermediate result image and the beautification strength value; wherein the beautification strength value is used to adjust the respective proportions of the original image and the second stage result image when they are mixed; The original image and the second-stage result image are mixed based on the second weight matrix to generate the final result image.

9. An image processing device, characterized in that: The device comprises: An original image acquisition module, used to acquire the original image to be processed; An original image detection module, used to perform skin color detection and face detection on the original image to obtain a first detection result image and a second detection result image; wherein the first detection result image is used to characterize the skin color area in the original image, and the second detection result image is used to characterize the face area in the original image; a first processing module, configured to perform a first filtering process on the original image based on the first detection result image and the second detection result image to obtain a first filtered image; wherein the first filtering process is used to perform filtering process on the face region while retaining the edge in the original image; generate a defect detection result image based on the first filtered image and the original image; wherein the defect detection result image is used to characterize the defect position in the face region; and, based on the defect detection result image and the second detection result image, mix the original image and the first filtered image to generate a first stage result image; A second processing module is used to generate a blur result map and an edge result map based on the first stage result map; generate an uneven skin color result map according to the blur result map and the first stage result map; and process the first stage result map using an inverse contrast enhancement method according to the uneven skin color result map to obtain a second stage result map; wherein the inverse contrast enhancement method is used to narrow the brightness between different pixels; The result generation module is used to generate a final result image based on the original image and the second-stage result image.

10. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the method according to any one of claims 1 to 8.

Images