Image Processing Method, Apparatus, Computer Device, and Storage Medium

Abstract

This application relates to the fields of artificial intelligence and computer vision, and to an image processing method, apparatus, computer device, and storage medium. The method comprises: smoothing an original image to obtain a smoothed image; determining an initial skin region in the smoothed image; determining pixels in the original image where pixel values significantly change to obtain texture edge information; removing the texture edge region in the initial skin region based on the texture edge information to obtain a target skin region; and fusing the image content in the smoothed image within the target skin region into the corresponding position of the target skin region in the original image to obtain a target image. This method can improve image quality.

Description

Technical Field

[0001] The present application relates to computer technology and image processing technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of artificial intelligence, especially image processing in the field of computer vision, image beautification technologies are emerging in an endless stream and are widely used. For example, facial enhancement in selfies, video calls, or live broadcasts can be applied to enhance facial features. Smoothing skin in images is a key area of research.

[0003] Traditional methods typically use edge-preserving filters, such as bilateral or guided filters, to smooth the image after skin refining. This allows for global skin refining while maintaining clear contours. However, global skin refining reduces the clarity of the image and degrades image quality. Summary of the Invention

[0004] Based on this, it is necessary to provide an image processing method, apparatus, computer equipment and storage medium that can improve image quality in response to the above technical problems.

[0005] An image processing method, comprising:

[0006] Smoothing the original image to obtain a smoothed image;

[0007] determining an initial skin region in the smoothed image;

[0008] Determining pixel points in the original image where pixel values change significantly to obtain texture edge information;

[0009] removing the texture edge region in the initial skin region based on the texture edge information to obtain a target skin region;

[0010] The image content in the target skin area of the smoothed image is fused into the corresponding position of the target skin area in the original image to obtain a target image.

[0011] In one embodiment, smoothing the original image to obtain a smoothed image includes:

[0012] Determine the downsampling multiple according to the resolution of the original image;

[0013] Downsampling the original image according to the downsampling multiple;

[0014] The downsampled image is smoothed to obtain a smoothed image.

[0015] In one embodiment, the method further comprises:

[0016] determining an edge mask according to the texture edge information;

[0017] The removing of the texture edge region in the initial skin region based on the texture edge information to obtain the target skin region includes:

[0018] fusing the initial mask image used to determine the initial skin area with the edge mask to obtain a target mask image; the target mask image is used to determine the target skin area;

[0019] The step of fusing the image content in the target skin area in the smoothed image into a corresponding position of the target skin area in the original image to obtain the target image comprises:

[0020] According to the target mask image, the image content in the target skin area in the smoothed image is fused into the corresponding position of the target skin area in the original image to obtain a target image.

[0021] In one embodiment, the method further comprises:

[0022] Acquiring a binary mask image for determining an initial skin area of the smoothed image;

[0023] Obtaining pixel values within the initial skin area corresponding to the binary mask image from a single color channel of the smoothed image;

[0024] The non-binary initial mask image for determining the initial skin area is generated based on the acquired pixel values; the pixel values of the points in the initial mask image corresponding to the initial skin area are determined based on the acquired pixel values.

[0025] In one embodiment, the step of fusing the image content in the target skin area in the smoothed image into a corresponding position of the target skin area in the original image based on the target mask image to obtain the target image includes:

[0026] Superimposing and fusing the first image and the second image to obtain a target image;

[0027] The first image is an image obtained by multiplying the pixel value corresponding to each point in the smoothed image by the pixel value corresponding to each point in the target mask image;

[0028] The second image is an image obtained by multiplying the pixel value corresponding to each point in the original image by the pixel value corresponding to each point in the inverse mask image of the target mask image.

[0029] In one embodiment, the method further comprises:

[0030] Determine the pixel value interval range to which the pixel value of each point in the target mask image belongs;

[0031] According to the significance coefficient corresponding to the pixel value interval range, the pixel value of the corresponding point in the target mask image is significance weighted to obtain a significance-weighted target mask image.

[0032] In one embodiment, determining the edge mask according to the texture edge information includes:

[0033] An inverse operation is performed on the texture edge information by an inversion function to obtain an edge mask.

[0034] In one embodiment, the texture edge information is a texture edge information map; and determining pixel points in the original image where pixel values significantly change to obtain the texture edge information includes:

[0035] determining difference information between the original image and the smoothed image;

[0036] The texture edge information map is obtained according to the difference information.

[0037] In one embodiment, the method further comprises:

[0038] extracting high-frequency information from the original image based on difference information between the original image and the smoothed image;

[0039] The high-frequency information is added to the target image to obtain a sharpened target image.

[0040] In one embodiment, the original image contains a human face; the method further includes:

[0041] Determining facial features of a person in the original image;

[0042] According to the facial feature areas, the target image is fused with the original image to obtain a target image that retains facial feature detail features.

[0043] In one embodiment, the original image contains a human face; the method further includes:

[0044] Obtaining a face region corresponding to a face in the original image;

[0045] According to the facial region, the target image is fused with the original image to obtain a target image with an optimized facial region.

[0046] In one embodiment, the method further comprises:

[0047] Get the smoothing parameters of the input;

[0048] Determining the superposition weights corresponding to the target image and the original image respectively according to the smoothing parameter;

[0049] Superimposing and fusing the target image and the original image according to corresponding superposition weights;

[0050] Output the superimposed fused image.

[0051] An image processing device, comprising:

[0052] A smoothing processing module is used to smooth the original image to obtain a smoothed image;

[0053] an initial region determination module, configured to determine an initial skin region in the smoothed image;

[0054] An edge determination module is used to determine pixel points in the original image where pixel values change significantly, and obtain texture edge information;

[0055] a target area determination module, configured to remove a texture edge area in the initial skin area based on the texture edge information to obtain a target skin area;

[0056] The fusion module is used to fuse the image content located in the target skin area in the smoothed image into the corresponding position of the target skin area in the original image to obtain a target image.

[0057] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the image processing method described in each embodiment of the present application.

[0058] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the image processing method described in each embodiment of the present application.

[0059] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps of the image processing method described in each embodiment of the present application.

[0060] The above-mentioned image processing method, device, computer equipment and storage medium smooth the original image to obtain a smoothed image, then determine the initial skin area in the smoothed image, and determine the pixel points in the original image where the pixel values change significantly to obtain texture edge information, and then remove the texture edge area in the initial skin area based on the texture edge information to obtain the target skin area, and fuse the image content located in the target skin area in the smoothed image into the corresponding position of the target skin area in the original image to obtain the target image, so that the image content within the target skin area in the fused target image is smoothed, while the image content outside the target skin area retains the clarity of the original image, avoiding the problem of reducing the transparency of the image by smoothing the original image as a whole, thereby improving the image quality, and fuse the image content located in the target skin area where the texture edge area is removed in the smoothed image into the corresponding position of the target skin area in the original image, also avoiding the smoothing of the texture edge area in the original image, making the texture edge details clear, and further improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A diagram showing an application environment of an image processing method in one embodiment;

[0062] Figure 2 A diagram showing an application environment of an image processing method in another embodiment;

[0063] Figure 3 1 is a flow chart of an image processing method according to an embodiment;

[0064] Figure 4 is a graph of an inversion function in one embodiment;

[0065] Figure 5 is a schematic diagram of a non-binary facial feature mask in one embodiment;

[0066] Figure 6 1 is a schematic diagram of the overall flow of an image processing method in one embodiment;

[0067] Figure 7 is a structural block diagram of an image processing device in one embodiment;

[0068] Figure 8is a structural block diagram of an image processing device in another embodiment;

[0069] Figure 9 is a diagram of the internal structure of a computer device in one embodiment;

[0070] Figure 10 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0072] The image processing method provided in this application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. Server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0073] Specifically, the user 106 can shoot the original video or original image through the terminal 102, or input the pre-stored original video or original image, and the terminal 102 can send the original video or original image to the server 104. The server 104 can execute the steps in the image processing method in each embodiment of the present application to obtain the target image based on each frame of the original image in the received original video or the received original image. The server 104 can send the obtained target image to the terminal 102, and the terminal 102 can display the target image, and the user 106 can see the displayed target image through the terminal 102.

[0074] In other embodiments, the terminal 102 may not send the original video or original image to the server 104 for image processing, but the terminal 102 may directly perform image processing itself and then directly display the target image obtained by the image processing.

[0075] It can be understood that the application environment in the above embodiment can be applied to scenarios such as real-time skin smoothing and beautification during selfie-taking, skin smoothing and beautification of images taken after selfie-taking, and skin smoothing and beautification of input local videos or images.

[0076] The image processing method provided by this application can also be applied to Figure 2 In the application environment shown, the first terminal 202 and the second terminal 204 communicate with the server 206 via a network. The first terminal 202 and the second terminal 204 can be, but are not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server 206 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0077] Specifically, the first user 208 can shoot the original video in real time through the first terminal 202. The first terminal 202 can send the original video or each frame of the original image in the original video to the server 206 in real time. The server 206 can perform the steps of the image processing method in each embodiment of the present application to obtain the target image based on each frame of the original image in the original video. The server 206 can send the obtained target image to the first terminal 202 and the second terminal 204. The first terminal 202 and the second terminal 204 can display the target image in real time. The first user 208 can see the displayed target image through the first terminal 202 used, and the second user 210 can see the displayed target image through the second terminal 204 used. There is at least one second terminal 204 and at least one second user 210, and the second user 210 corresponds to the second terminal 204 one-to-one.

[0078] In other embodiments, the first terminal 202 may not send the original video to the server 206 for image processing. Instead, the first terminal 202 itself directly performs image processing, displays the target image, and sends the target image to the second terminal 204 to achieve end-to-end communication between the first terminal 202 and the second terminal 204.

[0079] It is understood that the application environment in the above embodiment can be applied to scenarios such as video calls or live broadcasts. In the video call scenario, user 208 is one party in the video call, and user 210 is the other party in the video call with user 208. In the live broadcast scenario, user 208 is the host of the live broadcast, and user 210 is the audience watching the live broadcast.

[0080] It can be understood that the image processing method in each embodiment of the present application can adopt computer vision technology in artificial intelligence technology, etc., which can effectively improve the image quality of the target image.

[0081] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0082] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0083] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Further image processing is performed, allowing the computer to generate images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0084] In one embodiment, Figure 3 As shown, an image processing method is provided. The image processing method can be executed by a server or a terminal, or can be executed by the terminal and the server together. In the embodiment of the present application, the method is applied to Figure 1 or Figure 2 The following steps are used as an example to illustrate the server in the example:

[0085] Step 302: Smoothing the original image to obtain a smoothed image.

[0086] The original image is the unprocessed image containing the skin area. Smoothing (also known as blurring) is a process used to reduce noise or distortion in an image. The smoothed image is the image obtained after smoothing.

[0087] In one embodiment, the original image may be a pre-stored image or an image frame from a pre-stored video. In another embodiment, the original image may be an image captured in real time during a selfie, video call, or live broadcast.

[0088] Specifically, the server may perform filtering processing on the original image to obtain a smoothed image.

[0089] In one embodiment, the filtering process may be spatial domain filtering or frequency domain filtering.

[0090] In one embodiment, the server may use a sparse template as a convolution kernel to perform spatial filtering on the original image. Specifically, the server may use the sparse template as a convolution kernel to convolve the original image to obtain a smoothed image. A sparse template refers to a template in which non-zero pixel values are sparsely distributed, i.e., the non-zero pixel values in the template are not adjacent but are separated by zero pixel values. In this embodiment, by using a sparse template for spatial filtering, the amount of computation can be reduced, power consumption during image processing can be reduced, and processing efficiency can be improved.

[0091] In one embodiment, the server may use an X-shaped sparse template to perform spatial domain filtering, wherein the X-shaped sparse template refers to a template whose pixel value distribution is shaped like the letter "X".

[0092] In one embodiment, the number of layers of the X-shaped sparse template can be set arbitrarily. For example, the specific form of the four-layer X-shaped sparse template kernel can be as follows:

[0093]

[0094] As can be seen from the above formula, the distribution of pixel value 1 in the four-layer X-shaped sparse template is like the letter "X", and the pixel value 1 is sparsely distributed and separated by pixel value 0, so it is a sparse template.

[0095] In one embodiment, the server may first downsample the original image and then smooth the downsampled image to obtain a smoothed image, thereby improving processing efficiency and raising the upper limit of the smoothness. In one embodiment, the smoothing process of the downsampled image can be expressed by the following formula:

[0096] S ds =kernel*I ds ;

[0097] Among them, I ds The image is obtained by downsampling the original image, and kernel is the convolution kernel that performs spatial filtering on the image. ds is the smoothed image obtained after smoothing.

[0098] In other embodiments, the server may also use filtering methods such as Gaussian filtering or bilateral filtering to perform smoothing.

[0099] Step 304: Determine the initial skin area in the smoothed image.

[0100] The initial skin region is a skin region containing texture edge information in the smoothed image.

[0101] In one embodiment, the server may determine the initial skin area in the smoothed image according to the color distribution of the smoothed image.

[0102] In one embodiment, the server may determine, based on the color distribution of the smoothed image, an area in the smoothed image having the color of the skin surface as the initial skin area.

[0103] In one embodiment, the server may collect statistics on the distribution of pixel values of the smoothed image in each color channel related to the skin surface color, and determine the area in the smoothed image whose color is the skin surface color.

[0104] In one embodiment, the server may determine the area in the smoothed image whose color is the skin surface color based on the distribution of pixel values in the R channel (i.e., the red color channel) and the B channel (i.e., the blue color channel) of the smoothed image in the RGB color space.

[0105] In one embodiment, the server may determine the area in the smoothed image whose color is the skin surface color based on the pixel value of each point in the smoothed image in the R channel and the size relationship between the pixel value of each point in the R channel and the pixel value in the B channel.

[0106] In one embodiment, the server can determine the area composed of points in the smoothed image whose pixel values in the R channel belong to a preset pixel value range and whose pixel values in the R channel are greater than the pixel values in the B channel as an area with the skin surface color, as the initial skin area.

[0107] In one embodiment, the preset pixel value range can be set based on experience. For example, based on experience, the preset pixel value range can be set to [75, 255]. That is, the server can determine the area consisting of points in the smoothed image whose pixel values in the R channel are greater than or equal to 75 and less than or equal to 255, and whose pixel values in the R channel are greater than those in the B channel, as an area with the skin surface color.

[0108] In one embodiment, the initial skin area may be determined using the following formula:

[0109] S ds [R]>75I S ds [B]<S ds [R];

[0110] Among them, S ds is a smoothed image, S ds [R] is the R channel of the smoothed image, S ds [B] is the B channel of the smoothed image.

[0111] In one embodiment, the server may first downsample the original image, then smooth the downsampled image to obtain a smoothed image, and then determine the initial skin area based on the smoothed image.

[0112] Step 306: Determine the pixel points in the original image whose pixel values change significantly to obtain texture edge information.

[0113] Texture edge information refers to locations in an image where pixel values significantly change. It can be understood that texture edge information includes pixel points in the original image where pixel values significantly change. Therefore, texture edge information not only includes contour information in the original image, but also texture information in the original image.

[0114] In one embodiment, the server may determine pixel points in the original image where pixel values change significantly based on the difference between the original image and the smoothed image, and obtain texture edge information.

[0115] In another embodiment, the server may use an edge detection operator to determine pixel points in the original image where pixel values change significantly, and obtain texture edge information.

[0116] In other embodiments, the server may also use other edge detection methods to determine pixel points in the original image where pixel values change significantly to obtain texture edge information, without limitation.

[0117] Step 308: removing the texture edge region in the initial skin region based on the texture edge information to obtain the target skin region.

[0118] The target skin region is a skin region that does not include texture edge information, and the texture edge region is a region that only includes the position corresponding to the texture edge information.

[0119] Specifically, the server may locate the texture edge region corresponding to the texture edge information in the initial skin region according to the texture edge information, and then remove the texture edge region in the initial skin region to obtain the target skin region.

[0120] In one embodiment, the server may determine an edge mask based on the texture edge information, and mask the texture edge region in the initial skin region based on the edge mask to remove the texture edge region in the initial skin region and obtain the target skin region. The edge mask is used to mask the texture edge information in the original image.

[0121] In one embodiment, after the initial skin area is determined, a mask image may be used to represent the initial skin area, and then a target skin area excluding texture edge information may be determined based on the mask image representing the initial skin area and an edge mask.

[0122] In one embodiment, the mask image used to represent the initial skin area can be binary or non-binary.

[0123] In one embodiment, the server may multiply and fuse the mask image used to represent the initial skin area with the edge mask to obtain a target mask image used to determine the target skin area that does not include texture edge information.

[0124] In step 310 , the image content in the target skin area of the smoothed image is fused into the corresponding position of the target skin area in the original image to obtain the target image.

[0125] In one embodiment, if the smoothed image is obtained by directly smoothing the original image, the server may directly merge the image content in the target skin area in the smoothed image into the corresponding position of the target skin area in the original image to obtain the target image.

[0126] In another embodiment, if the smoothed image is obtained by smoothing a downsampled image, the server can upsample the smoothed image to obtain an upsampled smoothed image with the same resolution as the original image, and then merge the image content in the target skin area in the upsampled smoothed image into the corresponding position of the target skin area in the original image to obtain the target image.

[0127] In one embodiment, the fusion may be any one of normal fusion, color filter fusion, and bright light fusion.

[0128] In one embodiment, the server can locate the target skin area based on the target mask image used to determine the target skin area that does not include texture edge information, and merge the image content located in the target skin area in the smoothed image into the corresponding position of the target skin area in the original image to obtain the target image.

[0129] In one embodiment, the server may multiply and fuse the smoothed image with a target mask image used to determine a target skin area that does not include texture edge information to obtain a first image, and multiply and fuse the original image with an inverse mask image of the target mask image to obtain a second image. The server then superimposes and fuses the first and second images, thereby achieving a fusion process of the smoothed image and the original image. It will be understood that the fusion process in this embodiment is a normal fusion process.

[0130] In one embodiment, normal fusion can be expressed as follows:

[0131] SR=Blend(I in ,S↑,M)=S↑M+I in (1-M);

[0132] Among them, SR is the target image after fusion, Blend() is the fusion function, I in is the original image, S↑ is the smoothed image after upsampling, and M is the target mask used to determine the target skin area.

[0133] In one embodiment, color filter fusion can be to first perform color filter processing on the smoothed image, and then perform normal fusion of the smoothed image after color filter processing with the original image according to the target skin area. The color filter processing can be expressed by the following formula:

[0134] SR'=1-(1-I in )(1-SR);

[0135] Among them, SR' is the smoothed image after color filtering, I in is the original image and SR is the smoothed image.

[0136] In one embodiment, the strong light fusion can be to first perform strong light processing on the smoothed image, and then perform normal fusion of the smoothed image after the strong light processing with the original image according to the target skin area. The strong light processing can be expressed by the following formula:

[0137]

[0138] Among them, SR' is the smoothed image after strong light processing, I in is the original image and SR is the smoothed image.

[0139] It can be understood that because the image content within the target skin area in the smoothed image is fused into the corresponding position of the target skin area in the original image, the image content within the target skin area in the fused target image is mainly determined based on the image content corresponding to the target skin area in the smoothed image, so that the target skin area of the target image is smoothed, and the image content outside the target skin area in the target image is mainly determined based on the image content corresponding to the area outside the target skin area in the original image, so that the clarity of the original image is retained outside the target skin area of the target image.

[0140] In the above image processing method, the original image is smoothed to obtain a smoothed image, and then the initial skin area in the smoothed image is determined, and the pixel points where the pixel values in the original image change significantly are determined to obtain texture edge information, and then the texture edge area in the initial skin area is removed based on the texture edge information to obtain the target skin area, and the image content in the target skin area in the smoothed image is integrated into the corresponding position of the target skin area in the original image to obtain the target image, so that the image content within the target skin area in the fused target image is smoothed, while the image content outside the target skin area retains the clarity of the original image, avoiding the problem of reducing the transparency of the image by smoothing the original image as a whole, and improving the image quality, and the image content in the target skin area where the texture edge area is removed in the smoothed image is integrated into the corresponding position of the target skin area in the original image, and also avoiding the smoothing of the texture edge area in the original image, so that the texture edge details are clear, and the image quality is further improved.

[0141] In one embodiment, smoothing the original image to obtain a smoothed image includes: determining a downsampling factor according to the resolution of the original image; downsampling the original image according to the downsampling factor; and smoothing the downsampled image to obtain a smoothed image.

[0142] In one embodiment, downsampling the original image according to the downsampling factor can be expressed by the following formula:

[0143] I ds = Downsampling(I in ,r);

[0144] Among them, I ds It is the image after downsampling. Downsampling is the downsampling process. in is the original image, and r is the downsampling factor.

[0145] In one embodiment, the downsampling factor may be positively correlated with the resolution of the original image. The downsampling factors corresponding to the long side and the short side of the original image may be the same or different.

[0146] In one embodiment, the downsampling factor may be linearly related to the resolution of the original image. In one embodiment, the downsampling factor may be linearly related to the short side resolution of the original image. In this embodiment, the downsampling factor may be expressed as follows:

[0147] r=min(H,W) / L;

[0148] Where r is the downsampling factor, H and W are the height and width of the original image, min(H,W) represents the short side resolution of the original image, and L is a constant. For example, L can be set to 180 or any other value.

[0149] In another embodiment, the downsampling factor may be determined based on the resolution range of the original image. The larger the resolution range, the larger the downsampling factor. For example, when the resolution is between 720P (P, i.e., PPI, Pixels Per Inch, the unit of resolution) and 1080P, the downsampling factor is 3; when the resolution is between 1080P and 4KP, the downsampling factor is 6.

[0150] In other embodiments, the downsampling factor may be in an exponential relationship or a power relationship with the resolution of the original image, and is not limited thereto.

[0151] In the above embodiment, downsampling the original image before smoothing it reduces computational effort and improves efficiency. It also achieves a larger receptive field and smoothness during smoothing, raising the upper limit of smoothness. Furthermore, dynamically adjusting the downsampling factor based on the original image's resolution improves the solution's applicability, enabling effective smoothing for images of varying resolutions.

[0152] In one embodiment, the method further includes: determining an edge mask based on texture edge information. In this embodiment, removing the texture edge region in the initial skin region based on the texture edge information to obtain the target skin region includes: fusing the initial mask image used to determine the initial skin region with the edge mask to obtain a target mask image; the target mask image is used to determine the target skin region. In this embodiment, fusing the image content in the smoothed image that is within the target skin region into the corresponding position of the target skin region in the original image to obtain the target image includes: fusing the image content in the smoothed image that is within the target skin region into the corresponding position of the target skin region in the original image based on the target mask image to obtain the target image.

[0153] The initial mask image is used to determine the initial skin area. In one embodiment, the server can determine the area outside the texture edge area based on the texture edge information, and generate an edge mask based on the area outside the texture edge area.

[0154] In one embodiment, the edge mask may be binary, for example, the pixel value of a point at an edge is 0, and the pixel value of a point outside the edge is 1.

[0155] In another embodiment, the edge mask may be non-binary, for example, the pixel value of a point closer to the edge is closer to 0, and the pixel value of a point farther from the edge is closer to 1.

[0156] In one embodiment, the server may first determine an initial mask image for determining the initial skin area based on the smoothed image, and then multiply and fuse the initial mask image and the edge mask to obtain a target mask image.

[0157] In one embodiment, the initial mask image may be a binary initial mask image. A binary initial mask image refers to an initial mask image composed of only two pixel values. For example, a binary initial mask image may be composed of points with a pixel value of 0 and points with a pixel value of 1. For example, in a binary initial mask image, the pixel value of points within the initial skin area may be 1, while the pixel value of points outside the initial skin area may be 0. The reverse may also be true, that is, the pixel value within the initial skin area is 0, and the pixel value outside the initial skin area is 1.

[0158] In another embodiment, the initial mask image may be a non-binary initial mask image. A non-binary initial mask image refers to an initial mask image containing more than two pixel values. In this embodiment, using a non-binary initial mask image to determine the initial skin area can avoid sudden changes in pixel values in the initial mask image, which could result in hard edges in the target image obtained by fusion based on the mask image. This improves the naturalness of the target image and thus enhances the image quality of the target image.

[0159] In one embodiment, the server may first determine a binary initial mask image for determining the initial skin area according to the smoothed image, and then generate a non-binary initial mask image according to the binary initial mask image and the smoothed image.

[0160] In one embodiment, the server may perform blurring on the binary initial mask image to obtain a non-binary initial mask image. In another embodiment, the server may perform expansion processing on the binary initial mask image to obtain a non-binary initial mask image. In another embodiment, the server may perform automatic hardware interpolation processing to obtain a non-binary initial mask image. In other embodiments, the server may further obtain pixel values within the initial skin region of the smoothed image based on the binary initial mask image, and then generate a non-binary initial mask image based on the obtained pixel values.

[0161] In one embodiment, the server may multiply the pixel values of each point in the initial mask image and the edge mask respectively to obtain a target mask image. In one embodiment, if the initial mask image is obtained by smoothing the downsampled original image, the initial mask image is first upsampled to obtain an upsampled initial mask image with the same resolution as the original image, and then the upsampled initial mask image is multiplied and fused with the edge mask.

[0162] In one embodiment, the server can multiply and fuse the smoothed image with the target mask image to obtain a first image, and multiply and fuse the original image with the inverse mask image of the target mask image to obtain a second image, and superimpose and fuse the first image and the second image to fuse the image content in the smoothed image located in the target skin area into the corresponding position of the target skin area in the original image to obtain the target image.

[0163] In the above embodiment, the target mask image is obtained by fusing the initial mask image and the edge mask, and then the smoothed image and the original image are fused according to the target mask image to obtain the target image. The specified area in the image can be accurately determined in the form of the mask image. In particular, when a non-binary mask image is used, the problem of hard edges in the final target image can be avoided, thereby improving the image quality.

[0164] In one embodiment, the method further includes: obtaining a binary mask image for determining an initial skin area of a smoothed image; obtaining pixel values within the initial skin area corresponding to the binary mask image from a single color channel of the smoothed image; generating a non-binary initial mask image for determining the initial skin area based on the obtained pixel values; and determining the pixel values of points in the initial mask image corresponding to the initial skin area based on the obtained pixel values.

[0165] Here, a single color channel refers to a color channel in an image. For example, for an RGB image, the R channel, G channel, and B channel are all single color channels. That is, a single color channel can be any one of the R channel, G channel, and B channel.

[0166] Specifically, the server can first determine a binary initial mask image (i.e., a binary mask image used to determine the initial skin area of the smoothed image) based on the smoothed image, and then obtain the pixel values within the initial skin area corresponding to the binary mask image from the single color channel of the smoothed image, and then generate a non-binary initial mask image for determining the initial skin area based on the obtained pixel values.

[0167] In one embodiment, the server can take a single color channel of the smoothed image, and then multiply and fuse the binary mask image with the single color channel of the smoothed image to obtain a single color channel of the smoothed image that only retains the pixel values within the initial skin area. That is, the pixel values within the initial skin area in the single color channel of the smoothed image remain unchanged, while the pixel values outside the initial skin area are 0, thereby obtaining the pixel values within the initial skin area corresponding to the binary mask image.

[0168] In one embodiment, the server may amplify the pixel values within the obtained initial skin region to generate a non-binary initial mask image. That is, the pixel values within the initial skin region in the generated initial mask image are the amplified values of the obtained pixel values, and the pixel values outside the initial skin region are 0.

[0169] In one embodiment, the server may exponentially amplify the pixel values within the obtained initial skin area, for example, amplifying the pixel values to the fifth power of the original value.

[0170] In another embodiment, the server may amplify the pixel values in the acquired initial skin area by a multiple, for example, to 2 times the original value.

[0171] In other embodiments, the server may also perform both exponential amplification and multiple amplification on the pixel values in the initial skin area, for example, first amplifying the pixel values to the fifth power and then amplifying them by 2 times.

[0172] In one embodiment, the server may limit the amplified pixel value to between 0 and 255 to satisfy the pixel value range of the image.

[0173] In one embodiment, the steps of obtaining pixel values within the initial skin region corresponding to the binary mask image from a single color channel of the smoothed image and then generating a non-binary initial mask image for determining the initial skin region based on the obtained pixel values can be expressed as follows:

[0174] M coarse =clamp((S ds [R]·M color ) β 2,0,255);

[0175] Among them, Mcoarse is a non-binary initial mask image. clamp is a clipping function, which in this embodiment means that (S ds [R]·M color ) β Limit to 0 to 255, set values greater than 255 to 255, and set values less than 0 to 0. ds [R] represents the smoothed image S ds R channel, M color is a binary mask image. β is the coefficient of exponential enhancement, which is a constant.

[0176] In the above embodiment, pixel values located in the initial skin area corresponding to the binary mask image are obtained from a single color channel of the smoothed image to generate a non-binary initial mask image, and subsequent processing can be performed based on the non-binary initial mask image, thereby avoiding the problem of hard edges in the final target image due to sudden changes in pixel values in the mask image, improving the naturalness of the target image, and thus improving the image quality of the target image.

[0177] In one embodiment, based on the target mask image, the image content located in the target skin area in the smoothed image is fused into the corresponding position of the target skin area in the original image to obtain the target image, which includes: superimposing and fusing the first image and the second image to obtain the target image; wherein the first image is an image obtained by multiplying the pixel value corresponding to each point in the smoothed image with the pixel value corresponding to each point in the target mask image; the second image is an image obtained by multiplying the pixel value corresponding to each point in the original image with the pixel value corresponding to each point in the inverse mask image of the target mask image.

[0178] The reverse mask image is a mask image in which the pixel values of each point are opposite to the pixel values of the corresponding points in the target mask image. For example, if the pixel value of a point in the target mask image is 50, the pixel value of the corresponding point in the reverse mask image is 205 (i.e., 255-50).

[0179] Specifically, the server can multiply the pixel value corresponding to each point in the smoothed image with the pixel value corresponding to each point in the target mask image to obtain a first image, and multiply the pixel value corresponding to each point in the original image with the pixel value corresponding to each point in the inverse mask image of the target mask image to obtain a second image, and superimpose and fuse the first image and the second image to obtain the target image.

[0180] In one embodiment, the processing step of superimposing and fusing the first image and the second image to obtain the target image in the above embodiment can be expressed by the following formula:

[0181] SR=Blend(I in ,S↑,M)=S↑M+I in(1-M);

[0182] Among them, SR is the target image, Blend() is the fusion function, I in is the original image, S↑ is the smoothed image after upsampling, and M is the target mask.

[0183] In the above embodiment, the smoothed image and the original image are fused according to the target mask image to obtain the target image, which can ensure that the image content within the target skin area in the fused target image is smoothed, while the image content outside the target skin area retains the clarity of the original image, avoiding the problem of smoothing the original image as a whole and reducing the image transparency, thereby improving the image quality. The smoothed image and the original image are fused according to the target skin area that does not include texture edge information, which also avoids the smoothing of the edge position in the original image, making the edge details clear, and further improving the image quality.

[0184] In one embodiment, the method further includes: determining a pixel value interval to which the pixel value of each point in the target mask image belongs; and performing significance weighting on the pixel value of the corresponding point in the target mask image according to a significance coefficient corresponding to the pixel value interval to obtain a significance-weighted target mask image.

[0185] Among them, the significance coefficient is used to perform significance weighting on the pixel values in the target mask image.

[0186] Specifically, different significance coefficients can be pre-set for different pixel value intervals. The server can determine the pixel value interval to which the pixel value of each point in the target mask image belongs, and then perform significance weighting on the pixel value of the corresponding point in the target mask image based on the significance coefficient pre-set for the pixel value interval. After performing significance weighting on each point in the target mask image, a significance-weighted target mask image is obtained.

[0187] In one embodiment, the larger the pixel value in the pixel value interval range, the larger the corresponding significance coefficient.

[0188] In one embodiment, the significance coefficient may be around 1. For example, the significance coefficient corresponding to the pixel value range that needs to be dimmed may be greater than 0 and less than 1, and the greater the smaller the significance coefficient, the greater the degree of dimming. The significance coefficient corresponding to the pixel value range that needs to be brightened may be greater than 1, and the greater the greater the significance coefficient, the greater the degree of brightening.

[0189] In one embodiment, the significance-weighted pixel values may be limited to the range of [0, 255].

[0190] In one embodiment, the step of performing saliency weighting on the pixel values of each point in the target mask image to obtain a saliency-weighted target mask image can be implemented using the following piecewise function:

[0191]

[0192] Among them, T1, T2, T3 and T4 are the interval endpoints of each pixel value interval range, and T1<T2<T3<T4. R1 is the pixel value interval range T1≤M skin <T2 corresponding to the significance coefficient, R2 is the pixel value interval range T2≤M skin <T3 corresponding to the significance coefficient, R3 is the pixel value interval range T3≤M skin <T4 corresponding significance coefficient. M skin is the pixel value in the target mask image, and M is the saliency-weighted target mask image.

[0193] In the above embodiment, significance weighting is performed on pixel values within different pixel value threshold ranges in the target mask image, which can make the smoothing effect of the final target image more detailed, and different pixel value intervals have different brightness and darkness, thereby improving image quality.

[0194] In one embodiment, determining the edge mask according to the texture edge information includes: performing an inverse operation on the texture edge information by using an inversion function to obtain the edge mask.

[0195] The inversion function is used to perform an inverse operation on the texture edge information. The inverse operation is used to obtain an area outside the texture edge area corresponding to the texture edge information.

[0196] Specifically, the server may perform an inverse operation on the texture edge information through an inversion function to obtain an area outside the texture edge area corresponding to the texture edge information, and generate an edge mask according to the area outside the texture edge area.

[0197] In one embodiment, the server may obtain texture edge information in the original image based on difference information between the original image and the smoothed image.

[0198] In one embodiment, the texture edge information may be a texture edge information map. The server may perform an inverse operation on the texture edge information map using an inversion function to obtain an edge mask.

[0199] In one embodiment, the edge mask obtained by performing an inverse operation on the texture edge information map can be expressed as follows:

[0200] M edge =F(|E|);

[0201] Among them, Medge is the edge mask, F is the inversion function, and E is the texture edge information map.

[0202] In one embodiment, if the texture edge information is a texture edge information map and the edge mask is binary, the inversion function may be a function that sets the pixel values at the edge position in the texture edge information map to 0 and sets the pixel values outside the edge to 1.

[0203] In one embodiment, if the texture edge information is a texture edge information map, and the edge mask is non-binary, the inversion function may be a function that inverts the pixel values in the texture edge information map, that is, among the pixel values in the range of 0 to 1, the larger pixel values are converted to smaller pixel values, and the smaller pixel values are converted to larger pixel values, that is, the pixel values close to 1 are converted to pixel values close to 0, and the pixel values close to 0 are converted to pixel values close to 1. For example, Figure 4 The three functions shown are y(x)=1-x 0.2 y(x)=1-x 0.5 and All of them can be used as inversion functions. The figure shows the curve graphs of these three functions. It can be seen from the curve graphs that these three functions can change values close to 1 into values close to 0, and change values close to 0 into values close to 1.

[0204] In one embodiment, the inversion function may be in the form of the following formula:

[0205] F(E)=1.0-E γ ;

[0206] Where F(E) is the inversion function, E is the independent variable in the inversion function, and γ is a constant not exceeding 1.0. For example, Figure 4 The first two inversion functions are both in the form of the above formula, with γ being 0.2 and 0.5 respectively.

[0207] In the above embodiment, an inversion function is used to perform an inverse operation on the texture edge information to generate an edge mask. This eliminates the edge region from the target skin area, avoids smoothing of the edge region, and improves the image quality of the target image. Furthermore, particularly when a non-binary edge mask is used, hard edges in the target image are avoided, enhancing the naturalness of the target image and thus improving image quality.

[0208] In one embodiment, the texture edge information is a texture edge information map. In this embodiment, determining pixel points in the original image where pixel values significantly change to obtain the texture edge information includes: determining difference information between the original image and the smoothed image; and obtaining the texture edge information map based on the difference information.

[0209] In one embodiment, the server may subtract the original image from the smoothed image to determine difference information between the original image and the smoothed image.

[0210] In one embodiment, the server may average the difference information of each color channel obtained by subtracting the original image from the smoothed image to obtain a texture edge information map. It is understood that the obtained texture edge information map is a single-channel grayscale map.

[0211] In one embodiment, if the smoothed image is obtained by smoothing the downsampled original image, the smoothed image is first upsampled to obtain an upsampled smoothed image with the same resolution as the original image, and then a texture edge information map is obtained based on the difference information between the original image and the upsampled smoothed image.

[0212] In one embodiment, the texture edge information map can be obtained by the following formula:

[0213]

[0214] Among them, E is the texture edge information map, I in is the original image, S↑ is the upsampled smoothed image, R, G, and B represent the R channel, G channel, and B channel, respectively. The above formula represents the average of the difference information of the R channel, G channel, and B channel obtained by subtracting the original image from the upsampled smoothed image to obtain the texture edge information map.

[0215] In the above embodiment, the texture edge information map is obtained according to the difference information between the original image and the smoothed image, which can accurately determine the texture edge information in the image.

[0216] In one embodiment, the method further includes: extracting high-frequency information from the original image based on difference information between the original image and the smoothed image; and adding the high-frequency information to the target image to obtain a sharpened target image.

[0217] Specifically, the server may subtract the original image from the smoothed image to obtain difference information, then determine high-frequency information in the original image based on the difference information, and then add the high-frequency information to the target image to obtain a sharpened target image.

[0218] In one embodiment, the server can use a high-contrast preserving operator to extract high-frequency information from the original image. In this embodiment, using a high-contrast preserving operator to extract high-frequency information is more stable than traditional texture edge information extraction operators and can reduce the introduction of noise during the sharpening process.

[0219] In other embodiments, the server may use an operator such as the Laplacian operator for extracting texture edge information to extract high-frequency information from the original image. The extraction method is not limited.

[0220] In one embodiment, extracting high-frequency information from the original image using a high-contrast retention operator can be expressed by the following formula:

[0221] hPass = 2.0 · step(I in - S↑ + 127, 127) - 255;

[0222] where hPass represents high-frequency information. Step(x, e) is a truncation function. When x > e, step(x, e) = x; when x < e, step(x, e) = 0. I in is the original image, and S↑ is the smoothed image after upsampling.

[0223] In one embodiment, the server may superimpose the extracted high-frequency information on the target image to obtain a sharpened target image.

[0224] In one embodiment, the server may superimpose the extracted high-frequency information on the target image and limit the pixel values of each point after superimposition within the range of [0, 255] to obtain a sharpened target image. In one embodiment, the server may set the pixel values greater than 255 after superimposition to 255 and set the pixel values less than 0 after superimposition to 0, and the pixel values within the range of [0, 255] after superimposition remain unchanged.

[0225] In one embodiment, the server may add the high-frequency information to the target image through the following formula and limit the pixel values of each point after superimposition within the range of [0, 255] to obtain a sharpened target image:

[0226] I enhance = max(min(SR + hPass, 255), 0);

[0227] where I enhance is the sharpened target image, SR is the target image, and hPass represents high-frequency information. min(SR + hPass, 255) means that the pixel values of the points in SR + hPass that are less than or equal to 255 remain unchanged, and the pixel values of the points greater than 255 are set to 255. max(min(SR + hPass, 255), 0) means that the pixel values of the points in min(SR + hPass, 255) that are greater than or equal to 0 remain unchanged, and the pixel values of the points less than 0 are set to 0.

[0228] In the above embodiment, based on the difference information between the original image and the smoothed image, high-frequency information in the original image is extracted and added to the target image to obtain a sharpened target image. This can avoid a certain degree of damage to the image content of the environmental area in the target image, further improve the naturalness and realism of the target image, and improve the image quality.

[0229] In one embodiment, the original image contains a human face. In this embodiment, the method further includes: determining facial feature regions of the human face in the original image; and fusing the target image with the original image based on the facial feature regions to obtain a target image that retains detailed features of the facial features.

[0230] In one embodiment, the facial features area may include areas where facial features such as eyebrows, eyes, nose and mouth are located.

[0231] In one embodiment, the server may identify key points of facial features in the original image, and then determine the facial feature areas of the face in the original image based on the key points of facial features.

[0232] In one embodiment, the server may divide the facial features into grids based on the identified facial features, and then perform interpolation processing on the grids to obtain facial features regions. The interpolation method used in the interpolation processing is not limited.

[0233] In one embodiment, the divided mesh may be a triangular patch.

[0234] In one embodiment, the server can use a GPU (Graphics Processing Unit) to execute a hardware raster automation interpolation algorithm to perform interpolation processing. In this embodiment, the GPU automatically executes the interpolation algorithm, thereby improving processing efficiency.

[0235] In one embodiment, the server may generate a facial mask based on the facial features, and then fuse the target image with the original image based on the facial mask to obtain a target image that retains the facial features. The facial mask is used to block areas outside the facial features.

[0236] In one embodiment, the server may fuse the target image with the original image according to the facial feature mask using the following formula:

[0237] Res=Blend(I in ,SR,M contour );

[0238] Among them, Res is the target image obtained after fusion, which retains the detailed features of facial features. in is the original image, SR is the target image, M contour It is a facial feature mask.

[0239] In one embodiment, the facial feature mask may be binary, i.e., composed of pixel values 0 and 1. For example, the pixel value of the facial feature area in the facial feature mask is 0, and the pixel value of the non-facial feature area is 1, or vice versa, the pixel value of the facial feature area is 1, and the non-facial feature area is 0.

[0240] In another embodiment, the facial features mask can be non-binary, i.e., a grayscale image composed of more than two pixel values. For example, the pixel value at the junction between the facial features area and the non-facial features area in the facial features mask gradually transitions from 0 to 1. Figure 5 The figure shows a non-binary facial feature mask. It can be seen that the pixel value at the junction between the facial feature area and the non-facial feature area in the facial feature mask gradually transitions from 0 to 1.

[0241] In one embodiment, the server may first generate a binary facial feature mask based on the facial feature areas, and then generate a non-binary facial feature mask based on the binary facial feature mask.

[0242] In one embodiment, the server may blur the binary facial feature mask to obtain a non-binary facial feature mask. In another embodiment, the server may expand the facial feature area in the binary facial feature mask to obtain a non-binary facial feature mask. In other embodiments, the server may obtain the non-binary facial feature mask through automatic hardware interpolation. The method for generating the non-binary facial feature mask is not limited.

[0243] In one embodiment, the server may multiply and fuse the target image with the facial feature mask, and then superimpose and fuse the result with the result of multiplying and fusing the original image with the inverse mask of the facial feature mask to obtain a target image that retains the detailed features of the facial features.

[0244] It can be understood that the image content within the facial features area in the target image that retains the detailed features of the facial features is mainly determined based on the image content within the facial features area of the original image, and the image content outside the facial features area in the target image that retains the detailed features of the facial features is mainly determined based on the image content outside the facial features area of the target image.

[0245] In one embodiment, the server may fuse the sharpened target image with the original image based on facial features.

[0246] In the above embodiment, the target image is fused with the original image according to the facial feature areas to obtain a target image that retains the facial feature details, thereby avoiding the loss of facial feature details in the image during the smoothing process, improving the naturalness of the target image, and improving the image quality.

[0247] In one embodiment, the original image contains a human face. In this embodiment, the method further includes: obtaining a face region corresponding to the human face in the original image; and fusing the target image with the original image based on the face region to obtain a target image with an optimized face region.

[0248] The face area is the area corresponding to the face in the original image.

[0249] In one embodiment, the server may identify facial key points in the original image, and then determine the facial region in the original image based on the facial key points.

[0250] In one embodiment, the server may divide the face into grids based on the recognized key points, and then perform interpolation processing on the grids to obtain the face region. The interpolation method used in the interpolation processing is not limited.

[0251] In one embodiment, the divided mesh may be a triangular patch.

[0252] In one embodiment, the server can use a GPU (Graphics Processing Unit) to execute a hardware raster automation interpolation algorithm to perform interpolation processing. In this embodiment, the GPU automatically executes the interpolation algorithm, thereby improving processing efficiency.

[0253] In one embodiment, the server may generate a face mask based on the face area, and then fuse the target image with the original image based on the face mask to obtain a target image with an optimized face area. The face mask is used to block the face and determine the area outside the face.

[0254] In one embodiment, the face mask may be binary, i.e., composed of pixel values 0 and 1. For example, the pixel value of the face area in the face mask is 0, and the pixel value of the non-face area is 1, or vice versa, the face area is 1, and the non-face area is 0.

[0255] In another embodiment, the face mask may be non-binary, i.e., a grayscale image composed of more than two pixel values. For example, the pixel value at the boundary between the face area and the non-face area in the face mask gradually transitions from 0 to 1.

[0256] In one embodiment, the server may first generate a binary face mask according to the face area, and then generate a non-binary face mask according to the binary face mask.

[0257] In one embodiment, the server may blur the binary face mask to obtain a non-binary face mask. In another embodiment, the server may expand the facial region in the binary face mask to obtain a non-binary face mask. In other embodiments, the server may obtain the non-binary face mask through automatic hardware interpolation. The method for generating the non-binary face mask is not limited.

[0258] In one embodiment, the server may multiply and fuse the target image with the inverse mask of the face mask, and then superimpose and fuse the result with the result of multiplying and fusion of the original image and the face mask to obtain the target image with optimized face area.

[0259] It can be understood that the image content within the face area of the target image after the face area is optimized is mainly determined based on the image content of the target image within the face area, and the image content outside the face area of the target image after the face area is optimized is mainly determined based on the image content outside the face area of the original image.

[0260] In one embodiment, the server may fuse the target image retaining facial features with the original image based on the facial area.

[0261] In the above embodiment, the target image is fused with the original image according to the face area to obtain a target image with optimized face area, thereby avoiding the mistaken identification of the background in the original image with a color similar to that of the skin surface as skin and smoothing it, resulting in background blur. In the target image with optimized face area, only the face area is smoothed, and the background area outside the face area is not smoothed, thereby improving the quality of the target image.

[0262] In one embodiment, the method further includes: obtaining an input smoothing parameter; determining the superposition weights corresponding to the target image and the original image respectively according to the smoothing parameter; superimposing and fusing the target image and the original image according to the corresponding superposition weights; and outputting the superimposed and fused image.

[0263] The smoothing parameter is a parameter used to adjust the degree of smoothing.

[0264] In one embodiment, the terminal may display an adjustment entry for the smoothing parameter in the interface, and the user may adjust the smoothing parameter through the adjustment entry.

[0265] In one embodiment, the user may select a smoothing parameter through an adjustment portal, or may input a smoothing parameter through an adjustment portal, or may use other methods, which are not limited.

[0266] In other embodiments, if the user does not adjust the smoothing parameter, a preset default smoothing parameter may be used.

[0267] In one embodiment, the terminal can receive the smoothing parameters input by the user through adjustment and send the smoothing parameters to the server. The server can determine the superposition weights corresponding to the target image and the original image respectively based on the smoothing parameters, and superimpose and fuse the target image and the original image according to the corresponding superposition weights. The server can send the superimposed and fused image to the terminal, and the terminal can display the fused image.

[0268] In another embodiment, the terminal can receive a smoothing parameter input by the user through adjustment, and then the terminal can determine the superposition weights corresponding to the target image and the original image respectively according to the smoothing parameter, and superimpose and fuse the target image and the original image according to the corresponding superposition weights, and the terminal can display the fused image.

[0269] In one embodiment, the smoothing parameter may take values in the range of [0, 1].

[0270] In one embodiment, the server may use the smoothing parameter as the superposition weight corresponding to the target image, and use 1 minus the smoothing parameter as the superposition weight corresponding to the original image, and then superimpose and fuse the target image and the original image according to the corresponding superposition weights.

[0271] In one embodiment, the server may superimpose and fuse the target image and the original image according to the corresponding superposition weights according to the following formula:

[0272] P=Blend(I in ,Res,α)=Res·α+I in (1-α);

[0273] Among them, P is the superimposed fused image, Blend() is the fusion function, I in is the original image, Res is the target image, and α is the smoothing parameter.

[0274] It can be understood that the larger the value of the smoothing parameter is, the smoother the final superimposed and fused image will be, and the smoother the skin in the image will appear.

[0275] In the above embodiment, the target image and the original image are superimposed and fused according to the corresponding superposition weights based on the input smoothing parameters, thereby enabling the user to flexibly and conveniently adjust the smoothness of the target image, thereby improving operation efficiency.

[0276] like Figure 6, which is a schematic diagram of the overall process of the image processing method in various embodiments of the present application. First, the server can obtain an original image, downsample the original image, and smooth the downsampled image to obtain a smoothed image. The server can then determine the initial skin region based on the color distribution in the smoothed image and identify pixels in the original image with significant pixel value changes to obtain texture edge information. Next, the server can remove the texture edge region from the initial skin region based on the texture edge information to obtain a target skin region. The server can then fuse the image content within the target skin region in the smoothed image into the corresponding position of the target skin region in the original image to obtain a target image. The server can then extract high-frequency information from the original image to sharpen the fused target image. Based on the facial feature region, the server can then retain facial feature details in the sharpened target image. Based on the facial region, the server can then retain features outside the facial region and optimize only the image content within the facial region to obtain the final target image. Finally, the server can adjust the smoothness of the target image based on the input smoothing parameters and output the adjusted target image.

[0277] This application also provides an application scenario, which is a video call or live broadcast scenario, and the above-mentioned image processing method is applied to this scenario. Specifically, the application of the image processing method in this application scenario is as follows:

[0278] During a user's video call or live broadcast, the terminal used by the user can collect the user's facial image in real time and send the facial image to the server. The server can use the facial image as the original image to execute the image processing method in each embodiment of the present application to obtain the target image, and send the target image to the terminal used by the user, and to the terminals used by other users who are having video calls with the user or the terminals used by various viewers watching the user's live broadcast. The terminal that receives the target image can display the target image in real time.

[0279] In other embodiments, the terminal used by the user may not send the facial image to the server. Instead, the terminal itself directly executes the image processing method in each embodiment of the present application to obtain the target image, and sends the target image to the terminals used by other users who have video calls with the user or the terminals used by various viewers who watch the user's live broadcast, so as to achieve end-to-end communication between terminals.

[0280] During a video call or live broadcast, the user can also adjust the smoothing parameters in the interface to adjust the smoothness of the target image displayed by the terminal (ie, the degree of skin refining in the target image).

[0281] This application also provides an application scenario, which is image or video beautification, and the above-mentioned image processing method is applied to this application scenario. Specifically, the application of the image processing method in this application scenario is as follows:

[0282] The user can select an image or video pre-stored in the terminal, and the terminal can send the selected image or video to the server. The server can use the received image or each frame of the received video as the original image, execute the image processing method in each embodiment of the present application, obtain the target image corresponding to the image or each frame of the target image corresponding to the video, and send the target image to the terminal. The terminal can display the target image or the target video composed of each frame of the target image to achieve skin resurfacing in the image or video.

[0283] In other embodiments, the terminal may not send the image or video to the server, but instead the terminal itself directly executes the image processing method in each embodiment of the present application to obtain the target image corresponding to the image or each frame of target image corresponding to the video, and display the target image or the target video composed of each frame of target image.

[0284] During the beautification process of the selected image or video, the user can also adjust the smoothing parameters to adjust the smoothness of the skin in the beautified image or video.

[0285] It should be understood that, although the steps in each flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in each flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0286] In one embodiment, Figure 7 As shown, an image processing device 700 is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: a smoothing processing module 702, an initial area determination module 704, an edge determination module 706, a target area determination module 708 and a fusion module 710, wherein:

[0287] The smoothing processing module 702 is used to perform smoothing processing on the original image to obtain a smoothed image.

[0288] The initial region determination module 704 is configured to determine an initial skin region in the smoothed image.

[0289] The edge determination module 706 is used to determine the pixel points in the original image where the pixel values change significantly, and obtain texture edge information.

[0290] The target area determination module 708 is configured to remove the texture edge area in the initial skin area based on the texture edge information to obtain the target skin area.

[0291] The fusion module 710 is configured to fuse the image content in the target skin region in the smoothed image into the corresponding position of the target skin region in the original image to obtain the target image.

[0292] In one embodiment, the smoothing processing module 702 is further configured to determine a downsampling factor according to the resolution of the original image; downsample the original image according to the downsampling factor; and smooth the downsampled image to obtain a smoothed image.

[0293] In one embodiment, the edge determination module 706 is further configured to determine an edge mask based on texture edge information. In this embodiment, the target region determination module 708 is further configured to fuse the initial mask image used to determine the initial skin region with the edge mask to generate a target mask image; the target mask image is used to determine the target skin region. In this embodiment, the fusion module 710 is further configured to fuse the image content within the target skin region in the smoothed image into the corresponding position of the target skin region in the original image to generate the target image.

[0294] In one embodiment, the initial region determination module 704 is further configured to obtain a binary mask image for determining an initial skin region of a smoothed image; obtain pixel values within the initial skin region corresponding to the binary mask image from a single color channel of the smoothed image; generate a non-binary initial mask image for determining the initial skin region based on the obtained pixel values; and determine pixel values of points in the initial mask image corresponding to the initial skin region based on the obtained pixel values.

[0295] In one embodiment, the fusion module 710 is further used to superimpose and fuse the first image and the second image to obtain a target image; wherein the first image is an image obtained by multiplying the pixel value corresponding to each point in the smoothed image with the pixel value corresponding to each point in the target mask image; the second image is an image obtained by multiplying the pixel value corresponding to each point in the original image with the pixel value corresponding to each point in the inverse mask image of the target mask image.

[0296] In one embodiment, the target area determination module 708 is further configured to determine the pixel value interval to which the pixel values of each point in the target mask image belong; and perform significance weighting on the pixel values of the corresponding points in the target mask image based on the significance coefficient corresponding to the pixel value interval to obtain a significance-weighted target mask image.

[0297] In one embodiment, the edge determination module 706 is further configured to perform an inverse operation on the texture edge information by using an inversion function to obtain an edge mask.

[0298] In one embodiment, the texture edge information is a texture edge information map. In this embodiment, the edge determination module 706 is further configured to determine difference information between the original image and the smoothed image, and obtain the texture edge information map based on the difference information.

[0299] In one embodiment, the image processing apparatus 700 further includes:

[0300] The sharpening module 712 is configured to extract high-frequency information from the original image based on the difference between the original image and the smoothed image, and add the high-frequency information to the target image to obtain a sharpened target image.

[0301] In one embodiment, the original image contains a human face. In this embodiment, the image processing device 700 further includes:

[0302] The facial features processing module 714 is used to determine the facial features of the face in the original image; based on the facial features, the target image is merged with the original image to obtain a target image that retains the detailed features of the facial features.

[0303] In one embodiment, the original image contains a human face. In this embodiment, the image processing device 700 further includes:

[0304] The face region processing module 716 is used to obtain the face region corresponding to the face in the original image; based on the face region, the target image is merged with the original image to obtain the target image with the optimized face region.

[0305] In one embodiment, Figure 8 As shown, the image processing apparatus 700 further includes:

[0306] The smoothness adjustment module 718 is used to obtain an input smoothing parameter; determine the superposition weights corresponding to the target image and the original image respectively according to the smoothing parameter; superimpose and fuse the target image and the original image according to the corresponding superposition weights; and output the superimposed and fused image.

[0307] In the above-mentioned image processing device, the original image is smoothed to obtain a smoothed image, and then the initial skin area in the smoothed image is determined, and the pixel points in the original image where the pixel values change significantly are determined to obtain texture edge information. Then, based on the texture edge information, the texture edge area in the initial skin area is removed to obtain the target skin area, and the image content in the smoothed image located in the target skin area is integrated into the corresponding position of the target skin area in the original image to obtain the target image, so that the image content within the target skin area in the fused target image is smoothed, while the image content outside the target skin area retains the clarity of the original image, avoiding the problem of reducing the transparency of the image by smoothing the original image as a whole, thereby improving the image quality, and integrating the image content in the target skin area in the smoothed image where the texture edge area is removed into the corresponding position of the target skin area in the original image, also avoiding the smoothing of the texture edge area in the original image, making the texture edge details clear, and further improving the image quality.

[0308] For the specific definition of the image processing device, please refer to the definition of the image processing method above and will not be repeated here. Each module in the above-mentioned image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0309] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store raw image data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0310] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0311] Those skilled in the art will understand that Figure 9 and Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0312] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0313] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0314] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0315] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0316] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0317] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Smoothing the original image to obtain a smoothed image; determining an initial skin region in the smoothed image based on pixels of the smoothed image; Determining pixel points in the original image where pixel values significantly change to obtain texture edge information, comprising: determining pixel points in the original image where pixel values significantly change based on a difference between the original image and the smoothed image, to obtain the texture edge information, wherein the texture edge information includes not only contour information in the original image but also texture information in the original image; Determining an edge mask according to the texture edge information, wherein the edge mask is used to mask the texture edge information in the original image; Removing the texture edge region in the initial skin region based on the texture edge information to obtain a target skin region, comprising: obtaining pixel values in the initial skin region in the smoothed image according to a binary initial mask image, then generating a non-binary initial mask image according to the obtained pixel values, and multiplying and fusing the non-binary initial mask image used to determine the initial skin region with the edge mask to obtain a target mask image; the target mask image is used to determine the target skin region; The image content in the smoothed image located within the target skin area is integrated into the corresponding position of the target skin area in the original image to obtain the target image, including: according to the target mask image, the image content in the smoothed image located within the target skin area is integrated into the corresponding position of the target skin area in the original image to obtain the target image.

2. The method according to claim 1, characterized in that The smoothing process of the original image to obtain the smoothed image comprises: Determine the downsampling multiple according to the resolution of the original image; Downsampling the original image according to the downsampling multiple; The downsampled image is smoothed to obtain a smoothed image.

3. The method according to claim 1, characterized in that The edge mask is non-binary.

4. The method according to claim 3, characterized in that The method further comprises: Acquiring a binary mask image for determining an initial skin area of the smoothed image; Obtaining pixel values within the initial skin area corresponding to the binary mask image from a single color channel of the smoothed image; The non-binary initial mask image for determining the initial skin area is generated based on the acquired pixel values; the pixel values of the points in the initial mask image corresponding to the initial skin area are determined based on the acquired pixel values.

5. The method according to claim 4, characterized in that The step of fusing the image content in the target skin area of the smoothed image into a corresponding position of the target skin area in the original image according to the target mask image to obtain the target image includes: Superimposing and fusing the first image and the second image to obtain a target image; The first image is an image obtained by multiplying the pixel value corresponding to each point in the smoothed image by the pixel value corresponding to each point in the target mask image; The second image is an image obtained by multiplying the pixel value corresponding to each point in the original image by the pixel value corresponding to each point in the inverse mask image of the target mask image.

6. The method according to claim 4, characterized in that The method further comprises: Determine the pixel value interval range to which the pixel value of each point in the target mask image belongs; According to the significance coefficient corresponding to the pixel value interval range, the pixel value of the corresponding point in the target mask image is significance weighted to obtain a significance-weighted target mask image.

7. The method according to claim 3, characterized in that The determining of the edge mask according to the texture edge information comprises: An inverse operation is performed on the texture edge information by an inversion function to obtain an edge mask.

8. The method according to claim 1, characterized in that The texture edge information is a texture edge information map; and determining pixel points in the original image where pixel values significantly change to obtain the texture edge information includes: determining difference information between the original image and the smoothed image; The texture edge information map is obtained according to the difference information.

9. The method according to claim 1, characterized in that The method further comprises: extracting high-frequency information from the original image based on difference information between the original image and the smoothed image; The high-frequency information is added to the target image to obtain a sharpened target image.

10. The method according to claim 1, characterized in that The original image contains a human face; the method further includes: Determining facial features of a person in the original image; According to the facial feature areas, the target image is fused with the original image to obtain a target image that retains facial feature detail features.

11. The method according to claim 1, wherein The original image contains a human face; the method further includes: Obtaining a face region corresponding to a face in the original image; According to the facial region, the target image is fused with the original image to obtain a target image with an optimized facial region.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Get the smoothing parameters of the input; Determining the superposition weights corresponding to the target image and the original image respectively according to the smoothing parameter; Superimposing and fusing the target image and the original image according to corresponding superposition weights; Output the superimposed fused image.

13. An image processing device, characterized in that: The device comprises: A smoothing processing module is used to smooth the original image to obtain a smoothed image; an initial region determining module, configured to determine an initial skin region in the smoothed image based on pixels of the smoothed image; an edge determination module, configured to determine pixel points in the original image where pixel values significantly change to obtain texture edge information, comprising: determining pixel points in the original image where pixel values significantly change based on a difference between the original image and the smoothed image to obtain the texture edge information, wherein the texture edge information includes not only contour information in the original image but also texture information in the original image; a target region determination module, configured to determine an edge mask based on the texture edge information, the edge mask being used to mask the texture edge information in the original image, and to remove the texture edge region in the initial skin region based on the texture edge information to obtain a target skin region, comprising: obtaining pixel values in the initial skin region in the smoothed image based on a binary initial mask image, then generating a non-binary initial mask image based on the obtained pixel values, and multiplying and fusing the non-binary initial mask image used to determine the initial skin region with the edge mask to obtain a target mask image; the target mask image being used to determine the target skin region; A fusion module is used to fuse the image content in the smoothed image located in the target skin area into the corresponding position of the target skin area in the original image to obtain a target image, including: according to the target mask image, fusing the image content in the smoothed image located in the target skin area into the corresponding position of the target skin area in the original image to obtain a target image.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images