Image processing method, model, model training method, medium and equipment

Abstract

The invention relates to the field of image processing, and discloses an image processing method, a model, a model training method, a medium and equipment, which can perform partition image enhancement or partition image style migration on an image according to image scenes such as illumination performance of a human face, and greatly improve the image effect. The method comprises the following steps: acquiring a to-be-processed first image; determining a first semantic region and a second semantic region in the first image, and determining first scene classification information corresponding to the first image; adjusting the colors of the first semantic region and the second semantic region by using a first color parameter and a second color parameter corresponding to the first scene classification information of the first image; and obtaining a second image based on the adjusted first semantic region and the adjusted second semantic region.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, model, model training method, medium and device. Background Art

[0002] With the development of electronic technology, users are paying more and more attention to the image effects of photo shooting, especially the lighting effects of portrait shooting. For example, when shooting portraits, users tend to pay more attention to the facial state of the subject, while overexposure of the face, uneven distribution of facial lighting, blurring in backlight, etc. will affect the shooting experience.

[0003] At present, in order to improve the image effect, an image enhancement algorithm is usually performed on the image. The conventional image enhancement algorithm uses a set of enhancement curves to enhance the entire image. Specifically, the pixel values ​​of the pixels in different pixel value ranges in the image are adjusted using the numerical mapping relationship represented by different enhancement curves. That is, this enhancement method is to divide the entire image into different pixel value ranges and perform pixel-level adjustments. That is, this method focuses on the overall lighting effect of the entire image, but it is difficult to focus on the lighting effect of the face area in the entire image, so it is impossible to perform targeted enhancement on the lighting effect of the face area alone. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, model, model training method, medium and device, which can partition and enhance images according to image scenes such as the lighting performance of the face, thereby greatly improving the image effect.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes: acquiring a first image to be processed; determining a first semantic area and a second semantic area in the first image, and determining first scene classification information corresponding to the first image; adjusting the colors of the first semantic area and the second semantic area respectively using first color parameters and second color parameters corresponding to the first scene classification information of the first image; and obtaining a second image based on the adjusted first semantic area and the adjusted second semantic area.

[0006] For example, the first image is a portrait image. It can be understood that the first semantic area is different from the second semantic area, and the first color parameter is different from the second color parameter. The first image can be an image captured in real time or an image pre-acquired by the terminal device, and the first image can be in RGB format. When the first image is an image captured in real time during the shooting process, the terminal device can convert the captured image from RAW format to RGB format to obtain the first image. Specifically, the first color parameter and the second color parameter can be used to perform partition enhancement of the first image in combination with scene information, such as partition image enhancement, or partition image style transfer of the first image in combination with scene information. In this way, the present application can improve the classification ability of different image scenes in combination with scene classification information, such as improving the classification ability of different lighting conditions of the face in the portrait image, so as to perform targeted partition enhancement of the image, or partition image style transfer, so as to enhance the user's experience of viewing the adjusted image.

[0007] In a possible implementation of the first aspect, the first scene classification information is related to at least one of the following: the lighting conditions of the first image, the skin color of the person in the first image, and the color temperature of the first image. For example, when the first image is a portrait image, the first scene classification information corresponds to the lighting conditions of the face, such as front light, back light, color light, etc. As an example, the first scene classification information includes scene coding information corresponding to a plurality of lighting conditions.

[0008] In a possible implementation of the first aspect, the first semantic region is a foreground region in the first image and the second semantic region is a background region in the first image; or, the first semantic region is a different foreground region in the first image. It is understood that the first image may have multiple foreground regions and one background region.

[0009] In a possible implementation of the first aspect, determining a first semantic area and a second semantic area in a first image includes: obtaining a first segmentation mask image of the first image, the first segmentation mask image being a segmentation mask image of a foreground in the first image; determining a first semantic area based on the first segmentation mask image, wherein the first semantic area is a foreground area in the first image; determining a second semantic area in the first image based on the first semantic area, wherein the second semantic area is a background area in the first image. For example, the present application may identify a first segmentation mask in a first image through a semantic segmentation network, but is not limited thereto. Furthermore, the area in the first image excluding one or more first semantic areas may be the second semantic area.

[0010] In a possible implementation manner of the first aspect, the first image is a portrait image, and the first semantic region includes at least one of a human body region, a face region, and a mouth region.

[0011] In a possible implementation of the first aspect, determining first scene classification information corresponding to the first image includes: determining the first scene classification information according to image features of a first facial region in the first image. It can be understood that the present application can crop a facial image from the first image to obtain the first facial region. For example, the image features corresponding to the first scene classification information can be image latent codes, such as image latent codes corresponding to a variety of lighting conditions (i.e., lighting conditions).

[0012] In a possible implementation of the first aspect, the method further includes: obtaining a first depth image feature corresponding to the first image; determining a first color parameter and a second color parameter according to the first depth image feature and the first scene classification information, wherein the first color parameter and the second color parameter are different 3D LUT parameters. As an example, when the first scene classification information is a classification of different illumination scenes, the 3D LUT parameters corresponding to the facial region are also different. It can be understood that the 3D LUT parameters corresponding to the different first semantic regions and the second semantic regions in the first image are determined in real time in combination with the first scene classification information and the first depth image feature. In addition, when the first image has multiple first semantic regions (foreground regions), different first semantic regions may correspond to different first color parameters, i.e., different 3D LUT parameters. For example, when the illumination scene classification information indicates a backlight situation, the degree of enhancement of the brightness by the 3D LUT parameters corresponding to the facial region is greater than the degree of enhancement of the brightness by the 3D LUT parameters corresponding to the facial region when the illumination scene classification information indicates a front light situation.

[0013] In a possible implementation of the first aspect, the colors of the first semantic area and the second semantic area are adjusted respectively using the first color parameter and the second color parameter corresponding to the first scene classification information of the first image, including: adjusting the color of the first image respectively using the first color parameter and the second color parameter to obtain a third image containing the adjusted first semantic area and a fourth image containing the adjusted second semantic area; and obtaining the second image based on the adjusted first semantic area and the second semantic area, including: fusing the adjusted first semantic area in the third image and the adjusted second semantic area in the fourth image to obtain the second image. It can be understood that different semantic areas in the second image are obtained based on different color parameters (3D LUT parameters).

[0014] In a possible implementation of the first aspect, the number of first semantic regions is M, one first semantic region corresponds to one first color parameter, and the color of the first image is adjusted using the first color parameter and the second color parameter respectively to obtain a third image containing the adjusted first semantic region and a fourth image containing the adjusted second semantic region, including: adjusting the color of the first image using the first color parameters corresponding to each of the M first semantic regions respectively, generating a corresponding third image for each first semantic region, and obtaining M third images; adjusting the color of the first image using the second color parameter to obtain a fourth image containing the second semantic region; and fusing the adjusted first semantic region in the third image and the adjusted second semantic region in the fourth image to obtain the second image, including: fusing the adjusted first semantic region in each third image and the adjusted second semantic region in the fourth image to obtain the second image. As an example, the present application can replace the corresponding semantic region in the fourth image with the adjusted first semantic region in each third image based on the fourth image, so as to achieve that the second image contains different semantic regions adjusted by different color parameters. In addition, the present application may also use a first semantic region in a third image as a reference to replace the corresponding semantic region in the third image with each semantic region in other third images and the second semantic region in the fourth image.

[0015] In a possible implementation of the first aspect, determining a first color parameter and a second color parameter according to a first depth image feature and first scene classification information includes: obtaining N first weight values ​​corresponding to a first semantic area and N second weight values ​​corresponding to a second semantic area according to the first depth image feature and the first scene classification information, wherein the N first weight values ​​are related to the color of the first semantic area, and the N second weight values ​​are related to the color of the second semantic area; using the N first weight values ​​and the N second weight values ​​to perform weighted calculations on the preset N first 3D LUT parameters respectively to obtain a third 3D LUT parameter and a fourth 3D LUT parameter; performing trilinear interpolation on the third 3D LUT parameter and the fourth 3D LUT parameter respectively to obtain the first color parameter and the second color parameter. For example, the preset N first 3D LUT parameters may be a basic 3D LUT, the size of which may be 33*33*33, and the 3D LUT represented by the first color parameter and the second color parameter may be 256*256*256. It can be understood that the present application can use a pre-trained encoder to predict the weight values ​​corresponding to different semantic areas in the first image, such as predicting T*N weight values, where T is the number of semantic areas in the first image (i.e., the first semantic area and the second semantic area).

[0016] In a possible implementation manner of the first aspect, determining a first color parameter and a second color parameter according to a first depth image feature and first scene classification information includes: obtaining N first weight values ​​corresponding to a first semantic area and N second weight values ​​corresponding to a second semantic area according to the first depth image feature and the first scene classification information, where N is a positive integer; obtaining S first sampling intervals corresponding to the first semantic area and S second sampling intervals corresponding to the second semantic area according to the first depth image feature and the first scene classification information, where S is a positive integer; performing weighted calculation on N preset first 3D LUT parameters using the N first weight values ​​and the N second weight values, respectively, to obtain a third 3D LUT parameter and a fourth 3D LUT parameter; performing sampling processing on the third 3D LUT parameter using the S first sampling intervals to obtain a fifth 3D LUT parameter; performing sampling processing on the fourth 3D LUT parameter using the S second sampling intervals to obtain a sixth 3D LUT parameter; performing trilinear interpolation on the fifth 3D LUT parameter and the sixth 3D LUT parameter, respectively, to obtain the first color parameter and the second color parameter. It can be understood that the present application can use a pre-trained encoder to predict the sampling intervals corresponding to different semantic areas in the first image. It can be understood that the present application can predict T*S sampling intervals, where T is the number of semantic areas (i.e., the first semantic area and the second semantic area) in the first image. For example, S=32*3=96, where 32 represents the number of intervals of the basic 3D LUT of size 33*33*33), and 3 represents the number of channels of the three channels corresponding to the RGB format. Moreover, the above T*S sampling intervals are non-uniform sampling intervals. In this way, the present application can combine non-uniform interval sampling and trilinear interpolation to determine different 3D LUT parameters corresponding to different semantic areas, thereby improving the adaptability of semantic areas and 3D LUT parameters, which is beneficial to improving the effect of the final portrait partition enhancement.

[0017] In a second aspect, an embodiment of the present application provides an image processing model, including: a scene classification module, which is used to generate first scene classification information corresponding to the first image based on the image of the first facial area of ​​the first image to be processed; a parameter generation module, which is used to determine the first semantic area and the second semantic area in the first image; an image enhancement module, which is used to adjust the colors of the first semantic area and the second semantic area respectively according to the first color parameter and the second color parameter corresponding to the first scene classification information of the first image, and obtain the second image based on the adjusted first semantic area and the adjusted second semantic area. The above-mentioned image processing model can be a neural network model, such as the image enhancement network referred to below. The above-mentioned scene classification module can also include a scene encoding part.

[0018] In a possible implementation of the second aspect, the parameter generation module includes a first encoder, the first encoder is used to input a first image and a first segmentation mask image of the first image, the first segmentation mask image is a segmentation mask image of the foreground in the first image, and a first semantic area is determined based on the first segmentation mask image, wherein the first semantic area is the foreground area in the first image, and a second semantic area in the first image is determined based on the first semantic area, wherein the second semantic area is the background area in the first image. For example, the first encoder can be pre-trained. The parameter generation module can also be called a weight prediction part.

[0019] In a possible implementation of the second aspect, the scene classification module includes a second encoder, which is specifically configured to input an image of a first facial region in a first image, and determine first scene classification information based on image features of the first facial region in the first image. For example, the second encoder may be pre-trained.

[0020] In a possible implementation manner of the second aspect, the first color parameter and the second color parameter are determined according to a first depth image feature of the first image and first scene classification information, wherein the first color parameter and the second color parameter are different 3D LUT parameters.

[0021] In a possible implementation of the second aspect, a parameter generation module is used to obtain N first weight values ​​corresponding to the first semantic area and N second weight values ​​corresponding to the second semantic area according to the first depth image feature and the first scene classification information, wherein the N first weight values ​​are related to the color of the first semantic area, and the N second weight areas are related to the color of the second semantic area; an image enhancement module is used to use the N first weight values ​​and the N second weight values ​​to perform weighted calculations on the preset N first 3D LUT parameters respectively to obtain the third 3D LUT parameters and the fourth 3D LUT parameters, and perform trilinear interpolation on the third 3DLUT parameters and the fourth 3D LUT parameters respectively to obtain the first color parameters and the second color parameters. The image enhancement module can also be called the image enhancement part in the neural network model.

[0022] In a possible implementation manner of the second aspect, a parameter generation module is used to obtain, according to the first depth image feature and the first scene classification information, N first weight values ​​corresponding to the first semantic area and N second weight values ​​corresponding to the second semantic area, wherein the N first weight values ​​are related to the color of the first semantic area, and the N second weight areas are related to the color of the second semantic area, and according to the first depth image feature and the first scene classification information, S first sampling intervals corresponding to the first semantic area and S second sampling intervals corresponding to the second semantic area are obtained, where S is a positive integer; and an image enhancement module is used to use the N first weight values ​​and the N second weight values ​​to perform weighted calculation on the preset N first 3D LUT parameters respectively to obtain a third 3D LUT parameter and a fourth 3D LUT parameter, use the S first sampling intervals to sample the third 3D LUT parameter to obtain a fifth 3D LUT parameter, use the S second sampling intervals to sample the fourth 3DLUT parameter to obtain a sixth 3D LUT parameter, and perform trilinear interpolation on the fifth 3D LUT parameter and the sixth 3D LUT parameter respectively to obtain a first color parameter and a second color parameter. At this time, the parameter generation module may include an interval prediction part.

[0023] In a third aspect, an embodiment of the present application provides an image processing method applied to an electronic device, the method comprising: inputting a first image to be processed into an image processing model in the second aspect and any possible implementation thereof; and obtaining a second image output by the image processing model.

[0024] In a fourth aspect, an embodiment of the present application provides an image processing model training method, which is applied to an electronic device, and the method includes: inputting training sample data into a neural network model to be trained, the training sample data including a sample image, a scene type to which the sample image belongs, and a standard enhanced image corresponding to the sample image, the standard enhanced image being an image obtained by image processing multiple semantic regions of the sample image using different color parameters based on the scene type; obtaining a first enhanced image and a scene classification result output by the neural network model; adjusting the parameters of the neural network model based on the first enhanced image, the standard enhanced image, the scene classification result, and the scene type, so as to obtain an image processing model when the first enhanced image and the scene classification result output by the neural network model meet a first condition. For example, the sample image may be a portrait image, and the scene type to which the sample image belongs may be an illumination table corresponding to a facial region in the sample image, and the illumination table may be information on the illumination condition of the facial region image of the sample image pre-marked by the user. The first enhanced image and the scene classification are obtained by real-time prediction of the neural network model for the sample image. Moreover, the standard enhanced image can also be used for training enhanced portrait images. For example, the training enhanced portrait image is an image obtained by manually performing portrait enhancement processing on a training portrait image and focusing on portrait enhancement based on the illumination conditions of the face.

[0025] In a possible implementation manner of the fourth aspect, the scene classification result is related to at least one of the following: the lighting conditions under which the sample image is captured, the skin color of a person in the sample image, and the color temperature of the sample image.

[0026] In a possible implementation of the fourth aspect, based on the first enhanced image, the standard enhanced image, the scene classification result and the scene type, the parameters of the neural network model are adjusted, including: determining the value of the first loss function according to the scene classification result and the scene type; adjusting the first parameter of the neural network model according to the value of the first loss function; determining the value of the second loss function according to the first enhanced image and the standard enhanced image; adjusting the second parameter of the neural network model according to the value of the second loss function. It can be understood that the first parameter can be the network parameter corresponding to the first encoder, and the second parameter can be the network parameter corresponding to the second encoder.

[0027] In a possible implementation manner of the fourth aspect, the first condition includes: a value of the first loss function is within a first value range, and a value of the second loss function is within a second value range.

[0028] In a fifth aspect, an embodiment of the present application provides a readable medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the image processing method in the first aspect and any possible implementation thereof.

[0029] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing an image processing method as in the first aspect and any possible implementation thereof.

[0030] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising: a computer program / instructions, which, when executed by a processor, implements the image processing method in the first aspect and any possible implementation thereof.

[0031] The beneficial effects of the second to seventh aspects of the present application can be referred to the description of the first aspect and its various possible implementation methods, and will not be elaborated on here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 shows a schematic diagram of an image processing scenario according to some embodiments of the present application;

[0033] Figure 2 According to some embodiments of the present application, a schematic diagram of an image processing architecture based on an image enhancement network is shown;

[0034] Figure 3 According to some embodiments of the present application, a schematic diagram of a calculation process related to trilinear interpolation is shown;

[0035] Figure 4 According to some embodiments of the present application, a schematic diagram of a network architecture of an image enhancement network is shown;

[0036] Figure 5 According to some embodiments of the present application, a flow chart of an image processing method is shown;

[0037] Figure 6 According to some embodiments of the present application, a schematic diagram of a process of trilinear interpolation processing is shown;

[0038] Figure 7 According to some embodiments of the present application, a schematic diagram of a process of non-uniform interval processing and trilinear interpolation processing is shown;

[0039] Figure 8 According to some embodiments of the present application, a schematic diagram of a training and application process of an image enhancement network is shown;

[0040] Fig. 9 According to some embodiments of the present application, a structural schematic diagram of a mobile phone is shown. DETAILED DESCRIPTION

[0041] Illustrative embodiments of the present application include, but are not limited to, image processing methods, media, and electronic devices.

[0042] First, some terms provided in the embodiments of the present application are introduced.

[0043] Three-dimensional (3D) lookup table (LUT): A color mapping technology that achieves color correction of an image by mapping input color values ​​to output color values. It can be used to beautify and polish an image. Specifically, 3D LUT technology can perform color correction on images in red, green, and blue (RGB) format through a mapping relationship. For example, the mapping relationship can be expressed as (R, G, B) = f(r, g, b), where (r, g, b) represents the input color and (R, G, B) represents the output color.

[0044] Semantic segmentation: It is an important branch of image processing and machine vision, and its goal is to accurately understand the image scene and content. For example, the basic structure of a neural network for semantic segmentation may include an encoder and a decoder. The encoder can extract features from an image through filters. The decoder is responsible for generating the final output, which is usually a segmentation mask containing the contours of the object, that is, the output of image semantic segmentation can be multiple segmentation masks. For example, the semantic segmentation of a portrait image can obtain a human mask image (such as a human foreground image), a facial mask image, a human mouth mask image, etc.

[0045] Loss function: It is an important concept in machine learning and deep learning, which is used to measure the difference or error between the model prediction result and the actual result. In this application, loss function is also called loss function.

[0046] In order to solve the problem that the current image enhancement algorithm is difficult to focus on the lighting performance of the face, the present application provides an image processing method, which can identify the scene classification information of the portrait image for a portrait image, such as the lighting scene classification information corresponding to the lighting conditions such as the facial area in the portrait image and the front light or back light. In addition, the method can distinguish semantic areas such as the facial area, the mouth area, the human body area and the background area of ​​the portrait image, and then perform color enhancement on these semantic areas respectively in combination with the determined lighting scene classification information, such as image enhancement based on 3D LUT parameters. For example, the method can determine the 3D LUT parameters corresponding to each semantic area, and use these 3D LUT parameters to enhance the portrait image respectively to obtain multiple enhanced images, and then merge the multiple enhanced images with the different semantic areas corresponding to each of these 3D LUT parameters into a complete output image, thereby realizing image partition enhancement combined with scene classification. In this way, the present application can pay attention to the lighting performance of the face, and partition the image according to the lighting performance of the face, thereby greatly improving the image effect, so that the image effect meets user needs.

[0047] For example, when the lighting scene classification information indicates a backlit situation, the degree of enhancement of the brightness by the 3D LUT parameters corresponding to the facial region is greater than the degree of enhancement of the brightness by the 3D LUT parameters corresponding to the human mouth region. In addition, the 3D LUT parameters corresponding to the facial region under different lighting scene classifications are also different. For example, when the lighting scene classification information indicates a backlit situation, the degree of enhancement of the brightness by the 3D LUT parameters corresponding to the facial region is greater than the degree of enhancement of the brightness by the 3D LUT parameters corresponding to the facial region when the lighting scene classification information indicates a front-lit situation.

[0048] In some embodiments, after collecting an image, the present application may first convert the image into a red-green-blue (RGB) format, crop the facial region image in the image, and perform semantic segmentation on the image to obtain semantic segmentation mask images such as a face mask image, a mouth mask image, and a body mask image. On the one hand, the illumination scene classification information corresponding to the facial region image and the illumination conditions such as front light, back light, side light, and color light is determined, and the illumination scene classification information may be a set of image latent codes (or image features) corresponding to the illumination conditions such as front light, back light, side light, and color light. On the other hand, the method can identify the depth image features of the portrait image, such as semantic features or texture features. Furthermore, the method can combine the identified illumination scene classification information and depth image features to determine the 3D LUT parameters corresponding to each semantic region of the portrait image, wherein these semantic regions include the semantic regions corresponding to each semantic segmentation mask image and the background region of the portrait image. Furthermore, the method can use various 3D LUT parameters to perform color correction on the portrait image to obtain multiple enhanced images, and then fuse the different semantic areas corresponding to different 3D LUT parameters in the multiple enhanced images into a complete RGB format output image based on the various semantic segmentation mask images. Thus, the scene-based partitioning enhancement effect of the portrait image can be achieved.

[0049] In this way, compared with the image enhancement method in the background technology, the image processing method of the present application combines the lighting scene classification information of the facial lighting performance, and realizes different enhancement algorithms for different areas of the image based on different lighting scene classifications. That is, the facial area in the image can be targetedly enhanced according to the facial lighting performance, which is conducive to improving the image enhancement effect.

[0050] It can be understood that the portrait enhancement based on 3D LUT parameters in the present application can achieve image enhancement in RGB format in dimensions such as brightness, saturation, and contrast.

[0051] In some embodiments, the image scene classification provided by the embodiments of the present application is not limited to scene classification of the facial area in the image, but can also be scene classification of another area, scene classification of multiple areas at the same time, or scene classification of the entire image.

[0052] In some embodiments, the scene classification provided by the present application is not limited to the above-mentioned lighting scene classification, but can also be set according to the actual needs of the user. For example, scene classification is used to classify images of people with different skin colors, such as white, yellow, black and other skin color scenes. For another example, scene classification is used to classify images with different color temperatures, such as natural light sources, sunlight at sunrise, sunlight half an hour after sunrise, household incandescent lamps, cloudy days and other color temperature scenes.

[0053] In some embodiments, the processing object of the image processing method provided by the present application is not limited to portrait images, but can also be other types of images, such as animal images, scene images, or landscape images.

[0054] In some embodiments, the image processing method provided by the present application is not limited to image enhancement under different scene classifications, but can also be applied to image style migration under different scene classifications. At this time, the image style based on 3D LUT parameters refers to the image style in dimensions such as brightness, saturation, and contrast represented in RGB format.

[0055] The image processing method provided in this application can be applied to the process of post-processing image materials such as pictures or videos obtained by shooting. In this case, the method can use 3D LUT technology to perform color restoration and other processing on image materials such as photos or videos, such as portrait enhancement or image style transfer.

[0056] In this way, the present application can combine scene classification information to improve the ability to classify different image scenes, such as improving the ability to classify different lighting conditions of the face in a portrait image, so as to perform targeted partition enhancement of the image, or partition image style transfer, so as to enhance the user experience of viewing the adjusted image.

[0057] In some embodiments, the image processing method provided in the embodiments of the present application can be applied to the camera application of a mobile phone. For example, the camera application can provide a partition enhancement function and support users to turn the function on or off. In addition, when the camera application turns on the function, after the mobile phone captures a frame of image in shooting mode or video mode, the captured image can be post-processed to achieve partition portrait enhancement or image style transfer combined with scene classification.

[0058] like Figure 1A As shown, the user opens the camera application of the mobile phone and displays the shooting preview interface, and the shooting preview interface includes "Album" 200, "Personalized Enhancement" function switch 201, "Settings" 202, shooting controls 203, viewfinder 204, and shooting mode selection controls such as "Portrait", "Photo", and "Video". Among them, "Album" 200 is used to trigger the mobile phone to display the photos or videos taken during the most recent shooting process. The shooting control 203 is used to trigger the mobile phone to start shooting operations such as taking photos or recording videos. The viewfinder 204 is used to display the shooting preview image collected by the mobile phone in real time. The "Personalized Enhancement" function switch 201 is used to trigger the mobile phone to turn on or off the "Partition Enhancement" function, Figure 1A The function switch 201 in the figure indicates that the "partition enhancement" function is turned on. Figure 1AIn the "Portrait", "Photo", "Video" and other shooting modes, you can turn on the "Partition Enhancement" function to post-process the images collected during the shooting process, that is, to perform portrait enhancement or image style transfer in different semantic areas based on the collected images combined with scene classification.

[0059] Then, when the user Figure 1A After the shooting control 203 shown is clicked to trigger the mobile phone to take a photo, the mobile phone can capture the image A1 in the viewfinder 204, and perform the above-mentioned image processing on the image A1, such as the lighting scene classification based on the facial area, and the facial area and background area in the image A1 are enhanced using different 3DLUT parameters to obtain the photo A2, and then the photo A2 is saved to the album application. Figure 1A After clicking the photo album 200 shown, the mobile phone can enter Figure 1B The album interface shown displays photo A2, so that the user can view portrait photos with good facial enhancement effect, that is, with good facial lighting conditions.

[0060] In some embodiments of the present application, the subject that executes the image processing method in the present application is not limited to the mobile phone in the above example, but can also be other electronic devices with shooting functions or camera applications. For example, the electronic device can be a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc.

[0061] In some embodiments, the image processing method provided in the embodiments of the present application, after capturing a portrait image, the process of cropping a facial area image from the portrait image and segmenting each semantic segmentation mask image can be called image preprocessing.

[0062] In some embodiments, the present application may use a pre-trained semantic segmentation network to segment various semantic segmentation mask images in the portrait image, such as a face mask image, a mouth mask image, a body mask image, etc. Moreover, the size of these semantic segmentation mask images may be consistent with the size of the original portrait image.

[0063] In some embodiments, the present application may use a network model such as a neural network to perform scene classification and recognition and image partition enhancement on portrait images. For example, the neural network may be called an image enhancement network or an image processing model.

[0064] Next, refer to Figure 2 , which is a schematic diagram of an image processing architecture based on an image enhancement network provided in an embodiment of the present application. Figure 2 The image enhancement network shown mainly includes three parts: scene coding, weight prediction and image enhancement. The scene coding part mainly includes encoder 1 and a full connection (FC) layer, and the encoder 1 can also be a scene encoder. The weight prediction part mainly includes encoder 2, which can also be called a weight prediction encoder.

[0065] In some embodiments, Figure 2 The input of the image enhancement network in is T+1 images, including portrait image A1, human body mask image A11, human mouth mask image A12, human face mask image A13 and facial region image A1', and the output is portrait image A2 after portrait enhancement of portrait image A1. At this time, encoder 1 inputs A1', and encoder 2 inputs T images (i.e., A1, A11, A12 and A13).

[0066] In some embodiments, Figure 2 The image enhancement network shown may also input N base 3D LUTs, or the N base 3D LUTs may be pre-set in the image enhancement network.

[0067] Reference Figure 2 , the encoder 1 can identify a group of image latent codes (i.e., image classification information) corresponding to the facial area image A1' and the lighting conditions such as front light, back light, side light, and color light, and the group of image latent codes can include the image latent codes corresponding to each of these lighting conditions. For example, the image latent codes corresponding to the various lighting conditions can be image features such as light position features. At this time, the group of images can output the scene classification results after passing through the FC layer, such as the scene classification results indicating that A1' is in the front light condition.

[0068] In some embodiments, the encoder 1 can be implemented by a neural network. The above encoder 1 may include different types of network layers, such as convolutional layers, pooling layers, etc. Among them, the function of the convolutional layer is to extract information from the input image, which is called image features, and these image features are reflected by each pixel in the image in a combined or independent manner, such as the texture features and color features of the image. The function of the pooling layer is to select the image features extracted from the convolutional layer. The pooling layer is generally followed by the FC layer, which converts all feature matrices of the pooling layer into one-dimensional feature vectors. The fully connected layer is generally placed at the end of the convolutional neural network structure for classifying images, that is, the fully connected layer is used to output the classification results of the neural network. At this time, the output of the encoder 1 without the FC layer is a set of image latent codes corresponding to the image and various lighting conditions.

[0069] In some embodiments, the encoder 1 provided in the present application can be trained based on multiple training facial images and a lighting table corresponding to the lighting conditions of the multiple training facial images, and the lighting table may include lighting scene classification information corresponding to front light, back light, side light, and color light. Specifically, during the training process, the encoder 1 inputs a training facial image and outputs the lighting scene classification result of the training facial image, such as front light, and determines the value of the loss function based on the lighting scene classification result and the lighting scene classification information in the lighting table. If the value of the loss function is not in the preset numerical range 1, the network parameters of the encoder 1 are adjusted. For example, the network parameters may include the learning rate, the number of iterations, the size of the small batch data, etc. If the value of the loss function is in the numerical range 1, it means that the encoder 1 training is completed.

[0070] For example, Figure 2 The middle face area image A1' can be a training face image. At this time, after A1' enters encoder 1 (i.e., the second encoder), the loss function between the scene classification result output by encoder 1 through the FC layer and the illumination table is Loss1 (i.e., the first loss function). Encoder 1 can determine whether to adjust the network parameters (first parameters) of encoder 1 based on whether the area of ​​Loss1 is in value range 1 (i.e., the first value range).

[0071] In some embodiments, the encoder 2 provided in the present application can be trained based on multiple training portrait images, corresponding semantic segmentation mask images, and training enhanced portrait images, wherein the corresponding semantic segmentation mask images of each training portrait image include a human mask image, a face mask image, and a mouth mask image, and each training enhanced portrait image is a training portrait image that is manually enhanced and focuses on the image of portrait enhancement based on the face lighting conditions. Specifically, during the training process, the encoder 2 inputs a training portrait image and the corresponding image segmentation mask images and then outputs an enhanced portrait image, compares the enhanced portrait image with the training enhanced portrait image corresponding to the training portrait image, and thereby determines the value of the loss function related to image enhancement. If the value of the loss function is not in the preset numerical range 2, the network parameters of the encoder 2 are adjusted. If the value of the loss function is in the numerical range 2, it means that the encoder 2 training is completed.

[0072] For example, Figure 2The middle face area image A1 is a training portrait image and the image A3 is the corresponding training enhanced portrait image. At this time, after A1 and its corresponding A11-A13 enter the encoder 2, the encoder 2 (i.e., the first encoder) can output the enhanced portrait image A2 after a series of subsequent processing, and the loss function between A2 and A3 is Loss2 (i.e., the second loss function). The encoder 2 can determine whether to adjust the network parameters (i.e., the second parameters) of the encoder 2 based on whether the area of ​​Loss2 is in the value range 2 (i.e., the second value range).

[0073] Next, Figure 2 The weight prediction part and the image enhancement part shown are described in detail.

[0074] Reference Figure 2 In the weight prediction part, the T images of portrait image A1, human body mask image A11, human mouth mask image A12, and human face mask image A13 are down sampled and input into encoder 2. Encoder 2 can determine that portrait image A1 corresponds to T semantic regions, namely human body region, human mouth region, human face region, and background region. In addition, encoder 2 can identify the depth image features of portrait image A1, and based on the image depth features and a set of image latent codes corresponding to various lighting conditions from encoder 1 and not passing through the FC layer, predict T (such as T=4) groups of weight values ​​corresponding to N basic 3D LUTs, and each group of weight values ​​includes N (such as N=3) weight values ​​corresponding to the N basic 3D LUTs one by one. For example, Figure 2 The T groups of weight values ​​include W11-W1N, W21-W2N, ..., WT1-WTN, that is, the encoder 2 predicts a total of T*N weight values.

[0075] Reference Figure 2In the image enhancement part, T groups of weight values ​​including W11-W1N, W21-W2N, …, WT1-WTN can be weighted with N basic 3D LUTs respectively, and T weighted 3D LUTs can be obtained to realize the fusion of basic 3DLUTs. Moreover, T weighted 3D LUTs can be subjected to trilinear interpolation to obtain T fused 3D LUTs, that is, a fused 3DLUT corresponding to each of the T semantic areas. Thus, adaptive 3D LUTs for different semantic areas can be generated. At this time, the T fused 3D LUTs will be used to perform image enhancement on the T semantic areas in the human body image A1 respectively. For example, T groups of fused 3D LUTs can perform image enhancement on the entire human body image A1 to obtain T enhanced images. Then, based on the human body mask image A11, the human mouth mask image A12, and the human face mask image A13, the enhanced human body area image, the enhanced human mouth area image, the enhanced human face area image, and the enhanced background area image can be extracted from the T enhanced images respectively, and then these images can be fused into a complete enhanced human face image A2.

[0076] In some embodiments, the base 3D LUT is represented by the following formula (1):

[0077] T={(T r，(i，j，k) , T g，(i，j，k) , T b，(i，j，k) )}, i, j, k∈(0, 1, 2,...N-1) (1)

[0078] Among them, T r，(i，j，k) Indicates the R channel value, T g，(i，j，k) Indicates the G channel value, T b，(i，j，k) Indicates the B channel value.

[0079] In some embodiments, the weight prediction, i.e., the encoder 2 may use MobileNet-V2 (a neural network), predicting a total of T*N weight values, and generating the fused 3D LUT according to these weight values ​​by the following formula (2):

[0080] Lut t =w t1 *Lut1+w t2 *Lut2+…w tN *Lut N (2)

[0081] Wherein, t∈(1, 2, ...N) in formula (2). At this time, Lut in formula (2) t correspond Figure 23D LUT after weighted calculation.

[0082] In some embodiments, the present application searches the input portrait image A1 on T fused 3D LUTs according to the semantic segmentation results, and generates T enhanced image regions, which are recorded as pic0, pic1, pic2, ..., pic(t-1), where t is an integer less than or equal to T. Image fusion is performed according to a certain fusion method to generate an enhanced portrait image, which is converted from RGB format to blue-green-red (BGR) format and saved to obtain an enhanced portrait image A2 combined with scene coding information.

[0083] Understandably, Figure 2 In the weight prediction part shown, N basic 3D LUTs are pre-set, and T groups of weight values ​​are predicted in real time. However, in some other embodiments, the weight prediction part of the image enhancement network can pre-set T groups of weight values ​​and predict N basic 3D LUTs in real time, which is not specifically limited in this application. Figure 2 The output of the encoder 1 shown is N basic 3D LUTs, and the N basic 3D LUT parameters and the predetermined T groups of weight values ​​are still weighted calculated to perform basic 3D LUT fusion.

[0084] In some embodiments, the size of the basic 3D LUT generally used is 33*33*33, so the T weighted 3D LUTs are respectively subjected to trilinear interpolation to obtain an interpolated 3D LUT of size 256*256*256 (i.e., the fused 3D LUT), and then each semantic area of ​​the portrait image is searched on the interpolated 3D LUT for color adjustment. In addition, in other embodiments, the size of the basic 3D LUT may be 77*77*77, etc., which is not specifically limited here.

[0085] It can be understood that trilinear interpolation is mainly used in a 3D cube. It is a linear interpolation method that calculates the values ​​of other points in the cube by giving the value of a vertex, that is, calculating the surrounding 8 vertices in a cube.

[0086] For example, the weight prediction part and the interval prediction part in the above-mentioned image enhancement network can be called a parameter generation module, the scene encoding part can be called a scene classification module, and the image enhancement part can be called an image enhancement module.

[0087] Reference Figure 3 , which is a schematic diagram of the calculation process related to trilinear interpolation involved in an embodiment of the present application. Figure 3 The cube in (a) can be Figure 2 A cube in a weighted 3D LUT is shown, i.e., a periodic cubic grid with a step size of 1. And, Figure 3 The 8 vertices of the cube shown in (a) include C 000 , C 100 , C 110 , C 010 , C 001 , C 101 , C 111 , C 011 , and trilinear interpolation can be performed to obtain Figure 3 Point C, points C0, C1, C 01 , C 11 , C 001 , C 10 Specifically, assuming Figure 3 The coordinates of point C in (a) are (x, y, z), and the coordinates (x0, y0, z0) are the points with the smallest coordinates among the eight vertices of the cube (such as point C). 000 ), and the coordinates of these points are given by Figure 3 The coordinate system (xyz) is set as shown. Where x0 represents a grid point below x along the x-axis, x1 represents a grid point above x along the x-axis, and the relationship between y0, y1, z0, z1 is similar to the relationship between x0 and x1. In addition, xd, yd, zd are taken as the difference of the maximum integers of the points to be calculated (i.e., the points to be interpolated) less than x, y, z,.

[0088] In some embodiments, trilinear interpolation can be performed along the x-axis, y-axis, and z-axis, respectively. First, formula (3-6) can be used to interpolate Figure 3 The cube in (a) is interpolated along the x direction:

[0089] C 00 =V[x0,y0,z0](1-x d )+V[x0,y0,z0]x d (3)

[0090] C 01 =V[x0,y0,z1](1-x d )+V[x0,y0,z1]x d (4)

[0091] C 10 =V[x0,y1,z0](1-x d )+V[x0,y1,z0]x d (5)

[0092] C 11 =V[x0,y1,z1](1-x d )+V[x0,y1,z1]x d (6)

[0093] Among them, V[x0, y0, z0] represents the value of the function V[] at the point (x0, y0, z0).

[0094] Then, we can use formula (7-8) to Figure 3 The cube in (a) is interpolated along the y direction:

[0095] C0=C 00 (1-y d )+C 10 y d (7)

[0096] C1=C 01 (1-y d )+C 11 y d (8)

[0097] Finally, we can use formula (9) to Figure 3 The cube in (a) is interpolated along the z direction:

[0098] C=C0(1-z d )+C1z d (9)

[0099] In some embodiments, the trilinear interpolation in this application can be indexed by RGB values, which is equivalent to Figure 3 The XYZ axes in the.

[0100] Understandably, Figure 2 and Figure 3 The 3D LUT shown is a conventional uniformly sampled 3D LUT.

[0101] In some embodiments, the present application performs image enhancement using a 3D LUT at a non-uniform sampling interval. Figure 2 Based on the network architecture shown, Figure 4 FIG. 1 is a schematic diagram of a network architecture of another image enhancement network provided in an embodiment of the present application. Figure 2 Compared to the architecture shown, Figure 4 An interval prediction part is added, and the image enhancement part also needs to perform trilinear interpolation in combination with a 3D LUT with non-uniform sampling intervals to obtain T fused 3D LUTs.

[0102] like Figure 4As shown, the encoder 2 can not only predict T groups of weight values, but also predict T groups of sampling intervals, including W11-W1S, W21-W2S, ..., WT1-WTS, that is, the encoder 2 predicts a total of T*S sampling intervals. For example, S = 32*3 = 96, where 32 represents the number of intervals of the basic 3D LUT of size 33*33*33, and 3 represents the number of channels of the three channels corresponding to the RGB format. After T weighted 3D LUTs are obtained by weighted calculation using T groups of weight values ​​and N basic 3D LUTs, T groups of sampling intervals can be used to adjust the T weighted 3D LUTs to obtain T 3D LUTs with non-uniform sampling intervals. Then, the 3D LUTs with T non-uniform sampling intervals can be trilinearly interpolated to obtain T fused 3D LUTs, and then the T fused 3D LUTs are used to achieve image enhancement of different semantic areas of the input image (such as the portrait image A1).

[0103] It is understood that the Figure 2 and Figure 4 The relevant content mainly introduces the network architecture of regional portrait enhancement combined with lighting scene classification. In addition, the structure of the network architecture combined with character skin color scene classification or color temperature scene classification is similar to Figure 2 and Figure 4 Similarly, no detailed description is given here. For example, Figure 2 and Figure 4 The training enhanced image of the image enhancement part shown may be an image that is manually enhanced by the user with particular attention paid to the skin color of the person or the color temperature of the image.

[0104] Next, based on Figure 2 or Figure 4 The network architecture shown is used to describe in detail the image processing method provided by the application embodiment.

[0105] like Figure 5 FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application. The execution subject of the method can be an electronic device such as a mobile phone. In addition, the method can be applied to Figure 2 The image enhancement network shown is implemented. Specifically, Figure 5 The method shown includes the following steps:

[0106] S501: Acquire a portrait image.

[0107] For example, the first image may be a portrait image, such as Figure 2 Portrait image A1 is shown.

[0108] In some embodiments, the portrait image may be a frame of image captured during the shooting process of a mobile phone. At this time, the image is usually captured in a raw (RAW) format, and the image may be converted from the RAW format to an RGB format to obtain the portrait image.

[0109] S502: Obtain a facial region image of a portrait image and corresponding N-1 semantic segmentation mask images.

[0110] In some embodiments, the present application can crop a facial region image from a portrait image, and generate T-1 semantic segmentation mask images corresponding to the portrait image through a semantic segmentation network. For example, the T-1 semantic segmentation mask images can include a human body mask image, a face mask image, and a mouth mask image, such as Figure 2 The segmentation mask images A11-A13 are shown, and T is 3 at this time.

[0111] S503: Determine illumination scene classification information corresponding to the facial region image and various illumination conditions. For example, the illumination scene classification information may be a group of image latent codes corresponding to various illumination conditions.

[0112] In some embodiments, the present application may adopt Figure 2 and Figure 4 The encoder 1 shown identifies the illumination scene classification information of the facial region image, wherein the illumination scene classification information may be a set of image latent codes identified by the encoder 1 and not passed through the FC layer.

[0113] S504: Determine the depth image features of the portrait image and the number T of semantic regions according to the portrait image and T-1 semantic segmentation mask images.

[0114] In some embodiments, the T semantic regions corresponding to the portrait image, that is, the T-1 semantic regions corresponding to the T-1 semantic segmentation mask images and the background region in the portrait image. For example, the T-1 semantic regions are the human body region, the face region, and the mouth region. As an example, the human body region may not include the face region and the mouth region, and the face region may be a region that does not include the mouth region. At this time, T is 4, that is, 4 semantic regions corresponding to the face image.

[0115] It can be understood that deep image features can be divided into two categories: primary features (shallow features) and advanced features (structural features). For example, primary features include shape features, color features, and texture features. Advanced features include: semantic features extracted and learned from underlying features, which are very abstract, such as human face analysis.

[0116] In some embodiments, the present application may adopt Figure 2 or Figure 4The encoder 2 shown determines deep image features of a portrait image.

[0117] S505: Determine T first 3D LUT parameters corresponding to the T semantic regions respectively according to the illumination scene classification information and the depth image features of the facial region image.

[0118] In some embodiments, the T first 3D LUT parameters corresponding to each of the T semantic regions in the present application may be Figure 2 or Figure 4 The T fused 3D LUTs are shown. At this time, the T first 3D LUT parameters are adaptive 3D LUTs of T semantic regions.

[0119] S506: Perform image enhancement on the portrait images according to the T first 3D LUT parameters respectively to obtain T enhanced images corresponding to the T semantic regions.

[0120] For example, the size (ie, dimension) of the first 3D LUT parameter is (256*256*256).

[0121] In some embodiments, the present application can search for an RGB value having a mapping relationship with the RGB value of each pixel point of the portrait image from the first 3D LUT parameter to achieve color adjustment of the portrait image, i.e., image enhancement.

[0122] S507: Based on T-1 semantic segmentation mask images, fuse the images corresponding to the T semantic regions in the T enhanced images to obtain an enhanced portrait image. Figure 2 or Figure 4 The enhanced portrait image shown may be A2.

[0123] In some embodiments, the present application can convert the enhanced portrait image in RGB format into BGR format to adapt to image display and other processing of electronic devices such as mobile phones.

[0124] In some embodiments, it is assumed that the enhanced images corresponding to the background area, the human body area, the face area, and the mouth area of ​​T enhanced images are recorded as pic0, pic1, pic2, and pic3 respectively. Then, the present application can use an enhanced image as a reference image, and replace the corresponding pixels in the enhanced images corresponding to other semantic areas with the pixels in the reference image. For example, the present application can use pic0 as a reference, replace the pixels in the human body area of ​​pic1 with the pixels in the human body area of ​​pic0 according to the human body mask image, replace the pixels in the face area of ​​pic2 with the pixels in the face area of ​​pic0 according to the face mask image, and replace the pixels in the mouth area of ​​pic3 with the pixels in the mouth area of ​​pic0 according to the mouth mask image, thereby achieving the fusion of T enhanced images.

[0125] In this way, the image processing method of the present application can combine the lighting scene classification information of the facial lighting performance, and realize different enhancement algorithms for different areas of the image based on different lighting scene classifications. That is, the facial area in the image can be targetedly enhanced according to the facial lighting performance, which is conducive to improving the image enhancement effect.

[0126] In some embodiments, the image processing method provided by the present application can determine T first 3D LUT parameters based on trilinear interpolation processing. As an example, Figure 6 As shown, the above S505 can be implemented through S601-S603:

[0127] S601: Determine T groups of weight values ​​corresponding to T semantic regions respectively according to illumination scene classification information and depth image features of the facial region image.

[0128] In some embodiments, the T groups of weight values ​​corresponding to each of the T semantic regions in the present application can be Figure 2 or Figure 4 2 shows T groups of weight values ​​predicted by the encoder 2. When N basic 3D LUTs are provided in the present application, each of the T groups of weight values ​​includes N weight values.

[0129] S602: Perform weighted calculation on N basic 3D LUTs using T groups of weight values ​​to obtain T second 3D LUT parameters.

[0130] For example, the T second 3D LUTs may be Figure 2 or Figure 4 The illustrated 3D LUTs after weighted calculations are T. Furthermore, the present application may use formula (2) to implement weighted calculations on N basic 3D LUTs using T groups of weight values.

[0131] S603: Perform trilinear interpolation on the T second 3D LUT parameters respectively to obtain T first 3D LUT parameters.

[0132] It can be understood that the trilinear interpolation of the T second 3D LUT parameters in the present application can refer to the relevant description of formula (3-9) above, which will not be repeated here.

[0133] For example, the size of each basic 3D LUT may be (33*33*33), and the size of each first 3D LUT parameter may be (256*256*256).

[0134] In this way, the present application can predict N weight values ​​corresponding to different semantic areas in the portrait image, and calculate N basic 3D LUTs according to the N weight values ​​of different semantic areas, so as to obtain adaptive 3D LUTs corresponding to different semantic areas. Then, the adaptive 3D LUTs of different semantic areas are conducive to making the partition enhancement of the portrait image meet the actual needs of users.

[0135] In some embodiments, the present application provides an image processing method that can combine non-uniform spacing processing and trilinear interpolation processing to determine T first 3D LUT parameters. As an example, Figure 7 As shown, the above S505 can be implemented through S701-S703:

[0136] S701: According to the illumination scene classification information and depth image features of the facial region image, determine T groups of weight values ​​corresponding to each of the T semantic regions, and T groups of sampling intervals corresponding to each of the T semantic regions. The similarities between S701 and S601 are not described here, and the difference is that the T group sampling intervals are predicted. For example, the T group sampling intervals are T*32*3, that is, T*96 intervals.

[0137] S702: Perform weighted calculation on N basic 3D LUTs using T groups of weight values ​​to obtain T second 3D LUT parameters.

[0138] Among them, S702 is the same as S602 and will not be described in detail here.

[0139] S703: Using T groups of sampling intervals to perform non-uniform sampling on T second 3D LUT parameters respectively, to obtain T third 3D LUT parameters.

[0140] Among them, the present application adopts a set of sampling intervals corresponding to a semantic area, performs non-uniform interval sampling on the second 3D LUT parameters corresponding to the semantic area, and obtains the third 3D LUT parameters corresponding to the semantic area.

[0141] S704: Perform trilinear interpolation on the T third 3D LUT parameters respectively to obtain T first 3D LUT parameters.

[0142] The present application performs trilinear interpolation on a third 3D LUT parameter corresponding to a semantic region to obtain a first 3D LUT parameter corresponding to the semantic region.

[0143] In this way, the present application can combine non-uniform interval sampling and trilinear interpolation to determine different 3D LUT parameters corresponding to different semantic areas, thereby improving the adaptability of semantic areas and 3D LUT parameters, which is beneficial to improving the effect of the final portrait partition enhancement.

[0144] In some embodiments, the image processing method provided in the embodiments of the present application can be used Figure 2 or Figure 4 Image enhancement network implementation in . Figure 8 As shown in FIG. 1 , it is the training and application process of the image enhancement network, and the execution subject of the process is still a terminal such as a mobile phone.

[0145] S801: According to a plurality of training facial region images and illumination tables corresponding to the plurality of training facial region images, the encoder 1 in the image enhancement network is trained until the value of the loss function of the encoder 1 is within a preset value range 1 and the training is terminated.

[0146] The illumination table corresponding to the plurality of training facial region images may be information on illumination conditions of the training facial region images pre-marked by the user.

[0147] It can be understood that according to the scene classification recognition result (such as front light or back light, etc.) of the encoder 1 for a training face image, and the difference between the illumination conditions corresponding to the training face image in the illumination table, a value of the loss function of the encoder 1 can be determined. In addition, the specific value of the above-mentioned numerical range 1 can be set according to actual needs and is not specifically limited here.

[0148] S802: Based on multiple training portrait images, N-1 semantic segmentation mask images corresponding to each training portrait image, lighting scene classification information of multiple training facial images corresponding to the multiple training portrait images, and multiple training enhanced portrait images, train encoder 2 in the image enhancement network until the value of the loss function of encoder 2 is within a preset value range 2, and then terminate the training.

[0149] It can be understood that, according to the difference between the enhanced portrait image after the encoder 1 performs image enhancement on a portrait image and the corresponding training enhanced image, a value of the loss function of the encoder 2 can be determined. In addition, the specific value of the above-mentioned numerical range 2 can be set according to actual needs and is not specifically limited here.

[0150] S803: Input the portrait image to be processed, the corresponding T-1 segmentation mask images, and the facial region image into the image enhancement network, and output the enhanced portrait image corresponding to the portrait image to be processed through the image enhancement network.

[0151] It can be understood that the image enhancement network in S803 can be based on Figure 5-Figure 7 The image processing method in the embodiment can realize the portrait enhancement of the processed portrait image by scene and region.

[0152] Next, taking a mobile phone as an example of an electronic device that executes the image processing method of the present application, the structure of the electronic device is described.

[0153] like Fig. 9 As shown, the mobile phone 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, a button 101 and a display screen 102, etc.

[0154] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0155] The processor 110 may include one or more processing units, for example, a processing module or processing circuit including a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor DSP, a microprocessor (MCU), an artificial intelligence (AI) processor, or a field programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. A storage unit may be provided in the processor 110 for storing instructions and data. In some embodiments, the storage unit in the processor 110 is a cache memory 180. For example, the processor 110 may be configured to store instructions and data. Figure 2 or Figure 4 The image enhancement network shown performs the Figure 5-Figure 8 Image processing methods.

[0156] The power module 140 may include a power source, a power management component, etc. The power source may be a battery. The power management component is used to manage the charging of the power source and the power supply of the power source to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module is used to receive charging input from the charger; the power management module is used to connect the power source, the charging management module and the processor 110. The power management module receives input from the power source and / or the charging management module, and supplies power to the processor 110, the display screen 102, the camera 170, and the wireless communication module 120.

[0157] The mobile communication module 130 may include, but is not limited to, an antenna, a power amplifier, a filter, a low noise amplifier (LNA), etc. The mobile communication module 130 may provide a solution for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the mobile phone 10. The mobile communication module 130 may receive electromagnetic waves by an antenna, and perform filtering, amplification, and other processing on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 130 may also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be arranged in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be arranged in the same device as at least some of the modules of the processor 110.

[0158] The wireless communication module 120 may include an antenna, and transmit and receive electromagnetic waves via the antenna. The wireless communication module 120 may provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc., which are applied to the mobile phone 10. The mobile phone 10 can communicate with the network and other devices through wireless communication technology.

[0159] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the mobile phone 10 may also be located in the same module.

[0160] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc.

[0161] The sensor module 190 may include a proximity light sensor, a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0162] The audio module 150 is used to convert digital audio information into analog audio signal output, or convert analog audio input into digital audio signal. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be arranged in the processor 110, or some functional modules of the audio module 150 can be arranged in the processor 110. In some embodiments, the audio module 150 can include a speaker, an earpiece, a microphone, and an earphone interface.

[0163] The camera 170 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the image signal processing (ISP) to convert it into a digital image signal. The mobile phone 10 can implement the shooting function through the ISP, camera 170, video codec, graphic processing unit (GPU), display screen 102 and application processor. For example, in the present application, the mobile phone can collect images through the ISP, camera 170, etc., and perform the present application on these images in the post-processing stage. Figure 5-Figure 8 The relevant image processing flow.

[0164] The interface module 160 includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the mobile phone 10. The external memory card communicates with the processor 110 through the external memory interface to implement a data storage function. The universal serial bus interface is used for the mobile phone 10 to communicate with other electronic devices. The subscriber identification module card interface is used to communicate with a SIM card installed in the mobile phone 1010, for example, to read a phone number stored in the SIM card, or to write a phone number into the SIM card.

[0165] In some embodiments, the mobile phone 10 further includes a button 101, a motor, and an indicator. The button 101 may include a volume button, a power on / off button, etc. The motor is used to make the mobile phone 10 vibrate, for example, vibrate when the user's mobile phone 10 is called to prompt the user to answer the call. The indicator may include a laser indicator, a radio frequency indicator, an LED indicator, etc.

[0166] In some embodiments, the present application provides a readable medium having instructions stored thereon, which, when executed on an electronic device, causes the electronic device to perform the above-mentioned image processing method.

[0167] In some embodiments, the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing the image processing method described above.

[0168] In some embodiments, the present application provides a computer program product, including: a computer program / instruction, which implements the above image processing method when executed by a processor.

[0169] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0170] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0171] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0172] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0173] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0174] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0175] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0176] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a first image to be processed; Determining a first semantic area and a second semantic area in the first image, and determining first scene classification information corresponding to the first image; respectively adjusting the colors of the first semantic area and the second semantic area using a first color parameter and a second color parameter corresponding to the first scene classification information of the first image; A second image is obtained based on the adjusted first semantic area and the adjusted second semantic area.

2. The method according to claim 1, characterized in that The first scene classification information is related to at least one of the following: The lighting conditions under which the first image was captured, the skin color of the person in the first image, and the color temperature of the first image.

3. The method according to claim 2, characterized in that The first semantic region is a foreground region in the first image and the second semantic region is a background region in the first image; or, The first semantic regions are different foreground regions in the first image.

4. The method according to claim 3, characterized in that The determining the first semantic area and the second semantic area in the first image includes: Acquire a first segmentation mask image of the first image, where the first segmentation mask image is a segmentation mask image of a foreground in the first image; determining the first semantic region according to the first segmentation mask image, wherein the first semantic region is a foreground region in the first image; The second semantic region in the first image is determined according to the first semantic region, wherein the second semantic region is a background region in the first image.

5. The method according to claim 1, characterized in that The first image is a portrait image, and The first semantic region includes at least one of a human body region, a face region, and a mouth region.

6. The method according to claim 5, characterized in that The determining the first scene classification information corresponding to the first image includes: The first scene classification information is determined according to image features of a first facial area in the first image.

7. The method according to claim 6, characterized in that The method further comprises: Acquire a first depth image feature corresponding to the first image; The first color parameter and the second color parameter are determined according to the first depth image feature and the first scene classification information, wherein the first color parameter and the second color parameter are different 3D LUT parameters.

8. The method according to claim 7, characterized in that The adjusting the colors of the first semantic area and the second semantic area respectively by using the first color parameter and the second color parameter corresponding to the first scene classification information of the first image includes: Using the first color parameter and the second color parameter to adjust the color of the first image respectively, to obtain a third image including the adjusted first semantic area and a fourth image including the adjusted second semantic area; And obtaining the second image based on the adjusted first semantic area and second semantic area includes: The adjusted first semantic region in the third image and the adjusted second semantic region in the fourth image are fused to obtain the second image.

9. The method according to claim 8, characterized in that The number of the first semantic areas is M, one first semantic area corresponds to one first color parameter, and, The method of adjusting the color of the first image by using the first color parameter and the second color parameter to obtain a third image including the adjusted first semantic area and a fourth image including the adjusted second semantic area includes: Using the first color parameters corresponding to the M first semantic areas, respectively adjusting the color of the first image, generating the corresponding third image for each of the first semantic areas, and obtaining M third images; adjusting the color of the first image using the second color parameter to obtain the fourth image including the second semantic area; and fusing the adjusted first semantic region in the third image and the adjusted second semantic region in the fourth image to obtain the second image, comprising: The adjusted first semantic region in each of the third images and the adjusted second semantic region in the fourth image are fused to obtain the second image.

10. The method according to claim 8 or 9, characterized in that: The determining the first color parameter and the second color parameter according to the first depth image feature and the first scene classification information includes: Acquire, according to the first depth image feature and the first scene classification information, N first weight values ​​corresponding to the first semantic area and N second weight values ​​corresponding to the second semantic area, wherein the N first weight values ​​are related to the color of the first semantic area, and the N second weight values ​​are related to the color of the second semantic area; Using the N first weight values ​​and the N second weight values ​​to perform weighted calculation on the preset N first 3D LUT parameters respectively, to obtain a third 3D LUT parameter and a fourth 3D LUT parameter; The third 3D LUT parameter and the fourth 3D LUT parameter are respectively subjected to trilinear interpolation to obtain the first color parameter and the second color parameter.

11. The method according to claim 8 or 9, characterized in that: The determining the first color parameter and the second color parameter according to the first depth image feature and the first scene classification information includes: According to the first depth image feature and the first scene classification information, obtain N first weight values ​​corresponding to the first semantic area and N second weight values ​​corresponding to the second semantic area, where N is a positive integer; According to the first depth image feature and the first scene classification information, obtain S first sampling intervals corresponding to the first semantic area and S second sampling intervals corresponding to the second semantic area, where S is a positive integer; Using the N first weight values ​​and the N second weight values ​​to perform weighted calculation on the preset N first 3D LUT parameters respectively, to obtain a third 3D LUT parameter and a fourth 3D LUT parameter; Sampling the third 3D LUT parameter using the S first sampling intervals to obtain a fifth 3D LUT parameter; Sampling the fourth 3D LUT parameter using the S second sampling intervals to obtain a sixth 3D LUT parameter; Trilinear interpolation is performed on the fifth 3D LUT parameter and the sixth 3D LUT parameter respectively to obtain the first color parameter and the second color parameter.

12. An image processing model, characterized in that: include: A scene classification module, configured to generate first scene classification information corresponding to a first image to be processed based on an image of a first facial region of the first image; A parameter generation module, used to determine a first semantic area and a second semantic area in the first image; An image enhancement module is used to adjust the colors of the first semantic area and the second semantic area respectively according to the first color parameter and the second color parameter corresponding to the first scene classification information of the first image, and obtain a second image based on the adjusted first semantic area and the adjusted second semantic area.

13. The model according to claim 12, characterized in that The parameter generation module comprises a first encoder, the first encoder is used to input the first image and a first segmentation mask image of the first image, the first segmentation mask image is a segmentation mask image of a foreground in the first image, determining the first semantic region according to the first segmentation mask image, wherein the first semantic region is a foreground region in the first image, The second semantic region in the first image is determined according to the first semantic region, wherein the second semantic region is a background region in the first image.

14. The model according to claim 13, characterized in that The scene classification module includes a second encoder, The second encoder is specifically configured to input an image of a first facial region in the first image, and determine the first scene classification information according to image features of the first facial region in the first image.

15. The model according to claim 14, characterized in that The first color parameter and the second color parameter are determined according to a first depth image feature of the first image and the first scene classification information, wherein the first color parameter and the second color parameter are different 3D LUT parameters.

16. The model according to claim 15, characterized in that The parameter generation module is used to obtain N first weight values ​​corresponding to the first semantic area and N second weight values ​​corresponding to the second semantic area according to the first depth image feature and the first scene classification information, wherein the N first weight values ​​are related to the color of the first semantic area, and the N second weight values ​​are related to the color of the second semantic area; The image enhancement module is used to perform weighted calculation on the preset N first 3D LUT parameters by using the N first weight values ​​and the N second weight values ​​to obtain a third 3D LUT parameter and a fourth 3D LUT parameter, The third 3D LUT parameter and the fourth 3D LUT parameter are respectively subjected to trilinear interpolation to obtain the first color parameter and the second color parameter.

17. The model according to claim 15, characterized in that The parameter generation module is used to obtain N first weight values ​​corresponding to the first semantic area and N second weight values ​​corresponding to the second semantic area according to the first depth image feature and the first scene classification information, wherein the N first weight values ​​are related to the color of the first semantic area, and the N second weight values ​​are related to the color of the second semantic area, According to the first depth image feature and the first scene classification information, obtain S first sampling intervals corresponding to the first semantic area and S second sampling intervals corresponding to the second semantic area, where S is a positive integer; The image enhancement module is used to perform weighted calculation on the preset N first 3D LUT parameters by using the N first weight values ​​and the N second weight values ​​to obtain a third 3D LUT parameter and a fourth 3D LUT parameter, The third 3D LUT parameter is sampled using the S first sampling intervals to obtain a fifth 3D LUT parameter, The fourth 3D LUT parameter is sampled using the S second sampling intervals to obtain a sixth 3D LUT parameter, Trilinear interpolation is performed on the fifth 3D LUT parameter and the sixth 3D LUT parameter respectively to obtain the first color parameter and the second color parameter.

18. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Inputting a first image to be processed into the image processing model according to any one of claims 12 to 17; A second image output by the image processing model is obtained.

19. A method for training an image processing model, applied to an electronic device, the method comprising: Inputting training sample data into the neural network model to be trained, the training sample data including a sample image, a scene type to which the sample image belongs, and a standard enhanced image corresponding to the sample image, the standard enhanced image being an image obtained by performing image processing on multiple semantic areas of the sample image using different color parameters based on the scene type; Obtaining a first enhanced image and a scene classification result output by the neural network model; Based on the first enhanced image, the standard enhanced image, the scene classification result and the scene type, the parameters of the neural network model are adjusted to obtain the image processing model when the first enhanced image and the scene classification result output by the neural network model meet a first condition.

20. The method according to claim 19, characterized in that The scene classification result is related to at least one of the following: The lighting conditions under which the sample image was captured, the skin color of the person in the sample image, and the color temperature of the sample image.

21. The method according to claim 19 or 20, characterized in that The adjusting the parameters of the neural network model based on the first enhanced image, the standard enhanced image, the scene classification result and the scene type includes: Determining a value of a first loss function according to the scene classification result and the scene type; Adjusting a first parameter of the neural network model according to a value of the first loss function; Determining a value of a second loss function according to the first enhanced image and the standard enhanced image; Adjust a second parameter of the neural network model according to the value of the second loss function.

22. The method according to claim 21, characterized in that The first condition includes: The value of the first loss function is in a first value range, and the value of the second loss function is in a second value range.

23. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method according to any one of claims 1 to 11.

24. An electronic device, characterized in that: include: A memory for storing instructions executed by one or more processors of an electronic device, and a processor, which is one of the processors of the electronic device, for executing the image processing method as described in any one of claims 1 to 11.

Images