Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method. A light source MASK image corresponding to a to-be-processed image, a saturation image corresponding to the to-be-processed image and a depth image corresponding to the to-be-processed image are determined; wherein the to-be-processed image is a frame of image in the to-be-processed video; determining a saturation MASK image corresponding to the to-be-processed image based on the light source MASK image and the saturation image; based on the saturation MASK image and the depth image, determining a plurality of MASK images of a plurality of types corresponding to the to-be-processed image; and determining a processed image corresponding to the to-be-processed image based on the plurality of MASK images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] As mobile terminals gradually become the main tool for taking photos and recording videos in people's daily lives, the demand for image quality is also increasing. Tone and color are important factors in image perception. In order to obtain an image with a good atmosphere, mobile terminals usually need to adjust the tone of the image.

[0003] In related technologies, by adjusting image parameters such as brightness, contrast, and color balance, images can be made more artistic and aesthetic. However, the common problem of global image optimization solutions is that they cannot meet the needs of local image details, especially for high-contrast scenes, and often fail to achieve ideal image processing effects. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device and storage medium, which can improve the accurate recognition of dynamic photos, expand the application scenarios of dynamic photos, and enhance the intelligence of electronic devices.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Determine a light source MASK image corresponding to the image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame image in the video to be processed;

[0008] Based on the light source MASK image and the saturation image, determine the saturation MASK image corresponding to the image to be processed;

[0009] Based on the saturation MASK image and the depth image, determining that the image to be processed corresponds to multiple MASK images of multiple types;

[0010] Based on multiple MASK images, a processed image corresponding to the image to be processed is determined.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0012] A determination unit is used to determine a light source MASK image corresponding to an image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame of an image in a video to be processed; based on the light source MASK image and the saturation image, a saturation MASK image corresponding to the image to be processed is determined; based on the saturation MASK image and the depth image, it is determined that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, a processed image corresponding to the image to be processed is determined.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method of the first aspect is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method of the first aspect described above is implemented.

[0015] The embodiment of the present application provides an image processing method, an apparatus, a device and a storage medium, which determine the light source MASK image corresponding to the image to be processed, the saturation image corresponding to the image to be processed and the depth image corresponding to the image to be processed; wherein the image to be processed is a frame of the video to be processed; based on the light source MASK image and the saturation image, the saturation MASK image corresponding to the image to be processed is determined; based on the saturation MASK image and the depth image, it is determined that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, the processed image corresponding to the image to be processed is determined. That is to say, in the embodiment of the present application, during the image processing process, the light source MASK image representing the brightness information of the image to be processed, the saturation image representing the saturation information of the image to be processed and the depth image representing the depth information of the image can be determined respectively, and then the image to be processed is divided into different MASK images of different types by combining the light source MASK image, the saturation image and the depth image of the image to be processed, and finally the different MASK images are adjusted and processed respectively to obtain a processed image with a more ideal processing effect. It can be seen that in the embodiments of the present application, since the brightness information, saturation information and depth information of the image are introduced in the process of image processing, it is possible to more comprehensively consider the various parameters of the image and divide the image to be processed into more accurate and rich MASK images for subsequent processing to meet the detailed requirements of image processing, thereby effectively improving the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the implementation flow of the image processing method proposed in the embodiment of the present application;

[0017] Figure 2 A schematic diagram of the implementation flow of the image processing method proposed in the embodiment of the present application;

[0018] Figure 3 A schematic diagram of the implementation flow of the image processing method proposed in the embodiment of the present application;

[0019] Figure 4 A schematic diagram of the implementation flow of the image processing method proposed in the embodiment of the present application;

[0020] Figure 5 A schematic diagram of the implementation flow of the image processing method proposed in the embodiment of the present application;

[0021] Figure 6 A schematic diagram of the implementation framework of the image processing method proposed in the embodiment of the present application;

[0022] Figure 7 A schematic diagram of a background MASK image proposed in an embodiment of the present application;

[0023] Figure 8 A schematic diagram of a global tone mapping curve proposed in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram of the structure of the image processing device proposed in the embodiment of the present application;

[0025] Fig.10 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the difference application, rather than to limit the application. It should also be noted that, for the convenience of description, only the parts that are different from the related applications are shown in the drawings.

[0027] Tone, or the relationship between light and dark in an image, refers to the changes in different gray levels in a photo in photography. It is the relationship between light and shadow, light and dark tones in a photograph, and the layout of black, white and gray in a photo. For example, in a photograph, the different changes and combinations of light and dark distribution, light and dark contrast, and light and dark contrast form different tones.

[0028] Among them, according to the different relationships between light and dark, tones can be divided into light tones, dark tones and mid-tones.

[0029] As mobile terminals gradually become the main tool for taking photos and recording videos in people's daily lives, the demand for image quality is also increasing. Tone and color are important factors in image perception. In order to obtain an image with a good atmosphere, mobile terminals usually need to adjust the tone of the image.

[0030] In related technologies, by adjusting image parameters such as brightness, contrast, and color balance, the image can be made more artistic and aesthetically pleasing. For example, the brightness and contrast of the image can be optimized globally using histogram equalization (HE) or adaptive histogram equalization (AHE). However, the common problem of global image optimization solutions is that they cannot meet the needs of local details of the image, especially for high-contrast scenes, and often cannot obtain a more ideal image processing effect.

[0031] In other words, common image processing solutions reduce the image processing effect because they cannot meet the detailed requirements of image processing.

[0032] In order to solve the above problems, the embodiments of the present application provide an image processing method, an apparatus, a device and a storage medium, which determine the light source MASK image corresponding to the image to be processed, the saturation image corresponding to the image to be processed and the depth image corresponding to the image to be processed; wherein the image to be processed is a frame of the video to be processed; based on the light source MASK image and the saturation image, the saturation MASK image corresponding to the image to be processed is determined; based on the saturation MASK image and the depth image, it is determined that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, the processed image corresponding to the image to be processed is determined. That is to say, in the embodiments of the present application, during the image processing process, the light source MASK image representing the brightness information of the image to be processed, the saturation image representing the saturation information of the image to be processed and the depth image representing the depth information of the image can be determined respectively, and then the light source MASK image, the saturation image and the depth image of the image to be processed are combined to divide the image to be processed into different MASK images of different types, and finally the different MASK images are adjusted and processed respectively to obtain a processed image with a more ideal processing effect. It can be seen that in the embodiments of the present application, since the brightness information, saturation information and depth information of the image are introduced in the process of image processing, it is possible to more comprehensively consider the various parameters of the image and divide the image to be processed into more accurate and rich MASK images for subsequent processing to meet the detailed requirements of image processing, thereby effectively improving the image processing effect.

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0034] An embodiment of the present application provides an image processing method, which can be applied to an image processing device or an electronic device, and can also be applied to any terminal including the image processing device or the electronic device.

[0035] It can be understood that the image processing method proposed in the embodiment of the present application mainly includes a method for adjusting the tone of an image.

[0036] Below, taking an image processing device as an example, the image processing method proposed in the embodiment of the present application is exemplarily described.

[0037] Furthermore, in the embodiments of the present application, Figure 1 This is a schematic diagram of the image processing method implementation process proposed in the embodiment of the present application, such as Figure 1 As shown, the image processing method may include the following steps:

[0038] Step 101, determine a light source MASK image corresponding to an image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame image in a video to be processed.

[0039] In an embodiment of the present application, for an image to be processed, a light source MASK image corresponding to the image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed can be determined respectively. The light source MASK image can represent the brightness information of the image to be processed, the saturation image can represent the saturation information of the image to be processed, and the depth image can represent the depth information of the image to be processed.

[0040] In an embodiment of the present application, the image to be processed may be a frame of image in a video to be processed, that is, the image to be processed may be a video image frame.

[0041] Furthermore, in an embodiment of the present application, when determining a depth image corresponding to an image to be processed, the pixel positions of the first image and the pixel positions of the image to be processed can be determined first; then, based on the pixel positions of the image to be processed and the pixel positions of the first image, an initial depth image corresponding to the image to be processed can be determined; finally, the depth image can be determined based on the initial depth image and the scale of the first image.

[0042] It should be noted that, in the embodiment of the present application, the first image may be a frame of image before the image to be processed in the video to be processed.

[0043] That is to say, in an embodiment of the present application, the image to be processed and the first image may be different image frames in the same video to be processed. The timestamp corresponding to the first image is earlier than the timestamp corresponding to the image to be processed. For example, the first image may be the previous frame of the image to be processed in the video to be processed; or, the first image may be the second frame of the image before the image to be processed in the video to be processed. This application does not make any specific limitations.

[0044] It is understandable that in the embodiment of the present application, for the first image and the image to be processed in the video to be processed, the pixel positions can be determined separately, that is, the pixel position of the first image is determined, and the pixel position of the image to be processed is determined at the same time. The pixel position of the first image includes the position coordinates of each pixel point in the first image, and the pixel position of the image to be processed includes the position coordinates of each pixel point in the image to be processed.

[0045] In an embodiment of the present application, after respectively determining the pixel positions of the first image and the pixel positions of the image to be processed, the initial depth image corresponding to the image to be processed can be further determined based on the pixel positions of the image to be processed and the pixel positions of the first image. Among them, the offset of the pixel points in the image to be processed relative to the pixel points in the first image can be first determined based on the pixel positions of the image to be processed and the pixel positions of the first image, that is, the motion vector MV of the pixel points in the image to be processed is determined, and then the initial depth image corresponding to the image to be processed is further determined based on the motion vector between the pixel points in the two images.

[0046] Exemplarily, in some embodiments, a dense inverse search (DIS) algorithm may be used to determine the initial depth image corresponding to the image to be processed. The DIS optical flow algorithm is an optical flow estimation algorithm based on dense sampling, which estimates the optical flow by calculating the motion vector (motion vector) of each pixel in the image. For each pixel (pixel point), the algorithm uses a fixed-size window around it for sampling to obtain a set of sampling points around the pixel. Then, the motion vector of the current pixel is determined by calculating the similarity between the set of sampling points and the corresponding points in the target image.

[0047] Specifically, the DIS optical flow algorithm uses reverse search and gradient descent to seek optimization to solve the motion estimation problem. It is based on block-level correlation reverse search, which realizes the calculation of gradients once and uses them for multiple reverse searches, without reinitializing the calculation of gradients each time, thus saving a lot of calculations and improving performance. Among them, due to the use of optimization methods such as reverse search and gradient descent, the DIS optical flow algorithm has high real-time performance. At the same time, the DIS optical flow algorithm can accurately estimate the motion vector of the pixel point by accurately calculating the similarity between the sampling point and the corresponding point in the target image.

[0048] Exemplarily, in some embodiments, when the DIS optical flow algorithm is used to determine the initial depth image corresponding to the image to be processed, the image to be processed and the first image that need to perform optical flow estimation can be converted into grayscale images, and then the DIS optical flow algorithm function (such as cv::calcOpticalFlowDenseDIS in OpenCV) is used to perform optical flow estimation, traverse each pixel, obtain its corresponding motion vector, and draw corresponding arrows or other visual representations to finally obtain the corresponding initial depth image.

[0049] Furthermore, in an embodiment of the present application, after the initial depth image corresponding to the image to be processed is determined, the depth image corresponding to the image to be processed may be finally determined based on the initial depth image and the first image scale.

[0050] It is understandable that, in the embodiment of the present application, the first image scale can be used to divide and determine the image blocks of the image to be processed. The first image scale can be a scale configured according to actual needs, and the present application does not specifically limit the value of the first image scale. For example, the first image scale can be 32x32, or the first image scale can be 16x16.

[0051] Exemplarily, in some embodiments, assuming that the determined initial depth map can be represented as Depth0, the initial depth map Depth0 can be averaged according to the first image scale to obtain a depth image block Depth1 that meets the first image scale. For example, if the first image scale is 32x32, then the size of the obtained depth image block Depth1 is 32x32. The first image scale can also be reconfigured according to actual needs, and this application does not make specific restrictions. Finally, the final depth image Depth can be obtained according to the inversion formula shown below out :

[0052] Depth out =1-Depth1÷MAX(Depth1) (1)

[0053] Among them, MAX(Depth1) can be understood as the maximum depth in the depth image block Depth1.

[0054] It can be understood that in the embodiment of the present application, the depth image corresponding to the finally determined image to be processed may include depth information of at least one image block satisfying the first image scale, that is, includes depth information of image blocks in units of the first image scale.

[0055] Further, in the embodiments of the present application, a MASK image, also called a mask image, is a binary image in which pixels are marked as black or white (or other values ​​may be used in digital image processing, but are usually simplified to binary to represent a specific area). This type of image has a wide range of applications in the fields of computer vision and image processing, such as object segmentation, image segmentation, image fusion, background removal, etc. In image segmentation tasks, a MASK image is used to mark the area of ​​interest, thereby dividing the image into different parts for easier subsequent processing.

[0056] Accordingly, in the embodiments of the present application, the light source MASK image corresponding to the image to be processed can be understood as a MASK image that can reflect the ambient lighting conditions corresponding to the image to be processed.

[0057] Furthermore, in an embodiment of the present application, when determining the light source MASK image corresponding to the image to be processed, it is possible to choose to determine the brightness image block of the image to be processed based on the initial brightness image and the first image scale corresponding to the image to be processed; and then determine the light source MASK image based on the brightness image block of the image to be processed and the light source MASK curve.

[0058] For example, in some embodiments, the brightness information corresponding to the image to be processed may be determined first, thereby obtaining the corresponding initial brightness image; then, based on the initial brightness image, the image blocks may be divided and the average brightness of the image blocks may be determined in combination with the first image size, thereby obtaining brightness image blocks that meet the first image scale. For example, if the first image scale is 32x32, the size of the brightness image block obtained is 32x32. Finally, the light source MASK image may be further determined based on the brightness image block of the image to be processed and the light source MASK curve.

[0059] It can be understood that, in the embodiments of the present application, the light source MASK curve may be pre-configured or pre-acquired, and the present application does not make any specific limitation thereto.

[0060] In an embodiment of the present application, after determining the brightness image block of the image to be processed, the light source MASK curve can be used to perform mapping based on the brightness image blocks respectively, and finally the corresponding light source MASK image can be determined.

[0061] It can be understood that in the embodiment of the present application, the light source MASK image corresponding to the finally determined image to be processed may include MASK information of at least one image block satisfying the first image scale, that is, including MASK image blocks in units of the first image scale.

[0062] That is to say, in the embodiment of the present application, the depth image corresponding to the image to be processed and the light source MASK image corresponding to the image to be processed both satisfy the first image scale, that is, the scales of the two remain consistent.

[0063] Furthermore, in an embodiment of the present application, when determining a saturation image corresponding to an image to be processed, it is possible to first determine an initial saturation image corresponding to the image to be processed; and then determine the saturation image based on the initial saturation image and the first image scale.

[0064] Among them, in the embodiments of the present application, the saturation of the image is an important attribute of the color, which describes the purity or vividness of the color. Among them, saturation can be defined as chroma divided by lightness, which characterizes the degree to which the color deviates from the gray of the same brightness. Highly saturated colors are usually very bright and impactful, while low-saturated colors tend to be gray or brown, and look more stable and dull. It can also be simply understood that saturation refers to the vividness of the color, also known as the purity of the color.

[0065] For example, in some embodiments, the saturation information corresponding to the image to be processed may be determined first, thereby obtaining the corresponding initial saturation image; then, based on the initial saturation image, the image blocks may be divided and the average saturation of the image blocks may be determined in combination with the first image size, thereby obtaining saturation image blocks that meet the first image scale, and finally obtaining the corresponding saturation image. For example, if the first image scale is 32x32, then the size of the obtained saturation image block is 32x32.

[0066] It can be understood that in an embodiment of the present application, the saturation image corresponding to the finally determined image to be processed may include saturation information of at least one image block that satisfies the first image scale, that is, includes saturation image blocks in units of the first image scale.

[0067] That is to say, in the embodiment of the present application, the depth image corresponding to the image to be processed, the light source MASK image corresponding to the image to be processed, and the saturation image corresponding to the image to be processed all satisfy the first image scale, that is, the scales of the three remain consistent.

[0068] Step 102: Determine a saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image.

[0069] In an embodiment of the present application, after determining the light source MASK image corresponding to the image to be processed, the saturation image corresponding to the image to be processed, and the depth image corresponding to the image to be processed, the saturation MASK image corresponding to the image to be processed can be further determined based on the light source MASK image and the saturation image.

[0070] Furthermore, in an embodiment of the present application, after respectively determining the light source MASK image and saturation image corresponding to the image to be processed, fusion processing can be selected based on the light source MASK image and the saturation image, and finally the saturation MASK image corresponding to the image to be processed can be determined.

[0071] It is understandable that in the embodiments of the present application, the present application does not specifically limit the specific implementation of the fusion process, for example, including but not limited to using a mapping curve to fuse the light source MASK image and the saturation image.

[0072] For example, in some embodiments, the mapping curve may be a saturation MASK curve. For example, the light source MASK image and the saturation image may be fused based on the saturation MASK curve to determine the saturation MASK image.

[0073] In the embodiment of the present application, the saturation MASK curve can be used to implement different fusion strategies for image blocks with different saturations. For example, different fusion strategies can be selected for image blocks with higher saturation and image blocks with lower saturation.

[0074] It can be understood that, in the embodiments of the present application, the saturation MASK curve may be pre-configured or pre-acquired, and the present application does not make any specific limitation thereto.

[0075] For example, in some embodiments, it is assumed that the determined light source MASK image is MASK v , the determined saturation image is S global , MASK v and S global The scales of the two images are consistent and meet the scale of the first image. Then the MASK is fused according to the pre-configured saturation MASK curve. v and S global , get the corresponding saturation MASK image MASK s , where the fusion formula is as follows:

[0076] MASK s =MASK v * Lut(S global ) (2)

[0077] It can be understood that in an embodiment of the present application, the saturation MASK image corresponding to the finally determined image to be processed may include at least one saturation MASK image block that satisfies the first image scale, that is, includes a saturation MASK image block in units of the first image scale.

[0078] That is to say, in the embodiment of the present application, the depth image corresponding to the image to be processed and the saturation MASK image corresponding to the image to be processed both satisfy the first image scale, that is, the scales of the two remain consistent.

[0079] Furthermore, in an embodiment of the present application, the saturation MASK image corresponding to the image to be processed is determined in combination with the saturation image and the light source MASK image corresponding to the image to be processed. Therefore, the saturation MASK image can fully characterize the saturation information and brightness information of the image to be processed, that is, the saturation MASK image introduces the saturation information and brightness information of the image to be processed.

[0080] Step 103: Based on the saturation MASK image and the depth image, determine whether the image to be processed corresponds to multiple MASK images of multiple types.

[0081] In an embodiment of the present application, after determining the saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image, it is possible to further determine that the image to be processed corresponds to multiple MASK images of multiple types based on the saturation MASK image and the depth image.

[0082] It is understandable that in the embodiments of the present application, the determined saturation MASK image and depth image can be used to divide the image to be processed, so that it can be divided into multiple MASK images corresponding to multiple types. Among them, one type corresponds to one MASK image, and the number and definition of the types are not specifically limited in the present application.

[0083] Exemplarily, in some embodiments, the number of types may be 2, which may be defined as a foreground type and a background type, that is, multiple types may include a foreground type and a background type, and thus a foreground MASK image and a background MASK image corresponding to the image to be processed may be determined.

[0084] Exemplarily, in some embodiments, the number of types can be 3, which can be defined as building type, human body type and natural object type, that is, multiple types can include building type, human body type and natural object type, and then the building MASK image, human body MASK image and natural object MASK image corresponding to the image to be processed can be determined.

[0085] Furthermore, in an embodiment of the present application, when determining that the image to be processed corresponds to multiple MASK images of multiple types based on the saturation MASK image and the depth image, the saturation MASK image and the depth image can be fused respectively based on multiple scaling factors corresponding to the multiple types and multiple MASK thresholds corresponding to the multiple types, thereby determining multiple MASK images corresponding to the multiple types.

[0086] It can be understood that, in the embodiment of the present application, one type corresponds to one scaling factor and one MASK threshold.

[0087] Accordingly, in an embodiment of the present application, when the saturation MASK image and the depth image are fused respectively based on multiple scaling coefficients corresponding to multiple types and multiple MASK thresholds corresponding to multiple types, for any type, the scaling coefficient and MASK threshold corresponding to the type can be used, combined with the saturation MASK image and the depth image, to obtain a MASK image corresponding to the type; then each type is traversed, and finally multiple MASK images corresponding to multiple types can be determined.

[0088] For example, in some embodiments, it is assumed that the multiple types include a foreground type and a background type, wherein the scaling factor corresponding to the foreground type is α fg , the MASK threshold fgThd corresponding to the foreground type is 0.1, and the scaling factor corresponding to the background type is α bg , the MASK threshold bgThd corresponding to the background type is 0.4, then, for the foreground type, the saturation MASK image MASK can be fused according to the following formula s and depth image Depth out , get the MASK image Mask corresponding to the foreground type fg :

[0089] Mask fg = CLAMP(α fg *(fgThd-CLAMP(Depth out ,0.0,fgThd))*Mask s ,0.0,1.0) (3)

[0090] At the same time, for the background type, the saturation MASK image MASK can be fused according to the following formula s and depth image Depth out , get the MASK image Mask corresponding to the background type bg :

[0091] Mask bg = CLAMP(α bg *CLAMP(Depth out -bgThd,0.0,1.0)*Mask s ,0.0,1.0) (4)

[0092] That is to say, in an embodiment of the present application, a plurality of MASK images corresponding to a plurality of types may be determined based on a saturation MASK image and a depth image corresponding to the image to be processed.

[0093] Furthermore, in an embodiment of the present application, multiple MASK images corresponding to multiple types are determined in combination with the depth image and saturation MASK image corresponding to the image to be processed, wherein the saturation MASK image introduces the saturation information and brightness information of the image to be processed. Therefore, the multiple MASK images corresponding to multiple types can fully characterize the depth information, saturation information and brightness information of the image to be processed, that is, the multiple MASK images corresponding to multiple types introduce the depth information, saturation information and brightness information of the image to be processed.

[0094] Furthermore, in the embodiments of the present application, Figure 2 This is a schematic diagram of the image processing method implementation process proposed in the embodiment of the present application, such as Figure 2 As shown, after determining the saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image, that is, after step 102, the image processing method may further include the following steps:

[0095] Step 105: Based on the image to be processed and the saturation MASK image, determine whether the image to be processed corresponds to multiple MASK images of multiple types.

[0096] In an embodiment of the present application, after determining the saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image, it is also possible to further determine multiple MASK images corresponding to multiple types based on the image to be processed in combination with the saturation MASK image.

[0097] Further, in an embodiment of the present application, when determining that the image to be processed corresponds to multiple MASK images of multiple types based on the image to be processed and the saturation MASK image, multiple types can be first determined based on the image to be processed; and then, based on the multiple types and the saturation MASK image, multiple MASK images corresponding to the multiple types can be determined.

[0098] It is understandable that in the embodiments of the present application, the types of the images to be processed may be divided based on the image content and image information, so that the corresponding multiple types may be determined. The method of dividing the multiple types is not specifically limited in the present application.

[0099] For example, in some embodiments, multiple corresponding types can be determined by multi-semantic segmentation technology (such as U-Net network). For example, multi-semantic segmentation technology can be used to find different types of pixel areas in the image to be processed, thereby dividing and obtaining multiple types.

[0100] Accordingly, in an embodiment of the present application, when determining multiple MASK images corresponding to multiple types based on multiple types and saturation MASK images, the MASK images corresponding to each type can be extracted from the saturation MASK image according to the multiple types obtained by division, and finally multiple MASK images corresponding to multiple types are obtained.

[0101] That is to say, in an embodiment of the present application, a plurality of MASK images corresponding to a plurality of types may be first determined based on the image to be processed, and then determined according to the plurality of types from the saturation MASK image.

[0102] That is to say, in an embodiment of the present application, one implementation method of the image processing method is to determine that the image to be processed corresponds to multiple MASK images of multiple types based on the saturation MASK image and the depth image, and another implementation method is to determine that the image to be processed corresponds to multiple MASK images of multiple types based on the saturation MASK image and the image to be processed. Of course, it is also possible to determine that the image to be processed corresponds to multiple MASK images of multiple types in other ways, which is not specifically limited in the present application.

[0103] Step 104: Based on the multiple MASK images, determine a processed image corresponding to the image to be processed.

[0104] In an embodiment of the present application, after determining that the image to be processed corresponds to a plurality of MASK images of a plurality of types, a processed image corresponding to the image to be processed may be further determined based on the plurality of MASK images.

[0105] It can be understood that in an embodiment of the present application, for a plurality of MASK images corresponding to a plurality of types, different image processing methods can be used for image processing, for example, a plurality of global tone mapping curves corresponding to a plurality of types can be used to perform image processing on the corresponding MASK images, respectively, so as to obtain a processed image corresponding to the image to be processed.

[0106] It is understood that in the embodiments of the present application, multiple global tone mapping curves corresponding to multiple types can be used to improve the contrast and light and shadow levels of a specified brightness range of an image. The global tone mapping curve can be pre-configured or calculated based on a histogram statistics-related method such as AHE, which is not specifically limited in the present application.

[0107] It can be understood that, in the embodiment of the present application, one type corresponds to one global tone mapping curve, wherein different types may correspond to different global tone mapping curves.

[0108] Accordingly, in an embodiment of the present application, for any type, the global tone mapping curve corresponding to the type can be used to perform image processing on the MASK image corresponding to the type to determine the adjusted image corresponding to the type; then each type is traversed, and finally multiple adjusted images corresponding to multiple types can be determined, and then the processed image corresponding to the image to be processed can be obtained.

[0109] Further, in an embodiment of the present application, when determining a processed image corresponding to an image to be processed based on multiple MASK images, the first MASK image corresponding to the first type in the multiple MASK images and the initial brightness image corresponding to the image to be processed can be fused based on a first global tone mapping curve corresponding to a first type in the multiple types to determine a first adjusted image corresponding to the first MASK image; then, based on a kth global tone mapping curve corresponding to a kth type in the multiple MASK images, the kth MASK image corresponding to the kth type and the k-1th adjusted image are fused to determine a kth adjusted image corresponding to the kth MASK image; then, based on a k+1th global tone mapping curve corresponding to a k+1th type, the k+1th MASK image corresponding to the k+1th type in the multiple MASK images and the k adjusted image are fused to determine a k+1th adjusted image corresponding to the k+1th MASK image; finally, the processed image corresponding to the image to be processed can be determined based on the k+1th adjusted image.

[0110] It should be noted that, in the embodiments of the present application, k may be an integer greater than 1.

[0111] That is to say, in an embodiment of the present application, for any type, the global tone mapping curve corresponding to the current type can be used, and the MASK image corresponding to the current type can be processed in combination with the adjusted image corresponding to the previous type to determine the adjusted image corresponding to the current type; then continue to traverse the next type, and perform image processing on the MASK image corresponding to the next type in combination with the adjusted image corresponding to the current type. After traversing all types, multiple adjusted images corresponding to multiple types can be determined, and then the processed image corresponding to the image to be processed can be obtained.

[0112] It can be understood that in the embodiments of the present application, the traversal order of multiple types can be pre-set or randomly selected, and the present application does not make any specific limitation.

[0113] For example, in some embodiments, assuming that L out is the final processed image output, L inis the initial brightness image corresponding to the image to be processed, Lut is the global tone mapping curve corresponding to each type, then, for the first type of the multiple types, the corresponding first global tone mapping curve is Lut1, and the first MASK image Mask1 corresponding to the first type can be processed according to the following formula to obtain the corresponding first adjusted image L1:

[0114] L1=(1.0-Mask1)*Lin+Mask1*Lut1(Lin) (5)

[0115] Next, for the second type among the multiple types, the corresponding second global tone mapping curve is Lut2. The second MASK image Mask2 corresponding to the second type can be processed in combination with the first adjusted image L1 according to the following formula to obtain the corresponding second adjusted image L2:

[0116] L2=(1.0-Mask2)*L1+Mask2*Lut2(L1) (6)

[0117] Accordingly, for the kth type among the multiple types, the corresponding kth global tone mapping curve is Lut k , we can use the following formula, combined with the k-1th adjusted image L k-1 For the kth MASK image Mask corresponding to the kth type k Processing is performed to obtain the corresponding kth adjusted image L k :

[0118] Lk=(1.0-Maskk)*Lk -1 +Maskk*Lutk(Lk -1 ) (7)

[0119] Assume that the last type among multiple types is the k+1th type, and the corresponding k+1th global tone mapping curve is Lut k+1 , can be combined with the kth adjusted image L according to the following formula k For the k+1th type corresponding to the kth MASK image Mask k+1 Processing is performed to obtain the corresponding k+1th adjusted image L k+1 , that is, the processed image L corresponding to the final image to be processed out :

[0120] Lout=Lk +1 =(1.0-Maskk +1 )*Lk+Maskk +1 *Lutk +1 (Lk) (8)

[0121] Furthermore, in an embodiment of the present application, assuming that multiple types include a foreground type and a background type, when determining a processed image corresponding to an image to be processed based on multiple MASK images, the background MASK image and the initial brightness image corresponding to the image to be processed can be first fused based on a first global tone mapping curve corresponding to the background type to determine a first adjusted image corresponding to the background MASK image; and then, based on a second global tone mapping curve corresponding to the foreground type, the foreground MASK image and the first adjusted image can be fused to determine a processed image corresponding to the image to be processed.

[0122] It is understandable that in the embodiments of the present application, the traversal order of the foreground type and the background type may be pre-set or randomly selected, and the present application does not specifically limit it. For example, the background MASK image corresponding to the background type may be adjusted first, and then the foreground MASK image corresponding to the foreground type may be adjusted in combination with the first adjusted image corresponding to the background MASK image.

[0123] For example, in some embodiments, assuming that L out is the final processed image output, L in is the initial brightness image corresponding to the image to be processed, Lut is the global tone mapping curve corresponding to each type, then, for the background type, the corresponding first global tone mapping curve is Lut bg , the background MASK image Mask corresponding to the background type can be obtained according to the following formula bg Processing is performed to obtain the corresponding first adjusted image L bg :

[0124] L bg =(1.0-Mask bg )*L in +Mask bg *Lut bg (L in ) (9)

[0125] Correspondingly, for the foreground type, the corresponding second global tone mapping curve is Lut fg , can be combined with the first adjusted image L according to the following formula bg Foreground MASK image Mask corresponding to the foreground type fg Processing is performed to obtain the processed image L corresponding to the image to be processed out :

[0126] L out =(1.0-Mask fg )*L bg +Mask fg *Lutfg (L bg ) (10)

[0127] It can be understood that in the embodiments of the present application, considering that multiple MASK images corresponding to multiple types can fully characterize the depth information, saturation information and brightness information of the image to be processed, when adjusting different MASK images of different types, the corresponding global tone mapping curve can be selected based on the difference in one or more of the depth information, saturation information and brightness information, that is, based on the difference in one or more of the depth information, saturation information and brightness information, different global tone mapping curves are used to adopt different adjustment strategies for different MASK images.

[0128] For example, in some embodiments, for highlight areas, a global tone mapping curve that only enhances the transparency of highlight areas can be selected to adjust the MASK image. For example, based on the saturation MASK image, the highlight area of ​​the background MASK image and the highlight area of ​​the foreground MASK image in the image to be processed are extracted; finally, with the help of two global tone mapping curves corresponding to the background type and the foreground type, the MASK image is merged with the image to be processed, respectively enhancing the transparency of the highlight area, while avoiding the negative impact of global tone mapping in the dark area.

[0129] For example, in some embodiments, for a high saturation area, a global tone mapping curve that only adjusts the hue of the high saturation area can be selected to adjust the MASK image. For example, the saturation information of the reference image is used to allow the global tone mapping curve to act only on the high saturation area to avoid the phenomenon of dull colors in the low saturation area of ​​the image during the process of enhancing the light and shadow of the image.

[0130] Furthermore, in the embodiments of the present application, Figure 3 This is a schematic diagram of the image processing method implementation process proposed in the embodiment of the present application, such as Figure 3 As shown, the image processing method may further include the following steps:

[0131] Step 106: Scaling the image to be processed according to the second image scale.

[0132] In an embodiment of the present application, for an image to be processed in a video to be processed, the image to be processed may be first scaled, for example, the image to be processed may be scaled according to a second image scale.

[0133] It is understandable that in the embodiment of the present application, the second image scale can be used to adjust the size of the image to be processed. The second image scale can be a scale configured according to actual needs, and the present application does not specifically limit the value of the second image scale. For example, the second image scale can be 640x480, or the second image scale can be 480x320.

[0134] It can be understood that in the embodiments of the present application, assuming that the second image scale is 640x480, in order to reduce the amount of computation in the subsequent processing flow and improve the image processing efficiency, for the image to be processed whose image size is larger than the second image scale, the image size of the image to be processed can be scaled to 640x480 according to the second image scale.

[0135] It can be understood that in the embodiments of the present application, since the depth image corresponding to the image to be processed is determined in combination with the first image before the image to be processed in the video to be processed, for the first image, it is possible to choose to scale the image according to the scaling scale corresponding to the image to be processed, that is, scale the first image according to the same second image scale to ensure that the image sizes of the image to be processed and the first image used to determine the depth image corresponding to the image to be processed are the same.

[0136] To summarize, the image processing method proposed in the embodiment of the present application determines the depth image of the image to be processed through the optical flow information of the frame images before and after the video to be processed; then, the MASK image corresponding to the image to be processed, that is, the saturation MASK image, is extracted through light source detection and color detection, and the highlight area to be adjusted in the image to be processed is preliminarily confirmed; the depth image and the saturation MASK image are then fused to obtain the final MASK image, that is, different MASK images corresponding to different types; finally, with the help of global tone mapping curves corresponding to different types, the different MASK images corresponding to different types are fused with the image to be adjusted to obtain a processed image.

[0137] The embodiment of the present application proposes an image processing method, which determines a light source MASK image corresponding to the image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame of an image in a video to be processed; based on the light source MASK image and the saturation image, the saturation MASK image corresponding to the image to be processed is determined; based on the saturation MASK image and the depth image, it is determined that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, a processed image corresponding to the image to be processed is determined. That is to say, in the embodiment of the present application, during the image processing process, the light source MASK image representing the brightness information of the image to be processed, the saturation image representing the saturation information of the image to be processed, and the depth image representing the depth information of the image can be determined respectively, and then the image to be processed is divided into different MASK images of different types by combining the light source MASK image, the saturation image, and the depth image of the image to be processed, and finally the different MASK images are adjusted and processed respectively to obtain a processed image with a more ideal processing effect. It can be seen that in the embodiments of the present application, since the brightness information, saturation information and depth information of the image are introduced in the process of image processing, it is possible to more comprehensively consider the various parameters of the image and divide the image to be processed into more accurate and rich MASK images for subsequent processing to meet the detailed requirements of image processing, thereby effectively improving the image processing effect.

[0138] Based on the above embodiments, another embodiment of the present application proposes an image processing method, which can be applied to an image processing device or an electronic device, and can also be applied to any terminal including an image processing device or an electronic device.

[0139] Below, taking an electronic device as an example, the image processing method proposed in the embodiment of the present application is exemplarily described.

[0140] Furthermore, in the embodiments of the present application, Figure 4 This is a schematic diagram of the image processing method implementation process proposed in the embodiment of the present application, such as Figure 4 As shown, the image processing method may include the following steps:

[0141] Step 201: Determine an image to be processed.

[0142] The image to be processed may be a frame of image in the video to be processed.

[0143] Step 202: Scaling the image to be processed.

[0144] For the image to be processed in the video to be processed, the image to be processed may be first scaled, for example, the image to be processed may be scaled according to the second image scale.

[0145] For example, in order to reduce the amount of calculation in the subsequent processing flow and improve the image processing efficiency, assuming that the second image scale is 480x320, for the image to be processed whose image size is larger than the second image scale, you can choose to scale the image size of the image to be processed to 480x320 according to the second image scale.

[0146] Step 203: Determine a light source MASK image corresponding to the image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed.

[0147] In an embodiment of the present application, the image to be processed may be a frame of an image in a video to be processed. For the image to be processed, a light source MASK image representing brightness information of the image to be processed, a saturation image representing saturation information of the image to be processed, and a depth image representing depth information of the image to be processed may be determined respectively.

[0148] Step 204: Determine a saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image.

[0149] In an embodiment of the present application, a fusion process can be performed based on a light source MASK image and a saturation image, and finally a saturation MASK image corresponding to the image to be processed can be determined. For example, the light source MASK image and the saturation image can be fused based on a saturation MASK curve to determine a saturation MASK image. The saturation MASK curve can be used to perform different fusion strategies on image blocks with different saturations. For example, different fusion strategies can be selected for image blocks with higher saturation and image blocks with lower saturation.

[0150] In an embodiment of the present application, the saturation MASK image corresponding to the image to be processed is determined in combination with the saturation image and the light source MASK image corresponding to the image to be processed. Therefore, the saturation MASK image can fully characterize the saturation information and brightness information of the image to be processed, that is, the saturation MASK image introduces the saturation information and brightness information of the image to be processed.

[0151] Step 205 : Based on the saturation MASK image and the depth image, determine a foreground MASK image corresponding to the foreground type and a background MASK image corresponding to the background type of the image to be processed.

[0152] In an embodiment of the present application, the determined saturation MASK image and depth image may be used to divide the image to be processed, so that the image may be divided into a foreground MASK image corresponding to a foreground type and a background MASK image corresponding to a background type.

[0153] In an embodiment of the present application, multiple MASK images corresponding to multiple types are determined in combination with the depth image and saturation MASK image corresponding to the image to be processed, wherein the saturation MASK image introduces the saturation information and brightness information of the image to be processed. Therefore, the multiple MASK images corresponding to multiple types can fully characterize the depth information, saturation information and brightness information of the image to be processed, that is, the multiple MASK images corresponding to multiple types introduce the depth information, saturation information and brightness information of the image to be processed.

[0154] Step 206: Determine a processed image corresponding to the image to be processed based on the foreground MASK image corresponding to the foreground type and the background MASK image corresponding to the background type.

[0155] In an embodiment of the present application, different image processing methods can be used to perform image processing on the determined foreground MASK image corresponding to the foreground type and the background MASK image corresponding to the background type. For example, different global tone mapping curves corresponding to different types can be used to perform image processing on the corresponding MASK images respectively, so as to obtain a processed image corresponding to the image to be processed.

[0156] Exemplarily, the background MASK image and the initial brightness image corresponding to the image to be processed can be fused based on the first global tone mapping curve corresponding to the background type to determine the first adjusted image corresponding to the background MASK image; then, based on the second global tone mapping curve corresponding to the foreground type, the foreground MASK image and the first adjusted image can be fused to determine the processed image corresponding to the image to be processed.

[0157] Furthermore, in the embodiments of the present application, Figure 5 This is a schematic diagram of the image processing method implementation process proposed in the embodiment of the present application, such as Figure 5 As shown, the image processing method may include the following steps:

[0158] Step 201: Determine an image to be processed.

[0159] Step 202: Scaling the image to be processed.

[0160] Step 207: Determine a light source MASK image corresponding to the image to be processed and a saturation image corresponding to the image to be processed.

[0161] In an embodiment of the present application, the image to be processed may be a frame of an image in a video to be processed. For the image to be processed, a light source MASK image representing brightness information of the image to be processed and a saturation image representing saturation information of the image to be processed may be determined respectively.

[0162] Step 204: Determine a saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image.

[0163] Step 208: Determine multiple types based on the image to be processed.

[0164] In the embodiment of the present application, the types of images to be processed may be first divided based on the image content and image information, so that the corresponding multiple types may be determined.

[0165] Step 209: Based on the multiple types and saturation MASK images, determine multiple MASK images corresponding to the multiple types.

[0166] In an embodiment of the present application, when determining multiple MASK images corresponding to multiple types based on multiple types and saturation MASK images, the MASK images corresponding to each type can be extracted from the saturation MASK image according to the multiple types obtained by division, and finally multiple MASK images corresponding to multiple types are obtained.

[0167] In an embodiment of the present application, multiple MASK images corresponding to multiple types are determined in combination with a saturation MASK image corresponding to the image to be processed, wherein the saturation MASK image introduces the saturation information and brightness information of the image to be processed. Therefore, the multiple MASK images corresponding to multiple types can fully characterize the saturation information and brightness information of the image to be processed, that is, the multiple MASK images corresponding to multiple types introduce the saturation information and brightness information of the image to be processed.

[0168] Step 210: Based on the multiple MASK images, determine a processed image corresponding to the image to be processed.

[0169] In an embodiment of the present application, for multiple MASK images corresponding to multiple types, different image processing methods can be used to perform image processing respectively. For example, multiple global tone mapping curves corresponding to multiple types can be used to perform image processing on the corresponding MASK images respectively, so as to obtain a processed image corresponding to the image to be processed.

[0170] Exemplarily, for any type, the global tone mapping curve corresponding to the current type can be used to perform image processing on the MASK image corresponding to the current type in combination with the adjusted image corresponding to the previous type to determine the adjusted image corresponding to the current type; then continue to traverse the next type, and perform image processing on the MASK image corresponding to the next type in combination with the adjusted image corresponding to the current type. After traversing all types, multiple adjusted images corresponding to multiple types can be determined, and then the processed image corresponding to the image to be processed can be obtained.

[0171] To summarize, the image processing method proposed in the embodiment of the present application determines the depth image of the image to be processed through the optical flow information of the frame images before and after the video to be processed; then, the MASK image corresponding to the image to be processed, that is, the saturation MASK image, is extracted through light source detection and color detection, and the highlight area to be adjusted in the image to be processed is preliminarily confirmed; the depth image and the saturation MASK image are then fused to obtain the final MASK image, that is, different MASK images corresponding to different types; finally, with the help of global tone mapping curves corresponding to different types, the different MASK images corresponding to different types are fused with the image to be adjusted to obtain a processed image.

[0172] It can be understood that in the embodiments of the present application, considering that multiple MASK images corresponding to multiple types can fully characterize the depth information, saturation information and brightness information of the image to be processed, when adjusting different MASK images of different types, the corresponding global tone mapping curve can be selected based on the difference in one or more of the depth information, saturation information and brightness information, that is, based on the difference in one or more of the depth information, saturation information and brightness information, different global tone mapping curves are used to adopt different adjustment strategies for different MASK images.

[0173] For example, in some embodiments, for highlight areas, a global tone mapping curve that only enhances the transparency of highlight areas can be selected to adjust the MASK image. For example, based on the saturation MASK image, the highlight area of ​​the background MASK image and the highlight area of ​​the foreground MASK image in the image to be processed are extracted; finally, with the help of two global tone mapping curves corresponding to the background type and the foreground type, the MASK image is merged with the image to be processed, respectively enhancing the transparency of the highlight area, while avoiding the negative impact of global tone mapping in the dark area.

[0174] For example, in some embodiments, for a high saturation area, a global tone mapping curve that only adjusts the hue of the high saturation area can be selected to adjust the MASK image. For example, the saturation information of the reference image is used to allow the global tone mapping curve to act only on the high saturation area to avoid the phenomenon of dull colors in the low saturation area of ​​the image during the process of enhancing the light and shadow of the image.

[0175] The embodiment of the present application proposes an image processing method, which determines a light source MASK image corresponding to the image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame of an image in a video to be processed; based on the light source MASK image and the saturation image, the saturation MASK image corresponding to the image to be processed is determined; based on the saturation MASK image and the depth image, it is determined that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, a processed image corresponding to the image to be processed is determined. That is to say, in the embodiment of the present application, during the image processing process, the light source MASK image representing the brightness information of the image to be processed, the saturation image representing the saturation information of the image to be processed, and the depth image representing the depth information of the image can be determined respectively, and then the image to be processed is divided into different MASK images of different types by combining the light source MASK image, the saturation image, and the depth image of the image to be processed, and finally the different MASK images are adjusted and processed respectively to obtain a processed image with a more ideal processing effect. It can be seen that in the embodiments of the present application, since the brightness information, saturation information and depth information of the image are introduced in the process of image processing, it is possible to more comprehensively consider the various parameters of the image and divide the image to be processed into more accurate and rich MASK images for subsequent processing to meet the detailed requirements of image processing, thereby effectively improving the image processing effect.

[0176] Based on the above embodiments, another embodiment of the present application proposes an image processing method, which can be understood as an image light and shadow enhancement method.

[0177] Among them, in order to address the problem that ignoring the depth information of the image when performing light and shadow enhancement in the related art will affect the subject of the image and have a negative impact on the skin color and light and shadow of the subject, in an embodiment of the present application, the subject area and background area in the image can be found with the help of optical flow information of continuous video frames, so that the highlight areas to be adjusted in the background and subject can be processed separately, thereby enhancing the overall transparency of the background while enhancing the highlight details of the subject.

[0178] Among them, in view of the fact that the color saturation information of the image is ignored in the related technology, which easily causes the low-saturation area to appear dull in color, in an embodiment of the present application, the brightness information and color saturation information of the image can be considered at the same time, and the light source area to be processed in the image (that is, the highlight area with high color saturation) can be extracted, and the light and shadow of the highlight area with high color saturation can be adjusted, thereby improving the overall atmosphere of the image.

[0179] Figure 6 This is a schematic diagram of the implementation framework of the image processing method proposed in the embodiment of the present application, such as Figure 6As shown, an image to be adjusted (image to be processed) is obtained and scaled (step 301); a depth image of the image to be adjusted is calculated by combining the scaled current frame and the previous frame of the image to be adjusted (first image) (step 302); a light source MASK image of the scaled image to be adjusted is detected (step 303); a saturation image of the image to be adjusted is detected at the same time (step 304), and then a saturation MASK image is obtained by fusing the light source MASK image with the saturation image (step 305); the depth image and the saturation MASK image are subjected to MASK fusion to obtain a foreground MASK image and a background MASK image, respectively (step 306); a global tone mapping curve is used to fuse the brightness image of the image to be adjusted and the background MASK image, and then another global tone mapping curve is used to fuse the brightness image of the adjusted background and the foreground MASK image, that is, tone mapping is performed on the background MASK image and the foreground MASK image, respectively (step 307), to obtain a final output image (processed image).

[0180] In some embodiments, scaling the image to be adjusted can reduce the amount of computation of the depth calculation module and improve efficiency.

[0181] Exemplarily, the width and height of the image to be adjusted after scaling are 640x480, and correspondingly, the width and height of the previous frame image after scaling are also 640x480.

[0182] In some embodiments, the DIS optical flow algorithm can be used to determine the depth image in combination with the scaled current frame and the previous frame of the image to be adjusted. The algorithm is a dense optical flow algorithm that can accurately estimate the offset (dx, dy) of each pixel in the scaled image to be adjusted. For example, in the present application, the square root of the sum of dx and dy can be calculated pixel by pixel for the result of the DIS optical flow algorithm to obtain the initial depth image Depth0, and then the average value of the Depth0 blocks is taken to obtain Depth1 with a scale (such as the first image scale) of 32x32. The scale can be reconfigured according to actual needs, and finally the final depth image Depth is obtained according to the inversion formula shown in formula (1). out .

[0183] In some embodiments, when obtaining the scaled light source MASK image of the image to be adjusted, the average brightness of each block image of the brightness image (initial brightness image) of the image to be adjusted can be calculated to obtain a small-scale brightness image, which is the same as the calculated depth image Depth. out The scales of the light source MASK and the image MASK are kept consistent (for example, they all satisfy the first image scale), and then the light source MASK image MASK is obtained according to the pre-configured light source MASK curve. v .

[0184] In some embodiments, when the saturation MASK image is obtained by fusing the light source MASK image with the saturation image, the average saturation of each block image of the saturation image of the image to be adjusted can be calculated to obtain a small-scale saturation image S global , the scale is the same as the calculated light source MASK image MASK v The scales of the images are kept consistent (for example, they all meet the scale of the first image), and then S is fused according to the pre-configured saturation MASK curve. global and MASK v Get the saturation MASK image MASK s , where the fusion formula is shown in formula (2).

[0185] In some embodiments, when fusing the depth image Depth out and saturation MASK image MASK s , respectively get the foreground MASK image Mask fg and background MASK imageMask bg When α fg is the scaling factor of the foreground MASK, which is configured as 8.0 in the embodiment of the present application, fgThd is the MASK threshold of the foreground type, which is configured as 0.2 in the embodiment of the present application, α bg is the scaling factor of the background MASK, which is configured as 2.5 in the embodiment of the present application; bgThd is the MASK threshold of the background type, which is configured as 0.3 in the embodiment of the present application; the fusion formula is shown in formula (3) and formula (4).

[0186] For example, Figure 7 This is a schematic diagram of the background MASK image proposed in the embodiment of the present application, such as Figure 7 As shown, the upper image is the image to be processed, and the lower image is the MASK image Mask corresponding to the background type determined based on the image to be processed bg .

[0187] In some embodiments, the global tone mapping curve Lut corresponding to the background type is used. bg Fusion of the brightness image L of the image to be adjusted in and background MASK imageMask bg , improve the transparency of the light source area with high saturation, and then use another global tone mapping curve Lut corresponding to the foreground type fg Fusion of brightness image L with adjusted background brightness bg and foreground MASK image Mask fg , thereby enhancing the highlight details of the subject portrait area and obtaining the final output image L out , as shown in formula (9) and formula (10).

[0188] In some embodiments, if the foreground MASK image Mask fg and background MASK imageMask bg If the size of the MASK image is inconsistent with the size of the image to be processed, you need to first scale the size of the MASK image to the same size as the image to be processed.

[0189] For example, assuming that the obtained foreground MASK image Mask fg and background MASK imageMask bg The scale of the image is 32x32, which is inconsistent with the scale of the image to be adjusted. Therefore, the Mask needs to be adjusted before fusion. fg and Mask bg Zoom to the dimensions of the image to be adjusted.

[0190] In some embodiments, global tone mapping curves corresponding to different types may be pre-configured or calculated based on histogram statistics related methods such as AHE, with the goal of improving the contrast and light and shadow levels of a specified brightness range of an image.

[0191] For example, Figure 8 This is a schematic diagram of the global tone mapping curve proposed in the embodiment of the present application, such as Figure 8 As shown, among the global tone mapping curves used in the embodiment of the present application, one curve is a global tone mapping curve corresponding to the foreground type, and the other curve is a global tone mapping curve corresponding to the background type.

[0192] It can be understood that the image processing method proposed in the present application introduces a depth detection scheme and a light source detection scheme, estimates the depth information of the image based on the DIS optical flow algorithm, and fully combines the brightness information of the image to accurately distinguish the highlight areas of the portrait subject and the background, thereby ensuring that the range of the global tone mapping effect is controllable, thereby enhancing the sense of transparency in the highlight area while avoiding the negative impact of the global tone mapping in the dark area; at the same time, the embodiment of the present application introduces a color detection scheme, refers to the saturation information of the image, and allows the global tone mapping curve to act only on the high-saturation area, thereby avoiding the phenomenon of dull colors in the low-saturation area of ​​the image during the process of enhancing the light and shadow of the image.

[0193] It can be understood that in the process of image light and shadow enhancement based on the image processing method proposed in this application, the depth map of the current frame image is first calculated through the optical flow information of the previous and next frame images of the video stream; then the MASK image is extracted by using light source detection and color detection, and the highlight area to be adjusted in the current frame image is preliminarily confirmed; the depth map and the MASK image are then fused to obtain the final MASK image, and the highlight area of ​​the background light source and the highlight area of ​​the portrait subject in the current frame image are extracted; finally, with the help of two global tone mapping curves, the MASK image is fused with the image to be adjusted to enhance the transparency of the background highlight area and the portrait subject highlight area, respectively.

[0194] Furthermore, in addition to using depth information to segment the saturation MASK image into a foreground MASK image and a background MASK image through pre-set threshold parameters (such as MASK threshold), more categories of MASK images can also be generated through multi-semantic segmentation technology (such as U-Net network).

[0195] For example, multi-semantic segmentation technology can be used to find the building area in the image to be adjusted, and then the building MASK is extracted from the saturation MASK image. Finally, a global tone mapping curve for the building highlight area is simultaneously added to improve the highlight performance of the building in the image to be adjusted.

[0196] It can be seen that in the embodiments of the present application, since the brightness information, saturation information and depth information of the image are introduced in the process of image processing, it is possible to more comprehensively consider the various parameters of the image and divide the image to be processed into more accurate and rich MASK images for subsequent processing to meet the detailed requirements of image processing, thereby effectively improving the image processing effect.

[0197] Based on the above embodiment, in another embodiment of the present application, Fig. 9 This is a schematic diagram of the structure of the image processing device proposed in the embodiment of the present application. Fig. 9 As shown, the image processing device 110 proposed in the embodiment of the present application may include:

[0198] Determination unit 1101 is used to determine a light source MASK image corresponding to an image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame image in a video to be processed; based on the light source MASK image and the saturation image, determine the saturation MASK image corresponding to the image to be processed; based on the saturation MASK image and the depth image, determine that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, determine a processed image corresponding to the image to be processed.

[0199] In the embodiments of the present application, further, Fig.10 This is a schematic diagram of the structure of the electronic device proposed in the embodiment of the present application, such as Fig.10 As shown, the electronic device 120 proposed in the embodiment of the present application may include a processor 1201, a memory 1202, a communication interface 1203, and a bus 1204 for connecting the processor 1201, the memory 1202 and the communication interface 1203.

[0200] In the embodiment of the present application, the processor 1201 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiment of the present application is not specifically limited. The electronic device 120 can also include a memory 1202, which can be connected to the processor 1201, wherein the memory 1202 is used to store executable program code, the program code includes computer operation instructions, and the memory 1202 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0201] In the embodiment of the present application, the bus 1204 is used to connect the communication interface 1203, the processor 1201 and the memory 1202, as well as the mutual communication between these devices.

[0202] In practical applications, the memory 1202 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 1201.

[0203] Further, in an embodiment of the present application, the processor 1201 is used to determine a light source MASK image corresponding to the image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame image in the video to be processed;

[0204] Based on the light source MASK image and the saturation image, determine the saturation MASK image corresponding to the image to be processed; based on the saturation MASK image and the depth image, determine that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, determine the processed image corresponding to the image to be processed.

[0205] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0206] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0207] An embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the image processing method described above is implemented.

[0208] Specifically, the program instructions corresponding to an image processing method in this embodiment may be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to an image processing method in the storage medium are read or executed by an electronic device, the following steps are included:

[0209] Determine a light source MASK image corresponding to an image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame image in a video to be processed;

[0210] Determine a saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image;

[0211] Based on the saturation MASK image and the depth image, determining that the image to be processed corresponds to a plurality of MASK images of a plurality of types;

[0212] Based on the multiple MASK images, a processed image corresponding to the image to be processed is determined.

[0213] The embodiment of the present application also provides a computer program product.

[0214] In some embodiments, the computer program product may include a computer program or instructions.

[0215] In some embodiments, the computer program product can be applied to the computer device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the computer device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0216] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0217] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0220] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a light source MASK image corresponding to an image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame image in a video to be processed; Determine a saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image; Based on the saturation MASK image and the depth image, determining that the image to be processed corresponds to a plurality of MASK images of a plurality of types; Based on the multiple MASK images, a processed image corresponding to the image to be processed is determined.

2. The method according to claim 1, characterized in that The step of determining, based on the multiple MASK images, a processed image corresponding to the image to be processed includes: Based on a first global tone mapping curve corresponding to a first type among the multiple types, a first MASK image corresponding to the first type among the multiple MASK images and an initial brightness image corresponding to the image to be processed are merged to determine a first adjusted image corresponding to the first MASK image; Based on a kth global tone mapping curve corresponding to a kth type among the multiple types, a kth MASK image corresponding to the kth type and a k-1th adjusted image among the multiple MASK images are merged to determine a kth adjusted image corresponding to the kth MASK image; wherein k is an integer greater than 1; Based on a k+1th global tone mapping curve corresponding to the k+1th type, fusing a k+1th MASK image corresponding to the k+1th type among the multiple MASK images and the k+1th adjusted image to determine a k+1th adjusted image corresponding to the k+1th MASK image; Based on the (k+1)th adjusted image, a processed image corresponding to the image to be processed is determined.

3. The method according to claim 1, characterized in that: The multiple types include a foreground type and a background type, the multiple MASK images include a foreground MASK image corresponding to the foreground type and a background MASK image corresponding to the background type, and determining a processed image corresponding to the image to be processed based on the multiple MASK images includes: Based on a first global tone mapping curve corresponding to the background type, the background MASK image and the initial brightness image corresponding to the image to be processed are merged to determine a first adjusted image corresponding to the background MASK image; Based on a second global tone mapping curve corresponding to the foreground type, the foreground MASK image and the first adjusted image are fused to determine a processed image corresponding to the image to be processed.

4. The method according to any one of claims 1 to 3, characterized in that The step of determining the saturation MASK image corresponding to the image to be processed based on the light source MASK image and the saturation image includes: The light source MASK image and the saturation image are fused based on a saturation MASK curve to determine the saturation MASK image; wherein the saturation MASK curve is used to execute different fusion strategies for image blocks with different saturations.

5. The method according to any one of claims 1 to 3, characterized in that The step of determining, based on the saturation MASK image and the depth image, that the image to be processed corresponds to a plurality of MASK images of a plurality of types comprises: Based on a plurality of scaling coefficients corresponding to the plurality of types and a plurality of MASK thresholds corresponding to the plurality of types, the saturation MASK image and the depth image are fused respectively to determine the plurality of MASK images corresponding to the plurality of types.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Based on the image to be processed and the saturation MASK image, determining that the image to be processed corresponds to the multiple MASK images of the multiple types; wherein, The step of determining, based on the image to be processed and the saturation MASK image, that the image to be processed corresponds to the multiple MASK images of the multiple types comprises: Determining the multiple types based on the image to be processed; Based on the multiple types and the saturation MASK image, the multiple MASK images corresponding to the multiple types are determined.

7. The method according to claim 4, characterized in that The determining of the depth image corresponding to the image to be processed includes: Determine the pixel position of the first image and the pixel position of the image to be processed; wherein the first image is a frame of image before the image to be processed in the video to be processed; Determining an initial depth image corresponding to the image to be processed based on pixel positions of the image to be processed and pixel positions of the first image; The depth image is determined based on the initial depth image and a first image scale.

8. The method according to claim 4, characterized in that: The step of determining a light source MASK image corresponding to the image to be processed includes: Determining a brightness image block of the image to be processed based on an initial brightness image corresponding to the image to be processed and the first image scale; The light source MASK image is determined based on the brightness image block of the image to be processed and the light source MASK curve.

9. The method according to claim 4, characterized in that The step of determining a saturation image corresponding to the image to be processed includes: Determine an initial saturation image corresponding to the image to be processed; The saturation image is determined based on the initial saturation image and the first image scale.

10. An image processing device, characterized in that: The image processing device comprises: A determination unit, used to determine a light source MASK image corresponding to an image to be processed, a saturation image corresponding to the image to be processed, and a depth image corresponding to the image to be processed; wherein the image to be processed is a frame of an image in a video to be processed; based on the light source MASK image and the saturation image, determine the saturation MASK image corresponding to the image to be processed; based on the saturation MASK image and the depth image, determine that the image to be processed corresponds to multiple MASK images of multiple types; based on the multiple MASK images, determine a processed image corresponding to the image to be processed.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 9 is implemented.

Images