Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and particularly provides an image processing method and device, electronic equipment and a storage medium. The invention discloses an image processing method, which comprises the following steps of: performing image segmentation on an original image to obtain a target area sub-image containing a flare area; performing flare erasure on the target area sub-image to obtain a flare erased sub-image; performing fusion processing on the original image and the flare erasure sub-image to obtain a target fusion image; and performing image inpainting on the target fusion image to obtain a target inpainted image. In the embodiment of the invention, the flare region in the original image is segmented and the flare is erased, and the erased region is repaired based on the original image, so that the target repaired image is obtained, the loss of image details and textures is reduced, and the flare removal accuracy of the image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] When capturing an image, there may be light reflection and scattering inside the lens, which usually causes some unnecessary light to appear in the image, thus forming flare.

[0003] To remove flare from images, the physical processes of the image are usually modeled to estimate and remove flare and overexposed areas in the image.

[0004] However, conventional image deflare technology usually loses a large amount of image details and textures, and the image deflare accuracy is low, resulting in poor image deflare effect. Summary of the invention

[0005] In order to improve the accuracy of image de-glare, the embodiments of the present application provide an image processing method, device, electronic device and storage medium.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Perform image segmentation on the original image to obtain a sub-image of the target area including the glare area;

[0008] Performing glare erasing on the target area sub-image to obtain a glare-erased sub-image;

[0009] The original image and the glare-erased sub-image are fused to obtain a target fused image;

[0010] Perform image inpainting on the target fused image to obtain a target inpainted image.

[0011] In some embodiments, performing image segmentation on the original image to obtain a target area sub-image including a glare area includes:

[0012] The original image is segmented using a semantic segmentation model to obtain at least one sub-image of the target area; the semantic segmentation model is constructed based on a deep learning algorithm;

[0013] Alternatively, at least one target area sub-image meeting the pixel threshold condition is segmented from the original image according to the pixel value of each pixel in the original image.

[0014] In some embodiments, performing glare erasing on the target region sub-image to obtain the glare-erased sub-image includes:

[0015] Determine the glare category of the target area sub-image;

[0016] According to the glare erasing mode set corresponding to the glare category, the glare in the sub-image of the target area is removed to obtain the glare-erased sub-image.

[0017] In some embodiments, the glare categories include: light source area, light band area, and glare and sparkle area;

[0018] Remove flare from the sub-image of the target area according to the flare erasure method set for the flare category, including:

[0019] If the glare category is determined to be a light source area, each pixel value in the target area sub-image is set to a specified pixel value;

[0020] If the flare category is determined to be a light band area, a denoising model is used to erase the sub-image of the target area based on the image exposure of the sub-image of the target area. The light band area is constructed based on deep learning, and the erasing intensity of the sub-image of the target area is positively correlated with the image exposure.

[0021] If the glare category is determined to be a glare and shining area, the dehazing model is used to adjust the contrast and saturation of the sub-image in the target area; the dehazing model is built based on deep learning and is used to adjust the contrast and saturation of the image.

[0022] In some implementations, fusing the original image and the glare-erased sub-image to obtain a target fused image includes:

[0023] According to the mask of the flare-erased sub-image, the flare-erased sub-image is backfilled into the original image to obtain a target backfilled image;

[0024] The target backfill image is concatenated with the original image to obtain the target fused image.

[0025] In some implementations, performing image repair on the target fused image includes:

[0026] An image inpainting model is used to inpaint the target fusion image; the image inpainting model is built based on deep learning;

[0027] Alternatively, the target fused image is repaired based on texture synthesis technology; the texture synthesis technology is used to perform image repair by synthesizing background texture;

[0028] Alternatively, the target fused image is repaired based on pixel interpolation technology; the pixel interpolation technology is used to perform image repair by using pixel point interpolation.

[0029] In some embodiments, the method further comprises:

[0030] Perform weighted fusion of the original image and the target patch image to obtain a new target patch image;

[0031] Alternatively, based on a neural network fusion method, the original image and the target repair image are fused to obtain a new target repair image; the neural network fusion method is based on a neural network to perform image fusion.

[0032] In some embodiments, the method further comprises:

[0033] Based on the light source in the original image, the light source of the target patch image is backfilled to obtain a new target patch image;

[0034] Alternatively, based on the light source in the candidate image, the light source is backfilled on the target patch image to obtain a new target patch image.

[0035] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0036] A segmentation unit, used for performing image segmentation on the original image to obtain a sub-image of a target area including a glare area;

[0037] A removal unit, used for performing glare erasing on the target area sub-image to obtain a glare-erased sub-image;

[0038] A fusion unit, used for fusing the original image and the glare-erased sub-image to obtain a target fused image;

[0039] The patching unit is used to perform image patching on the target fused image to obtain a target patched image.

[0040] In some embodiments, the segmentation unit is used to:

[0041] The original image is segmented using a semantic segmentation model to obtain at least one sub-image of the target area; the semantic segmentation model is constructed based on a deep learning algorithm;

[0042] Alternatively, at least one target area sub-image meeting the pixel threshold condition is segmented from the original image according to the pixel value of each pixel in the original image.

[0043] In some embodiments, the removal unit is used to:

[0044] Determine the glare category of the target area sub-image;

[0045] According to the glare erasing mode set corresponding to the glare category, the glare in the sub-image of the target area is removed to obtain the glare-erased sub-image.

[0046] In some embodiments, the glare categories include: light source area, light band area, and glare and sparkle area;

[0047] Removal units are used to:

[0048] If the glare category is determined to be a light source area, each pixel value in the target area sub-image is set to a specified pixel value;

[0049] If the flare category is determined to be a light band area, a denoising model is used to erase the sub-image of the target area based on the image exposure of the sub-image of the target area. The light band area is constructed based on deep learning, and the erasing intensity of the sub-image of the target area is positively correlated with the image exposure.

[0050] If the glare category is determined to be a glare and shining area, the dehazing model is used to adjust the contrast and saturation of the sub-image in the target area; the dehazing model is built based on deep learning and is used to adjust the contrast and saturation of the image.

[0051] In some embodiments, the fusion unit is used to:

[0052] According to the mask of the flare-erased sub-image, the flare-erased sub-image is backfilled into the original image to obtain a target backfilled image;

[0053] The target backfill image is concatenated with the original image to obtain the target fused image.

[0054] In some embodiments, the repair unit is used to:

[0055] An image inpainting model is used to inpaint the target fusion image; the image inpainting model is built based on deep learning;

[0056] Alternatively, the target fused image is repaired based on texture synthesis technology; the texture synthesis technology is used to perform image repair by synthesizing background texture;

[0057] Alternatively, the target fused image is repaired based on pixel interpolation technology; the pixel interpolation technology is used to perform image repair by using pixel point interpolation.

[0058] In some embodiments, the repair unit is further configured to:

[0059] Perform weighted fusion of the original image and the target patch image to obtain a new target patch image;

[0060] Alternatively, based on a neural network fusion method, the original image and the target repair image are fused to obtain a new target repair image; the neural network fusion method is based on a neural network to perform image fusion.

[0061] In some embodiments, the repair unit is further configured to:

[0062] Based on the light source in the original image, the light source of the target patch image is backfilled to obtain a new target patch image;

[0063] Alternatively, based on the light source in the candidate image, the light source is backfilled on the target patch image to obtain a new target patch image.

[0064] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0065] Processor; and

[0066] A memory stores computer instructions, wherein the computer instructions are used to enable a processor to execute the method of any embodiment of the first aspect.

[0067] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a method of any embodiment of the first aspect.

[0068] The image processing method of the embodiment of the present application includes segmenting the original image to obtain a sub-image of the target area including the flare area; erasing the flare on the sub-image of the target area to obtain a flare-erased sub-image; fusing the original image and the flare-erased sub-image to obtain a target fused image; and inpainting the target fused image to obtain a target inpainted image. In the embodiment of the present application, the flare area in the original image is segmented and the flare is erased, and the erased area is inpainted based on the original image to obtain the target inpainted image, thereby reducing the loss of image details and texture and improving the accuracy of image flare removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 is a flowchart of an image processing method according to some embodiments of the present application.

[0071] Figure 2 This is an example diagram of image segmentation comparison in some implementation methods of the present application.

[0072] Figure 3 It is a schematic diagram of the structure of the glare erasing system in some embodiments of the present application.

[0073] Figure 4It is a schematic diagram of the structure of an image repair system in some implementation modes of the present application.

[0074] Figure 5 It is a structural schematic diagram of a light source backfilling system in some implementation modes of the present application.

[0075] Figure 6 It is a structural block diagram of an image processing device in some embodiments of the present application.

[0076] Figure 7 It is a structural block diagram of an electronic device in some implementation modes of the present application. DETAILED DESCRIPTION

[0077] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described implementation methods are part of the implementation methods of the present application, rather than all of the implementation methods. Based on the implementation methods in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, the technical features involved in the different implementation methods of the present application described below can be combined with each other as long as they do not conflict with each other.

[0078] In actual application scenarios, when shooting images, due to the possible reflection and scattering of light inside the lens, some unnecessary light usually appears in the image, thus forming glare.

[0079] Traditionally, image flare removal is usually based on physical models. Specifically, the physical process of the image is modeled to estimate and remove flare and overexposed areas in the image. However, this approach may result in loss of image details and generation of artifacts, and the accuracy of image flare removal is poor.

[0080] Furthermore, an image de-flare model can be constructed based on a deep convolutional neural network, and the reflection and shadow information in the image can be learned based on the image de-flare model, and separated to remove the flare and over-exposed areas in the image. However, due to the high complexity of the image de-flare model, a large amount of training data and computing resources are required, and overfitting problems may occur, so there is also a problem of poor image de-flare effect.

[0081] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a method, device, electronic device and storage medium for image processing, aiming to improve the accuracy of image de-glare.

[0082] On the first aspect, an embodiment of the present application provides a method for image processing, which can be applied to electronic devices. The present application does not limit the type of electronic device, which can be any type of device suitable for implementation, such as a smartphone, a tablet computer, etc., and the present application will not go into details.

[0083] See also Figure 1 As shown, it is a flowchart of the method of image processing in some embodiments of the present application. Figure 1 The method is described below. The specific implementation process of the method is as follows:

[0084] Step 101: performing image segmentation on the original image to obtain a sub-image of the target area including the glare area.

[0085] It should be noted that, in order to facilitate flexible image glare removal in subsequent steps, the glare category of the glare area can be divided into a light source area, a streak area, and a glare and shimmer area, and different target area sub-images are divided for glare areas of different glare categories. Optionally, the target area sub-image can be one or more.

[0086] In one implementation, when executing step 101, any of the following methods may be used:

[0087] Method 1: Use a semantic segmentation model to segment the original image and obtain at least one sub-image of the target area. The semantic segmentation model is built based on a deep learning algorithm.

[0088] In one embodiment, a semantic segmentation model can be constructed based on a deep learning algorithm, and the flare area in the input original image can be divided through the semantic segmentation model to obtain at least one flare area, and based on the mask of each flare area, a corresponding target area sub-image is generated.

[0089] It should be noted that the purpose of generating a sub-image of the target region based on the masking technology is to facilitate the backfilling of the segmented image (i.e., the sub-image of the target region) into the original image in the subsequent steps. Segmentation and backfilling are inverse processes. Segmentation refers to segmenting a sub-image from an image, and backfilling refers to restoring the segmented sub-image to its original position in the original image.

[0090] In this way, the semantic segmentation technology based on deep learning can achieve efficient and accurate segmentation of the glare area, so as to obtain a clear glare area, and then the glare can be accurately removed in the subsequent steps.

[0091] Method 2: According to the pixel value of each pixel in the original image, at least one target area sub-image that meets the pixel threshold condition is segmented from the original image.

[0092] The pixel threshold condition is set according to the pixel value of each pixel point. In practical applications, the pixel threshold condition can be set according to the actual application scenario, and is not limited here.

[0093] In one implementation, different pixel value thresholds may be set for different glare categories, so that target area sub-images meeting the pixel value thresholds of the respective glare categories may be divided from the original image.

[0094] In this way, the segmentation method based on pixel value threshold can be used to segment the glare area.

[0095] See also Figure 2 The following is an example of image segmentation comparison in some embodiments of the present application. Figure 2 Each flare category is described with examples. Figure 2 The left image in is the original input image. Figure 2 The right figure in the figure is an example of image segmentation, that is, three target area sub-images are obtained based on the masks of each glare area. Among them, the glare areas are the light source area, the light band area and the glare area.

[0096] Step 102: performing glare erasing on the target area sub-image to obtain a glare-erased sub-image.

[0097] Specifically, the implementation process of step 102 may include:

[0098] S1021: Determine the glare category of the target area sub-image.

[0099] The glare categories may include but are not limited to: light source area, light band area, and glare area.

[0100] S1022: removing glare from the target region sub-image according to the glare erasing method set corresponding to the glare category, to obtain a glare-erased sub-image.

[0101] In one implementation, when executing S1022, any of the following methods may be used:

[0102] Method 1: If the glare category is determined to be a light source area, each pixel value in the target area sub-image is set to a specified pixel value.

[0103] As an example, if the glare category is determined to be a light source area, the light source area in the target area sub-image is directly erased, that is, all are set to white, and the specified pixel value is the pixel value corresponding to white.

[0104] This is because the exposure of the light source area is very high and there is almost no background information. Therefore, the light source area can be completely erased quickly and accurately, thereby avoiding the influence of the light source area on subsequent processing and further improving the accuracy of subsequent image de-glare.

[0105] Method 2: If the flare category is determined to be a light band area, a denoising model is used to erase the sub-image of the target area based on the image exposure of the sub-image of the target area; the light band area is constructed based on deep learning, and the erasing intensity of the sub-image of the target area is positively correlated with the image exposure.

[0106] Generally speaking, in a light band, the exposure of the end close to the light source is higher, and the exposure of the end far from the light source is lower. If the exposure is low, then this area will carry more background information, so weaker erasure is required, and vice versa. Therefore, in the embodiment of the present application, a denoising model is used for erasing, which can automatically and accurately perform different intensities of erasing for different exposure levels, thereby retaining the information on the image background to the greatest extent, so as to facilitate the processing of subsequent modules, and further improve the accuracy of subsequent image glare removal.

[0107] Method 3: If the glare category is determined to be a glare area, a dehazing model is used to adjust the contrast and saturation of the sub-image in the target area; the dehazing model is built based on deep learning and is used to adjust the contrast and saturation of the image.

[0108] Taking into account that the exposure in the glare area is relatively low and there is usually a large amount of background information, the whole area looks like it is covered with a layer of mist. Therefore, in an embodiment of the present application, a dehazing model is constructed based on deep learning, so that the contrast, saturation, etc. of the glare area can be adjusted through the powerful feature extraction capability of the dehazing model, thereby achieving the effect of accurate erasing.

[0109] In the embodiment of the present application, different erasing methods can be used to erase sub-images of target areas of different glare categories, thereby adapting to different image glare removal requirements and improving the flexibility, efficiency and accuracy of glare erasure.

[0110] Step 103: Fusing the original image and the glare-erased sub-image to obtain a target fused image.

[0111] Specifically, the implementation process of step 103 may include:

[0112] S1031: backfilling the flare-erased sub-image into the original image according to the mask of the flare-erased sub-image to obtain a target backfilled image.

[0113] In an embodiment of the present application, glare is erased from a sub-image of a target area segmented from an original image to obtain a glare-erased sub-image. Then, based on a mask-based regional fusion method, the glare-erased sub-image after glare erasure is backfilled into the original image. The glare-erased sub-image can be seamlessly merged with the original image, thereby erasing the glare area while retaining the available information in the image background as much as possible, thereby obtaining a clear and natural glare area erasing effect.

[0114] See also Figure 3 FIG. 1 is a schematic diagram of the structure of the glare erasing system in some embodiments of the present application. Figure 3 The above-mentioned flare erasing and image backfilling methods are described with examples. Figure 3 It includes an original image input module, an image segmentation module, a light source area module, a total erasure module, a light band area module, an exposure erasure module, a glare module, an adjustment module, a regional fusion module and a fused image output module.

[0115] Among them, the original image input module is used to input the original image. The image segmentation module is used to perform semantic segmentation on the original image to obtain the target area sub-module. The light source module is used to filter out the target area sub-image whose glare category is the light source area. The all-erasing module is used to erase the light source area completely. The light band area module is used to filter out the target area sub-module whose glare category is the light band area. The exposure erasing module is used to erase the glare according to the exposure. The glare shine module is used to filter out the target area sub-module whose glare category is the glare shine area. The adjustment module is used to adjust the contrast and saturation of the image. The area fusion module is used to backfill the glare erased into the original image. The fused image output module is used to output the obtained target backfilled image.

[0116] S1032: Perform channel stitching on the target backfill image and the original image to obtain a target fused image.

[0117] In this way, the original image information can be introduced so that the glare erasure area can be determined more accurately and flexibly in subsequent steps.

[0118] Step 104: Perform image inpainting on the target fused image to obtain a target inpainted image.

[0119] Specifically, when executing step 104, any of the following methods may be used:

[0120] Method 1: Use an image inpainting model to inpaint the target fused image; the image inpainting model is built based on deep learning.

[0121] In one implementation, an image inpainting model is constructed based on deep learning, and image inpainting is performed on a target fused image based on the image inpainting model.

[0122] Since the image inpainting model can automatically learn how to perform image inpainting, it can adapt to different images and glare situations.

[0123] Method 2: Based on texture synthesis technology, repair the target fused image; texture synthesis technology is used to perform image repair by synthesizing background texture.

[0124] In one implementation, the background texture in the target fused image is synthesized to generate a target patched image. In this way, image patching can be achieved by synthesizing the background texture.

[0125] Method three: based on pixel interpolation technology, repair the target fused image; pixel interpolation technology is used to perform image repair by pixel point interpolation.

[0126] In one implementation, based on an interpolation method, pixel points around the flare area in the target fused image are interpolated to generate a target patched image. In this way, image patching can be achieved based on the interpolation method.

[0127] In the embodiment of the present application, the original image is introduced, so that the area erased by the glare can be judged more accurately and flexibly based on the original image information, and the information originally destroyed and erased by the glare can be restored, making the output target repaired image more realistic.

[0128] See also Figure 4 FIG. 1 is a schematic diagram of the structure of an image repair system in some embodiments of the present application. Figure 4 The above-mentioned image repair method is described. Figure 4 The system includes an original image input module, an erased image input module, a channel stitching module, a patching module and a patched image output module.

[0129] Among them, the original image input module is used to input the original image. The erased image input module is used to input the target backfill image. The channel stitching module is used to stitch the original image and the target backfill image to obtain the target fused image. The patching module is used to patch the target fusion module. The patched image output module is used to output the target patched image.

[0130] Furthermore, in order to make the output image more realistic and natural, the target repaired image may be fused again.

[0131] In one implementation, when the images are fused again, any of the following methods may be used:

[0132] Method 1: Perform weighted fusion of the original image and the target patch image to obtain a new target patch image.

[0133] In one implementation, a first weight of the original image and a second weight of the target patch image are determined, and based on the first weight and the second weight, the original image and the target patch image are weighted fused to obtain a new target patch image.

[0134] The first weight and the second weight may be pre-set or determined according to the glare intensity. In practical applications, the first weight and the second weight may be set according to the actual application scenario, and are not limited here. The first weight may generally be 0.1-0.3. As an example, the first weight is 0.2 and the second weight is 0.8.

[0135] Furthermore, different second weights can be set in advance for different flare categories, so that the first weight of the original image and the second weights set corresponding to the flare categories of each flare area in the target patch image can be obtained, and based on the first weight and the second weight, the original image and the target patch image are weightedly fused to obtain a new target patch image.

[0136] In the embodiment of the present application, the original image and the target patched image are weightedly fused according to the weight coefficient, so that the information in the original image can be retained and the output image is more natural.

[0137] Method 2: Based on the neural network fusion method, the original image and the target repair image are fused to obtain a new target repair image; the neural network fusion method is based on neural network image fusion.

[0138] In an embodiment of the present application, a neural network fusion method based on deep learning is used to perform pixel-level weighted fusion of the original image and the target patched image to obtain a more natural output image.

[0139] Furthermore, the light source may be backfilled on the target patched image after the image patching or the target patched image after the fusion again.

[0140] The backfilled light source may be a light source in the original image or a light source in the alternative image. In practical applications, the backfilled light source may be set according to the actual application scenario and is not limited here.

[0141] In one implementation, when backfilling the light source, any of the following methods may be used:

[0142] Method 1: Based on the light source in the original image, the light source of the target patch image is backfilled to obtain a new target patch image.

[0143] In one implementation, a target region sub-image whose glare category is a light source region may be screened out, and light source backfilling may be performed based on the light source in the target region sub-image.

[0144] Method 2: Based on the light source in the candidate image, the light source of the target patch image is backfilled to obtain a new target patch image.

[0145] In one implementation, a candidate image is selected from a database, and light source backfilling is performed based on the light source in the candidate image.

[0146] Furthermore, light source backfill may be performed based on light sources obtained in other ways.

[0147] In the embodiments of the present application, different light sources can be flexibly selected for light source backfilling, so that it can be suitable for different usage scenarios and meet different user needs.

[0148] In the embodiment of the present application, the target patched image obtained by light source backfilling retains the naturalness of the original image and the patched image, and looks more real and natural, greatly improving the user experience.

[0149] See also Figure 5 FIG. 1 is a schematic diagram of the structure of the light source backfill system in some embodiments of the present application. Figure 5 The method of re-fusion of the above images and backfilling of light sources is illustrated by way of example. Figure 5 It includes an original image input module, a patched image input module, a weighted mixing module, a light source module, a light source backfill module and a backfill image output module.

[0150] Among them, the original image input module is used to input the original image. The patch image input module is used to input the target patch image after image patching. The weight blending module is used to weightedly fuse the original image and the target patch image to obtain a new target patch image. The light source module is used to obtain the light source to be backfilled, such as the light source in the original image or the light source in the alternative image. The light source backfill module is used to backfill the light source into the above-mentioned new target patch image. The backfill image output module is used to output the new target patch image obtained after the light source is backfilled.

[0151] In the embodiment of the present application, the first stage and the second stage are mainly included. In the first stage, the original image is segmented to obtain the target area sub-image containing glare, and different methods are used to erase the glare of the target area sub-images of different glare categories. In the second stage, the image is repaired and the information is backfilled for the area erased in the first stage. Optionally, a third stage can also be included to perform weighted fusion and / or light source backfill on the target repair image obtained in the second stage, so that the target backfill image outputted at the end is more realistic.

[0152] In this way, in the first stage, different glare erasing methods can be selected for different glare categories for glare erasing, and in the third stage, different light sources can be selected for backfilling according to different scene requirements, which has high flexibility and can adapt to different image glare removal requirements. In addition, combined with technologies such as deep learning, image glare can be automatically removed, reducing labor costs and time costs, and improving image processing efficiency. Furthermore, different glare areas can be accurately divided, and glare areas can be accurately erased, improving the accuracy of image area glare, and information such as light sources can be backfilled into the target patch image, making the output image look more realistic, reducing the loss of image details and the generation of artifacts, and the glare removal effect for images containing complex textures and large color changes is also better. Further, it can be applied to various types of images, including but not limited to natural landscape images, character images, and architectural images, etc., and has high versatility. Finally, the image glare removal process is decomposed instead of using a complex model for image glare removal, which does not require a large amount of training data and computing resources, and reduces the overfitting problem.

[0153] Based on the same inventive concept, the embodiment of the present application also provides an image processing device. Since the principle of solving the problem by the above device and equipment is similar to that of an image processing method, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present application does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as a smart phone, a tablet computer, etc., and the present application will not repeat it.

[0154] See also Figure 6 , which is a structural block diagram of an image processing device in some embodiments of the present application. In some embodiments, the image processing device exemplified in the present application includes:

[0155] The segmentation unit 601 is used to segment the original image to obtain a sub-image of the target area including the glare area;

[0156] A removal unit 602, configured to perform glare erasure on the target area sub-image to obtain a glare-erased sub-image;

[0157] A fusion unit 603 is used to fuse the original image and the glare-erased sub-image to obtain a target fused image;

[0158] The patching unit 604 is used to perform image patching on the target fused image to obtain a target patched image.

[0159] In some implementations, the segmentation unit 601 is used to:

[0160] The original image is segmented using a semantic segmentation model to obtain at least one sub-image of the target area; the semantic segmentation model is constructed based on a deep learning algorithm;

[0161] Alternatively, at least one target area sub-image meeting the pixel threshold condition is segmented from the original image according to the pixel value of each pixel in the original image.

[0162] In some embodiments, the removal unit 602 is configured to:

[0163] Determine the glare category of the target area sub-image;

[0164] According to the glare erasing mode set corresponding to the glare category, the glare in the sub-image of the target area is removed to obtain the glare-erased sub-image.

[0165] In some embodiments, the glare categories include: light source area, light band area, and glare and sparkle area;

[0166] The removal unit 602 is used to:

[0167] If the glare category is determined to be a light source area, each pixel value in the target area sub-image is set to a specified pixel value;

[0168] If the flare category is determined to be a light band area, a denoising model is used to erase the sub-image of the target area based on the image exposure of the sub-image of the target area. The light band area is constructed based on deep learning, and the erasing intensity of the sub-image of the target area is positively correlated with the image exposure.

[0169] If the glare category is determined to be a glare and shining area, the dehazing model is used to adjust the contrast and saturation of the sub-image in the target area; the dehazing model is built based on deep learning and is used to adjust the contrast and saturation of the image.

[0170] In some embodiments, the fusion unit 603 is used to:

[0171] According to the mask of the flare-erased sub-image, the flare-erased sub-image is backfilled into the original image to obtain a target backfilled image;

[0172] The target backfill image is concatenated with the original image to obtain the target fused image.

[0173] In some embodiments, the repair unit 604 is configured to:

[0174] An image inpainting model is used to inpaint the target fusion image; the image inpainting model is built based on deep learning;

[0175] Alternatively, the target fused image is repaired based on texture synthesis technology; the texture synthesis technology is used to perform image repair by synthesizing background texture;

[0176] Alternatively, the target fused image is repaired based on pixel interpolation technology; the pixel interpolation technology is used to perform image repair by using pixel point interpolation.

[0177] In some implementations, the repair unit 604 is further configured to:

[0178] Perform weighted fusion of the original image and the target patch image to obtain a new target patch image;

[0179] Alternatively, based on a neural network fusion method, the original image and the target repair image are fused to obtain a new target repair image; the neural network fusion method is based on a neural network to perform image fusion.

[0180] In some implementations, the repair unit 604 is further configured to:

[0181] Based on the light source in the original image, the light source of the target patch image is backfilled to obtain a new target patch image;

[0182] Alternatively, based on the light source in the candidate image, the light source is backfilled on the target patch image to obtain a new target patch image.

[0183] The image processing method of the embodiment of the present application includes segmenting the original image to obtain a sub-image of the target area including the flare area; erasing the flare on the sub-image of the target area to obtain a flare-erased sub-image; fusing the original image and the flare-erased sub-image to obtain a target fused image; and inpainting the target fused image to obtain a target inpainted image. In the embodiment of the present application, the flare area in the original image is segmented and the flare is erased, and the erased area is inpainted based on the original image to obtain the target inpainted image, thereby reducing the loss of image details and texture and improving the accuracy of image flare removal.

[0184] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0185] Processor; and

[0186] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0187] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute a method of any of the above embodiments.

[0188] Figure 7 The electronic device structure in some embodiments of the present application is shown in FIG. Figure 7Electronic devices according to some embodiments of the present application are described.

[0189] Reference Figure 7 , the electronic device 7000 may include one or more of the following components: a processing component 7002 , a memory 7004 , a power component 7006 , a multimedia component 7008 , an audio component 7010 , an input / output (I / O) interface 7012 , a sensor component 7016 , and a communication component 7070 .

[0190] The processing component 7002 generally controls the overall operation of the electronic device 7000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 7002 may include one or more processors 7020 to execute instructions. In addition, the processing component 7002 may include one or more modules to facilitate interaction between the processing component 7002 and other components. For example, the processing component 7002 may include a multimedia module to facilitate interaction between the multimedia component 7008 and the processing component 7002. For another example, the processing component 7002 may read executable instructions from a memory to implement electronic device related functions.

[0191] The memory 7004 is configured to store various types of data to support operations on the electronic device 7000. Examples of such data include instructions for any application or method operating on the electronic device 7000, contact data, phone book data, messages, pictures, videos, etc. The memory 7004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0192] The power supply component 7006 provides power to various components of the electronic device 7000. The power supply component 7006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 7000.

[0193] The multimedia component 7008 includes a display screen that provides an output interface between the electronic device 7000 and the user. In some embodiments, the multimedia component 7008 includes a front camera and / or a rear camera. When the electronic device 7000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0194] The audio component 7010 is configured to output and / or input audio signals. For example, the audio component 7010 includes a microphone (MIC), and when the electronic device 7000 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 7004 or sent via the communication component 7070. In some embodiments, the audio component 7010 also includes a speaker for outputting audio signals.

[0195] I / O interface 7012 provides an interface between processing component 7002 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0196] The sensor assembly 7016 includes one or more sensors for providing various aspects of status assessment for the electronic device 7000. For example, the sensor assembly 7016 can detect the open / closed state of the electronic device 7000, the relative positioning of the components, such as the display and keypad of the electronic device 7000, and the sensor assembly 7016 can also detect the position change of the electronic device 7000 or a component of the electronic device 7000, the presence or absence of contact between the user and the electronic device 7000, the orientation or acceleration / deceleration of the electronic device 7000, and the temperature change of the electronic device 7000. The sensor assembly 7016 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 7016 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 7016 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0197] The communication component 7070 is configured to facilitate wired or wireless communication between the electronic device 7000 and other devices. The electronic device 7000 can access a wireless network based on a communication standard, such as Wi-Fi, 2G, 3G, 4G, 5G or 6G, or a combination thereof. In an exemplary embodiment, the communication component 7070 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 7070 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0198] In an exemplary embodiment, the electronic device 7000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0199] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the embodiments. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A method for image processing, characterized in that: The method comprises: Perform image segmentation on the original image to obtain a sub-image of the target area including the glare area; Performing glare erasing on the target area sub-image to obtain a glare-erased sub-image; Fusing the original image and the glare-erased sub-image to obtain a target fused image; Perform image repair on the target fused image to obtain a target repaired image.

2. The method according to claim 1, characterized in that The image segmentation is performed on the original image to obtain a sub-image of the target area including the glare area, including: Using a semantic segmentation model to segment the original image to obtain at least one target area sub-image; the semantic segmentation model is constructed based on a deep learning algorithm; Alternatively, at least one target area sub-image meeting a pixel threshold condition is segmented from the original image according to the pixel value of each pixel in the original image.

3. The method according to claim 1, characterized in that The step of performing glare erasing on the target area sub-image to obtain the glare-erased sub-image includes: Determining a glare category of the target area sub-image; The glare in the target area sub-image is removed according to the glare erasing mode set corresponding to the glare category to obtain the glare-erased sub-image.

4. The method according to claim 3, characterized in that The glare categories include: light source area, light band area and glare shining area; The removing of glare in the target area sub-image according to the glare erasing mode set corresponding to the glare category includes: If it is determined that the glare category is the light source area, each pixel value in the target area sub-image is set to a specified pixel value; If it is determined that the glare category is the light band area, a denoising model is used to erase the sub-image of the target area based on the image exposure of the sub-image of the target area; the light band area is constructed based on deep learning, and the erasing intensity of the sub-image of the target area is positively correlated with the image exposure; If it is determined that the glare category is the glaring area, a defogging model is used to adjust the contrast and saturation of the sub-image of the target area; the defogging model is constructed based on deep learning and is used to adjust the contrast and saturation of the image.

5. The method according to any one of claims 1 to 4, characterized in that: The fusing the original image and the glare-erased sub-image to obtain a target fused image includes: backfilling the flare-erased sub-image into the original image according to the mask of the flare-erased sub-image to obtain a target backfilled image; The target backfill image and the original image are channel-joined to obtain a target fused image.

6. The method according to any one of claims 1 to 4, characterized in that: The performing image repair on the target fused image comprises: Using an image patching model to patch the target fused image; the image patching model is constructed based on deep learning; Alternatively, the target fused image is repaired based on a texture synthesis technology; the texture synthesis technology is used to perform image repair by synthesizing background texture; Alternatively, the target fused image is repaired based on a pixel interpolation technique; the pixel interpolation technique is used to perform image repair in a pixel point interpolation manner.

7. The method according to claim 1, characterized in that The method further comprises: Performing weighted fusion on the original image and the target patched image to obtain a new target patched image; Alternatively, based on a neural network fusion method, the original image and the target repair image are fused to obtain a new target repair image; the neural network fusion method is based on a neural network to perform image fusion.

8. The method according to claim 1 or 7, characterized in that The method further comprises: Based on the light source in the original image, backfilling the light source of the target patch image to obtain a new target patch image; Alternatively, based on the light source in the candidate image, the light source is backfilled on the target patch image to obtain a new target patch image.

9. An image processing device, characterized in that: The device comprises: A segmentation unit, used for performing image segmentation on the original image to obtain a sub-image of a target area including a glare area; a removal unit, configured to perform glare erasure on the target area sub-image to obtain a glare-erased sub-image; A fusion unit, used for fusing the original image and the glare-erased sub-image to obtain a target fused image; The patching unit is used to perform image patching on the target fused image to obtain a target patched image.

10. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 8.

Images