Image processing method, device, storage medium and electronic device

Abstract

The embodiments of the present invention provide an image processing method, device, storage medium and electronic device. The method includes: separating a mask image of a region to be enhanced from an original image; obtaining a boundary map of the mask image; filtering the mask image with the boundary map as a guide to obtain a weighted mask image; separating a sub-image corresponding to the mask image from the original image, and enhancing the sub-image; fusing the original image and the enhanced sub-image according to the weight value of each pixel in the weighted mask image. In the present invention, a mask image is separated from the original image, the original image is enhanced by the mask image, and then the enhanced image is fused according to the weight and boundary information of the mask image, thereby effectively solving the problem of poor effects of some scenes shot due to hardware limitations of the camera equipment.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] Due to the physical characteristics of the shooting equipment, such as dynamic range and noise, the image quality is often not ideal. Some areas of these images need to be enhanced, while others need to be weakened. For example, the blue sky outside can be processed to be bluer and the grass to be greener. In some scenes, such as outdoor backlighting, the dark scene is not ideal even with High Dynamic Range (HDR) processing.

[0003] Therefore, how to effectively solve the problem of poor shooting effects of some scenes due to hardware limitations such as lenses or sensors of camera equipment is a technical problem currently existing in the field of image processing. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method, a device storage medium and an electronic device to at least solve the problem in the related art that some scenes captured are poor in effect due to limitations on the physical characteristics of the shooting equipment.

[0005] According to one embodiment of the present invention, there is provided an image processing method, the method comprising: separating a mask image of a region to be enhanced from an original image (i.e., a mask image, an image obtained by binarizing a portion of image data); obtaining a boundary map of the mask image; filtering the mask image with the boundary map as a guide to obtain a weighted mask image; separating a sub-image corresponding to the mask image from the original image, and enhancing the sub-image; and fusing the original image and the enhanced sub-image according to the weight value of each pixel point of the weighted mask image.

[0006] According to another embodiment of the present invention, an image processing device is provided, which includes a memory and a processor, and the processor performs the following operations by running a program stored in the memory: separating a mask image of a region to be enhanced from an original image; obtaining a boundary map of the mask image; filtering the mask image based on the boundary map to obtain a weighted mask image; separating a sub-image corresponding to the mask image from the original image, and enhancing the sub-image; and fusing the original image and the enhanced sub-image according to the weight value of each pixel in the weighted mask image.

[0007] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] In the above-mentioned embodiment of the present invention, a mask image is separated from an original image, the original image is enhanced by means of the mask image, and then the enhanced image is fused according to the weight and boundary information of the mask image, thereby effectively solving the problem of poor shooting effects of some scenes due to the hardware limitations of the camera equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of the hardware structure of a mobile terminal according to an embodiment of the present invention;

[0011] Figure 2 is a flowchart of an image processing method according to an embodiment of the present invention;

[0012] Figure 3 is a structural block diagram of an image processing device according to an embodiment of the present invention;

[0013] Figure 4 is a schematic structural diagram of an image processing device according to another embodiment of the present invention;

[0014] Figure 5 is a flowchart of an image processing method according to another embodiment of the present invention;

[0015] Figure 6 is a schematic diagram of a Mask for separating an enhanced area according to an embodiment of the present invention;

[0016] Figure 7 is a schematic diagram of obtaining the boundary of a Mask according to an embodiment of the present invention;

[0017] Figure 8 It is a schematic diagram of the effect after the original image and the enhanced image are fused according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0020] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal, a smart watch, a camera or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal in which the method embodiment of the present invention is executed. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0021] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the method steps in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0022] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0023] Example 1

[0024] This embodiment provides an image processing method that can be run in the above hardware device. Figure 2 is a flowchart of an image processing method according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0025] Step S201, separating a mask image of a region to be enhanced from an original image;

[0026] Step S202, obtaining a boundary map of the mask image;

[0027] Step S203, filtering the mask image with the boundary image as a guide to obtain a weighted mask image;

[0028] Step S204, separating a sub-image corresponding to the mask image from the original image, and performing enhancement processing on the sub-image;

[0029] Step S205 , fusing the original image and the enhanced sub-image according to the weight value of each pixel point of the weight mask image.

[0030] In step S201 of this embodiment, in order to increase the calculation speed, the mask image may be reduced to reduce the amount of calculation.

[0031] In this embodiment, after step S201, the mask image may be repaired by dilation and / or corrosion.

[0032] In step S202 of this embodiment, the mask image may be dilated and eroded respectively, and a subtraction operation is performed between the mask image after the dilation process and the mask image after the erosion process to obtain a boundary map of the mask image.

[0033] In step S204 of this embodiment, the separated sub-images may be enhanced, that is, the useful information in the image may be enhanced to improve the visual effect of the sub-images, for example, contrast enhancement, color enhancement, noise reduction, etc. may be performed on the sub-images.

[0034] In step S205 of this embodiment, the weight mask image may be enlarged to a size corresponding to the original image before image fusion processing is performed.

[0035] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0036] In this embodiment, an image processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0037] Example 2

[0038] Figure 3 is a structural block diagram of an image processing device according to an embodiment of the present invention. Figure 3 As shown, the image processing device includes a memory 10 and a processor 20. The processor 20 performs the following operations by running a program stored in the memory:

[0039] Mask separation: Separate the mask image of the area to be enhanced from the original image;

[0040] Boundary acquisition: acquiring a boundary map of the mask image;

[0041] Mask filtering: filtering the mask image based on the boundary image to obtain a weighted mask image;

[0042] Image enhancement: separating a sub-image corresponding to the mask image from the original image, and performing enhancement processing on the sub-image;

[0043] Image fusion: fusing the original image and the enhanced sub-image according to the weights of the weight mask image.

[0044] Example 3

[0045] Figure 4 is a structural block diagram of an image processing device according to another embodiment of the present invention. Figure 4 As shown, the processor in the device executes Figure 3 In addition to all the operations shown, you can also perform mask inpainting, image reduction, and image enlargement.

[0046] Mask repair: Before the acquisition module acquires the boundary map of the mask image, the mask image is repaired by dilation and / or corrosion.

[0047] Image reduction: reducing the mask image separated from the original image to a set size;

[0048] Image enlargement: enlarge the weight mask image obtained after filtering to a size corresponding to the original image.

[0049] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above modules can all be located in the same processor; or, the above modules can also be located in different processors in any combination.

[0050] In order to facilitate the understanding of the technical solution provided by the present invention, the following will be described in detail with reference to embodiments of specific scenarios.

[0051] Example 4

[0052] The present embodiment provides an image processing method. In the present embodiment, the sub-region mask information can be extracted by image feature separation technology and other methods, and the grayscale guide map is separated from the image processing. After that, the grayscale guide map is subjected to boundary filtering to obtain the boundary map of the guide map, and then the boundary map is enhanced with a certain radius to expand the boundary area. In the next step, the enhanced boundary map is blurred to obtain the blurred guide map, solve the boundary problem, and finally the original image is enhanced with the blurred guide map as the guide, and the grayscale value of the blurred guide map is used as the enhanced weight to enhance the original image. In order to improve the calculation speed, the Mask map can be taken at a small resolution to reduce the amount of calculation. After the above steps are completed, it is enlarged to the size of the original image for image fusion. It should be noted that in the present embodiment, the Mask map, the grayscale guide map and the guide map all represent the same concept.

[0053] like Figure 5 As shown, the image processing method of this embodiment may include the following steps:

[0054] Step S501: Separate the area to be enhanced and obtain a small-sized grayscale-guided Mask image of the area.

[0055] Specifically, in this embodiment, the original image of size (W, H) is subjected to preliminary separation processing to separate the area to be enhanced, that is, the area to be enhanced is separated from the background area to obtain a guide mask image of grayscale value and small size (w, h). Figure 6 As shown, Figure 6 The sub-region to be enhanced in the right box is identified and separated to obtain the Mask map of the large tree.

[0056] Step S502: Perform boundary repair on the separated Mask image to obtain Mask rep .

[0057] Specifically, in this embodiment, when the separated Mask boundary is inaccurate or there is a hole in the middle, the Mask image may be further repaired. For example, the Mask image may be closed as follows to obtain a repaired Mask image:

[0058] Mask dila =Dilater(Mask,element1) (1)

[0059] Mask rep =Erode(Mask dila ,element2) (2)

[0060] Among them, Dilater is the dilation process, Mask is the image to be dilated, element1 represents the kernel of the dilation process (or called convolution template), Mask dila The image is obtained by dilating the Mask using the convolution template element1; Erode is the erosion process, Mask dila is the image to be eroded, element2 represents the erosion kernel (or convolution template), Mask rep To use the convolution template element2 to Mask dila Image after erosion.

[0061] In this embodiment, the above expansion and erosion can be repeated multiple times to obtain a more complete Mask image.

[0062] Step S503: Mask rep Perform boundary-oriented processing to obtain the boundary map Mask of Mask edge .

[0063] Specifically, the boundary map of Mask can be obtained according to the following operation: edge :

[0064] Mask edge = dilate(Mask rep ,element1)-erode(Mask rep ,element2)

[0065] That is, for Mask repThe dilation and erosion processes are performed on the image, and the mask image after the dilation process is subtracted from the mask image after the erosion process to obtain the following: Figure 7 The boundary map Mask of the big tree Mask shown edge .

[0066] Step S504: Mask edge Mask rep Perform filtering to obtain Mask weight .

[0067] Specifically, in this embodiment, Mask edge Mask rep Perform low-pass filtering to obtain Mask weight In this step, the boundary part of the Mask image is mainly filtered, and the filtering algorithm used includes but is not limited to the following boxfilter algorithm:

[0068] Kernel is denoted as F(x,y), and its size is (2r+1)*(2r+1), where r can be adjusted. Let K=2r+1,

[0069]

[0070] Elements in F(x,y): x i,j ,(i,j≤K), can be obtained by the following formula:

[0071]

[0072] Among them, δ is an adjustable parameter.

[0073] Mask weight The calculation is as follows:

[0074]

[0075] Step S505: Use the mask image obtained in step S501 to separate the sub-image (subimg) of the area to be enhanced, and perform enhancement processing on the separated sub-image. For example, the following contrast enhancement method can be used:

[0076] enhanced(i,j)=k*log(1+subimg(i,j))

[0077] Among them, K is an adjustable parameter. After the above processing, it is normalized to the normal grayscale range to obtain the enhanced image:

[0078]

[0079] It should be noted that, in addition to the contrast enhancement of this embodiment, other forms of enhancement processing, such as color enhancement, noise reduction processing, etc. are also applicable.

[0080] Step S506: Mask weight Upscale to get upMask weight .

[0081] Specifically, in this embodiment, Mask weight The image is enlarged (upscaled) to the original image size (W, H) to obtain the upMask of the original image size weight .

[0082] Step S507, according to upMask weight The weights of each point in the image fuse the original image and the enhanced sub-image.

[0083] Specifically, in this embodiment, according to upMask weight The weights of each point in the original image (srcimg) and the enhanced sub-image (enSubimg) are fused as follows to obtain the processed image:

[0084] dstimg(i,j)=(1-upMask weight (i,j))*srcimg(i,j)+upMask weight (i,j)*enSubimg(i,j)

[0085] Among them, the effect after fusion processing is as follows Figure 8 shown.

[0086] In this embodiment, when there are multiple images in the original image that need to be enhanced, a Mask image containing multiple sub-areas is generated, and then the Mask is processed in the same way as above, and different optimization parameters are selected to finally obtain the same optimization effect.

[0087] In this embodiment, the guide graph can be separated through features such as neural networks or chromaticity data, and the original image can be enhanced through the guide graph. The enhanced image is fused according to the location and boundary information of the guide graph, which can solve the above problems well. With the application of artificial intelligence methods in image processing, such as semantic segmentation of images, some specific sub-images can be separated from the original image, and the separated images can be processed and then fused back into the original image, which can effectively solve the problem of poor shooting effects of some scenes due to limitations of the lens, sensor, etc. of the camera equipment.

[0088] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0089] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0090] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0091] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0092] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0093] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: A mask image is used to separate the area to be enhanced from the original image; Acquire a boundary map of the mask image; Performing filtering processing on the mask image based on the boundary image to obtain a weighted mask image; Separating a sub-image corresponding to the mask image from the original image, and performing enhancement processing on the sub-image; The original image and the enhanced sub-image are fused according to the weight value of each pixel in the weight mask image.

2. The method according to claim 1, characterized in that Before obtaining the boundary map of the mask map, the method further includes: The boundary or the hole of the mask image is repaired by at least one of the following processes: dilation processing and corrosion processing.

3. The method according to claim 1, characterized in that Acquiring the boundary map of the mask image includes: The mask image is respectively subjected to dilation processing and erosion processing, and a subtraction operation is performed between the mask image after the dilation processing and the mask image after the erosion processing to obtain a boundary image of the mask image.

4. The method according to claim 1, characterized in that After separating the mask image of the area to be enhanced from the original image, it also includes: Reduce the mask image to a set size.

5. The method according to claim 4, characterized in that Before fusing the original image and the enhanced sub-image based on the weight of the weight mask image, the method further includes: The weight mask image is enlarged to a size corresponding to the original image.

6. An image processing device, characterized in that: The system comprises a memory and a processor, wherein the processor executes the following operations by running a program stored in the memory: A mask image is used to separate the area to be enhanced from the original image; Acquire a boundary map of the mask image; Performing filtering processing on the mask image based on the boundary image to obtain a weighted mask image; Separating a sub-image corresponding to the mask image from the original image, and performing enhancement processing on the sub-image; The original image and the enhanced sub-image are fused according to the weight value of each pixel in the weight mask image.

7. The device according to claim 6, characterized in that The processor is also used to perform the following operations: The mask image separated from the original image is reduced to a set size.

8. The device according to claim 7, characterized in that The processor is also used to perform the following operations: Before performing the fusion operation of the original image and the enhanced sub-image, the weight mask image obtained after filtering is enlarged to a size corresponding to the original image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 5 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 5 are implemented.

Images