Image processing method, device, electronic device and computer readable storage medium

Abstract

The present invention provides an image processing method, device, electronic device and computer-readable storage medium, the method comprising: obtaining multiple foreground focus images and multiple background focus images obtained by an image acquisition device for the same scene; generating a clear foreground image based on the multiple foreground focus images; generating a clear background image based on the multiple background focus images; fusing the clear foreground image and the clear background image to obtain a clear image of the scene. The present invention can better balance the foreground clarity and the background clarity, and effectively improve the image shooting effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] Image acquisition devices such as mobile phones and cameras often have the problem of balancing the clarity of the foreground and background when taking photos. Take portrait photography as an example. Either the portrait is clear but the background is blurred, or the background is clear but the portrait is blurred. The clarity of the portrait and the background cannot be met at the same time. This problem is particularly obvious on mobile phones and other devices. Even autofocus cameras will have the above problem due to the limited depth of field of the lens, resulting in poor shooting effects. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image processing method, device, electronic device and computer-readable storage medium, which can better balance the foreground clarity and background clarity and effectively improve the image shooting effect.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, comprising: acquiring multiple foreground focus images and multiple background focus images taken by an image acquisition device for the same scene; generating a clear foreground image based on the multiple foreground focus images; generating a clear background image based on the multiple background focus images; and fusing the clear foreground image and the clear background image to obtain a clear image of the scene.

[0006] Furthermore, the exposure values ​​of different foreground focused images are the same; the step of generating a clear foreground image based on the multiple foreground focused images includes: using a multi-frame noise reduction algorithm to perform noise reduction and synthesis processing on the multiple foreground focused images, and using the processed image as a clear foreground image.

[0007] Furthermore, different background focused images have different exposure values; the step of generating a clear background image based on the multiple background focused images includes: synthesizing the multiple background focused images using an HDR algorithm, and using the synthesized image as a clear background image.

[0008] Furthermore, the step of fusing the clear foreground image and the clear background image to obtain a clear image of the scene includes: based on the clear foreground image and the clear background image, segmenting an object map of the foreground object in the scene and a background map of the background environment excluding the foreground object; optimizing the object map to obtain a processed object map; the optimization processing includes one or more of brightness adjustment operations, face beautification operations, detail enhancement operations, light and shadow adjustment operations, and noise reduction operations; fusing the processed object map with the background map to obtain a clear image of the scene.

[0009] Furthermore, the step of segmenting an object map of the foreground object in the scene and a background map of the background environment excluding the foreground object based on the clear foreground image and the clear background image includes: fusing the clear foreground image and the clear background image to obtain a fused image, and segmenting the object map of the foreground object in the scene and the background map of the background environment excluding the foreground object from the fused image; or, segmenting the object map of the foreground object in the scene from the clear foreground image, and obtaining a background map of the background environment excluding the foreground object in the scene based on the clear background image.

[0010] Furthermore, the step of obtaining a background map of the background environment in the scene excluding the foreground object based on the clear background image includes: segmenting a first background image of the background environment excluding the foreground object from the clear foreground image; segmenting a second background image of the background environment excluding the foreground object from the clear background image; and fusing the first background image and the second background image to obtain a background map of the background environment in the scene excluding the foreground object.

[0011] Further, the object image includes a face image and / or a portrait image; the step of optimizing the object image to obtain a processed object image includes: determining the lighting environment in which the foreground object is located based on the brightness difference between the object image and the background image; the lighting environment includes a front-lit environment, a side-lit environment or a back-lit environment; if the object image includes a face image, performing a brightness adjustment operation on the face image according to the lighting environment in which the foreground object is located to obtain a processed face image; if the object image includes a portrait image, performing a brightness adjustment operation on the portrait image according to the lighting environment in which the foreground object is located to obtain a processed portrait image.

[0012] Furthermore, the step of fusing the processed object image with the background image to obtain a clear image of the scene includes: performing sharpening and / or brightness processing on the background image to obtain a processed background image; and fusing the processed object image with the processed background image to obtain a clear image of the scene.

[0013] In a second aspect, an embodiment of the present invention further provides an image processing device, comprising: an image acquisition module, used to acquire multiple foreground focus images and multiple background focus images captured by an image acquisition device for the same scene; a foreground image generation module, used to generate a clear foreground image based on the multiple foreground focus images; a background image generation module, used to generate a clear background image based on the multiple background focus images; and an image fusion module, used to fuse the clear foreground image and the clear background image to obtain a clear image of the scene.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a storage device; the storage device stores a computer program, and when the computer program is executed by the processor, it executes the method as described in any one of the first aspects.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any one of the above-mentioned first aspects are executed.

[0016] The embodiments of the present invention provide an image processing method, device, electronic device and computer-readable storage medium, which first obtains multiple foreground focus images and multiple background focus images taken by an image acquisition device for the same scene, then generates a clear foreground image based on the multiple foreground focus images, and generates a clear background image based on the multiple background focus images, and finally fuses the clear foreground image and the clear background image to obtain a clear image of the scene. This method generates a clear foreground image and a clear background image respectively by acquiring multiple foreground focus images and multiple background focus images, and then fuses them to obtain a clear image. The final clear image can take into account both foreground clarity and background clarity, effectively improving the image shooting effect.

[0017] Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown;

[0021] Figure 2 A flow chart of an image processing method provided by an embodiment of the present invention is shown;

[0022] Figure 3 A flow chart of another image processing method provided by an embodiment of the present invention is shown;

[0023] Figure 4 A flow chart of another image processing method provided by an embodiment of the present invention is shown;

[0024] Figure 5 A structural block diagram of an image processing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be described below in conjunction with the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0026] Taking into account the problem in the prior art that foreground clarity and background clarity cannot be satisfied at the same time, in order to improve this problem, an embodiment of the present invention provides an image processing method, device, electronic device and computer-readable storage medium. This technology can be applied to any occasion where taking pictures is required. The embodiments of the present invention are introduced in detail below.

[0027] Embodiment 1:

[0028] First, refer to Figure 1 An example electronic device 100 for implementing an image processing method, apparatus, electronic device, and computer-readable storage medium according to an embodiment of the present invention is described.

[0029] like Figure 1The electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, and an image acquisition device 110. These components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0030] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 102 can be a central processing unit (CPU), a graphics processing unit (GPU), or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0031] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0032] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0033] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0034] The image acquisition device 110 can capture images desired by the user (eg, photos, videos, etc.), and store the captured images in the storage device 104 for use by other components.

[0035] Exemplarily, an example electronic device for implementing the image processing method, apparatus, electronic device, and computer-readable storage medium according to the embodiments of the present invention may be implemented as an intelligent terminal such as a smart phone, a camera, a computer, or the like.

[0036] Embodiment 2:

[0037] This embodiment provides an image processing method, which can be executed by an image processing device such as a mobile phone, a camera, etc., that is, the image processing device directly executes the image processing method after capturing an image, and provides the user with the processed image as the final captured image. The method can also be executed by a device with image processing capabilities such as a computer, for example, a computer can receive an image to be processed transmitted by an image acquisition device or an image to be processed uploaded by a user, and execute the image processing method. Figure 2 The image processing method flow chart shown mainly includes the following steps S202 to S208:

[0038] Step S202: acquiring a plurality of foreground focus images and a plurality of background focus images captured by an image acquisition device for the same scene.

[0039] When shooting the same scene, the image acquisition device can perform a foreground focus operation to obtain multiple foreground focus images, and perform a background focus operation to obtain multiple background focus images. When performing a focus operation, the focus point can be manually selected for focusing, or the image acquisition device can achieve automatic focus, such as using face detection technology to detect a face frame in the shooting picture, and determining a focus point based on the face frame for focusing. Assuming that there are multiple people in the shooting scene, that is, multiple face frames are detected, the focus operation can be performed with the largest face frame as the focus point. In actual applications, the exposure values ​​of different foreground focus images are the same or different, and the exposure values ​​of different background focus images are the same or different.

[0040] Step S204: generating a clear foreground image according to the multiple foreground focused images.

[0041] In one embodiment, the exposure values ​​of different foreground focused images are the same, usually normal exposure; for example, multiple foreground focused images are captured with exposure values ​​of EV0, where EV (Exposure Values) reflects the combination of aperture size and shutter speed, and each increase of 1 in the exposure value will change the exposure by one stop, that is, the exposure amount is halved, which can be achieved by changing the exposure time or aperture area. When generating a clear foreground image based on multiple foreground focused images, a multi-frame noise reduction algorithm can be used to perform noise reduction and synthesis processing on the multiple foreground focused images, and the image obtained after processing is used as a clear foreground image. In this way, a clear foreground image with noise removed can be obtained. It should be understood that since the clear foreground image is generated based on multiple foreground focused images, the foreground in the image is clearer than the background, and the background may be blurry relative to the foreground.

[0042] Step S206, generating a clear background image according to the multiple background focus images.

[0043] In one embodiment, different background focus images have different exposure values, usually including normal exposure and underexposure; for example, three background focus images with exposure values ​​of EV0, EV-2, and EV-4 can be taken, or three background focus images with exposure values ​​of EV 0, EV-3, and EV-6 can be taken, or multiple background focus images with other exposure values ​​can be taken. In practical applications, the exposure value of each image can be adjusted according to the light intensity in the shooting scene, which is not limited here. When a clear background image is generated based on multiple background focus images, an HDR (High-Dynamic Range) algorithm can be used to synthesize the multiple background focus images, and the synthesized image is used as a clear background image. The HDR algorithm can be used to achieve a larger exposure dynamic range (i.e., a larger difference in light and dark) than ordinary image technology, which can provide more dynamic range and image details. In this way, a clear background image with rich background details can be obtained, and the details of the highlight area and the details of the non-highlight area in the background focus image are retained. It should be understood that, since the clear background image is generated based on a plurality of background focus images, the background in the image is clearer than the foreground, and the foreground may be blurry compared to the background.

[0044] Step S208: Fusing the clear foreground image and the clear background image to obtain a clear image of the above scene. Fusing the clear foreground image and the clear background image can obtain a clear image with both a clear foreground and a clear background.

[0045] It should be noted that the above method does not Figure 2The specific order is a limitation, for example, step S204 and step S206 can be performed simultaneously, or step S206 can be performed first and then step S204. In other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted.

[0046] The above-mentioned image processing method provided by the embodiment of the present invention generates a clear foreground image and a clear background image respectively by collecting multiple foreground focus images and multiple background focus images, thereby fusing them to obtain a clear image. The final clear image can take into account both foreground clarity and background clarity, thereby effectively improving the image shooting effect.

[0047] In some embodiments, the step of fusing the clear foreground image and the clear background image to obtain a clear image of the scene may be performed with reference to the following steps 1 to 3:

[0048] Step 1: Based on the clear foreground image and the clear background image, an object map of the foreground object in the scene and a background map of the background environment excluding the foreground object are segmented.

[0049] In practical applications, the object map can be an image of the target object in the foreground (i.e., the foreground object), and the foreground object can also be understood as an object of interest in the shooting scene; for example, when the foreground object is a person in the shooting scene, the object map is a portrait. Of course, the object map can be not only a portrait, but also only include a local area of ​​the target object, such as a face map. If the foreground object is a person, in order to be able to perform targeted processing on the face and the portrait respectively during the image processing process, so as to obtain a foreground portrait with a better shooting effect, the object map can include both a face map and a portrait map, that is, the portrait map and the face map are respectively segmented from the image for subsequent corresponding processing. Optionally, one of the images in the portrait map and the face map can be selected as the object map, for example, the portrait map is used as the object map and the portrait map is processed, or the face map is used as the object map and the face map is processed. This embodiment provides two segmentation methods, which can be specifically referred to as the following method one and method two:

[0050] Method 1: A clear foreground image and a clear background image are fused to obtain a fused image, and an object map of the foreground object in the scene and a background map of the background environment excluding the foreground object are segmented from the fused image.

[0051] In one embodiment, a guided filtering algorithm can be used to fuse the clear foreground image and the clear background image. The guided filtering algorithm (also known as the guide filter algorithm) can effectively select the clear image in the image to be fused for fusion, so that the fused image has both the clear foreground object in the clear foreground image and the clear background environment in the clear background image. In addition, the guided filtering algorithm can effectively maintain the edge characteristics of the image when fusing the images, and can also make the obtained fused image have an image enhancement effect. Method 1 First, the clear foreground image and the clear background image are fused to obtain a fused image. The fused image has a clear foreground and a clear background, but there may still be problems such as insufficient foreground brightness. Therefore, it is necessary to segment the object image and the background image from the fused image so that the object image can be processed later, such as brightening the portrait or beautifying the face. It can be understood that the fused image is clear, so the segmented object image and background image are both clear.

[0052] Method 2: Segment the object map of the foreground object in the scene from the clear foreground image, and obtain the background map of the background environment in the scene except the foreground object based on the clear background image. Method 2 is implemented by segmenting the two images separately.

[0053] The embodiment of the present invention provides an implementation method for obtaining a background image of a background environment other than a foreground object in a scene based on a clear background image. First, a first background image of the background environment other than the foreground object is segmented from the clear foreground image, and then a second background image of the background environment other than the foreground object is segmented from the clear background image, so that the first background image and the second background image are fused to obtain a background image of the background environment other than the foreground object in the scene. The first background image is also the background image segmented from the clear foreground image. If the foreground object is a portrait, the portrait in the clear foreground image can be detected using a portrait detection algorithm, and then the non-portrait area in the clear foreground image is used as the first background image; the second background image is also the background image segmented from the clear background image. Similarly, the portrait in the clear background image can be detected using a portrait detection algorithm, and then the non-portrait area in the clear background image is used as the second background image. After the first background image and the second background image are segmented, the image after the first background image and the second background image are fused can be used as the background image of the background environment other than the foreground object in the scene.

[0054] Optionally, an embodiment of the present invention further provides another implementation method for obtaining a background map of the background environment excluding the foreground object in the scene based on a clear background image, segmenting an object map of the foreground object in the scene from the clear foreground image, and segmenting a background map of the background environment excluding the foreground object from the clear background image. In practical applications, a clear object map can be segmented from a clear foreground image, and a clear background map can be segmented from a clear background image. When segmenting the background map, a portrait detection algorithm can be first used to detect the portrait, and then the non-portrait area can be segmented as the background environment to obtain the background map.

[0055] Regardless of the first or second method, when the object image includes a portrait image and a face image, a portrait detection algorithm and a face detection algorithm can be used to implement portrait image segmentation and face image segmentation respectively, which will not be described in detail here.

[0056] Step 2, optimizing the object graph to obtain a processed object graph; the optimization processing includes one or more of brightness adjustment operation, face beautification operation, detail enhancement operation, light and shadow adjustment operation, and noise reduction operation.

[0057] This embodiment provides a specific implementation method for performing a brightness adjustment operation on an object image. First, the lighting environment in which the foreground object is located can be determined based on the brightness difference between the object image and the background image. The lighting environment includes a front-lit environment, a side-lit environment, or a back-lit environment. Then, if the object image includes a face image, a brightness adjustment operation is performed on the face image according to the lighting environment in which the foreground object is located to obtain a processed face image. If the object image includes a portrait image, a brightness adjustment operation is performed on the portrait image according to the lighting environment in which the foreground object is located to obtain a processed portrait image.

[0058] This embodiment provides a simple and quick determination method for determining the light environment based on the brightness difference, wherein the brightness difference between the object image and the background image can be reflected in the form of a ratio. When the ratio is greater than the first threshold, it indicates that the light environment in which the foreground object is located is a backlight environment. When the ratio is between the second threshold and the first threshold, the light environment in which the foreground object is located is a sidelight environment. When the ratio is less than the second threshold, the light environment in which the foreground object is located is a frontlight environment. After determining the light environment, the brightness of the face image and the portrait image can be adjusted accordingly. The brightness compensation amounts of the face image and the portrait image can be the same or different, and differential adjustments can be made as needed.

[0059] In addition, different optimization processes can be performed on the face image and the portrait image, such as performing brightness adjustment, face beautification, detail enhancement, light and shadow adjustment, and noise reduction on the face image, and only performing brightness adjustment on the portrait image. In actual applications, different optimization processes can be selected for the face image and the portrait image according to needs, and there is no limitation here.

[0060] Step 3: Fuse the processed object image with the background image to obtain a clear image of the scene.

[0061] In one embodiment, the processed object image can be directly fused with the background image obtained above to obtain a clear image of the shooting scene, such as directly covering the processed object image on the area where the object is located on the background image to obtain a clear image of the scene. In another embodiment, the background image can be further sharpened and / or brightness processed to obtain a processed background image; then the processed object image is fused with the processed background image to obtain a clear image of the scene. In this embodiment, the background image is subsequently processed to obtain a background image with better effect, and then the processed object image and the processed background image are fused to further improve the shooting effect.

[0062] Based on the above image processing method, this embodiment further provides two specific examples of applying the above image processing method, which are described in detail as follows:

[0063] Example 1:

[0064] See also Figure 3 The flowchart of an image processing method shown mainly includes the following steps S301 to S316:

[0065] Step S301: Execute a portrait focus operation to obtain three portrait focus images (ie, the foreground focus images mentioned above) with exposure values ​​all being EV0, and then execute step S303.

[0066] Step S302: Execute a background focus operation to obtain three background focus images with exposure values ​​of EV0, EV-2 and EV-4 respectively, and then execute step S304.

[0067] Step S303: performing noise reduction and synthesis processing on the three portrait focus images to obtain a portrait focus image after noise reduction, wherein the portrait focus image after noise reduction is the aforementioned clear foreground image, and then executing step S305.

[0068] Step S304: Perform HDR synthesis processing on the three background focus images to obtain an HDR image, wherein the HDR image is the aforementioned clear background image, and then execute step S305.

[0069] Step S305: Perform guided filtering fusion on the denoised portrait focus image and the HDR image to obtain a fused image, and then execute steps S306, S307 and S308 respectively.

[0070] Step S306: Perform face segmentation based on the fused image to obtain a face map. In specific implementation, a face detection algorithm can be used to detect the face in the fused image, and then the face map is extracted, and then step S309 is executed.

[0071] Step S307: performing portrait segmentation based on the fused image to obtain a portrait image; in specific implementation, a portrait detection algorithm may be used to detect the portrait subject in the fused image, and then the portrait image is extracted, and then step S312 is executed.

[0072] Step S308: Enhance the background image in the fused image to obtain a processed background image, and then execute step S315. The enhancement process includes sharpening, brightness processing, etc. The background image can be a background environment image other than the face image and the portrait image in the fused image.

[0073] Step S309: Determine whether the face is taken against the light. If so, execute step S310: process the backlit face image to obtain a processed face image. If not, execute step S311: process the non-backlit face image to obtain a processed face image. Regardless of executing step S310 or step S311, step S315 is executed afterwards.

[0074] Step S312: Determine whether the portrait is taken against light. If yes, execute step S313: process the backlight portrait to obtain a processed portrait. If no, execute step S314: process the non-backlight portrait to obtain a processed portrait. No matter whether step S313 or step S314 is executed, step S315 is executed afterwards.

[0075] Step S315: perform an image fusion operation and use the fused image as the final captured image, that is, fuse the processed face image, the processed portrait image and the processed background image to obtain the final captured image.

[0076] Step S316: Output the final captured image.

[0077] In the above example, the portrait focus image that has undergone multi-frame denoising is fused with the background focus image that has been synthesized by HDR. The resulting fused image has a clear portrait and a clear background environment, and the highlight suppression is better after guided filtering fusion. However, there may be problems such as poor brightness of the face and portrait due to the lighting environment. Therefore, portrait segmentation, face segmentation and other operations will be further performed to facilitate subsequent targeted optimization processing.

[0078] See also Figure 4 The flowchart of an image processing method shown in FIG. 1 mainly includes the following steps:

[0079] Step S401: Execute a portrait focus operation to obtain three portrait focus images (ie, the foreground focus images mentioned above) with exposure values ​​all being EV0, and then execute step S403.

[0080] Step S402: Execute a background focus operation to obtain three background focus images with exposure values ​​of EV0, EV-2 and EV-4 respectively, and then execute step S404.

[0081] Step S403: Perform noise reduction and synthesis processing on the three portrait focus images to obtain a portrait focus image after noise reduction. The portrait focus image after noise reduction is the clear foreground image mentioned above, and then perform steps S405 and S406;

[0082] Step S404: Perform HDR synthesis processing on the three background focus images to obtain an HDR image. The HDR image is the clear background image mentioned above, and then step S413 is performed:

[0083] Step S405: performing portrait segmentation on the portrait focused image after noise reduction to obtain a portrait image and a first background image, executing step S407 for the portrait image, and executing step S414 for the first background image;

[0084] Step S406: performing face segmentation on the portrait focus image after noise reduction to obtain a face map; then executing step S410;

[0085] Step S407: Determine whether the portrait is taken against light. If yes, execute step S408: process the backlight portrait to obtain a processed portrait. If no, execute step S409: process the non-backlight portrait to obtain a processed portrait. Regardless of whether step S408 or step S409 is executed, step S415 is executed afterwards.

[0086] Step S410: determine whether the face is backlit, if yes, execute step S411: process the backlit face image to obtain a processed face image; if no, execute step S412: process the non-backlit face image to obtain a processed face image. No matter whether step S411 or step S412 is executed, step S415 is executed afterwards.

[0087] Step S413: Segment the second background image consisting of the non-portrait area from the HDR image. Then execute step S414. In one embodiment, the HDR image can be processed using a portrait segmentation algorithm, and it is determined whether the processed image is a non-portrait area. When it is determined that the processed image is a non-portrait area, step S414 is executed, that is, the portrait segmentation method can be used to detect the portrait area in the HDR image, and the non-portrait area in the HDR image is used as the second background image and step S414 is executed for the second background image.

[0088] Step S414, performing an image fusion operation to fuse the first background image and the second background image to obtain a background image, and then executing step S415.

[0089] Step S415: perform an image fusion operation and use the fused image as the final captured image, that is, fuse the processed face image, the processed portrait image and the background image to obtain the final captured image.

[0090] Step S416: Output the final captured image.

[0091] In the above example, portrait segmentation and face segmentation operations are performed directly on the multi-frame denoised portrait focus image, and the segmented portraits and faces are optimized according to the lighting environment. The optimized portraits and faces are merged with the clear background synthesized by HDR technology, and finally an image with good foreground (portrait, face) and background is obtained.

[0092] It should be noted that the above method does not Figure 3 and Figure 4 The specific order described is a limitation. The above flowchart does not limit the order of execution of the steps. The order of some steps in the above method can be interchanged according to actual needs, or some steps can be omitted or deleted.

[0093] In summary, the above-mentioned image processing method provided in this embodiment can be well applied to image acquisition devices such as mobile phones, and can take into account the clarity of the foreground and the clarity of the background at the same time. Through multi-frame denoising, HDR technology and the fusion of clear foreground and clear background, the image noise is effectively reduced and the image details are retained as much as possible, such as retaining the details of the highlight area while retaining the details of the non-highlight area. At the same time, through optimization processing operations such as brightness adjustment, it can improve the problems of the background being easily overexposed or the portrait being insufficiently bright due to the dynamic range limitation of the camera sensor when the foreground such as the portrait accounts for a large proportion in the prior art. In addition, the portrait, face, and background environment can be separated out for targeted optimization processing, such as performing face beautification operations, light and shadow adjustment operations on the face, and sharpening operations on the background. Even for the same operation, the operating parameters of the portrait, face, and background environment can be different, such as when performing the brightness adjustment operation, the brightness compensation amount required for the face, portrait, and background environment is different. Through the above method, the captured image finally presented to the user can be more in line with the user's wishes, and the user experience is effectively improved.

[0094] Embodiment three:

[0095] For the image processing method provided in the second embodiment, the present invention provides an image processing device, see Figure 5 The structure block diagram of an image processing device shown in FIG. 1 includes the following modules:

[0096] An image acquisition module 52 is used to acquire multiple foreground focus images and multiple background focus images captured by an image acquisition device for the same scene;

[0097] A foreground image generation module 54, used to generate a clear foreground image according to a plurality of foreground focus images;

[0098] A background image generation module 56, used to generate a clear background image according to a plurality of background focus images;

[0099] The image fusion module 58 is used to fuse the clear foreground image and the clear background image to obtain a clear image of the scene.

[0100] The above-mentioned image processing device provided by the embodiment of the present invention generates a clear foreground image and a clear background image respectively by collecting multiple foreground focus images and multiple background focus images, thereby fusing them to obtain a clear image. The final clear image can take into account both foreground clarity and background clarity, thereby effectively improving the image shooting effect.

[0101] In one embodiment, different foreground focused images have the same exposure value; the foreground image generation module 54 is used to: use a multi-frame noise reduction algorithm to perform noise reduction and synthesis processing on multiple foreground focused images, and use the processed images as clear foreground images.

[0102] In one implementation, different background focus images have different exposure values; the background image generation module 56 is used to: synthesize multiple background focus images using an HDR algorithm, and use the synthesized image as a clear background image.

[0103] In one embodiment, the image fusion module 58 is used to: segment an object map of a foreground object in a scene and a background map of a background environment excluding the foreground object based on a clear foreground image and a clear background image; optimize the object map to obtain a processed object map; the optimization processing includes one or more of brightness adjustment operations, face beautification operations, detail enhancement operations, light and shadow adjustment operations, and noise reduction operations; and fuse the processed object map with the background map to obtain a clear image of the scene.

[0104] In one embodiment, the image fusion module 58 is further used to: fuse the clear foreground image and the clear background image to obtain a fused image, and segment from the fused image an object map of the foreground object in the scene and a background map of the background environment excluding the foreground object; or segment from the clear foreground image an object map of the foreground object in the scene, and obtain a background map of the background environment excluding the foreground object in the scene based on the clear background image.

[0105] In one embodiment, the image fusion module 58 is further used to: segment a first background image of the background environment excluding the foreground object from the clear foreground image; segment a second background image of the background environment excluding the foreground object from the clear background image; and fuse the first background image and the second background image to obtain a background image of the background environment excluding the foreground object in the scene.

[0106] In one embodiment, the object image includes a face image and / or a portrait image; the image fusion module 58 is further used to: determine the lighting environment of the foreground object based on the brightness difference between the object image and the background image; the lighting environment includes a front lighting environment, a side lighting environment or a backlighting environment; if the object image includes a face image, a brightness adjustment operation is performed on the face image according to the lighting environment of the foreground object to obtain a processed face image; if the object image includes a portrait image, a brightness adjustment operation is performed on the portrait image according to the lighting environment of the foreground object to obtain a processed portrait image.

[0107] In one embodiment, the image fusion module 58 is further used to: perform sharpening processing and / or brightness processing on the background image to obtain a processed background image; and fuse the processed object image with the processed background image to obtain a clear image of the scene.

[0108] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0109] Furthermore, this embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned image processing methods are executed.

[0110] The computer program product of the image processing method, device, electronic device and computer-readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0112] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire multiple foreground focused images and multiple background focused images captured by an image acquisition device for the same scene; Generating a clear foreground image according to the multiple foreground focused images; wherein the exposure values ​​of different foreground focused images are the same; the step of generating a clear foreground image according to the multiple foreground focused images comprises: performing noise reduction and synthesis processing on the multiple foreground focused images using a multi-frame noise reduction algorithm, and using the processed image as the clear foreground image; Generating a clear background image according to the multiple background focus images; wherein different background focus images have different exposure values; the step of generating a clear background image according to the multiple background focus images comprises: synthesizing the multiple background focus images using an HDR algorithm, and using the synthesized image as the clear background image; The clear foreground image and the clear background image are fused to obtain a clear image of the scene.

2. The method according to claim 1, characterized in that: The step of fusing the clear foreground image and the clear background image to obtain a clear image of the scene includes: Segmenting an object map of a foreground object in the scene and a background map of a background environment excluding the foreground object based on the clear foreground image and the clear background image; Performing optimization processing on the object graph to obtain a processed object graph; the optimization processing includes one or more of a brightness adjustment operation, a face beautification operation, a detail enhancement operation, a light and shadow adjustment operation, and a noise reduction operation; The processed object image is fused with the background image to obtain a clear image of the scene.

3. The method according to claim 2, characterized in that The step of segmenting an object graph of a foreground object in the scene and a background graph of a background environment excluding the foreground object based on the clear foreground image and the clear background image comprises: Fusing the clear foreground image and the clear background image to obtain a fused image, and segmenting an object image of a foreground object in the scene and a background image of a background environment excluding the foreground object from the fused image; or, An object map of a foreground object in the scene is segmented from the clear foreground image, and a background map of a background environment in the scene excluding the foreground object is obtained based on the clear background image.

4. The method according to claim 3, characterized in that The step of obtaining a background image of the background environment in the scene excluding the foreground object according to the clear background image comprises: Segmenting a first background image of a background environment excluding the foreground object from the clear foreground image; Segmenting a second background image of the background environment excluding the foreground object from the clear background image; The first background image and the second background image are fused to obtain a background image of the background environment in the scene excluding the foreground object.

5. The method according to claim 2, characterized in that: The object graph includes a face graph and / or a portrait graph; The step of optimizing the object graph to obtain a processed object graph includes: Determining the light environment of the foreground object according to the brightness difference between the object image and the background image; the light environment includes a front light environment, a side light environment or a back light environment; If the object image includes a face image, a brightness adjustment operation is performed on the face image according to the light environment in which the foreground object is located to obtain a processed face image; if the object image includes a portrait image, a brightness adjustment operation is performed on the portrait image according to the light environment in which the foreground object is located to obtain a processed portrait image.

6. The method according to claim 2, characterized in that The step of fusing the processed object image with the background image to obtain a clear image of the scene includes: Performing sharpening processing and / or brightness processing on the background image to obtain a processed background image; The processed object image is fused with the processed background image to obtain a clear image of the scene.

7. An image processing device, characterized in that: include: An image acquisition module is used to acquire multiple foreground focus images and multiple background focus images taken by an image acquisition device for the same scene; A foreground image generation module is used to generate a clear foreground image according to the multiple foreground focused images; wherein the exposure values ​​of different foreground focused images are the same; the foreground image generation module is specifically used to: use a multi-frame noise reduction algorithm to perform noise reduction and synthesis processing on the multiple foreground focused images, and use the processed image as a clear foreground image; A background image generation module is used to generate a clear background image according to the multiple background focus images; wherein different background focus images have different exposure values; the background image generation module is specifically used to: synthesize the multiple background focus images using an HDR algorithm, and use the synthesized image as a clear background image; The image fusion module is used to fuse the clear foreground image and the clear background image to obtain a clear image of the scene.

8. An electronic device, characterized in that: include: processor and storage device; The storage device stores a computer program, which, when executed by the processor, executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.

Images