Image processing method, device, electronic device and storage medium

Abstract

Embodiments of the present invention relate to the field of image processing technology and disclose an image processing method, including: acquiring a color image and a depth image of a photographed object; determining overexposed areas and non-overexposed areas in the color image; determining the boundaries of the object within the overexposed area based on the depth image; and repairing the overexposed object image within the object boundary based on the color information of the non-overexposed area. The present invention also discloses an image processing device, electronic device, and storage medium. The image processing method, device, electronic device, and storage medium disclosed in the present invention can improve the image quality of captured images in the event of overexposure.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] Existing devices, including mobile phones and tablets, often experience overexposure when taking photos. (Overexposure refers to the presence of overexposed areas in a photo, resulting in excessive brightness and a whitened image.) This degrades the quality of the photos. Traditional photography techniques capture color information (RGB information) in photos but are unable to address the information lost in overexposed areas. Traditional methods to avoid overexposure rely solely on reducing the exposure during shooting. This method results in underexposure of dark areas, resulting in poor quality photos.

[0003] Currently, high-dynamic range (HDR) photography, based on multi-frame technology, is widely used to avoid overexposure when taking photos. HDR captures highlights while also synthesizing dark areas for better results. However, HDR also presents many inherent issues with multi-frame technology. For example, dynamic scenes can be affected by ghosting, and the synthesized dark areas can have a significantly reduced effect. Furthermore, HDR requires multiple photos and a significant amount of computation, resulting in longer processing times and higher power consumption.

[0004] Since the HDR method still has the above-mentioned defects, the solution to the problem of how to improve the quality of the captured image when overexposure occurs during the shooting process still needs to be optimized. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide an image processing method, device, server and storage medium, which can improve the image quality of the captured image when overexposure occurs.

[0006] To solve the above technical problems, an embodiment of the present invention provides an image processing method, including: acquiring a color image and a depth image of a photographed object; determining overexposed areas and non-overexposed areas in the color image; determining the boundary of an object located within the overexposed area based on the depth image; and repairing the overexposed object image within the object boundary based on the color information of the non-overexposed area.

[0007] An embodiment of the present invention also provides an image processing device, comprising: an acquisition module for acquiring a color image and a depth image of a photographed object; a first determination module for determining an overexposed area and a non-overexposed area in the color image; a second determination module for determining an object boundary within the overexposed area based on the depth image; and a repair module for repairing the overexposed object image within the object boundary based on the color information of the non-overexposed area.

[0008] An embodiment of the present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned image processing method.

[0009] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned image processing method when executed by a processor.

[0010] Compared to the prior art, the embodiments of the present invention acquire a color image and a depth image of a photographed object, determine overexposed and non-overexposed areas in the color image, determine the object boundary within the overexposed area based on the depth image, and repair the overexposed object image within the object boundary based on the color information of the non-overexposed area. By acquiring and determining the object boundary of the overexposed area based on the depth image of the photographed object, and then repairing the overexposed image within the object boundary based on the color information of the non-overexposed area, the image quality of the photographed object can be improved even when overexposure occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0012] Figure 1 is a flowchart of an image processing method according to a first embodiment of the present invention;

[0013] Figure 2 is a flowchart of an image processing method according to a second embodiment of the present invention;

[0014] Figure 3 is a flowchart of an image processing method according to a third embodiment of the present invention;

[0015] Figure 4 is a schematic structural diagram of an image processing apparatus according to a fourth embodiment of the present invention;

[0016] Figure 5FIG. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention. Specific embodiments

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in the embodiments of the present invention, many technical details are provided to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with each other and referenced to each other under the premise that there is no contradiction.

[0018] A first embodiment of the present invention relates to an image processing method, comprising: acquiring a color image and a depth image of a photographed object; determining overexposed and non-overexposed regions in the color image; determining the object boundary within the overexposed region based on the depth image; and repairing the overexposed object image within the object boundary based on the color information of the non-overexposed region. This embodiment can improve the quality of captured images even when overexposure occurs.

[0019] The following describes the implementation details of the image processing method of this embodiment. The following content is only provided for ease of understanding and is not necessary for implementing this solution.

[0020] The specific process of this embodiment is as follows Figure 1 As shown, the specific steps include:

[0021] Step 101: Acquire a color image and a depth image of the photographed object.

[0022] Specifically, a color image (RGB image) refers to an image in which each pixel is composed of red, green, and blue components. A color image can be captured by an ordinary camera. A depth image (also known as a range image) refers to an image that uses the distance from the image collector to each point in the scene as the pixel value. The depth image directly reflects the geometric shape of the visible surface of the scene. The depth image can be captured by a stereo camera or a TOF (time of flight) camera. Preferably, the depth image is captured by a TOF camera.

[0023] Step 102: Determine overexposed areas and non-overexposed areas in the color image.

[0024] Specifically, the color image is analyzed by a preset algorithm to determine overexposed areas in the color image, and at the same time, other areas in the color image are determined to be non-overexposed areas.

[0025] For example, the following algorithm can be used to determine overexposed areas: Since overexposed areas are bright white areas, the area consisting of continuous bright white pixels can be determined as overexposed areas. For an 8-bit RGB image, the values ​​of all pixels are tested. When a continuous area with pixel values ​​of (255, 255, 255) (in an 8-bit RGB image, this value is a bright white pixel value) appears, the continuous area is determined to be overexposed.

[0026] Step 103: Determine the boundary of the object located in the overexposed area according to the depth image.

[0027] Specifically, the overexposed area obtained in the previous step is matched with the depth image to obtain a corresponding area of ​​the overexposed area in the depth image, and the object boundary in the corresponding area is determined.

[0028] Step 104: Repair the overexposed object image within the object boundary according to the color information of the non-overexposed area.

[0029] Specifically, a filling image is generated based on the color information of the non-overexposed area of ​​the object (which belongs to the same object as the "overexposed object image to be repaired and located within the object boundary"), and the filling image is used to cover the overexposed object image within the object boundary.

[0030] Furthermore, after determining the object boundary within the overexposed region of the color image in step 103, the entire object's outline can be obtained. For the same object, a fill image with the same color information, such as the color and texture of the non-overexposed region, can be generated and filled into the overexposed portion of the object's outline in the color image to complete the restoration of the overexposed region.

[0031] In one example, a convolutional neural network is used to extract features from non-overexposed areas, generate a fill image with similar color and texture to the non-overexposed areas, and fill the fill image into the overexposed portion within the outline of the object in the color image.

[0032] Compared to the prior art, this embodiment acquires a color image and a depth image of the object being photographed, determines overexposed and non-overexposed areas in the color image, determines the object boundary within the overexposed area based on the depth image, and repairs the overexposed image of the object within the object boundary based on the color information of the non-overexposed area. By acquiring and determining the object boundary of the overexposed area based on the depth image of the object being photographed, and then repairing the overexposed image within the object boundary based on the color information of the non-overexposed area, the quality of the image of the object being photographed can be improved even when overexposure occurs.

[0033] The second embodiment of the present invention relates to an image processing method. The second embodiment is a further improvement of the first embodiment. The improvement is that, in this embodiment, after executing the step of repairing the overexposed object image within the object boundary based on the color information of the non-overexposed area, it also includes the step of performing light and shadow repair on the repaired overexposed object image to further improve the image quality.

[0034] The specific flow chart of the image processing method in this embodiment is as follows Figure 2 As shown, the specific steps include:

[0035] Step 201: Acquire a color image and a depth image of the photographed object.

[0036] Step 202: Determine overexposed areas and non-overexposed areas in the color image.

[0037] Step 203: Determine the boundary of the object located in the overexposed area according to the depth image.

[0038] Step 204: Repair the overexposed object image within the object boundary according to the color information of the non-overexposed area.

[0039] Steps 201 to 204 in this embodiment are substantially the same as steps 101 to 104 in the first embodiment, and are not described again herein.

[0040] Step 205: Perform light and shadow restoration on the restored overexposed object image.

[0041] Specifically, after executing step 204, the brightness and shadow variation trends of the repaired overexposed object image may still differ from those of the surrounding area, causing the repaired image to appear noticeably abnormal. In this step, light and shadow restoration is performed on the repaired overexposed object image so that the brightness and shadow variation trends of the repaired overexposed object image are consistent with those of the surrounding area.

[0042] Furthermore, light and shadow restoration includes:

[0043] (1) Brightness adjustment: Adjusting the brightness of the restored overexposed object image. This step can adjust the brightness of the restored overexposed object image based on the brightness of the surrounding area of ​​the restored overexposed object image so that it is consistent with the brightness of the surrounding area.

[0044] (2) Shadow filling: The overexposed object image is shadow filled based on the shadows of the surrounding area of ​​the repaired overexposed object image. In this step, the shadow filling is performed so that the shadow of the overexposed object image tends to be consistent with the surrounding area.

[0045] It should be noted that, in actual application scenarios, the above steps (1) and (2) can be selectively performed according to the application scenario, implementation conditions, and implementation effects. Only (1), only (2), or both (1) and (2) can be performed. Moreover, when performing light and shadow restoration, there is no restriction on the order in which the above steps (1) and (2) are performed.

[0046] Compared with the first embodiment, after repairing the overexposed object image within the object boundary, this embodiment also performs brightness adjustment and shadow filling on the repaired overexposed object image, so that the brightness and shadow change trends of the repaired overexposed object image are consistent with the surrounding area, further improving the image quality.

[0047] The third embodiment of the present invention relates to an image processing method. The third embodiment is a further improvement of the second embodiment. The improvement is that, in this embodiment, before performing the step of performing light and shadow restoration on the repaired overexposed object image, the method also includes the step of determining the position of the repaired overexposed object image at the photographed object, and subsequently, based on the position of the repaired overexposed object image at the photographed object, light and shadow restoration is performed on the repaired overexposed object image located at the edge of the object, thereby optimizing the light and shadow restoration scheme.

[0048] The specific flow chart of the image processing method in this embodiment is as follows Figure 3 As shown, the specific steps include:

[0049] Step 301: Acquire a color image and a depth image of the photographed object.

[0050] Step 302: Determine overexposed areas and non-overexposed areas in the color image.

[0051] Step 303: Determine the boundary of the object located in the overexposed area according to the depth image.

[0052] Step 304: Repair the overexposed object image within the object boundary according to the color information of the non-overexposed area.

[0053] Steps 301 to 304 in this embodiment are substantially the same as steps 101 to 104 in the first embodiment, and are not described again herein.

[0054] Step 305: Determine the position of the repaired overexposed object image at the photographed object.

[0055] Specifically, the repaired overexposed object image is determined to be located at the edge of the photographed object. For overexposed images located near the object's edge, the brightness and shadow changes are generally consistent with those of the surrounding area, so no light and shadow repair is required. However, for overexposed images located at the object's edge, the brightness and shadow changes are significantly different from those of the surrounding area, so light and shadow repair is required.

[0056] Step 306: Perform light and shadow restoration on the filled over-exposed area at the position of the photographed object according to the restored over-exposed object image.

[0057] Specifically, according to the execution result of step 305, light and shadow restoration is performed on the repaired overexposed object image located at the edge of the object, while light and shadow restoration is not performed on the repaired overexposed object image located at the non-edge position of the object.

[0058] It should be noted that, in this step, the specific implementation method of light and shadow restoration may adopt the specific implementation method of light and shadow restoration in step 205 of the second embodiment.

[0059] Compared to the second embodiment, this embodiment first determines the position of the repaired overexposed object image relative to the photographed object before performing light and shadow restoration on the repaired overexposed object image. Subsequently, light and shadow restoration is performed on the repaired overexposed object image located at the edge of the object based on the position of the repaired overexposed object image relative to the photographed object. By specifically adjusting the brightness and filling the shadows of the overexposed image located at the edge of the object based on the relative position of the overexposed object image and the photographed object, this further improves image quality while avoiding wasted computing resources during light and shadow restoration.

[0060] In addition, those skilled in the art will understand that the division of steps in the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0061] A fourth embodiment of the present invention relates to an image processing device, such as Figure 4 Shown, including:

[0062] An acquisition module 401 is used to acquire a color image and a depth image of the photographed object;

[0063] A first determining module 402 is configured to determine an overexposed area and a non-overexposed area in the color image;

[0064] A second determining module 403 is configured to determine a boundary of an object located in the overexposed area according to the depth image;

[0065] The restoration module 404 is configured to restore the overexposed object image within the object boundary according to the color information of the non-overexposed area.

[0066] In one example, the restoration module 404 is specifically configured to generate a filling image according to the color information of the non-overexposed area, and use the filling image to cover the overexposed object image within the object boundary.

[0067] In one example, the restoration module 404 is further configured to perform light and shadow restoration on the restored overexposed object image.

[0068] In one example, the second determination module 403 is also used to determine the position of the repaired overexposed object image at the photographed object; the repair module 404 is specifically used to perform light and shadow repair on the repaired overexposed object image located at the edge of the object based on the position of the repaired overexposed object image at the photographed object.

[0069] In one example, the restoration module 404 is specifically configured to adjust the brightness of the restored exposed object image.

[0070] In one example, the restoration module 404 is specifically configured to perform shadow filling on the restored overexposed object image according to shadows of surrounding areas of the restored overexposed object image.

[0071] In one example, the restoration module 404 is specifically configured to adjust the brightness of the restored overexposed object image; and perform shadow filling on the restored overexposed object image according to the shadow of the surrounding area of ​​the restored overexposed object image.

[0072] It is not difficult to find that this embodiment is a device embodiment corresponding to the first, second, or third embodiment, and this embodiment can be implemented in conjunction with the first, second, or third embodiment. The relevant technical details mentioned in the first, second, or third embodiment are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first, second, or third embodiment.

[0073] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.

[0074] A fifth embodiment of the present invention relates to an electronic device, such as Figure 5 As shown, it includes: at least one processor 501; and a memory 502 communicatively connected to the at least one processor 501; wherein the memory 502 stores instructions that can be executed by the at least one processor 501, and the instructions are executed by the at least one processor 501 to enable the at least one processor 501 to perform the above-mentioned image processing method.

[0075] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0076] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0077] A sixth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0078] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0079] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Obtain a color image and a depth image of the photographed object; determining an overexposed area and a non-overexposed area in the color image; Matching the overexposed area with the depth image, obtaining an area corresponding to the overexposed area in the depth image, and determining an object boundary within the corresponding area; Repairing an overexposed object image of the same object within the object boundary according to color information of a non-overexposed area of ​​the object; Determine the position of the repaired overexposed object image at the photographed object; Performing light and shadow restoration on the restored overexposed object image; The step of performing light and shadow restoration on the restored overexposed object image comprises: Light and shadow restoration is performed on the restored overexposed object image located at the edge of the object.

2. The image processing method according to claim 1, wherein: Repairing an overexposed object image of the same object within the object boundary according to color information of a non-overexposed area of ​​the object, including: Generate a filling image based on the color information of the non-overexposed area of ​​the object; The filling image is used to cover the overexposed object image of the same object within the object boundary.

3. The image processing method according to claim 1 or 2, characterized in that: The performing light and shadow restoration on the restored overexposed object image includes: The brightness of the restored overexposed object image is adjusted and / or shadow filling is performed on the restored overexposed object image according to the shadow of the surrounding area of ​​the restored overexposed object image.

4. The image processing method according to claim 1, wherein: The acquiring of a depth image of the photographed object comprises: A TOF camera is used to obtain a depth image of the photographed object.

5. An image processing device, characterized in that: include: An acquisition module is used to acquire a color image and a depth image of the photographed object; A first determining module, configured to determine an overexposed area and a non-overexposed area in the color image; a second determining module, configured to match the overexposed area with the depth image, obtain an area corresponding to the overexposed area in the depth image, and determine an object boundary within the corresponding area; a restoration module, configured to restore an overexposed object image of the same object within the object boundary based on color information of a non-overexposed area of ​​the object; The second determining module is further configured to determine a position of the repaired overexposed object image at the photographed object; The restoration module is further configured to perform light and shadow restoration on the restored overexposed object image; The repair module is specifically used to perform light and shadow repair on the repaired overexposed object image located at the edge of the object.

6. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 4 is implemented.

Images