Image processing method and electronic device

Abstract

The present application provides an image processing method and an electronic device, which relates to the field of image processing technology and can solve the problem of over-compensation or under-compensation when grayscale value compensation is performed on each pixel in an original RAW domain image; the method comprises: an electronic device acquires a RAW domain image acquired by an image sensor, and the RAW domain image includes a plurality of target objects with different depths of field; the electronic device determines the grayscale compensation coefficient corresponding to each target object based on the grayscale value of each pixel in the RAW domain image; the grayscale compensation coefficients corresponding to target objects with different depths of field are different. Furthermore, the electronic device uses the grayscale compensation coefficient to perform grayscale value compensation on the target object.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] As mobile phone photography gradually becomes the preferred method of taking photos, users are no longer satisfied with simple shooting records, but the ultimate pursuit of image details and light and shadow artistic sense. However, when users use their mobile phones to take photos facing strong light sources (such as the sun, street lights, etc.), the photos obtained may contain crosstalk artifacts, which will affect the clarity of the photos and thus affect the shooting effect. Summary of the invention

[0003] The present application provides an image processing method and an electronic device, which are used to solve the problem of over-compensation or under-compensation when grayscale value compensation is performed on each pixel in an original RAW domain image.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0005] In the first aspect, an image processing method is provided, which can be applied to electronic devices such as mobile phones, tablet computers, and laptop computers that include cameras, and can also be applied to chips, chip systems, or processors that can implement the image processing method, and can also be applied to logic modules or software executions that can implement all or part of the functions of electronic devices. The scheme of the present application is introduced by applying the method to electronic devices. The electronic device includes a camera and an image sensor; the image sensor includes a plurality of microlenses, a filter, and a plurality of photoelectric conversion elements, and the filter includes a plurality of filter units; wherein each microlens is arranged relative to a filter unit and p photoelectric conversion elements. P≥2, p is an integer.

[0006] Optionally, p=4.

[0007] The method includes: the electronic device acquires a RAW domain image acquired by an image sensor, wherein the RAW domain image includes a plurality of target objects with different depths of field; the electronic device determines the grayscale compensation coefficient corresponding to each target object based on the grayscale value of each pixel in the RAW domain image; the target objects with different depths of field correspond to different grayscale compensation coefficients. Then, the electronic device uses the grayscale compensation coefficient to compensate the grayscale value of the target object.

[0008] Based on the first aspect, the solution of the present application is adopted, and for target objects with different depths of field in the RAW domain image, the electronic device determines the grayscale compensation coefficient corresponding to the target object. Since the target objects with different depths of field correspond to different grayscale compensation coefficients, the electronic device compensates for the grayscale values ​​of different target objects respectively through different grayscale compensation coefficients, thereby avoiding the problems of over-compensation or under-compensation while eliminating crosstalk artifacts.

[0009] Optionally, the electronic device uses a grayscale compensation coefficient to compensate the grayscale value of the target object, and then the image sensor of the electronic device outputs a three-primary color RGB domain image, wherein the RGB domain image does not include crosstalk artifacts (such as a grid).

[0010] In a possible implementation manner of the first aspect, the target object includes a plurality of pixel blocks of different color channels, the pixel block includes a plurality of pixels in a matrix form of n×m, where n and m are positive integers; n×m=p.

[0011] The electronic device determines the grayscale compensation coefficient corresponding to each target object based on the grayscale value of each pixel in the RAW domain image, including: the electronic device determines the difference coefficient corresponding to the pixel block of each color channel in multiple different color channels based on the grayscale value of each pixel in the RAW domain image; wherein the difference coefficient is used to indicate the brightness difference between the pixel blocks of the same color channel. Furthermore, the electronic device determines the grayscale compensation coefficient corresponding to the target object based on the difference coefficient corresponding to the pixel block of each color channel.

[0012] Optionally, the multiple color channels include a red channel, a blue channel, a green channel and a white channel.

[0013] Optionally, the multiple color channels include a red channel, a blue channel, and a green channel.

[0014] In a possible implementation of the first aspect, the electronic device determines a grayscale compensation coefficient corresponding to the target object based on the difference coefficient corresponding to the pixel block of each color channel, including: determining a first difference coefficient corresponding to the pixel block of the red channel, a second difference coefficient corresponding to the pixel block of the blue channel, and a third difference coefficient corresponding to the pixel block of the green channel; the electronic device determines the grayscale compensation coefficient corresponding to the target object based on the first difference coefficient, the second difference coefficient and the third difference coefficient.

[0015] Optionally, the grayscale compensation coefficient is the average value of the first difference coefficient, the second difference coefficient and the third difference coefficient; or, the grayscale compensation coefficient is the variance of the first difference coefficient, the second difference coefficient and the third difference coefficient; or, the grayscale compensation coefficient is the standard deviation of the first difference coefficient, the second difference coefficient and the third difference coefficient; or, the grayscale compensation coefficient is the maximum value among the first difference coefficient, the second difference coefficient and the third difference coefficient; or, the grayscale compensation coefficient is the minimum value among the first difference coefficient, the second difference coefficient and the third difference coefficient.

[0016] In a possible implementation of the first aspect, the grayscale compensation coefficients of pixel blocks of different color channels of the same target object are different; wherein the electronic device uses the grayscale compensation coefficient to compensate the grayscale value of the target object, including: the electronic device uses the grayscale compensation coefficient of the pixel blocks of each color channel of the target object to compensate the grayscale value of the pixel blocks corresponding to each color channel in multiple different color channels.

[0017] Optionally, the grayscale compensation coefficients of the pixel blocks of different color channels of the same target object are the same, and the electronic device uses the grayscale compensation coefficients to compensate the grayscale values ​​of the pixel blocks of each color channel.

[0018] In a possible implementation of the first aspect, the electronic device uses a grayscale compensation coefficient to compensate for the grayscale value of the target object, including: the electronic device uses the first difference coefficient as the grayscale compensation coefficient to compensate for the grayscale value of the pixel block of the first color channel; the electronic device uses the second difference coefficient as the grayscale compensation coefficient to compensate for the grayscale value of the pixel block of the second color channel; the electronic device uses the third difference coefficient as the grayscale compensation coefficient to compensate for the grayscale value of the pixel block of the third color channel.

[0019] In a possible implementation of the first aspect, the electronic device determines a first difference coefficient corresponding to a pixel block of a first color channel, including: the electronic device determines a difference value corresponding to each pixel block in a plurality of pixel blocks based on a grayscale value of each pixel in a RAW domain image, wherein the difference value is used to indicate a brightness difference between the plurality of pixels; and determines the first difference coefficient corresponding to the pixel block of the first color channel according to the difference value corresponding to each pixel block.

[0020] Optionally, the first difference coefficient is the average value of the difference values ​​corresponding to each pixel block; or, the first difference coefficient is the variance of the difference values ​​corresponding to each pixel block; or, the first difference coefficient is the standard deviation of the difference values ​​corresponding to each pixel block; or, the first difference coefficient is the maximum value of the difference values ​​corresponding to each pixel block; or, the first difference coefficient is the minimum value of the difference values ​​corresponding to each pixel block.

[0021] In a possible implementation of the first aspect, the electronic device determines a difference value corresponding to each pixel block in a plurality of pixel blocks, including: the electronic device obtains a grayscale value of each pixel in a plurality of pixels included in the pixel block; the electronic device determines a grayscale difference value of each pixel based on the grayscale value of each pixel and a grayscale reference value of the pixel block; the electronic device determines a difference value corresponding to the pixel block based on the grayscale difference value of each pixel.

[0022] Optionally, the grayscale reference value is the average value of the grayscale value of each pixel; or, the grayscale reference value is the variance of the grayscale value of each sub-pixel; or, the grayscale reference value is the standard deviation of the grayscale value of each sub-pixel; or, the grayscale reference value is the maximum value of the grayscale value of each sub-pixel; or, the grayscale reference value is the minimum value of the grayscale value of each sub-pixel.

[0023] In a possible implementation of the first aspect, the electronic device uses a grayscale compensation coefficient to compensate for the grayscale value of the target object, including: the electronic device inputs the grayscale compensation coefficient and the RAW domain image into a preset model, and outputs an RGB domain image; wherein the RGB domain image is obtained after the preset model compensates for the grayscale value of the target object in the RAW domain image.

[0024] In a second aspect, an electronic device is provided, which has the function of implementing any one of the functions in the first aspect, and the function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0025] In a third aspect, an electronic device is provided, the electronic device comprising: a camera, a memory and one or more processors; the camera comprises an image sensor. The memory stores a computer program code, the computer program code comprises a computer instruction; when the computer instruction is executed by the processor, the electronic device executes any one of the methods in the first aspect.

[0026] In a possible implementation manner of the third aspect, the image sensor includes a plurality of microlenses, a filter, and a plurality of photoelectric conversion elements, and the filter includes a plurality of filter units; wherein each microlens is arranged relative to a filter unit and p photoelectric conversion elements. P ≥ 2, where p is an integer.

[0027] Optionally, p=4.

[0028] In a fourth aspect, a chip system is provided, which includes: at least one processor and an interface, the interface being used to receive instructions and transmit them to at least one processor; at least one processor executes the instructions so that the electronic device executes any one of the methods described in the first aspect above.

[0029] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute any of the methods described in the first aspect.

[0030] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0031] Among them, the technical effects brought about by any implementation method in the above-mentioned second to sixth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a pixel unit provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of an imaging principle provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the structure of a camera provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of the structure of an image sensor provided in an embodiment of the present application;

[0037] Figure 6 A schematic diagram of the principle of a crosstalk artifact provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of another principle of crosstalk artifact provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of a crosstalk artifact in an RGB domain image provided by an embodiment of the present application;

[0040] Fig. 9 A flowchart of an image processing method provided in an embodiment of the present application;

[0041] Fig.10 A schematic diagram of difference coefficients corresponding to pixel blocks of different color channels included in a target object provided in an embodiment of the present application;

[0042] Fig.11 A schematic diagram of a grayscale compensation diagram provided in an embodiment of the present application;

[0043] Fig.12 A schematic diagram of another grayscale compensation diagram provided in an embodiment of the present application;

[0044] Fig.13 A schematic diagram of the principle of generating a grayscale compensation map provided in an embodiment of the present application;

[0045] Fig.14 A schematic diagram of difference values ​​of various pixel blocks included in a red channel of a target object provided by an embodiment of the present application;

[0046] Fig.15 A schematic diagram of grayscale difference values ​​corresponding to each pixel included in a red pixel block provided in an embodiment of the present application;

[0047] Fig.16 A schematic diagram of the principle of grayscale value compensation of an electronic device provided in an embodiment of the present application;

[0048] Fig.17 A schematic diagram of a model training provided in an embodiment of the present application;

[0049] Fig.18 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to facilitate understanding of the solutions provided in the embodiments of the present application, some terms involved in the embodiments of the present application are explained below.

[0051] Crosstalk

[0052] In the field of photography, crosstalk usually refers to the signal interference caused by the interaction between adjacent photosensitive elements on the image sensor. Crosstalk can occur at multiple levels, including optical, electronic and signal processing.

[0053] Optical crosstalk: Usually occurs between adjacent photosensitive elements. When light passes through the filter block of one pixel, it sometimes shines into adjacent pixels, resulting in color crossover or loss of image detail.

[0054] Electronic crosstalk: involves the mutual influence of electrons between adjacent pixels. This is because after light passes through the photosensitive element, the photosensitive element converts the light signal into an electrical signal. In the process of converting the light signal into an electrical signal, the electrical signal moves on the photosensitive element, thus causing interference between adjacent pixels.

[0055] Signal processing crosstalk: In the process of photosensitive elements converting light signals into electrical signals, crosstalk may be caused by improper circuit design or defects in signal processing algorithms.

[0056] Crosstalk Artifacts

[0057] Crosstalk artifacts are defects in the image caused by crosstalk on the photosensitive element, which usually appears as a grid-like texture or artifact in the image. In digital photography, one of the common crosstalk situations is the interaction between the arrangement of the camera's photosensitive elements and the fine patterns or textures in the object being photographed. This interaction can cause frequency aliasing and produce obvious grid-like artifacts.

[0058] QPD Array

[0059] The QPD array is also called the Quad Bayer array, which means that on the basis of the Bayer array, a pixel block of a color channel is divided into four pixels, and four adjacent pixels share a color channel filter (color filter array, CFA). For example, Figure 1 As shown, a QPD array is a pixel unit including a red pixel block (R), two green pixel blocks (G) and a blue pixel block (B), which can also be called an RGGB pixel unit. Each pixel block forms a 2×2 matrix, that is, a pixel block includes four pixels.

[0060] For example, Figure 1 As shown, the red pixels R1, R2, R3 and R4 included in the red pixel block R form a 2×2 matrix; the green pixels G1, G2, G3 and G4 included in the green pixel block G form a 2×2 matrix; the green pixels G5, G6, G7 and G8 included in the green pixel block G form a 2×2 matrix; the blue pixels B1, B2, B3 and B4 included in the blue pixel block B form a 2×2 matrix.

[0061] Understandably, Figure 1 The four red pixels R1, R2, R3 and R4 share a red channel filter; the four green pixels G1, G2, G3 and G4 share a green channel filter; the four green pixels G5, G6, G7 and G8 share a green channel filter; the four blue pixels B1, B2, B3 and B4 share a blue filter.

[0062] RAW domain

[0063] Also known as RAW domain images, the original image contains the data processed from the image sensor of a digital camera, scanner or film scanner. It is so named because the RAW domain image has not been processed, printed or used for editing. The RAW domain image contains the most original information of the image and has not been processed by the nonlinear processing of the image signal processor (ISP).

[0064] Depth of Field

[0065] When an electronic device (such as a mobile phone) is shooting, the process of making a clear image of the object at a certain distance from the camera is called focusing. Among them, the point where the object is located is called the focus point. Within a certain range before and after the focus point, the mobile phone can still obtain a clear image, that is, within a certain range before and after the focus point, the image of the object is still clear, so the range of clear image before and after the focus point can be called the clear range. This clear range is called depth of field. It should be noted that within the clear range, the range where the focus point is close to the camera is called the front depth of field, and the range where the focus point is far from the camera is called the back depth of field.

[0066] It is understandable that during the imaging process of the mobile phone, the light beam reflected by the photographed object will propagate to the imaging surface, thereby forming an image of the photographed object on the imaging surface. Usually, the light beam reflected by the photographed object is focused on a point (i.e., the focus) after passing through the camera; wherein the focus may be located before the imaging surface, or may be located after the imaging surface, or may also be located on the imaging surface. Taking the focus being located on the imaging surface as an example, for example, Figure 2 As shown in FIG. 1 , it is a schematic diagram of an electronic device photographing a photographed object. Among them, O is the optical axis of the camera L, F is the focus, and f is the focus point. In the range between M1 and M2 before and after the focus point f, the electronic device can still obtain a clear image, so the range between M1 and M2, that is, the distance S is the depth of field. Figure 2 As shown, within the range between M1 and M2, there are near point A and far point B. The light beams reflected by near point A and far point B can both pass through camera L to the imaging surface. Figure 2 A' is the imaging point of the near point A, and B' is the imaging point of the far point B.

[0067] The technical solution provided by the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0068] An image processing method provided in an embodiment of the present application can be applied to an electronic device with a shooting function, such as a mobile phone, a sports camera (GoPro), a digital camera, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, etc. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device.

[0069] For example, Figure 3 A schematic structural diagram of an electronic device 100 is shown.

[0070] Among them, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 193 and a display screen 194, etc.

[0071] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0072] The processor 110 may include one or more processing units, for example, the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0073] Among them, the controller can be the nerve center and command center of the electronic device 100.

[0074] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0075] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an I2C interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface 130, etc.

[0076] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0077] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0078] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the above instructions stored in the internal memory 121, so that the electronic device 100 performs the methods provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory; it may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, universal flash storage (UFS), etc. In other embodiments, the processor 110 executes the instructions stored in the internal memory 121, and / or the instructions stored in the memory provided in the processor, so that the electronic device 100 performs the methods provided in the embodiments of the present application, as well as various functional applications and data processing.

[0079] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0080] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 can receive input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.

[0081] The power management module 141 may be used to monitor performance parameters such as battery capacity, battery cycle times, battery charging voltage, battery discharging voltage, and battery health status (eg, leakage, impedance).

[0082] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0083] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0084] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include one or more filters, switches, power amplifiers, low noise amplifiers (LAN), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be placed in the processor 110. In some embodiments, at least some of the functions of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0085] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating one or more communication processing modules. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0086] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0087] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0088] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0089] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. Among them, the camera 193 is used to capture static images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. The ISP is used to process the data fed back by the camera 193.

[0090] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0091] Of course, the electronic device 100 provided in the embodiment of the present application may also include one or more components such as a positioning module 181, a button 190, a motor 191, an indicator 192 and a SIM card interface 195, without limitation.

[0092] In some embodiments of the present application, the camera 193 can be used as a rear camera and arranged on the side where the housing of the electronic device 100 is located. Alternatively, in other embodiments of the present application, the camera 193 can be used as a front camera and arranged on the side where the electronic device 100 displays images.

[0093] For example, the camera 193 may include Figure 4The image sensor 20 shown in the figure and the optical lens 30 disposed on the light incident side of the image sensor 20. The optical lens 30 can allow external light to be incident on the image sensor 20. The image sensor 20 may be a 2×2 optical clear lens (OCL) type sensor, which may be called a QPD sensor.

[0094] For example, the image sensor 20 may include: Figure 5 The filter 200 shown in the figure may include filter units 210 arranged in an array. The filter unit 210 may include at least three color filter blocks for transmitting three primary colors of light, such as red (red, R) light, blue (blue, B) light, and green (green, G) light. Since the human eye is more sensitive to green, in the same filter unit 210, the number of green filter blocks G may be greater than the number of other color filter blocks. For example, when a filter unit 210 is arranged in the form of a 2×2 matrix, any of the filter units 210 may include a red filter block R, a blue filter block B, and two green filter blocks G.

[0095] Based on this, the red filter block R is used to convert the Figure 4 The red light in the light from the optical lens 30 is transmitted, and the rest of the light is filtered out. The blue filter block B is used to transmit the blue light in the light from the optical lens 30, and filter out the rest of the light. The green filter block G is used to transmit the green light in the light from the optical lens 30, and filter out the rest of the light. In this way, the light from the optical lens 30 can be separated into three primary colors (RGB) after passing through the above-mentioned color filter blocks, so that the electronic device 100 can obtain an RGB domain image.

[0096] Exemplarily, the image sensor 20 may also include: Figure 5 The position of each filter unit 210 may correspond to the position of a micro lens 220, so that each micro lens 220 may cover a filter unit 210. The micro lens 220 may Figure 4 The external light from the optical lens 30 is converged and incident on the filter unit 210 .

[0097] It should be noted that due to the influence of the shooting environment, there may be some stray light in the light passing through the optical lens 30, and the incident direction and angle of the stray light are different. Therefore, a microlens 220 is set in the image sensor 20. The purpose is to converge these stray lights and make them incident on the filter unit 210, thereby improving the performance of the optical system and improving the imaging contrast and picture quality.

[0098] For example, Figure 5 As shown, the image sensor 20 also includes a plurality of photoelectric conversion elements 230 (or photosensitive elements). In the one filter unit 210, the position of each filter block can correspond to p (such as p=4) photoelectric conversion elements 230, so that each filter block can cover four photoelectric conversion elements 230. The photoelectric conversion element 230 can be a photodiode, which is used to convert the light passing through the filter block into an electrical signal.

[0099] Exemplarily, the photodiode can be manufactured by a charge coupled device (CCD) process, and the photodiode can collect the light signal ( Figure 4 The photodiode is a semiconductor device that converts light (indicated by arrows in the figure) into an electrical signal, which is then converted into a digital image signal through an amplification and analog-to-digital conversion circuit. Alternatively, the photodiode can be manufactured using a complementary metal oxide semiconductor (CMOS) process. The photodiode manufactured using the CMOS process has both N-type and P-type semiconductors. The current generated by the complementary effect of these two semiconductors can be recorded and interpreted by the processing chip, and converted into a digital image signal through a digital-to-analog conversion circuit.

[0100] As can be seen from the above, the image sensor 20 may include a filter 200 and a plurality of photoelectric conversion elements 230, and one filter block in the filter 200 corresponds to the positions of the four photoelectric conversion elements 230. When the filter block is the color filter block, for example, the red filter block R, the blue filter block B, and the green filter block G, of the light incident on the filter block, only the light consistent with the color of the color filter block can pass through and enter the photoelectric conversion element 230 corresponding to the position of the filter block, and is converted into an electrical signal under the action of the photoelectric conversion element 230.

[0101] In this case, the above-mentioned one filter block and the photoelectric conversion element 230 corresponding to the position of the filter block can constitute the above-mentioned Figure 1 The pixel blocks shown. The digital signals obtained after analog-to-digital conversion of the electrical signals obtained by each pixel block are the raw (RAW) domain images output by the image sensor 20. Since the color filter block, for example, the blue filter block B, only allows one color of light (for example, B light) to pass through, in the RAW domain image, the data obtained by the pixels of the image sensor 20 composed of the blue filter block of the image sensor 20 and the photoelectric conversion element 230 corresponding to the position of the blue filter block only has one color information (or one color channel), such as the blue channel. The color channel is consistent with the color of the blue filter block.

[0102] It can be understood that, since one filter block covers four photoelectric conversion elements 230, that is, one filter block and four photoelectric conversion elements 230 corresponding to the position of the filter block can constitute four pixels. In this way, the number of pixels of the image sensor 20 can be increased without changing the size of the image sensor 20, thereby improving the resolution of the image. In addition, compared with the traditional image sensor, since only the structure of the photoelectric conversion element is changed, and the structure of the microlens and the filter is not changed, the purpose of reducing costs can be achieved under the premise of increasing the number of pixels.

[0103] In summary, the image sensor 20 provided in the embodiment of the present application comprises a filter 200, and a filter unit 210 as the minimum repeating unit of the filter 200, whose structure is as follows: Figure 1 As shown, it includes a plurality of color filter blocks for obtaining color information. Each color filter block in the plurality of color filter blocks and the photoelectric conversion element 230 corresponding to the color filter block constitute a color channel. The color channel includes four pixels. Figure 5 As shown, four pixels of one color channel share one microlens 220. Thus, the microlens 220 will Figure 4 After the external light from the optical lens 30 is gathered, it enters the photoelectric conversion element 230 through the filter block.

[0104] In the image sensor 20 provided in the embodiment of the present application, four photoelectric conversion elements 230 correspond to the position of one filter block, that is, four pixels are formed without changing the size of the filter block, so that the size of the pixel becomes smaller; and because the four pixels of one color channel share one microlens 220, after the microlens 220 converges the external light, it cannot be uniformly incident on the corresponding four photoelectric conversion elements 230 (that is, four pixels) after passing through one filter block, thereby causing crosstalk between pixels.

[0105] For example, Figure 6 As shown in (a) of FIG. 1 , after being converged by the microlens 220, the external light is incident on the positions corresponding to the photoelectric conversion element a and the photoelectric conversion element b. In this way, due to the different amounts of light incident on the four photoelectric conversion elements 230 and the different light intensities of the incident light, the optical signals received by the four photoelectric conversion elements 230 are different, thereby causing crosstalk between adjacent pixels. For example, referring to Figure 6In (a), the light signals received by photoelectric conversion element a and photoelectric conversion element b are greater than those received by photoelectric conversion element c and photoelectric conversion element d, resulting in brighter pixels corresponding to photoelectric conversion element a and photoelectric conversion element b. The brightness of the pixels corresponding to photoelectric conversion element c and photoelectric conversion element d is darker, resulting in obvious crosstalk artifacts (such as stripes) in the RAW domain image output by the image sensor 20. For example, Figure 6 As shown in (b) in FIG. 1 , the RAW domain image has obvious and regular stripes. For example, each pixel in the RAW domain image has obvious alternating light and dark stripes.

[0106] In some embodiments of the present application, the phenomenon of crosstalk artifacts can be improved by qualified sensitivity calibration (QSC), wherein QSC calibration refers to changing the angle at which the microlens 220 in the image sensor 20 receives external light, that is, changing the incident angle of external light incident on the microlens 220, so as to improve the phenomenon of crosstalk artifacts.

[0107] Among them, the incident angles of external light are different at positions corresponding to different depths of field. Incident angle A can be calibrated by the QSC calibration data corresponding to incident angle A, but the QSC calibration data corresponding to incident angle A cannot calibrate other incident angles (such as incident angle B). After the external light at incident angle B is incident on the microlens 220, the RAW domain image output by the image sensor 20 still has the phenomenon of crosstalk artifacts.

[0108] For example, Figure 7 As shown in (a), for the external light with an incident angle B, the external light will be incident on the position corresponding to the photoelectric conversion element a after being converged by the microlens 220. In this way, the light signal received by the photoelectric conversion element a will be larger than the light signals received by other photoelectric conversion elements (such as photoelectric conversion elements b, photoelectric conversion c, and photoelectric conversion d). Therefore, the brightness of the pixel corresponding to the photoelectric conversion element a is brighter, while the brightness of the pixels corresponding to the photoelectric conversion elements b, photoelectric conversion c, and photoelectric conversion d is darker, so that obvious crosstalk artifacts (such as a grid shape) will still appear in the RAW domain image output by the image sensor 20. For example, as Figure 7 As shown in (b) in FIG. 1 , there is an obvious and regular grid in the RAW domain image. For example, there is an obvious grid on each pixel in the RAW domain image.

[0109] In other embodiments of the present application, for RAW domain images with obvious crosstalk artifacts, a preset grayscale compensation coefficient can be used to compensate the grayscale values ​​of pixels in the RAW domain image. It can be understood that the phenomenon of crosstalk artifacts is mainly caused by the brightness difference between pixels in the same color channel, that is, there are differences in the grayscale values ​​between pixels in the same color channel. Therefore, by compensating the grayscale values ​​of pixels in the RAW domain image through the grayscale compensation coefficient, the purpose of reducing crosstalk artifacts can be achieved.

[0110] In some cases, such as when there are multiple depth of field problems, out-of-focus problems, or point light sources with high contrast to the background (such as sunlight or night lights), the severity of crosstalk artifacts in the same RAW domain image will also be different.

[0111] For example, in the same RAW domain image, the phenomenon of crosstalk artifacts in the highlight area is more serious than that in the dark area. Based on this, after the grayscale value compensation of the pixels in the RAW domain image is performed using the preset grayscale compensation coefficient, there may be problems of over-compensation or under-compensation. For example, for areas with more serious crosstalk artifacts, there is a problem of under-compensation, while for areas with less serious crosstalk artifacts, there is a problem of over-compensation.

[0112] For example, for a RAW domain image without eliminating crosstalk artifacts, Figure 8 As shown in (a) in FIG. 1 , an obvious grid can be seen in the RAW domain image. After the grayscale value of the pixels in the RAW domain image is compensated by using the preset grayscale compensation coefficient, as shown in FIG. Figure 8 As shown in (b), in the RAW domain image, the grid in a part of the area is obviously eliminated, while the grid can still be clearly seen in another part of the area.

[0113] In some embodiments of the present application, the severity of the crosstalk artifact is strongly correlated with the depth of field. This is because the brightness differences between pixels of each color channel corresponding to the same depth of field position can be considered to be the same, so the severity of the crosstalk artifact at the same depth of field position is roughly the same.

[0114] Based on this, the image processing method provided in the embodiment of the present application determines the grayscale compensation coefficient corresponding to the target object for multiple target objects with different depths of field in the RAW domain image. The grayscale compensation coefficients corresponding to the target objects with different depths of field are different. Therefore, different grayscale compensation coefficients can be used to compensate the grayscale values ​​of the target objects with different depths of field, so as to distinguishably eliminate crosstalk artifacts of different severity and minimize the loss of details of the RAW domain image.

[0115] For example, using the solution of the embodiment of the present application, after differentiating and eliminating crosstalk artifacts of different severity, as shown in FIG. Figure 8 As shown in (c), no obvious network is seen in the RAW domain image, and the details of the RAW domain image are stronger.

[0116] The image processing method provided by the embodiment of the present application is described in detail below in conjunction with the drawings of the specification. The image processing method provided by the embodiment of the present application can be performed by an electronic device including a camera, such as a mobile phone, a tablet computer, a laptop computer, etc., or can be performed by a chip, a chip system or a processor that can implement the image processing method, or can be performed by a logic module or software that can implement all or part of the functions of the electronic device, without limitation. The image processing method provided by the embodiment of the present application is described in detail below with the electronic device as the execution subject.

[0117] It should be noted that in the embodiments of the present application, the logic module for implementing the functions of the image processing method can be integrated into the ISP or exist independently of the ISP without limitation.

[0118] Fig. 9 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the method may include the following steps.

[0119] S301: An electronic device acquires a RAW domain image, where the RAW domain image includes a plurality of target objects with different depths of field.

[0120] The RAW domain image is collected by an image sensor of an electronic device.

[0121] Exemplarily, the target object may be various types of objects, for example, the target object may be a task, an animal, a doll, a building, or a plant, etc. The present application does not specifically limit the type of the target object.

[0122] For example, the target object may include a building (hereinafter referred to as target object A) and the sky (hereinafter referred to as the second target object B). When target object A is the focus object, the electronic device uses the focus function to make the focus object (i.e., target object A) fall on the focus of the camera focal length. On this basis, target object A is a clear image on the imaging surface, while the second target object B is a blurred image (which can be understood as a blurred image) on the imaging surface.

[0123] It is understandable that the depth of field is related to the distance between the object being photographed and the electronic device. For example, the closer the distance between the object being photographed and the electronic device is, the smaller the depth of field is; correspondingly, the farther the distance between the object being photographed and the electronic device is, the larger the depth of field is.

[0124] Exemplarily, in the RAW domain image, the distance between the target object A and the electronic device is smaller than the distance between the second target object B and the electronic device, so the depth of field of the target object A is smaller than the depth of field of the second target object B.

[0125] In some embodiments of the present application, when the target object A just falls on the focus of the camera focal length, the target object A is clearly imaged on the imaging plane, and theoretically, the depth of field of the target object A can be considered to be 0. On this basis, after the light reflected by the target object A is incident on the image sensor 20, in the RAW domain image output by the image sensor 20, the crosstalk artifact (i.e., grid) at the position of the screen where the target object A is located is less, or even non-existent.

[0126] Since the target object B is blurred on the non-imaging surface, the depth of field of the target object B is relatively large. On this basis, after the light reflected by the target object B is incident on the image sensor 20, in the RAW domain image output by the image sensor 20, the crosstalk artifact (i.e., grid) at the position of the image where the target object B is located is relatively serious.

[0127] S302: The electronic device determines a grayscale compensation coefficient corresponding to each target object based on the grayscale value of each pixel in the RAW domain image; wherein the grayscale compensation coefficients corresponding to target objects with different depths of field are different.

[0128] In some embodiments of the present application, the strength of the brightness difference at the screen position where the target object is located is strongly correlated with the depth of field of the target object, and the grayscale compensation coefficient corresponding to the target object is strongly correlated with the strength of the brightness difference at the screen position where the target object is located. Therefore, the grayscale compensation coefficient corresponding to the target object is strongly correlated with the depth of field of the target object.

[0129] Exemplarily, the smaller the depth of field of the target object, the weaker the brightness difference at the position of the target object on the screen, and the smaller the grayscale compensation coefficient corresponding to the target object. Correspondingly, the larger the depth of field of the target object, the stronger the brightness difference at the position of the target object on the screen, and the larger the grayscale compensation coefficient corresponding to the target object.

[0130] Exemplarily, the target object includes pixel blocks of multiple different color channels. The electronic device determines a difference coefficient corresponding to the pixel block of each color channel in the multiple different color channels, and determines a grayscale compensation coefficient corresponding to the target object based on the difference coefficient corresponding to the pixel block of each color channel. The difference coefficient is used to indicate the brightness difference between pixel blocks of the same color channel.

[0131] In some embodiments of the present application, the plurality of different color channels include a red channel, a blue channel, a green channel, and a white channel. On this basis, the electronic device determines the difference coefficient corresponding to the pixel block of the red channel, the difference coefficient corresponding to the pixel block of the blue channel, the difference coefficient corresponding to the pixel block of the green channel, and the difference coefficient corresponding to the pixel block of the white channel. Furthermore, the electronic device determines the grayscale compensation coefficient corresponding to the target object based on the difference coefficient corresponding to the pixel block of the red channel, the difference coefficient corresponding to the pixel block of the blue channel, the difference coefficient corresponding to the pixel block of the green channel, and the difference coefficient corresponding to the pixel block of the white channel.

[0132] In some other embodiments of the present application, the plurality of different color channels include a red channel (or first color channel), a blue channel (or second color channel), and a green channel (or third color channel). On this basis, the electronic device determines the first difference coefficient corresponding to the pixel block of the red channel, the second difference coefficient corresponding to the pixel block of the blue channel, and the third difference coefficient corresponding to the pixel block of the green channel. Furthermore, the electronic device determines the grayscale compensation coefficient of the target object based on the first difference coefficient, the second difference coefficient, and the third difference coefficient.

[0133] It can be understood that the first difference coefficient is used to indicate the brightness difference between pixel blocks of the red channel; the second difference coefficient is used to indicate the brightness difference between pixel blocks of the blue channel; and the third difference coefficient is used to indicate the brightness difference between pixel blocks of the green channel.

[0134] For example, Fig.10 As shown, the target object includes four pixel blocks of the red channel (i.e., four pixel blocks R), four pixel blocks of the blue channel (i.e., four pixel blocks B), and eight pixel blocks of the green channel (i.e., eight pixel blocks G). For example, the first difference coefficient corresponding to the four pixel blocks R is expressed as k1, the second difference coefficient corresponding to the four pixel blocks B is expressed as k2, and the third difference coefficient corresponding to the eight pixel blocks G is expressed as k3. The grayscale compensation coefficient w can be determined according to the first difference coefficient k1, the second difference coefficient k2, and the third difference coefficient k3.

[0135] In some embodiments of the present application, the grayscale compensation coefficient w is the average value of the first difference coefficient k1, the second difference coefficient k2, and the third difference coefficient k3; or, the grayscale compensation coefficient w is the variance of the first difference coefficient k1, the second difference coefficient k2, and the third difference coefficient k3; or, the grayscale compensation coefficient w is the standard deviation of the first difference coefficient k1, the second difference coefficient k2, and the third difference coefficient k3; or, the grayscale compensation coefficient w is the maximum value of the first difference coefficient k1, the second difference coefficient k2, and the third difference coefficient k3; or, the grayscale compensation coefficient w is the minimum value of the first difference coefficient k1, the second difference coefficient k2, and the third difference coefficient k3, etc., without limitation.

[0136] In actual implementation, the electronic device can determine the grayscale compensation coefficient corresponding to each target object according to the target objects with different depths of field included in the RAW domain image, thereby generating a grayscale compensation map (or diff map). In the diff map generated by the electronic device, each pixel block included in the target object corresponds to the grayscale compensation coefficient w. For example, Fig.11 As shown, the pixel block R, the pixel block B and the pixel block G included in the target object all correspond to the grayscale compensation coefficient w.

[0137] In some other embodiments of the present application, the grayscale compensation coefficient w includes a first difference coefficient k1, a second difference coefficient k2, and a third difference coefficient k3. Fig.12 As shown, in the diff map generated by the electronic device, the grayscale compensation coefficient w corresponding to the pixel block R of the red channel is the first difference coefficient k1; the grayscale compensation coefficient w corresponding to the pixel block B of the blue channel is the second difference coefficient k2, and the grayscale compensation coefficient w corresponding to the pixel block G of the green channel is the third difference coefficient k3.

[0138] In some embodiments, Fig.13 As shown, the electronic device generates a diffmap according to the depth map and the difference coefficient map. The depth map is used to indicate the depths of field corresponding to different target objects in the RAW domain image. Fig.13 As shown, the RAW domain image includes target object A (such as a building) and target object B (such as the sky), and the depth of field corresponding to target object A is H a , the depth of field corresponding to the target object B is H b .

[0139] Exemplarily, the electronic device is based on the depth of field H corresponding to the target object A. a, select pixel blocks of different color channels at the position of the screen where the target object A is located, such as the pixel block R of the red channel, the pixel block B of the blue channel and the pixel block G of the green channel, and determine the difference coefficient of the pixel block of each color channel. b , select pixel blocks of different color channels at the screen position where the target object B is located, and determine the difference coefficient of the pixel blocks of each color channel.

[0140] For example, Fig.13 As shown, R(k1-a) represents the difference coefficient corresponding to the pixel block R of the red channel included in the target object A; B(k2-a) represents the difference coefficient corresponding to the pixel block B of the blue channel included in the target object A; G(k3-a) represents the difference coefficient corresponding to the pixel block G of the green channel included in the target object A. Correspondingly, R(k1-b) represents the difference coefficient corresponding to the pixel block R of the red channel included in the target object B; B(k2-b) represents the difference coefficient corresponding to the pixel block B of the blue channel included in the target object B; G(k3-b) represents the difference coefficient corresponding to the pixel block G of the green channel included in the target object B.

[0141] For example, it is assumed that the grayscale compensation coefficient corresponding to the target object A determined by the electronic device is w1, and the grayscale compensation coefficient corresponding to the target object B is w2. Fig.13 As shown, in the diff map generated by the electronic device, the grayscale compensation coefficient corresponding to each pixel block in the target object A is w1, and the grayscale compensation coefficient corresponding to each pixel block in the target object B is w2.

[0142] The following describes in detail the difference coefficient corresponding to the pixel block of each color channel determined by the electronic device. Taking the first difference coefficient corresponding to the pixel block of the red channel determined by the electronic device as an example, illustratively, the red channel includes multiple pixel blocks, each pixel block includes multiple pixels in the form of an n×m matrix, n and m are positive integers, and n×m=p. Optionally, n and m can be the same or different, and there is no restriction on the values ​​of n and m. Optionally, n=m=2, p=4, that is, each pixel block includes four pixels in the form of a 2×2 matrix.

[0143] Exemplarily, the electronic device determines a difference value corresponding to each pixel block in the plurality of pixel blocks, the difference value being used to indicate a brightness difference between the plurality of pixels, and further determines a first difference coefficient corresponding to the pixel block of the red channel according to the difference value corresponding to each pixel block.

[0144] For example, Fig.14 As shown, it is assumed that the red channel of the target object includes four pixel blocks, namely pixel blocks R a , pixel block R b , pixel block Rc and pixel block R d Among them, the pixel block R a The corresponding difference value is x a , pixel block R b The corresponding difference value is x b , pixel block R c The corresponding difference value is x c , pixel block R d The corresponding difference value is x d On this basis, the electronic device can calculate the difference value x a , the difference value x b , the difference value x c , the difference value x d , determine the first difference coefficient corresponding to the pixel block of the red channel.

[0145] Exemplarily, the first difference coefficient is the average value of the difference values ​​corresponding to each pixel block, such as the difference value x a , the difference value x b , the difference value x c , the difference value x d or the first difference coefficient is the variance of the difference value corresponding to each pixel block, such as the difference value x a , the difference value x b , the difference value x c , the difference value x d Variance; or, the first difference coefficient is the standard deviation of the difference value corresponding to each pixel block, such as the difference value x a , the difference value x b , the difference value x c , the difference value x d or the first difference coefficient is the maximum value of the difference values ​​corresponding to each pixel block, such as the difference value x a , the difference value x b , the difference value x c , the difference value x d or the first difference coefficient is the minimum value of the difference values ​​corresponding to each pixel block; for example, the difference value x a , the difference value x b , the difference value x c , the difference value x d The minimum value in is not restricted.

[0146] Taking the first difference coefficient as the average value of the difference values ​​corresponding to each pixel block as an example, illustratively, the electronic device can remove the maximum value and the minimum value of the multiple difference values, and use the average value of the difference values ​​after removing the maximum value and the minimum value as the first difference coefficient.

[0147] In actual implementation, the difference values ​​corresponding to each pixel block are not much different. Considering that there may be calculation errors when the difference values ​​corresponding to the pixel blocks are too large or too small, after removing the maximum and minimum values ​​of the multiple difference values, it can be ensured that no error values ​​(or isolated values) will appear in the multiple difference values. Therefore, the average value of the difference values ​​after removing the maximum and minimum values ​​is used as the first difference coefficient to ensure the accuracy of the first difference coefficient.

[0148] It should be noted that for the examples of the electronic device determining the second difference coefficient corresponding to the pixel block of the blue channel, and the electronic device determining the third difference coefficient corresponding to the pixel block of the green channel, reference can be made to the specific implementation process of the electronic device determining the first difference coefficient, which will not be repeated here.

[0149] The following describes in detail the difference coefficients corresponding to the pixel blocks of each color channel determined by the electronic device. a Taking the corresponding first difference coefficient as an example, for example, Fig.15 As shown, the pixel block R of the red channel a The electronic device includes four pixels in a 2×2 matrix form, namely R0, R1, R2 and R3. On this basis, the electronic device obtains the grayscale value of each pixel in the pixel R0, the pixel R1, the pixel R2 and the pixel R3. The electronic device obtains the grayscale value of each pixel according to the grayscale value of each pixel and the pixel block R a The grayscale reference value is used to determine the grayscale difference value of each pixel. Then, the electronic device determines the grayscale difference value of the pixel block R according to the grayscale difference value of each pixel. a The difference value of .

[0150] For example, Fig.15 As shown, the grayscale value of pixel R0 is x0, the grayscale value of pixel R1 is x1, the grayscale value of pixel R2 is x2, and the grayscale value of pixel R3 is x3. a The grayscale reference value is X. On this basis, the grayscale difference value of pixel R0 is y0=x0-X; the grayscale difference value of pixel R1 is y1=x1-X; the grayscale difference value of pixel R2 is y2=x2-X; and the grayscale difference value of pixel R3 is y3=x3-X.

[0151] Then, the electronic device determines the pixel block R according to the grayscale difference value of each pixel, such as grayscale difference value y0, grayscale difference value y1, grayscale difference value y2, and grayscale difference value y3. a The corresponding difference value.

[0152] Optionally, the difference value corresponding to the pixel block is the average value of the grayscale difference value of each pixel; or, the difference value corresponding to the pixel block is the variance of the grayscale difference value of each pixel; or, the difference value corresponding to the pixel block is the standard deviation of the grayscale difference value of each pixel; or, the difference value corresponding to the pixel block is the maximum value of the grayscale difference value of each pixel; or, the difference value corresponding to the pixel block is the minimum value of the grayscale difference value of each pixel, etc., without limitation.

[0153] In some embodiments of the present application, the grayscale reference value of the pixel block may be preset by the electronic device. Alternatively, in other embodiments of the present application, the grayscale reference value of the pixel block may be determined by the electronic device based on the grayscale value of each pixel in the pixel block.

[0154] Optionally, the grayscale reference value may be the average value of the grayscale value of each pixel; or, the grayscale reference value may be the variance of the grayscale value of each pixel; or, the grayscale reference value may be the standard deviation of the grayscale value of each pixel; or, the grayscale reference value may be the maximum value among the grayscale values ​​of each pixel; or, the grayscale reference value may be the minimum value among the grayscale values ​​of each pixel, etc., without limitation.

[0155] S303: The electronic device uses the grayscale compensation coefficient to compensate the grayscale value of the target object.

[0156] It is understandable that the target object includes pixel blocks of multiple different color channels, each pixel block is a plurality of pixels in the form of an n×m matrix. On this basis, the electronic device performs grayscale value compensation on the pixel blocks of each color channel in the multiple different color channels according to the grayscale compensation coefficient.

[0157] Exemplarily, the target object includes a pixel block R of a red channel, a pixel block B of a blue channel, and a pixel block G of a green channel. The electronic device performs grayscale value compensation on the pixel block R of the red channel, the pixel block B of the blue channel, and the pixel block G of the green channel based on a determined grayscale compensation coefficient w.

[0158] In some embodiments of the present application, the RAW domain image includes a depth of field H a The target object A and the depth of field H b The electronic device determines that the grayscale compensation coefficient corresponding to the target object A is w1, and the grayscale compensation coefficient corresponding to the target object B is w2. On this basis, the electronic device performs grayscale value compensation on the pixel blocks of each color channel included in the target object A according to the grayscale compensation coefficient w1; correspondingly, the electronic device performs grayscale value compensation on the pixel blocks of each color channel included in the target object B according to the grayscale compensation coefficient w2.

[0159] It should be noted that since the phenomenon of crosstalk artifacts (such as grids) is mainly caused by the brightness difference between different pixels, that is, there are differences in the grayscale values ​​between different pixels, therefore, compensating the grayscale values ​​of pixels in the RAW domain image through the grayscale compensation coefficient can achieve the purpose of improving (or even eliminating) the crosstalk artifacts.

[0160] To summarize, by adopting the solution of the embodiment of the present application, the electronic device determines the grayscale compensation coefficient corresponding to the target object with different depths of field in the RAW domain image. Since the target objects with different depths of field have different corresponding grayscale compensation coefficients, the electronic device compensates for the grayscale values ​​of different target objects respectively through different grayscale compensation coefficients, thereby avoiding the problem of over-compensation or under-compensation while eliminating crosstalk artifacts.

[0161] In the above embodiment, the electronic device uses the same grayscale compensation coefficient to perform grayscale value compensation on each pixel included in the target object.

[0162] In some embodiments of the present application, the electronic device may also use the first difference coefficient as the grayscale compensation coefficient to compensate the grayscale value of the pixel block of the red channel. Correspondingly, the electronic device may also use the second difference coefficient as the grayscale compensation coefficient to compensate the pixel block of the blue channel. Correspondingly, the electronic device may also use the third difference coefficient as the grayscale compensation coefficient to compensate the pixel block of the green channel.

[0163] That is to say, for the pixel blocks of the same color channel, the electronic device uses the same difference coefficient to perform grayscale value compensation on the pixel blocks of the color channel. Fig.12 As shown, the electronic device uses the first difference coefficient k1 to compensate the grayscale value of the pixel block R of the red channel. Correspondingly, the electronic device uses the second difference coefficient k2 to compensate the grayscale value of the pixel block B of the blue channel. Correspondingly, the electronic device uses the third difference coefficient k3 to compensate the grayscale value of the pixel block G of the green channel.

[0164] In this way, by using the same difference coefficient to perform grayscale value compensation on the pixel blocks of each color channel, the grayscale value compensation can be made more precise and accurate, and the problem of over-compensation or under-compensation can be further avoided.

[0165] In some other embodiments of the present application, the electronic device may also use the grayscale difference value of the pixel as the grayscale compensation coefficient to perform grayscale value compensation on the pixel. Fig.15As shown, the electronic device uses the grayscale difference value y0 to compensate the grayscale value of pixel R0; accordingly, the electronic device uses the grayscale difference value y1 to compensate the grayscale value of pixel R1; accordingly, the electronic device uses the grayscale difference value y2 to compensate the grayscale value of pixel R2; accordingly, the electronic device uses the grayscale difference value y3 to compensate the grayscale value of pixel R3.

[0166] It should be noted that, for multiple pixels included in other pixel blocks, the electronic device can also perform grayscale value compensation on the pixel according to the grayscale difference value corresponding to the pixel. The specific implementation method can refer to the above embodiment and will not be described in detail here.

[0167] In this way, by using the corresponding grayscale difference value to perform grayscale value compensation for each pixel, the grayscale value compensation can be made more precise and accurate, and the problem of over-compensation or under-compensation can be further avoided.

[0168] It should be noted that in the embodiment of the present application, grayscale value compensation refers to the grayscale value of each pixel in the target object using a grayscale compensation coefficient. The reason why compensating the grayscale value of the pixel can eliminate the phenomenon of crosstalk artifacts is that in a backlit environment such as sunlight or night lights, when using an electronic device to shoot, stray light will be generated, and the stray light will be incident on the microlens 220 at a relatively large incident angle, resulting in different grayscale values ​​of the four pixels in the same pixel block, that is, the grayscale value is unbalanced. This unbalanced grayscale value will cause regular crosstalk artifacts (such as grids) on the RAW domain image, so using a grayscale compensation coefficient to compensate for the grayscale value of the pixel can make the grayscale values ​​of the four pixels in the same pixel block balanced, thereby achieving the purpose of eliminating crosstalk artifacts.

[0169] As an example, the electronic device generates a corresponding diff map for each frame of RAW domain image, and then uses the diff map to process the RAW domain image to eliminate crosstalk artifacts. Or, as another example, the electronic device generates a corresponding diff map based on N consecutive (N is a positive integer greater than 1) frames of RAW domain images, and then uses the diffmap to process the N frames of RAW domain images to eliminate crosstalk artifacts. In this way, by processing N consecutive frames of RAW domain images, the power consumption of the device can be effectively reduced.

[0170] It is understandable that the diff map generated by the electronic device includes the grayscale compensation coefficient corresponding to each pixel block. On this basis, in some embodiments of the present application, the electronic device inputs the diff map and the RAW domain image into a preset model for processing, and finally outputs a three-primary color RGB domain image. Among them, the RGB domain image is obtained after the preset model compensates the grayscale value of the target object in the RAW domain image. In other words, the preset model is used to compensate the grayscale value of each target object in the RAW domain image according to the diff map.

[0171] Exemplarily, the preset model can be any neural network model, for example, the preset model can be a neural network model such as U-net, res-net, mobile-net, efficient-net, etc., or other suitable neural network models, without limitation.

[0172] For example, Fig.16 As shown, the electronic device inputs the diff map and N consecutive RAW domain images into a preset model, and the preset model compensates the grayscale value of each target object in the RAW domain image and finally outputs an RGB domain image.

[0173] In some embodiments of the present application, the electronic device also has the function of training a preset model. Alternatively, in other embodiments of the present application, after the preset model is trained by other devices (such as a model training device), the preset model is saved in the electronic device so that the electronic device can perform the grayscale value compensation function based on the preset model.

[0174] Exemplarily, the preset model is trained by M sample data. The M sample data include M RAW domain images, each of which includes crosstalk artifacts to varying degrees, where M≥1.

[0175] Optional, such as Fig.17 As shown, the training process of the preset model includes: the electronic device randomly generates M diff maps, and simulates different degrees of crosstalk artifacts on the RAW domain image according to the M diff maps to obtain M sample data. Then, the electronic device uses the M sample data as the input of the model and the corresponding RGB domain image as the output of the model to perform model training to obtain the preset model. It can be understood that the above RGB domain image does not include a regular grid.

[0176] It is understandable that, during the model training process, the larger the value of M, that is, the more sample data there is, the more accurate the model training result is, and the more mature the preset model obtained through training is.

[0177] It should be noted that the contents recorded in each embodiment of the present application can explain the technical solutions in other embodiments of the embodiments of the present application, and the technical features recorded in each embodiment can also be applied in other embodiments to form new solutions by combining the technical features in other embodiments. The present application only exemplarily lists several embodiments for illustration, and does not mean that the present application is limited to this.

[0178] The present application provides an electronic device, which may include a camera, a memory, and one or more processors; the memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the functions or steps in the above embodiments. The structure of the electronic device can refer to the above Figure 3 The structure of the electronic device 100 is shown.

[0179] The present application also provides a chip system for use in electronic devices. Fig.18 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 may be Figure 3 The processor 110 is shown. On this basis, the interface circuit 1102 can be, for example, an interface circuit between the processor 1101 and an external memory; or an interface circuit between the processor 1101 and an internal memory.

[0180] The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of the electronic device 100). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various functions or steps executed by the mobile phone in the above embodiment. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0181] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each function or step executed by the electronic device in the above method embodiment.

[0182] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the electronic device in the above method embodiment.

[0183] It should be noted that the terms "first" and "second" in the specification, claims and drawings of this application are used to distinguish different objects rather than to describe a specific order. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more.

[0184] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0185] It should be understood that in the present application, "at least one (item)" refers to one or more. "Multiple" refers to two or more. "At least two (items)" refers to two or three and more than three. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. “When” and “if” both mean that corresponding measures will be taken under certain objective circumstances. It does not limit the time, nor does it require any judgment when it is implemented, nor does it mean that there are other limitations.

[0186] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0187] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0190] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0192] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a camera, the camera includes an image sensor, and the method includes: Acquire an original RAW domain image captured by the image sensor, wherein the RAW domain image includes a plurality of target objects with different depths of field; Determining a grayscale compensation coefficient corresponding to each target object based on the grayscale value of each pixel in the RAW domain image; wherein the grayscale compensation coefficients corresponding to target objects with different depths of field are different; The grayscale compensation coefficient is used to compensate the grayscale value of the target object.

2. The method according to claim 1, characterized in that The image sensor includes a plurality of micro lenses, a filter, and a plurality of photoelectric conversion elements, wherein the filter includes a plurality of filter units; Each microlens is arranged relative to a filter unit and p photoelectric conversion elements, where p≥2 and p is an integer.

3. The method according to claim 2, characterized in that p＝4。 4. The method according to claim 2 or 3, characterized in that: The target object includes a plurality of pixel blocks of different color channels, and the pixel block includes a plurality of pixels in the form of an n×m matrix, where n and m are positive integers; n×m=p; Wherein, determining the grayscale compensation coefficient corresponding to each target object based on the grayscale value of each pixel in the RAW domain image includes: Based on the grayscale value of each pixel in the RAW domain image, determining a difference coefficient corresponding to a pixel block of each color channel in the multiple different color channels; wherein the difference coefficient is used to indicate a brightness difference between pixel blocks of the same color channel; Based on the difference coefficient corresponding to the pixel block of each color channel, a grayscale compensation coefficient corresponding to the target object is determined.

5. The method according to claim 4, characterized in that The plurality of different color channels include a first color channel, a second color channel, and a third color channel; Wherein, determining the grayscale compensation coefficient corresponding to the target object based on the difference coefficient corresponding to the pixel block of each color channel includes: Determine a first difference coefficient corresponding to the pixel block of the first color channel, a second difference coefficient corresponding to the pixel block of the second color channel, and a third difference coefficient corresponding to the pixel block of the third color channel; Based on the first difference coefficient, the second difference coefficient and the third difference coefficient, a grayscale compensation coefficient corresponding to the target object is determined.

6. The method according to claim 5, characterized in that The grayscale compensation coefficient is an average value of the first difference coefficient, the second difference coefficient and the third difference coefficient; or, The grayscale compensation coefficient is the variance of the first difference coefficient, the second difference coefficient and the third difference coefficient; or, The grayscale compensation coefficient is the standard deviation of the first difference coefficient, the second difference coefficient and the third difference coefficient; or, The grayscale compensation coefficient is the maximum value among the first difference coefficient, the second difference coefficient and the third difference coefficient; or The grayscale compensation coefficient is a minimum value among the first difference coefficient, the second difference coefficient and the third difference coefficient.

7. The method according to any one of claims 1 to 3 or 5, characterized in that: The target object includes a plurality of pixel blocks of different color channels, the pixel blocks include a plurality of pixels in a matrix form of n×m, where n and m are positive integers; wherein the grayscale compensation coefficients of the pixel blocks of different color channels of the same target object are different; wherein the grayscale compensation coefficients are used to compensate the grayscale value of the target object, including: The grayscale compensation coefficients of the pixel blocks of each color channel of the target object are used to perform grayscale value compensation on the pixel blocks corresponding to each color channel in the multiple different color channels.

8. The method according to claim 5, characterized in that The using the grayscale compensation coefficient to perform grayscale value compensation on the target object includes: Using the first difference coefficient as the grayscale compensation coefficient, performing grayscale value compensation on the pixel block of the first color channel; Using the second difference coefficient as the grayscale compensation coefficient, performing grayscale value compensation on the pixel block of the second color channel; The third difference coefficient is used as the grayscale compensation coefficient to perform grayscale value compensation on the pixel block of the third color channel.

9. The method according to any one of claims 5, 6 or 8, characterized in that The first color channel includes a plurality of pixel blocks, each pixel block includes a plurality of pixels in a matrix form of n×m, where n and m are positive integers; The step of determining a first difference coefficient corresponding to the pixel block of the first color channel includes: Determine, based on the grayscale value of each pixel in the RAW domain image, a difference value corresponding to each pixel block in the plurality of pixel blocks, wherein the difference value is used to indicate a brightness difference between the plurality of pixels; The first difference coefficient corresponding to the pixel block of the first color channel is determined according to the difference value corresponding to each pixel block.

10. The method according to claim 9, characterized in that The first difference coefficient is an average value of the difference values ​​corresponding to each pixel block; or The first difference coefficient is the variance of the difference value corresponding to each pixel block; or The first difference coefficient is the standard deviation of the difference value corresponding to each pixel block; or The first difference coefficient is the maximum value of the difference values ​​corresponding to each pixel block; or The first difference coefficient is the minimum value of the difference values ​​corresponding to each pixel block.

11. The method according to claim 10, characterized in that The determining a difference value corresponding to each pixel block in the plurality of pixel blocks comprises: Obtaining a grayscale value of each pixel in a plurality of pixels included in the pixel block; Determining a grayscale difference value of each pixel according to the grayscale value of each pixel and the grayscale reference value of the pixel block; According to the grayscale difference value of each pixel, a difference value corresponding to the pixel block is determined.

12. The method according to claim 11, characterized in that The grayscale reference value is the average grayscale value of each pixel; or The grayscale reference value is the variance of the grayscale value of each pixel; or, The grayscale reference value is the standard deviation of the grayscale value of each pixel; or, The grayscale reference value is the maximum value of the grayscale values ​​of each pixel; or The grayscale reference value is the minimum value of the grayscale values ​​of each pixel.

13. The method according to any one of claims 1-3, 5-6, 8 or 10-12, characterized in that: The using the grayscale compensation coefficient to perform grayscale value compensation on the target object includes: The grayscale compensation coefficient and the RAW domain image are input into a preset model, and a three-primary color RGB domain image is output; wherein the RGB domain image is obtained after the preset model performs grayscale value compensation on the target object in the RAW domain image.

14. An electronic device, characterized in that: include: A camera, a memory and one or more processors; the camera includes an image sensor; The memory stores computer program code, which includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1 to 13.

15. The electronic device according to claim 14, characterized in that: The image sensor includes a plurality of micro lenses, a filter, and a plurality of photosensitive elements, wherein the filter includes a plurality of filter units; Each microlens is arranged relative to a filter unit and p photosensitive elements, where p≥2 and p is an integer.

16. The electronic device according to claim 15, characterized in that: p＝4。 17. A chip system, characterized in that: Applied in electronic equipment, the chip system includes: at least one processor and an interface; The interface is used to receive instructions and transmit them to the at least one processor; the at least one processor executes the instructions so that the electronic device executes the method as described in any one of claims 1-13.

18. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 13.

Images