Image processing method and device, electronic equipment and medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a medium. The image processing method comprises: determining an exposure group when an image sensor performs image acquisition, the image sensor comprising a plurality of pixel unit groups, each pixel unit group comprising a plurality of pixel units, the exposure group comprising a plurality of exposures, the plurality of exposures corresponding to each pixel unit in each pixel unit group, and the plurality of exposures corresponding to each pixel unit in each pixel unit group; at least part of the exposures are different; performing image acquisition according to the exposure group to obtain a first image; extracting the first image to obtain a second image corresponding to each exposure; and fusing to generate a third image according to the second images. The second images with different exposure can be acquired at the same time, and the acquisition time difference of the second images is reduced, so that the blurring degree of the images is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of shooting technologies, and in particular, to an image processing method, apparatus, electronic device, and medium. Background Art

[0002] Currently, when an electronic device captures an image, through the High Dynamic Range (HDR) function, the image has a larger brightness range and contrast. However, after the high dynamic range function is turned on, the captured image has a blurring problem. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an image processing method, apparatus, electronic device, and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the image processing method including:

[0005] Determine an exposure amount group when an image sensor performs image acquisition, where the image sensor includes a plurality of pixel unit groups, each of the pixel unit groups includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each of the pixel units in each of the pixel unit groups, and at least some of the exposure amounts are different;

[0006] Perform image acquisition according to the exposure amount group to obtain a first image;

[0007] Extract the first image to obtain a second image corresponding to each of the exposure amounts;

[0008] Fuse and generate a third image according to each of the second images.

[0009] In some embodiments of the present disclosure, the performing image acquisition according to the exposure amount group to obtain a first image includes:

[0010] Determine parameter values of exposure parameters corresponding to each of the exposure amounts according to the exposure amount group;

[0011] Write the parameter values of the exposure parameters into the image sensor;

[0012] Perform image acquisition in a line exposure manner to obtain the first image.

[0013] In some embodiments of the present disclosure, the exposure parameters include an exposure time parameter and a gain parameter; the writing the parameter values of the exposure parameters into the image sensor includes:

[0014] Sort the parameter values of the exposure time parameters;

[0015] Write the parameter values of the sorted exposure time parameters and the corresponding parameter values of the gain parameters into the image sensor in units of the pixel unit groups;

[0016] Among them, the parameter values of the exposure time parameters corresponding to the pixel units at the same position in each pixel unit group are the same, and the pixel units at different positions in each pixel unit group respectively correspond to a parameter value of the exposure time parameter and a parameter value of the gain parameter.

[0017] In some embodiments of the present disclosure, the image acquisition is performed in a row exposure manner to obtain the first image, including:

[0018] Perform image acquisition in a multi-row exposure manner to obtain the first image;

[0019] Among them, the number of rows exposed each time is the same as the number of rows of the pixel units in the pixel unit group.

[0020] In some embodiments of the present disclosure, the photosensitive colors of the pixel units in each pixel unit group are the same, and the photosensitive colors of the pixel units in adjacent pixel unit groups are different.

[0021] In some embodiments of the present disclosure, each pixel unit group includes a first pixel unit, a second pixel unit, a third pixel unit, and a fourth pixel unit that are distributed at the four corners and located in adjacent rows. The first pixel unit and the fourth pixel unit are diagonally arranged, and the second pixel unit and the third pixel unit are diagonally arranged;

[0022] The multiple exposure amounts include a first exposure amount corresponding to the first pixel unit, a second exposure amount corresponding to the second pixel unit, a third exposure amount corresponding to the third pixel unit, and a fourth exposure amount corresponding to the fourth pixel unit. The second exposure amount is the same as the third exposure amount, the second exposure amount is greater than the first exposure amount, and the third exposure amount is less than the fourth exposure amount.

[0023] In some embodiments of the present disclosure, the fusion of the third image according to each second image includes:

[0024] Convert each of the second images in the first format into each of the fourth images in the second format;

[0025] Determine multiple to-be-fused images with different exposure amounts from each of the fourth images;

[0026] Fuse each of the to-be-fused images to generate the third image.

[0027] In some embodiments of the present disclosure, determining a plurality of images to be fused from each of the fourth images includes:

[0028] For each exposure amount, if there are the same exposure amounts in the exposure amount group, then among the fourth images with the same exposure amount, the fourth image with the maximum sharpness is used as the image to be fused;

[0029] If there are no same exposure amounts in the exposure amount group, then the fourth image at this exposure amount is used as the image to be fused.

[0030] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, which includes:

[0031] A determination module configured to determine an exposure amount group when an image sensor performs image acquisition, where the image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each pixel unit in each pixel unit group, and at least some of the exposure amounts are different;

[0032] An acquisition module configured to perform image acquisition according to the exposure amount group to obtain a first image;

[0033] An extraction module configured to extract the first image to obtain a second image corresponding to each exposure amount;

[0034] A generation module, the determination module is configured to fuse and generate a third image according to each of the second images.

[0035] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, which includes:

[0036] A processor;

[0037] A memory for storing executable instructions of the processor;

[0038] Wherein, the processor is configured to execute:

[0039] Determine an exposure amount group when an image sensor performs image acquisition, where the image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each pixel unit in each pixel unit group, and at least some of the exposure amounts are different;

[0040] Perform image acquisition according to the exposure amount group to obtain a first image;

[0041] Extract the first image to obtain a second image corresponding to each exposure amount;

[0042] Generate a third image by fusing according to each of the second images.

[0043] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute:

[0044] Determine an exposure amount group during image acquisition by an image sensor, where the image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each pixel unit in each pixel unit group, and at least some of the exposure amounts are different;

[0045] Perform image acquisition according to the exposure amount group to obtain a first image;

[0046] Extract the first image to obtain a second image corresponding to each exposure amount;

[0047] Generate a third image by fusing according to each of the second images.

[0048] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0049] During the process of taking an image, determine the exposure amount group during image acquisition, and at least some of the exposure amounts in the exposure amount group are different so that the captured image has a larger brightness range and contrast. Since at least some of the pixel units in each pixel unit group correspond to different exposure amounts, in the first image obtained by performing image acquisition, there are a plurality of second images with different exposure amounts and at least some of the second images are acquired simultaneously. Extract the first image to obtain a second image corresponding to each exposure amount so that at least some of the second images have different exposure amounts. Generate a third image with a larger brightness range and contrast by fusing according to each of the second images. Since the second images with different exposure amounts can be acquired simultaneously, the time difference between the acquisitions of each second image is reduced, thereby reducing the blurring degree of the image. And, since the second images with different exposure amounts can be acquired simultaneously, it is avoided to adjust the exposure amount to acquire different second images, thereby improving the efficiency of image processing.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0052] Figure 1-1 is a schematic diagram of an image acquisition process;

[0053] Figure 1-2 is a flowchart of an image processing method;

[0054] Figure 2 is a schematic diagram of another image acquisition process;

[0055] Figure 3 is a flowchart of an image processing method shown according to an exemplary embodiment;

[0056] Figure 4 is a schematic structural diagram of an image sensor shown according to an exemplary embodiment;

[0057] Figure 5 is a flowchart of an image processing method shown according to another exemplary embodiment;

[0058] Figure 6 is a flowchart of an image processing method shown according to another exemplary embodiment;

[0059] Figure 7 is a flowchart of an image processing method shown according to another exemplary embodiment;

[0060] Figure 8 is a flowchart of an image processing method shown according to another exemplary embodiment;

[0061] Figure 9 is a block diagram of an image processing apparatus shown according to an exemplary embodiment;

[0062] Figure 10 is a block diagram of an electronic device shown according to an exemplary embodiment.

[0063] In the figure:

[0064] 100 - determination module; 150 - acquisition module; 200 - extraction module; 250 - generation module; 400 - electronic device; 402 - processing component; 404 - memory; 406 - power supply component; 408 - multimedia component; 410 - audio component; 412 - input / output interface; 414 - sensor component; 416 - communication component; 420 - processor. Detailed implementation manners

[0065] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0066] Currently, with the development of electronic devices and the improvement of user requirements, users have increasingly high requirements for the images captured by electronic devices. In order to improve the brightness range and contrast of images, a high dynamic range function has been added to the shooting function of electronic devices to improve the image quality. As Figure 1-1 shown, after the high dynamic range function is turned on during the shooting process, the image sensor will sequentially receive multiple exposure amounts, that is, N, N + 1, N + 2 corresponding to different exposure amounts, and N + 3, N + 4, N + 5 corresponding to different exposure amounts. At each exposure amount, the image sensor will collect multiple images, and each image will be synthesized or screened to obtain an image at one exposure amount. The processed images at multiple exposure amounts are subjected to format conversion and fusion to obtain an image with high dynamic range. Among them, the abscissa t represents time, and the ordinate f represents the framelength parameter of the image sensor. The framelength parameter can reflect the size of the exposure amount. As Figure 1-2 shown, the processing process of the high dynamic range image may include the image sensor collecting an image, performing image signal processing (ISP) on the image to perform format conversion of the image, performing high dynamic range fusion on the format-converted image, and generating a high dynamic range image. However, due to a certain time difference when each image is collected at each exposure amount, and a certain time difference when the image sensor receives different exposure amounts, the fused generated image is blurred.

[0067] In the related art, an image processing method is provided, such as Figure 2As shown in the figure, after enabling the high dynamic range function during the shooting process, the image sensor receives multiple exposure levels at once, that is, N, N+1, and N+2 correspond to different exposure levels, and N+3, N+4, and N+5 correspond to different exposure levels. At each exposure level, the image sensor captures an image, and the images at multiple exposure levels are subjected to format conversion and fusion to obtain an image with high dynamic range. Among them, the abscissa t represents time, and the ordinate f represents the framelength parameter of the image sensor. Since the image sensor receives multiple exposure levels at once, the time difference caused by receiving different exposure levels is eliminated, making the fused image clearer. However, since there is still a certain time difference when capturing each image with different exposure levels, the fused image still has a blurring problem.

[0068] To solve the above technical problems, the present disclosure provides an image processing method. Since at least some of the pixel units in each pixel unit group have different corresponding exposure levels and at least some of the second images are captured simultaneously, the time difference between the captures of the second images is reduced, thereby reducing the blurring degree of the image. And, since the second images with different exposure levels can be captured and extracted according to the exposure level group, there is no need to adjust the exposure level, thereby improving the efficiency of image processing.

[0069] The embodiments of the present disclosure provide an image processing method, as Figure 3 shown, the method includes:

[0070] S100. Determine the exposure level group when the image sensor performs image capture. Among them, the image sensor includes multiple pixel unit groups, each pixel unit group includes multiple pixel units, the exposure level group includes multiple exposure levels, and the multiple exposure levels correspond to each pixel unit in each pixel unit group, and at least some of the exposure levels are different.

[0071] S200. Perform image capture according to the exposure level group to obtain a first image.

[0072] S300. Extract the first image to obtain a second image corresponding to each exposure level.

[0073] S400. Generate a third image by fusing each second image.

[0074] In this embodiment, during the process of capturing an image, an exposure amount group at the time of image acquisition is determined, and at least some of the exposure amounts in the exposure amount group are different so that the captured image has a larger brightness range and contrast. Since at least some of the pixel units corresponding to each pixel unit group have different exposure amounts, in the first image obtained by image acquisition, a plurality of second images with different exposure amounts are included and at least some of the second images are acquired simultaneously. The first image is extracted to obtain the second image corresponding to each exposure amount, so that at least some of the second images have different exposure amounts. According to each second image, a third image with a larger brightness range and contrast is generated by fusion. Since the second images with different exposure amounts can be acquired simultaneously, the time difference between the acquisitions of each second image is reduced, thereby reducing the blurring degree of the image. Moreover, since the second images with different exposure amounts can be acquired simultaneously, the need to adjust the exposure amount to acquire different second images is avoided, thereby improving the efficiency of image processing.

[0075] Exemplarily, each pixel unit group in the image sensor may be as Figure 4 shown. Each pixel unit group includes a plurality of pixel units, which may be 4, 9, 16, etc. For example, when the number of pixel units in each pixel unit group is 4, the pixel units at positions 1, 2, 3, and 4 form a pixel unit group, the pixel units at positions 5, 6, 7, and 8 form a pixel unit group, the pixel units at positions 9, 10, 11, and 12 form a pixel unit group, the pixel units at positions 13, 14, 15, and 16 form a pixel unit group, and so on. In each pixel unit group, at least some of the pixel units have different exposure amounts. For example, the pixel units at positions 1, 5, 9, and 13 correspond to a low exposure amount, the pixel units at positions 2, 6, 10, and 14 correspond to a medium exposure amount, the pixel units at positions 3, 7, 11, and 15 correspond to a medium exposure amount, and the pixel units at positions 4, 8, 12, and 16 correspond to a high exposure amount. In the exposure amount group, the number of different exposure amounts depends on the number of images required for generating the third image by fusion. For example, when three images are required to be fused to generate the third image, there are three different exposure amounts in the exposure amount group. When four images are required to be fused to generate the third image, there are four different exposure amounts in the exposure amount group, and so on.

[0076] Exemplarily, the exposure value group when determining image acquisition by the image sensor in step S100 may be that in response to a user's shooting operation on the electronic device, the electronic device calculates the exposure value group required to generate the third image through a preset algorithm. Since the exposure value group includes multiple exposure values, the exposure value group can be represented by an array EVLIST. When there are 4 pixel units in each pixel unit group, the exposure values in the array EVLIST can be expressed as (EV0, EV0, EV-, EV+). Among them, EV0 represents the exposure value when the ISO is 100, the aperture value is F1, and the exposure time is 1 s. EV- is the exposure value with a reduced exposure level based on EV0. EV+ is the exposure value with an increased exposure level based on EV0. Each pixel unit in each pixel unit group corresponds one-to-one with EV0, EV0, EV-, and EV+.

[0077] In one embodiment, as Figure 5 shown, the first image obtained by performing image acquisition according to the exposure value group in step S200 is determined in the following manner:

[0078] S210. Determine the parameter values of the exposure parameters corresponding to each exposure value according to the exposure value group.

[0079] S220. Write the parameter values of the exposure parameters into the image sensor.

[0080] S230. Perform image acquisition in a line exposure manner to obtain the first image.

[0081] In this embodiment, since the image sensor cannot directly recognize the exposure values in the exposure value group, the parameter values of the exposure parameters corresponding to each exposure value are determined according to the exposure value group. The parameter values of the exposure parameters are written into the image sensor so that each pixel unit in the image sensor can perform exposure according to the corresponding exposure parameters. Under the exposure parameters, image acquisition is performed in a line exposure manner to obtain the first image. By performing image acquisition in a line exposure manner under the exposure parameters, images with different exposure values can be acquired simultaneously after each line or every multiple lines of exposure, thereby reducing the blurring degree of the image and improving the efficiency of image processing.

[0082] Exemplarily, determining the parameter values of the exposure parameters corresponding to each exposure amount in step S210 may be to convert the exposure amount into the parameter values of the exposure parameters through an Auto Exposure (AE) module. For example, the exposure amount can be converted into the parameter values of the exposure time parameter and the gain parameter. Since the exposure time parameter and the gain parameter are two different parameters, when the parameter value of the gain parameter is fixed, the difference in the exposure amount can be reflected by the parameter value of the exposure time parameter. For example, the exposure amounts in the array EVLIST are (EV0, EV0, EV-10, EV+8), and the ratio of the parameter values of the exposure time parameter and the gain parameter obtained can be (10000000:1.2, 10000000:1.2, 8000000:1.0, 15000000:2.0).

[0083] Exemplarily, due to the difference in the type of image sensor, the parameter values of the exposure parameters and the gain parameters can be further converted for identification. For example, the parameter values of the exposure time parameter and the gain parameter can be converted into the parameter values of the FrameLength parameter and the LineCount parameter. At this time, writing the parameter values of the exposure parameters into the image sensor in step S220 may be to write the parameter values of the FrameLength parameter and the LineCount parameter into the image sensor.

[0084] Exemplarily, in addition to using the line exposure method in step S230 to collect an image to obtain a first image, the global exposure method can also be used to collect an image to obtain a first image.

[0085] In one embodiment, as Figure 6 shown, the image acquisition according to the exposure amount group in step S220 to obtain the first image is determined by the following method:

[0086] S221. Sort the parameter values of the exposure time parameters.

[0087] S222. Write the sorted parameter values of the exposure time parameters and the corresponding parameter values of the gain parameters into the image sensor in units of pixel unit groups.

[0088] Among them, the parameter values of the exposure time parameters corresponding to the pixel units at the same position in each pixel unit group are the same, and the pixel units at different positions in each pixel unit group respectively correspond to a parameter value of the exposure time parameter and a parameter value of the gain parameter.

[0089] In this embodiment, since at least some of the exposure amounts in the exposure amount group are different, if the parameter values of each exposure time parameter and the parameter values of the corresponding gain parameter are directly written into the image sensor, it is difficult to determine the exposure amount corresponding to each second image. Since the parameter value of the gain parameter is kept constant when determining the parameter value of the exposure parameter corresponding to the exposure amount, the parameter values of each exposure time parameter can reflect the magnitude of the exposure amount, and the parameter values of each exposure time parameter are sorted. The sorted parameter values of each exposure time parameter and the parameter values of the corresponding gain parameter are written into the image sensor in units of pixel unit groups, so that the pixel units at the same position in each pixel unit group have the same exposure amount, and the pixel units at different positions in each pixel unit correspond to an exposure amount respectively. By sorting the parameter values of each exposure time parameter and then writing them into the image sensor, the exposure amount corresponding to each second image can be determined for image fusion, thereby improving the reliability of image processing.

[0090] Exemplarily, sorting the parameter values of each exposure time parameter in step S221 may be sorting the parameter values of each exposure time parameter from small to large, or sorting the parameter values of each exposure time parameter from large to small, or sorting the parameter values of each exposure time parameter according to a preset rule. For example, as Figure 4 shown, when sorting the parameter values of each exposure time parameter from small to large, after writing the sorted parameter values of each exposure time parameter and the parameter values of the corresponding gain parameter into the image sensor in units of pixel unit groups in step S222, the pixel units at positions 1, 5, 9, and 13 correspond to low exposure amounts, the pixel units at positions 2, 6, 10, and 14 correspond to medium exposure amounts, the pixel units at positions 3, 7, 11, and 15 correspond to medium exposure amounts, and the pixel units at positions 4, 8, 12, and 16 correspond to high exposure amounts.

[0091] Exemplarily, when sorting the parameter values of each exposure time parameter in step S221, after converting the parameter values of the exposure time parameter and the parameter values of the gain parameter into the parameter values of the FrameLength parameter and the parameter values of the LineCount parameter, it may be sorting the parameter values of the FrameLength parameter. The step of writing the sorted parameter values of each exposure time parameter and the parameter values of the corresponding gain parameter into the image sensor in units of pixel unit groups in step S222 may be writing the parameter values of the FrameLength parameter and the parameter values of the corresponding LineCount parameter into the image sensor in units of pixel unit groups.

[0092] In one embodiment, the first image obtained by performing image acquisition in the form of line exposure in step S230 can be determined by the following method:

[0093] Image acquisition is performed in a multi-line exposure manner to obtain a first image.

[0094] Among them, the number of rows exposed each time is the same as the number of rows of pixel units in the pixel unit group.

[0095] In this embodiment, during the process of image acquisition in a line exposure manner, multiple lines are exposed each time to obtain a first image. Since the number of rows exposed each time is the same as the number of rows of pixel units in the pixel unit group, images with different exposure amounts can be acquired simultaneously, thereby reducing the blurring degree of the image and improving the efficiency of image processing.

[0096] Exemplarily, the working principle of image acquisition in a multi-line exposure manner to obtain a first image in the above steps is described. The pixel units in each pixel unit group are two rows, and image acquisition is performed in a double-line exposure manner. After one double-line exposure, the pixel units at positions 1, 2, 3, 4, the pixel units at positions 5, 6, 7, 8, and the remaining pixel units in these two rows can all acquire images simultaneously. Since when generating the third image by fusion, it is necessary to rely on the second image corresponding to each exposure amount, at least part of the images acquired by the pixel units at positions 1, 2, 3, 4 will be fused. Since the pixel units at positions 1, 2, 3, 4 acquire images simultaneously, the time difference during image acquisition of each image is eliminated, thereby reducing the blurring degree of the image. Similarly, there is no time difference in the images acquired by the pixel units in other pixel unit groups, thereby reducing the blurring degree of the image.

[0097] Exemplarily, during the process of image acquisition in a multi-line exposure manner, the number of rows exposed each time can also be different from the number of rows of pixel units in the pixel unit group. When the number of rows exposed each time is an integer multiple of the number of rows of pixel units in the pixel unit group, the achieved effect is the same as the above method, which will not be elaborated here. When the number of rows exposed each time is not an integer multiple of the number of rows of pixel units in the pixel unit group, during the process of writing the parameter value of the exposure parameter into the image sensor in step S220, the corresponding relationship between the pixel unit and the parameter value of the exposure parameter needs to be set. For example, when image acquisition is performed in a progressive exposure manner, the parameter values of the exposure parameters of at least part of the pixel units in each row of pixel units in each pixel unit group are different. The parameter values of the exposure parameters corresponding to the pixel units at positions 1, 2 are different, and the parameter values of the exposure parameters corresponding to the pixel units at positions 3, 4 are different. When image acquisition is performed in a progressive exposure manner, the pixel units at positions 1, 2 can acquire images simultaneously, and the pixel units at positions 3, 4 can acquire images simultaneously, reducing the time difference during image acquisition of each image to reduce the blurring degree of the image.

[0098] In one embodiment, the photosensitive colors of the pixel units in each pixel unit group are the same, and the photosensitive colors of the pixel units in adjacent pixel unit groups are different.

[0099] In this embodiment, since there is an exposure amount corresponding to each pixel unit in the pixel unit group, in order to extract the second images corresponding to each exposure amount for fusing to generate the third image, the photosensitive colors of the pixel units in each pixel unit group need to be the same. In order to enable the second images to reflect various colors, it is necessary to make the photosensitive colors of the pixel units in adjacent pixel unit groups different. By making the photosensitive colors of the pixel units in each pixel unit group the same and the photosensitive colors of the pixel units in adjacent pixel unit groups different, the contents of the extracted second images are the same and the corresponding exposure amounts are different for fusing to generate the third image, thereby reducing the blurring degree of the image and improving the efficiency of image processing.

[0100] Exemplarily, the photosensitive colors of the pixel units at positions 1, 2, 3, and 4 are the same, the photosensitive colors of the pixel units at positions 5, 6, 7, and 8 are the same, the photosensitive colors of the pixel units at positions 9, 10, 11, and 12 are the same, and the photosensitive colors of the pixel units at positions 13, 14, 15, and 16 are the same. The photosensitive colors of the pixel units at positions 1, 2, 3, and 4 are different from the photosensitive colors of the pixel units at positions 5, 6, 7, and 8 and the photosensitive colors of the pixel units at positions 9, 10, 11, and 12, and the photosensitive colors of the pixel units at positions 13, 14, 15, and 16 are different from the photosensitive colors of the pixel units at positions 5, 6, 7, and 8 and the photosensitive colors of the pixel units at positions 9, 10, 11, and 12. For the photosensitive colors of the pixel units in each pixel unit group, they can be set in units of pixel unit groups. For example, the photosensitive color of the pixel units at positions 1, 2, 3, and 4 is red, the photosensitive color of the pixel units at positions 5, 6, 7, and 8 is the first green, the photosensitive color of the pixel units at positions 9, 10, 11, and 12 is the second green, and the photosensitive color of the pixel units at positions 13, 14, 15, and 16 is blue. Among them, the first green and the second green can be the same or different. For the photosensitive colors of the pixel units in other pixel unit groups, they are still set in the same positions in the manner of red, the first green, the second green, and blue, which will not be elaborated here. It can be understood that the photosensitive colors of the pixel units in the pixel unit group can be set not only in the manner of red, the first green, the second green, and blue, but also in the manner of blue, the first green, the second green, and red or other manners, which are not limited here.

[0101] In one embodiment, each pixel unit group includes a first pixel unit, a second pixel unit, a third pixel unit, and a fourth pixel unit that are distributed at the four corners and located in adjacent rows. The first pixel unit and the fourth pixel unit are diagonally arranged, and the second pixel unit and the third pixel unit are diagonally arranged. The multiple exposure amounts include a first exposure amount corresponding to the first pixel unit, a second exposure amount corresponding to the second pixel unit, a third exposure amount corresponding to the third pixel unit, and a fourth exposure amount corresponding to the fourth pixel unit. The second exposure amount is the same as the third exposure amount, the second exposure amount is greater than the first exposure amount, and the third exposure amount is less than the fourth exposure amount.

[0102] In this embodiment, since the number of second images with different required exposure amounts is generally within four when generating the third image by fusion, each pixel unit group only needs to include four pixel units to avoid blurring of the extracted second images. Since the pixel units in each pixel unit group are distributed at the four corners and located in two rows, the images collected by the pixel units in the same pixel unit group reflect substantially the same content and at least some of the images have different exposure amounts to generate the third image by fusion, and it is convenient to use the row exposure method for image acquisition to reduce the blurring degree of the image. Since the first exposure amount, the second exposure amount, the third exposure amount, and the fourth exposure amount are sorted in order of magnitude, it is convenient to extract each second image from the corresponding position of the first image, thereby reducing the complexity of image processing.

[0103] Exemplarily, the first pixel unit is a pixel unit located at positions corresponding to the first exposure amount such as 1, 5, 9, 13, etc., the second pixel unit is a pixel unit located at positions corresponding to the second exposure amount such as 2, 6, 10, 14, etc., the third pixel unit is a pixel unit located at positions corresponding to the third exposure amount such as 3, 7, 11, 15, etc., and the fourth pixel unit is a pixel unit located at positions corresponding to the fourth exposure amount such as 4, 8, 12, 16, etc.

[0104] Exemplarily, for the extraction of the first image in step S300 to obtain the second image corresponding to each exposure amount, the images collected by the pixel units corresponding to each exposure amount in the first image can be extracted respectively to obtain the second image corresponding to each exposure amount. For example, the image collected by the pixel units located at positions corresponding to the first exposure amount such as 1, 5, 9, 13, etc. can be extracted to obtain the second image, the image collected by the pixel units located at positions corresponding to the second exposure amount such as 2, 6, 10, 14, etc. can be extracted to obtain the second image, the image collected by the pixel units located at positions corresponding to the third exposure amount such as 3, 7, 11, 15, etc. can be extracted to obtain the second image, and the image collected by the pixel units located at positions corresponding to the fourth exposure amount such as 4, 8, 12, 16, etc. can be extracted to obtain the second image.

[0105] In one embodiment, as Figure 7As shown, the generation of the third image by fusing according to each second image in step S400 is determined as follows:

[0106] S410. Convert each second image in the first format into each fourth image in the second format.

[0107] S420. Determine multiple to-be-fused images with different exposure amounts from each fourth image.

[0108] S430. Fuse each to-be-fused image to generate the third image.

[0109] In this embodiment, since the first format of the first image collected by the image sensor cannot be recognized, and the first formats of the second images extracted from the first image also cannot be recognized, it is necessary to convert the format of the second images. Convert each second image in the first format into each fourth image in the second format to generate the third image by fusing according to each fourth image. Since the number of fourth images may be greater than the number required to generate the third image by fusing, determine multiple to-be-fused images with different exposure amounts from each fourth image to generate the third image by fusing. By converting the format of the second images and screening the fourth images, the third image with a larger brightness range and contrast is generated by fusing, thereby improving the reliability of image processing.

[0110] Exemplarily, the first format can be, for example, the RAW format, and the second format can be, for example, the YUV format.

[0111] Exemplarily, if the first format can be recognized and the third image is generated by fusing according to the second images in the first format, step S410 does not need to be executed, and multiple to-be-fused images with different exposure amounts are determined from each second image to generate the third image by fusing.

[0112] Exemplarily, the conversion of each second image in the first format into each fourth image in the second format in step S410 can be to perform image signal processing on each second image to perform the format conversion of the second image. In the process of converting each second image in the first format into each fourth image in the second format, PD data correction (PDPC), demosaic (Demosaic), color correction (CC), local brightening (GTM), overall brightening (Gamma), grid noise reduction (HNR), downsampling (Down Scaler), etc. can be performed on each second image in the first format to obtain each fourth image in the second format.

[0113] In one embodiment, the determination of multiple to-be-fused images from each fourth image in step S420 is determined as follows:

[0114] For each exposure amount, if there are the same exposure amounts in the exposure amount group, among the fourth images with the same exposure amount, the fourth image with the maximum sharpness is used as the image to be fused.

[0115] If there are no same exposure amounts in the exposure amount group, the fourth image corresponding to that exposure amount is used as the image to be fused.

[0116] In this embodiment, since images to be fused with different exposure amounts are required when generating the third image, and the exposure amounts of some fourth images are the same, it is necessary to screen out the images to be fused from multiple fourth images with the same exposure amount. Since when collecting images, the sharpness of the images collected by different pixel units may be different, among the fourth images with the same exposure amount, the fourth image with the maximum sharpness is used as the image to be fused to improve the quality of the third image. When an exposure amount is not the same as any other exposure amount, the corresponding fourth image is unique, and the fourth image corresponding to that exposure amount is used as the image to be fused. By screening the fourth images, each image to be fused has a high sharpness, thereby improving the effect of image processing.

[0117] Exemplarily, when the exposure amounts in the exposure amount group are (EV0, EV0, EV-, EV+), the number of fourth images with the exposure amount of EV0 is two, and the fourth image with the maximum sharpness is selected from the two fourth images as the image to be fused. The number of fourth images with the exposure amount of EV- and EV+ is one respectively, and both are used as the images to be fused.

[0118] Exemplarily, as the number of the same exposure amounts in the exposure amount group increases, the number of available fourth images increases. Since the number of available fourth images is large, the sharpness of the selected image to be fused can be improved, thereby improving the quality of the image.

[0119] Exemplarily, in step S430, fusing each image to be fused to generate the third image may be to fuse each image to be fused by a Gaussian pyramid or Laplacian pyramid fusion method to generate the third image.

[0120] Exemplarily, after generating the third image, the third image in the third format may be converted into a fifth image in the fourth format. The fourth format may be, for example, the JPEG format.

[0121] The embodiments of the present disclosure provide an image processing method, as Figure 8 shown, the method includes:

[0122] S500. Determine the exposure value group for image acquisition by the image sensor. The image sensor includes multiple pixel unit groups, and each pixel unit group includes multiple pixel units. The exposure value group includes multiple exposure values, and the multiple exposure values correspond to each pixel unit in each pixel unit group, and at least some of the exposure values are different.

[0123] S510. Determine the parameter values of the exposure parameters corresponding to each exposure value according to the exposure value group.

[0124] S520. Sort the parameter values of the respective exposure time parameters.

[0125] S530. Write the sorted parameter values of the respective exposure time parameters and the parameter values of the corresponding gain parameters into the image sensor in units of pixel unit groups.

[0126] S540. Perform image acquisition in a multi-line exposure manner to obtain a first image, where the number of lines for each exposure is the same as the number of lines of the pixel units in the pixel unit group.

[0127] S550. Extract the first image to obtain second images corresponding to each exposure value.

[0128] S560. Convert the second images in the first format into fourth images in the second format.

[0129] S570. Determine multiple unfused images with different exposure values from the respective fourth images.

[0130] S580. Fuse the respective unfused images to generate a third image.

[0131] In this embodiment, during the process of capturing an image, an exposure amount group during image acquisition by an image sensor is determined, so as to generate an image with a larger brightness range and contrast through image fusion corresponding to different exposure amounts. According to the exposure amount group, each exposure amount is converted to obtain a parameter value of a corresponding exposure parameter. Since the parameter value of the gain parameter is fixed, the magnitude of the exposure amount is reflected by the parameter value of the exposure time parameter. The parameter values of the exposure time parameters are sorted to sort the exposure amounts. The sorted parameter values of the exposure time parameters and the parameter values of the corresponding gain parameters are written into the image sensor in units of pixel unit groups. Each pixel unit group corresponds to the same exposure amount group, and each pixel unit in each pixel unit group corresponds to an exposure amount. Image acquisition is performed in a multi-line exposure manner, and images with different exposure amounts can be acquired simultaneously to obtain a first image including corresponding multiple exposure amounts. The first image is extracted to obtain a second image corresponding to each exposure amount, and the format of the second image is converted. Since there may be multiple fourth images with the same exposure amount, multiple second images with different exposure amounts are determined from the fourth images for fusion to generate a third image with a high brightness range and contrast. Since the second images with different exposure amounts can be acquired simultaneously, the time difference between the acquisitions of the second images is reduced, thereby reducing the blurring degree of the image. Moreover, since the second images with different exposure amounts can be acquired simultaneously, adjusting the exposure amount to acquire different second images is avoided, thereby improving the efficiency of image processing.

[0132] In an exemplary embodiment, an image processing apparatus is provided. The image processing apparatus is used to implement the above method. Refer to Figure 9 As shown, the image processing apparatus may include a determination module 100, an acquisition module 150, an extraction module 200, and a generation module 250. Among them, during the process of implementing the above method,

[0133] The determination module 100 is configured to determine an exposure amount group during image acquisition by an image sensor. The image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each pixel unit in each pixel unit group, and at least some of the exposure amounts are different.

[0134] The acquisition module 150 is configured to perform image acquisition according to the exposure amount group to obtain a first image.

[0135] The extraction module 200 is configured to extract the first image to obtain a second image corresponding to each exposure amount.

[0136] The generation module 250 is configured to fuse and generate a third image according to the second images.

[0137] In an exemplary embodiment, an image processing apparatus is provided. In this apparatus, an acquisition module 150 is configured to:

[0138] Determine the parameter values of the exposure parameters corresponding to each exposure amount according to the exposure amount group.

[0139] Write the parameter values of the exposure parameters into the image sensor.

[0140] Perform image acquisition in a line exposure manner to obtain a first image.

[0141] In an exemplary embodiment, an image processing apparatus is provided. In this apparatus, an acquisition module 150 is configured to:

[0142] Sort the parameter values of the respective exposure time parameters.

[0143] Write the sorted parameter values of the respective exposure time parameters and the parameter values of the corresponding gain parameters into the image sensor in units of pixel cell groups.

[0144] In an exemplary embodiment, an image processing apparatus is provided. In this apparatus, an acquisition module 150 is configured to:

[0145] Perform image acquisition in a multi-line exposure manner to obtain a first image.

[0146] In an exemplary embodiment, an image processing apparatus is provided. In this apparatus, a generation module 250 is configured to:

[0147] Convert the respective second images in the first format into the respective fourth images in the second format.

[0148] Determine a plurality of to-be-fused images with different exposure amounts from the respective fourth images.

[0149] Fuse the respective to-be-fused images to generate a third image.

[0150] In an exemplary embodiment, an image processing apparatus is provided. In this apparatus, a generation module 250 is configured to:

[0151] For each exposure amount, if there are the same exposure amounts in the exposure amount group, use the fourth image with the maximum sharpness among the fourth images with the same exposure amount as the to-be-fused image.

[0152] If there are no same exposure amounts in the exposure amount group, use the fourth image at this exposure amount as the to-be-fused image.

[0153] In an exemplary embodiment, an electronic device is provided. The electronic device is, for example, a mobile phone, a laptop computer, a tablet computer, a wearable device, etc.

[0154] Reference Figure 10 As shown, the electronic device 400 may include one or more of the following components: a processing component 402, a memory 404, a power supply component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 416.

[0155] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0156] The memory 404 is configured to store various types of data to support the operation of the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, videos, and the like. The memory 404 may be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks.

[0157] The power supply component 406 provides power to various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 400.

[0158] The multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 408 includes a front camera module and / or a rear camera module. When the electronic device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera module and / or the rear camera module can receive external multimedia data. Each of the front camera module and the rear camera module can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0159] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 404 or transmitted via the communication component 416. In some embodiments, the audio component 410 further includes a speaker for outputting audio signals.

[0160] The I / O interface 412 provides an interface between the processing component 402 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0161] The sensor component 414 includes one or more sensors for providing a status assessment of various aspects of the electronic device 400. For example, the sensor component 414 can detect the on / off state of the electronic device 400, the relative positioning of components, such as the display and the keypad of the electronic device 400. The sensor component 414 can also detect a change in the position of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor component 414 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 414 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 414 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0162] The communication component 416 is configured to facilitate communication between the electronic device 400 and other terminals in a wired or wireless manner. The electronic device 400 can access a communication standard-based wireless network, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0163] In an exemplary embodiment, the electronic device 400 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described method.

[0164] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the above instructions can be executed by a processor 420 of the electronic device 400 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the method shown in the above embodiments.

[0165] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0166] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0167] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, the image processing method includes: determining an exposure amount group when an image sensor performs image acquisition, wherein the image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each pixel unit in each pixel unit group, and at least some of the exposure amounts are different; performing image acquisition according to the exposure amount group to obtain a first image; extracting the first image to obtain a second image corresponding to each exposure amount; fusing and generating a third image according to each second image.

2. The image processing method according to claim 1, characterized in that, the performing image acquisition according to the exposure amount group to obtain a first image includes: determining parameter values of exposure parameters corresponding to each exposure amount according to the exposure amount group; writing the parameter values of the exposure parameters into the image sensor; performing image acquisition in a line exposure manner to obtain the first image.

3. The image processing method according to claim 2, characterized in that, the exposure parameters include an exposure time parameter and a gain parameter; the writing the parameter values of the exposure parameters into the image sensor includes: sorting the parameter values of the exposure time parameters; writing the sorted parameter values of the exposure time parameters and the corresponding parameter values of the gain parameters into the image sensor in units of the pixel unit groups; wherein, the parameter values of the exposure time parameters corresponding to the pixel units at the same position in each pixel unit group are the same, and each pixel unit at a different position in each pixel unit group corresponds to a parameter value of an exposure time parameter and a parameter value of a gain parameter.

4. The image processing method according to claim 2, characterized in that, the performing image acquisition in a line exposure manner to obtain the first image includes: performing image acquisition in a multi-line exposure manner to obtain the first image; wherein, the number of lines exposed each time is the same as the number of lines of the pixel units in the pixel unit group.

5. The image processing method according to claim 1, characterized in that, the photosensitive colors of the pixel units in each pixel unit group are the same, and the photosensitive colors of the pixel units in adjacent pixel unit groups are different.

6. The image processing method according to any one of claims 1 to 5, characterized in that, each pixel unit group includes a first pixel unit, a second pixel unit, a third pixel unit, and a fourth pixel unit that are distributed at the four corners and are located in adjacent rows, the first pixel unit and the fourth pixel unit are diagonally arranged, and the second pixel unit and the third pixel unit are diagonally arranged; The multiple exposure amounts include a first exposure amount corresponding to the first pixel unit, a second exposure amount corresponding to the second pixel unit, a third exposure amount corresponding to the third pixel unit, and a fourth exposure amount corresponding to the fourth pixel unit. The second exposure amount is the same as the third exposure amount. The second exposure amount is greater than the first exposure amount. The third exposure amount is less than the fourth exposure amount.

7. The image processing method according to any one of claims 1 to 5, wherein, the fusing the second images to generate a third image includes: converting each of the second images in a first format into a fourth image in a second format; determining, from each of the fourth images, a plurality of to-be-fused images with different exposure amounts; fusing each of the to-be-fused images to generate the third image.

8. The image processing method according to claim 7, wherein, the determining, from each of the fourth images, a plurality of to-be-fused images includes: for each of the exposure amounts, if there are identical exposure amounts in the exposure amount group, taking, as the to-be-fused image, the fourth image with the highest sharpness among the fourth images with the same exposure amount; if there are no identical exposure amounts in the exposure amount group, taking, as the to-be-fused image, the fourth image corresponding to that exposure amount.

9. An image processing apparatus, wherein, the image processing apparatus includes: a determining module configured to determine an exposure amount group when an image sensor performs image acquisition. The image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each of the pixel units in each pixel unit group, and at least some of the exposure amounts are different; an acquisition module configured to perform image acquisition according to the exposure amount group to obtain a first image; an extraction module configured to extract the first image to obtain a second image corresponding to each exposure amount; a generation module configured to fuse the second images to generate a third image.

10. An electronic device, wherein, the electronic device includes: a processor; a memory for storing executable instructions of the processor; wherein, the processor is configured to execute: determining an exposure amount group when an image sensor performs image acquisition. The image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each of the pixel units in each pixel unit group, and at least some of the exposure amounts are different; performing image acquisition according to the exposure amount group to obtain a first image; extracting the first image to obtain a second image corresponding to each exposure amount; fusing the second images to generate a third image.

11. A non-transitory computer-readable storage medium, wherein, When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform: Determine an exposure amount group during image acquisition by an image sensor, where the image sensor includes a plurality of pixel unit groups, each pixel unit group includes a plurality of pixel units, the exposure amount group includes a plurality of exposure amounts, the plurality of exposure amounts correspond to each of the pixel units in each pixel unit group, and at least some of the exposure amounts are different; Perform image acquisition according to the exposure amount group to obtain a first image; Extract the first image to obtain a second image corresponding to each exposure amount; Fuse and generate a third image according to each of the second images.

Images