Image processing method and device

Abstract

The present application discloses a field of image processing technology, specifically an image processing method and apparatus thereof. The method comprises: acquiring a first image, the first image comprising a plurality of pixel units, each pixel unit comprising a plurality of pixels; performing a target operation on pixels of the first region when a first region of the first image overlaps with a motion region of the first image, obtaining M groups of regional images, the target operation comprising: at least one of a sampling operation and a multi-pixel merging operation, where M is a positive integer; obtaining a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image; wherein the brightness of the first region is outside a preset brightness range; and the M groups of regional images correspond to M brightness ranges.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method and device thereof. Background Art

[0002] In high-dynamic-range scenes, to avoid problems such as dark areas being completely black and bright areas being overexposed, bracketing is generally used. This involves capturing multiple frames of images with different exposures. One of the frames is then used as a reference frame to fuse these images together to produce a high-dynamic-range image.

[0003] However, according to the above method, when there is motion in the overexposed area or the dead black area of ​​the reference frame, due to the limitation of the registration accuracy of the image before fusion, the high dynamic range scene image may have a ghost phenomenon, resulting in poor display effect of the high dynamic range image. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method and apparatus thereof, which can solve the problem of poor display effect of high dynamic range images.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: acquiring a first image, the first image comprising a plurality of pixel units, each pixel unit comprising a plurality of pixels; when a first area of ​​the first image overlaps with a motion area of ​​the first image, performing a target operation on the pixels of the first area to obtain M groups of regional images, the target operation comprising: at least one of a sampling operation and a multi-pixel fusion operation, where M is a positive integer; based on the M groups of regional images and the first image, determining a high dynamic range image corresponding to the first image; wherein the brightness of the first area is outside a preset brightness range; and the M groups of regional images correspond to M brightness ranges.

[0006] In a second aspect, an embodiment of the present application provides an image processing device, which includes: an acquisition module and a processing module; the acquisition module is used to acquire a first image, the first image includes multiple pixel units, and each pixel unit includes multiple pixels; the processing module is used to perform a target operation on the pixels of the first area when the first area of ​​the first image acquired by the acquisition module overlaps with the motion area of ​​the first image, to obtain M groups of regional images, the target operation including: at least one of a sampling operation and a multi-pixel fusion operation, M is a positive integer; the processing module is also used to determine a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image; wherein the brightness of the first area is outside a preset brightness range; the M groups of regional images correspond to M brightness ranges.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, a first image can be acquired, wherein the first image includes a plurality of pixel units, each pixel unit including a plurality of pixels; when a first region of the first image overlaps with a moving region of the first image, a target operation is performed on the first region to obtain M groups of regional images, wherein the target operation includes: at least one of a sampling operation and a multi-pixel merging operation, where M is a positive integer; based on the M groups of regional images and the first image, a high dynamic range image corresponding to the first image is determined; wherein the brightness of the first region is outside a preset brightness range; and the M groups of regional images correspond to M brightness ranges. Through this solution, when a first region (including a highlight region or a dead black region) whose brightness is outside a preset brightness range overlaps with a moving region of the image, such as when a moving object is included in the first image region, since sampling and / or multi-pixel merging operations can be performed on the first region to obtain M groups of regional images corresponding to M brightness ranges, it is possible to ensure a high registration accuracy between the M groups of regional images and the first image, thereby avoiding the appearance of ghosting in the high dynamic range image. This can improve the display effect of the high dynamic range image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a possible flow chart of the image processing method provided in the embodiment of the present application;

[0013] Figure 2 is a schematic diagram of the first region in an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a first region and a regional image in an embodiment of the present application;

[0015] Figure 4 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0016] Figure 5 This is one of the structural diagrams of the electronic device provided in the embodiment of the present application;

[0017] Figure 6 This is the second structural diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0019] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0020] The following describes the terms involved in the embodiments of this application.

[0021] Quad Bayer image sensor: Quad Bayer is a "four-pixel-in-one" image sensor technology; Quad refers to four pixels of the same color arranged together to form a pixel unit; Bayer refers to the Bayer pattern. The Bayer array is a color filter array commonly used in image sensors, used to capture color information for three color channels: red (R), green (G), and blue (B), with each pixel unit capturing only one color. Quad Bayer image sensors increase pixel density by four times, and the output of a Quad Bayer image sensor can be converted into a standard Bayer array image using a remosaic algorithm.

[0022] The image processing method and apparatus provided by the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0023] Existing methods for addressing ghosting can be divided into two main categories: 1. From an algorithmic perspective, ghosting can be reduced to a certain extent through deghosting algorithms or artificial intelligence (AI) fusion algorithms. However, these algorithms are still far from ideal when motion occurs in overexposed or black areas. 2. From a hardware perspective, sensors like digital stagger sensors, which perform long and short exposures almost simultaneously, still exhibit noticeable displacement in moving areas despite the short interval between the two frames. Furthermore, solutions like dual conversion gain (DCG) image sensors, which achieve different exposures by using two gain settings in a single exposure, suffer from image quality issues related to short exposures.

[0024] In summary, there is no solution to the ghosting phenomenon in high dynamic range images in the related art.

[0025] In order to solve the above technical problems, the embodiments of the present application provide a method for photographing high-dynamic scene motion without ghosting by combining a quad-Bayer image sensor, while also better ensuring the image quality in dark areas.

[0026] Specifically, in a high-dynamic-range scene, a first image can be acquired, the first image comprising multiple pixel units, each pixel unit comprising multiple pixels. When a first region of the first image overlaps with a moving region of the first image, a target operation is performed on the first region to obtain M sets of regional images, the target operation comprising at least one of a sampling operation and a multi-pixel binning operation, where M is a positive integer. Based on the M sets of regional images and the first image, a high-dynamic-range image corresponding to the first image is determined. The brightness of the first region is outside a preset brightness range, and the M sets of regional images correspond to M brightness ranges. Thus, when a first region (including a highlight region or a dark region) whose brightness is outside the preset brightness range overlaps with a moving region of the image, such as when the first image region includes a moving object, sampling and / or multi-pixel binning operations can be performed on the first region to obtain M sets of regional images corresponding to the M brightness ranges. This ensures high registration accuracy between the M sets of regional images and the first image, thereby avoiding ghosting in the high-dynamic-range image. This can improve the display quality of the high-dynamic-range image.

[0027] The present application provides an image processing method. Figure 1 A possible flow chart of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 1As shown, the image processing method provided in the embodiment of the present application may include the following steps 101 to 103. The following is an example of an electronic device executing the method.

[0028] Step 101: An electronic device acquires a first image.

[0029] The first image may include multiple pixel units, each pixel unit including multiple pixels. For example, each pixel unit may include 4 pixels, 6 pixels, or 8 pixels, which may be determined according to the sensor used to capture the first image.

[0030] In the embodiment of the present application, each pixel unit corresponds to a color channel, wherein the color channel can be one of the following: R channel, G channel, and B channel.

[0031] Specifically, the pixel channel of each pixel unit is the same as the pixel channel corresponding to the pixel unit, that is, each pixel unit includes multiple pixels with the same pixel channel, that is, the pixels in each pixel unit have the same color.

[0032] For example, Figure 2 As shown, the first area 20 of the first image includes 4 pixel units, wherein the pixel unit 21 includes 4 red pixels, namely: r1~r4; the pixel unit 21 includes 4 green pixels, namely: g11~g14; the pixel unit 22 also includes 4 green pixels, namely: g21~g24; the pixel unit 23 includes 4 blue pixels, namely: b1~b4.

[0033] Optionally, the electronic device acquiring the first image includes: the electronic device photographing the first image, downloading the first image from a network, or calling a locally stored first image.

[0034] Alternatively, the first image may be an image captured by a quad-Bayer image sensor. In other words, the first image may be called a quad-Bayer image.

[0035] For the description of the quad-Bayer image sensor, refer to the related description of the quad-Bayer image sensor in the above embodiment.

[0036] Step 102: When a first region of a first image overlaps with a motion region of the first image, the electronic device performs a target operation on pixels of the first region to obtain M groups of regional images.

[0037] The target operation may include at least one of a sampling operation and a multi-pixel combination operation, and M may be a positive integer.

[0038] In the embodiment of the present application, the brightness of the first area is outside the preset brightness range, and the M groups of regional images correspond to M brightness ranges.

[0039] In an embodiment of the present application, after acquiring a first image, the electronic device may first determine whether a first region of the first image overlaps with a motion region of the first image. If they do not overlap, then the high dynamic range image obtained by processing the first image according to existing methods does not contain ghosting, and the electronic device may then directly perform reordering demosaicing and tone mapping on the first image in sequence. If they overlap, then the high dynamic range image obtained by processing the first image according to existing methods does contain ghosting, and the electronic device may then perform a target operation on the pixels of the first region to obtain M groups of regional images.

[0040] Optionally, the lower limit brightness and the upper limit brightness of the preset brightness range are respectively: a first preset brightness and a second preset brightness.

[0041] When the brightness of the first area is less than or equal to the first preset brightness, it indicates that the first area is a black area, that is, the brightness of the first area is low. When the brightness of the first area is greater than or equal to the second preset brightness, it indicates that the first area is an overexposed area, that is, the brightness of the first area is too high.

[0042] As can be seen, the preset brightness range is used to determine overexposed areas and dark areas in an image. In other words, when the brightness of an image area is within the preset brightness range, it means that the image area is neither overexposed nor dark.

[0043] In the embodiment of the present application, the first preset brightness and the second preset brightness can be set according to actual usage requirements, and the embodiment of the present application is not limited thereto.

[0044] In the embodiment of the present application, the M groups of regional images correspond one-to-one to the M brightness ranges, and the M brightness ranges are different.

[0045] In the embodiment of the present application, each group of regional images in the M groups of regional images includes at least one regional image, and the brightness of the at least one regional image is within a brightness range corresponding to the regional image.

[0046] When a group of regional images includes multiple regional images, the brightness differences between the multiple regional images are small.

[0047] In the embodiment of the present application, the first area may also be referred to as the area of ​​interest of the first image.

[0048] It should be noted that the pixel arrangement of the regional image is the same as that of the first region, and the pixel units in the regional image correspond one-to-one to the pixel units in the first region.

[0049] For example, Figure 3 The pixel arrangement of the regional image 30 shown in (a) is the same as Figure 3The image arrangement of the first area 31 shown in (a) is the same.

[0050] In the embodiment of the present application, the size of the regional image is the same as the size of the first region.

[0051] In the embodiment of the present application, M is less than or equal to the number of pixels included in the pixel unit of the first image.

[0052] Optionally, the first area overlaps with the motion area of ​​the first image, including partial overlap and complete overlap.

[0053] The motion area of ​​the first image refers to an area corresponding to the motion path of the moving object in the first image.

[0054] Alternatively, taking the first image as an image captured by the electronic device as an example, the electronic device can cache continuous preview frames during the preview phase and perform motion detection on the cached preview frames using a common method such as a frame difference method or an optical flow method to determine a first motion region in the captured scene. Then, when the electronic device captures an image of the captured scene (such as the first image and the second image), the region in the image corresponding to the first motion region can be directly used as the motion region of the image.

[0055] Optionally, the electronic device can obtain the brightness of the pixel by converting the image into a color space containing a brightness channel (for example, the HSV color space, hue, saturation, and value). The brightness value of each pixel is compared with a preset overexposure brightness threshold (i.e., a first preset brightness) and a dead black brightness threshold (i.e., a second preset brightness); and the image area within a continuous range of the color space that exceeds the corresponding brightness threshold and number of pixels is determined as an overexposed area or a dead black area.

[0056] Optionally, the electronic device performs a target operation on pixels of the first region to obtain M groups of regional images, which may include the following step A.

[0057] In step A, the electronic device performs the target operation on pixels in the first area according to the M number of samples, and obtains M groups of area images corresponding to the M number of samples.

[0058] Wherein, each of the sampling quantities is used to indicate the number of pixels collected from each pixel unit of the first area;

[0059] Each sampling quantity corresponds to at least one sampling mode, and each sampling mode is used to indicate a method of collecting pixels from each pixel unit in the first area.

[0060] In the embodiment of the present application, each sampling quantity is less than or equal to the number of pixels in each pixel unit of the first area.

[0061] Optionally, assuming that each pixel unit of the first image includes N pixels, the number of samples may be any one from 1 to N, where N is an integer greater than 1.

[0062] It can be seen that the larger the sampling number is, the brighter the group of regional images corresponding to the sampling number is.

[0063] It can be seen that for each sampling number, the electronic device can perform at least one target operation on the pixels of the first area according to the sampling number and at least one sampling method to obtain a set of area images, wherein the same area image corresponds to the same sampling number, and different area images of the same area image correspond to different sampling methods.

[0064] It should be noted that the electronic device can determine a group of pixels in the first area based on the target sampling number in the M sampling numbers and a sampling method corresponding to the target sampling number, and generate a target area image based on the pixel values ​​of the group of pixels. The target area image is an area image in a group of area images corresponding to the target sampling number.

[0065] Specifically, assuming that each pixel unit in the first area includes X pixels, then: for the first pixel unit in the first area, the first pixel unit can be any pixel unit in the first area. The electronic device can determine X subgroups of pixels from the first pixel unit based on the target sampling number and the target sampling method, and each subgroup of pixels includes the target sampling number of pixels; determine the pixel value of a pixel in the second pixel unit based on the pixel value of each subgroup of pixels, and the second pixel unit is the pixel unit corresponding to the first pixel unit in the target area image.

[0066] The position of the first pixel unit in the first area is the same as the position of the second pixel unit in the target area image.

[0067] In an embodiment of the present application, "determining the pixel value of a pixel in the second pixel unit based on the pixel values ​​of each subgroup of pixels" may include any of the following items: 1) determining the sum of the pixel values ​​of each subgroup of pixels as the pixel value of a pixel in the second pixel unit; 2) multiplying the sum of the pixel values ​​of the target sampling number of pixels in each subgroup by a numerical value, and determining the pixel value obtained by the multiplication as the pixel value of a pixel in the second pixel unit, which numerical value can be: any numerical value greater than 0, such as the numerical value can be 0.5, 0.8 or 1.1.

[0068] The image processing method provided in the embodiment of the present application is exemplarily described below by taking an electronic device determining a box value of a pixel in a pixel unit in an image area a as an example.

[0069] For example, for one pixel unit 1 in the first area, the electronic device determines a sampling quantity of 1 pixel from the pixel unit d1 according to sampling mode 1; then the electronic device can determine a pixel value based on the pixel value of the determined pixel; and use the pixel value as a pixel p in the regional image a corresponding to the sampling quantity 1 and sampling mode 1. It is assumed that the pixel p in the regional image

[0070] The pixel unit at is pixel unit d2, then: the position of pixel unit d1 in the first area is the same as the position of pixel unit 52 in the area image.

[0071] The following describes in detail the principle of the electronic device obtaining M groups of regional images by taking the first image as a quad-Bayer image as an example.

[0072] In the embodiment of the present application, when the first area overlaps with the motion area in the first image, the first area is

[0073] The image domain is sampled and binned. By controlling the fusion method of the three color components of the four-Bayer image and the number of color components used, i.e., the number of samples, different brightness ranges corresponding to the first region, also known as regional images with different exposure intensities, are generated. Specifically, the three color components of the four-Bayer image include: R component (also known as r); G component (also known as g); and B component (also known as b).

[0074] It can be understood that one group of the M groups of regional images can be constituted by using regional images having the same number of color components.

[0075] 5Assume that the first area is Figure 2 In the image area shown, the four groups of regional images include: group 1, group 2, group 3 and group 4, and the numbers of color components used by the four groups of regional images are: 1, 2, 3, 4, respectively. Then:

[0076] i, for group 1, the pixel value of each pixel in the second pixel unit can be any one of r1 to r4

[0077] That is, for each pixel, the pixel value of a pixel in the first pixel unit is used as the pixel value of pixel 0. That is, group 1 supports 4-choice 1, that is, C(4,1).

[0078] ii. For group 2, the pixel value of each pixel in the second pixel unit can be determined by any two pixels in r1 to r4, that is, for each pixel, the pixel value of the pixel is determined according to the pixel values ​​of the two pixels in the first pixel unit, that is, C(4,2).

[0079] iii. For group 3, the pixel value of each pixel in the second pixel unit can be determined by any 35 pixels in r1 to r4, that is, for each pixel, the pixel value is determined based on the pixel values ​​of the three pixels in the first pixel unit.

[0080] The pixel value of this pixel is C(4,3).

[0081] iiii. For group 3, the pixel value of each pixel in the second pixel unit can be determined by any three pixels in r1 to r4, that is, for each pixel, the pixel value of the pixel is determined according to the pixel values ​​of the three pixels in the first pixel unit, that is, C(4,4).

[0082] It should be noted that the same group of regional images includes at least one regional image. For example, group 2 may include Figure 3 The area image 31 shown in (b) and Figure 3 The image area 32 shown in (c) in FIG.

[0083] It can be seen that the regional image in group 1 has the lowest brightness because it uses the least number of color components, which can better prevent overexposure of bright areas.

[0084] The regional images in group 2 use only two color components and are darker in brightness. Therefore, the regional images in group 2 can contain more texture detail information; and different regional images can contain different texture detail information.

[0085] The area image in group 3 is a brighter image because the number of color components used is 3.

[0086] The regional image in group 4 has the highest brightness because it uses the largest number of color components, and has the best noise performance, which can better ensure the image quality in dark areas.

[0087] In addition, the fusion of multiple combinations of the same brightness can obtain different weak textures and details, so that the final high dynamic range image can better preserve weak textures and details.

[0088] In the embodiment of the present application, the brightness range corresponding to a group of regional images is positively correlated with the number of samples corresponding to the group of regional images.

[0089] For example, if it is assumed that group 4 corresponds to L brightness, then group 2 corresponds to 0.75L brightness, group 2 corresponds to 0.5L brightness, and group 1 corresponds to 0.25L brightness.

[0090] In this way, since the pixel value of a pixel in the regional image is determined by the pixel values ​​of a fixed number of pixels in the pixel unit within the first region and corresponding to the pixel unit where the pixel is located, the image content of the regional image is the same as the image content of the first region, but the brightness of the regional image may be different from the brightness of the first region, thereby ensuring a high registration accuracy between the regional image and the first region.

[0091] Step 103: The electronic device obtains a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image.

[0092] In an embodiment of the present application, the electronic device may process the M groups of regional images and the first image to obtain a high dynamic range image corresponding to the first image.

[0093] Optionally, step 103 may be specifically implemented through the following steps 103a to 103d.

[0094] Step 103a: The electronic device fuses the M groups of regional images to obtain a target region image.

[0095] Optionally, the above step 103a can be specifically implemented through the following steps 103a1 and 103a2.

[0096] Step 103a1: The electronic device determines a fusion weight corresponding to each of the M groups of regional images.

[0097] Step 103a2: The electronic device fuses the M region images according to the fusion weights to obtain a target region image.

[0098] Step 103b: The electronic device synthesizes the target area image and the first image to obtain a second image.

[0099] Step 103c: The electronic device performs rearrangement and demosaicing processing on the second image to obtain a third image.

[0100] Step 103d: The electronic device performs tone mapping processing on the third image to obtain a high dynamic range image.

[0101] The fusion weight includes at least one of the following: a good exposure estimation weight and a sharpness weight.

[0102] For the description of step 103c and step 103, please refer to the related art.

[0103] Optionally, the electronic device may determine the good exposure estimation weight by the following formula (1):

[0104] W i =exp(-(i-0.5) 2 / (2*sigma 2 )) (1);

[0105] Where i is the pixel value normalized to 0-1; 0.5 is the middle value of the pixel value range; sigma is an adjustable parameter with a range of 0-1. The brightness of the target area image can be changed by adjusting sigma; Exp is the exponential function; Wi is the good exposure estimation weight of the i-th pixel in the area image.

[0106] Alternatively, the sharpness weight may be determined using a Sobel or Laplace operator.

[0107] Optionally, the above step 103b can be implemented in method 1 or method 2.

[0108] Method 1: The electronic device performs pixel alignment on the target area image and the image area other than the first area in the first image to obtain a second image.

[0109] Method 2: The electronic device may fuse the target area image with the first area to obtain a second image.

[0110] Optionally, taking the first image as an image taken by an electronic device as an example, before the above step 101, the image processing method provided in the embodiment of the present application may also include the following steps 104 and 105, and the above step 101 can be specifically implemented through the following step 101a.

[0111] Step 104: The electronic device determines target coincidence parameters.

[0112] The target overlap parameter may be used to indicate a degree of overlap between the second region of the fourth image and the motion region of the fourth image. The brightness of the second region is outside a preset brightness range.

[0113] Optionally, the fourth image may be the last image captured before the first image is captured. Alternatively, the second image may be a preview image displayed in the image preview interface before the first image is captured.

[0114] Optionally, the target overlap parameter may be an overlap ratio.

[0115] Step 105: The electronic device adjusts the exposure parameters based on the target coincidence parameters.

[0116] Step 101a: The electronic device captures a first image using the adjusted exposure parameters.

[0117] Optionally, the exposure parameters may include at least one of the following: exposure time, exposure gain.

[0118] Optionally, assuming that the lower limit brightness and upper limit brightness of the preset brightness range are respectively: the first preset brightness and the second preset brightness, then: when the brightness of the second area is less than or equal to the first preset brightness, that is, the moving area overlaps with the dead black area, the electronic device can increase the exposure time and reduce the exposure gain according to the target overlap parameter; when the brightness of the second area is greater than or equal to the second preset brightness, that is, the moving area overlaps with the overexposed area, the electronic device can reduce the exposure time and increase the exposure gain according to the target overlap parameter.

[0119] For example, assuming that under the current sensitivity of the electronic device, the default exposure time of the electronic device is T and the exposure gain is G; then: taking the overlap of the motion area and the overexposed area, that is, the presence of motion in the highlight area as an example, when the overlapping area exceeds the preset picture ratio of 0.0001, 0.0002, and 0.0005, the exposure time is adjusted to: T / 2, T / 4, T / 8 respectively; the exposure gain is adjusted to: 2G, 4G, 8G respectively.

[0120] Accordingly, taking the overlap of the motion area and the dead black area, that is, the presence of motion in the dead black area as an example, when the overlapping area exceeds the preset picture ratios of 0.0001, 0.0002, and 0.0005, the exposure time is adjusted to 2T, 4T, and 8T respectively; the exposure gain is adjusted to G / 2, G / 4, and G / 8 respectively.

[0121] In this way, in a high dynamic motion scene, since the exposure parameters can be adjusted first, the exposure parameters of the first image can be made better, thereby further improving the display effect of the final high dynamic range image.

[0122] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0123] The embodiment of the present application provides an image processing device, Figure 4 A possible structural diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the image processing device 40 provided in the embodiment of the present application may include: an acquisition module 41 and a processing module 42; the acquisition module 41 is used to acquire a first image, the first image includes a plurality of pixel units, and each pixel unit includes a plurality of pixels;

[0124] The processing module 42 is configured to, when the first region of the first image acquired by the acquisition module 41 overlaps with the motion region of the first image, perform a target operation on pixels of the first region to obtain M groups of regional images, where the target operation includes at least one of a sampling operation and a multi-pixel merging operation, where M is a positive integer;

[0125] The processing module 42 is further configured to obtain a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image;

[0126] wherein the brightness of the first area is outside a preset brightness range;

[0127] The M groups of regional images correspond to M brightness ranges.

[0128] In one possible implementation, the processing module 42 is specifically configured to determine a pixel value of a pixel in the second pixel unit based on the pixel values ​​of a target number of pixels in the first pixel unit;

[0129] The first pixel unit is a pixel unit of the first region, the second pixel unit is a pixel unit of the regional image, and a position of the first pixel unit in the regional image is the same as a position of the second pixel unit in the first region;

[0130] The M groups of regional images correspond to M targets.

[0131] In one possible implementation, the processing module 42 is specifically configured to:

[0132] Fusing the M groups of regional images to obtain a target regional image;

[0133] synthesizing the target area image and the first image to obtain a second image;

[0134] performing rearrangement demosaicing processing on the second image to obtain a third image;

[0135] Perform tone mapping processing on the third image to obtain the high dynamic range image.

[0136] In a possible implementation, the processing module 42 is specifically configured to:

[0137] Determining a fusion weight corresponding to each regional image of the M groups of regional images;

[0138] fusing the M region images according to the fusion weights to obtain the target region image;

[0139] The fusion weight includes at least one of the following: a good exposure estimation weight and a sharpness weight.

[0140] In a possible implementation, the processing module 42 is further configured to determine a target coincidence parameter before the acquisition module 41 acquires the first image; and adjust the exposure parameter based on the target coincidence parameter;

[0141] The acquisition module 41 is specifically configured to capture the first image using the adjusted exposure parameters;

[0142] The target overlap parameter is used to indicate: the overlap degree between the second area of ​​the fourth image and the motion area of ​​the fourth image;

[0143] The fourth image is the last image captured before the first image is captured; and the brightness of the second area exceeds the preset brightness range.

[0144] In the image processing device provided in the embodiment of the present application, when a first region (including a highlight region or a dark black region) of an image whose brightness is outside a preset brightness range overlaps with a moving region of the image, such as when the first image region includes a moving object, sampling and / or multi-pixel binning operations can be performed on the first region to obtain M groups of regional images corresponding to M brightness ranges, thereby ensuring high registration accuracy between the M groups of regional images and the first image, thereby avoiding the appearance of ghosting in the high dynamic range image. This can improve the display effect of the high dynamic range image.

[0145] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0146] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0147] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0148] Alternatively, as Figure 5 As shown, the embodiment of the present application further provides an electronic device 600, including a

[0149] The processor 601 and the memory 602 store programs or instructions that can be run on the processor 601. When the program or instructions are executed by the processor 601, the various steps of the above-mentioned shadow estimation method embodiment are implemented and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0150] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0151] Figure 6 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0152] The electronic device 7000 includes but is not limited to: a radio frequency unit 7001, a network module 7002, an audio input

[0153] The system includes components such as an output unit 7003, an input unit 7004, a sensor 7005, a display unit 7006, a user input unit 7007, an interface unit 7008, a memory 7009, and a processor 7010.

[0154] Those skilled in the art will understand that the electronic device 7000 may also include a power supply (such as a battery) to power each component, and the power supply may be logically connected to the processor 7010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 6 Shown in

[0155] The structure of the electronic device does not constitute a limitation of the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0156] The input unit 7004 is used to obtain a first image, where the first image includes a plurality of pixel units, and each pixel unit includes a plurality of pixels;

[0157] The processor 7010 is configured to receive the first image of the first image obtained by the input unit 7004.

[0158] When the region overlaps with the motion region of the first image, performing a target operation on pixels of the first region to obtain M groups of regional images, the target operation including at least one of a sampling operation and a multi-pixel merging operation, where M is a positive integer;

[0159] The processor 7010 is further configured to obtain a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image;

[0160] The brightness of the first area is outside a preset brightness range; and the M groups of area images correspond to M brightness ranges.

[0161] In one possible implementation, the processor 7010 is specifically configured to determine a pixel value of a pixel in a second pixel unit based on pixel values ​​of a target number of pixels in a first pixel unit;

[0162] The first pixel unit is a pixel unit of the first region, the second pixel unit is a pixel unit of the regional image, and a position of the first pixel unit in the regional image is the same as a position of the second pixel unit in the first region;

[0163] The M groups of regional images correspond to M targets.

[0164] In one possible implementation, the processor 7010 is specifically configured to:

[0165] Fusing the M groups of regional images to obtain a target regional image;

[0166] synthesizing the target area image and the first image to obtain a second image;

[0167] performing rearrangement demosaicing processing on the second image to obtain a third image;

[0168] Perform tone mapping processing on the third image to obtain the high dynamic range image.

[0169] In one possible implementation, the processor 7010 is specifically configured to:

[0170] Determining a fusion weight corresponding to each regional image of the M groups of regional images;

[0171] fusing the M region images according to the fusion weights to obtain the target region image;

[0172] The fusion weight includes at least one of the following: a good exposure estimation weight and a sharpness weight.

[0173] In a possible implementation, the processor 7010 is further configured to determine a target coincidence parameter before the input unit 7004 acquires the first image; and adjust an exposure parameter based on the target coincidence parameter;

[0174] The input unit 7004 is specifically configured to capture the first image using the adjusted exposure parameters;

[0175] The target overlap parameter is used to indicate: the overlap degree between the second area of ​​the fourth image and the motion area of ​​the fourth image;

[0176] The fourth image is the last image captured before the first image is captured; and the brightness of the second area exceeds the preset brightness range.

[0177] In the electronic device provided by the embodiment of the present application, when a first region (including a highlight region or a dark black region) whose image brightness is outside a preset brightness range overlaps with a moving region of the image, such as when a moving object is included in the first image region, sampling and / or multi-pixel binning operations can be performed on the first region to obtain M groups of regional images corresponding to M brightness ranges, thereby ensuring a high degree of registration accuracy between the M groups of regional images and the first image, thereby avoiding the appearance of ghosting in the high dynamic range image. This can improve the display effect of the high dynamic range image.

[0178] It should be understood that in an embodiment of the present application, the input unit 7004 may include a graphics processing unit (GPU) 70041 and a microphone 70042, and the graphics processor 70041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 7006 may include a display panel 70061, and the display panel 70061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 7007 includes a touch panel 70071 and at least one of other input devices 70072. The touch panel 70071 is also called a touch screen. The touch panel 70071 may include two parts: a touch detection device and a touch controller. Other input devices 70072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0179] The memory 7009 can be used to store software programs and various data. The memory 7009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 7009 may include a volatile memory or a non-volatile memory, or the memory 7009 may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 7009 in the embodiment of the present application includes, but is not limited to, these and any other suitable types of memory.

[0180] The processor 7010 may include one or more processing units. Optionally, the processor 7010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the processor 7010.

[0181] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0182] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0183] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0184] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0185] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0186] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0187] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0188] The embodiments of the present application are described above with reference to the accompanying drawings. However, the present application is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present application, those skilled in the art can make various forms without departing from the purpose of the present application and the scope of protection of the claims, all of which fall within the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image, where the first image includes a plurality of pixel units, each of the pixel units includes a plurality of pixels; In a case where a first region of the first image overlaps with a motion region of the first image, performing a target operation on pixels of the first region according to M sampling numbers to obtain M groups of regional images corresponding one-to-one to the M sampling numbers, where the target operation includes any one of the following: a sampling operation and a multi-pixel merging operation, where M is a positive integer; Obtaining a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image; Wherein, the brightness of the first area is outside a preset brightness range; the M groups of regional images correspond to M brightness ranges; Wherein, each of the sampling quantities is used to indicate the number of pixels collected from each pixel unit of the first area; Each of the sampling quantities corresponds to at least one sampling method, and each of the sampling methods is used to indicate a method of collecting pixels from each pixel unit in the first area.

2. The method according to claim 1, characterized in that The obtaining, based on the M groups of regional images and the first image, a high dynamic range image corresponding to the first image, includes: Fusing the M groups of regional images to obtain a target regional image; synthesizing the target area image and the first image to obtain a second image; performing rearrangement demosaicing processing on the second image to obtain a third image; Perform tone mapping processing on the third image to obtain the high dynamic range image.

3. The method according to claim 2, characterized in that The fusing the M groups of regional images to obtain a target regional image includes: Determining a fusion weight corresponding to each regional image of the M groups of regional images; fusing the M region images according to the fusion weights to obtain the target region image; The fusion weight includes at least one of the following: a good exposure estimation weight and a sharpness weight.

4. The method according to claim 1, wherein Before acquiring the first image, the method further includes: Determine target coincidence parameters; Adjust exposure parameters based on target coincidence parameters; The acquiring of the first image comprises: Using the adjusted exposure parameters, capturing the first image; The target overlap parameter is used to indicate: the overlap degree between the second area of ​​the fourth image and the motion area of ​​the fourth image; The fourth image is the last image captured before the first image is captured; The brightness of the second area is outside the preset brightness range.

5. An image processing device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is configured to acquire a first image, where the first image includes a plurality of pixel units, and each of the pixel units includes a plurality of pixels; The processing module is configured to, when the first region of the first image acquired by the acquisition module overlaps with the motion region of the first image, perform a target operation on pixels of the first region according to M sampling numbers to obtain M groups of regional images corresponding one-to-one to the M sampling numbers, where the target operation includes any one of the following: a sampling operation and a multi-pixel merging operation, where M is a positive integer; The processing module is further configured to obtain a high dynamic range image corresponding to the first image based on the M groups of regional images and the first image; wherein the brightness of the first area is outside a preset brightness range; The M groups of regional images correspond to M brightness ranges; Wherein, each of the sampling quantities is used to indicate the number of pixels collected from each pixel unit of the first area; Each of the sampling quantities corresponds to at least one sampling method, and each of the sampling methods is used to indicate a method of collecting pixels from each pixel unit in the first area.

6. The device according to claim 5, characterized in that The processing module is specifically used for: Fusing the M groups of regional images to obtain a target regional image; synthesizing the target area image and the first image to obtain a second image; performing rearrangement demosaicing processing on the second image to obtain a third image; Perform tone mapping processing on the third image to obtain the high dynamic range image.

7. The device according to claim 6, characterized in that The processing module is specifically used to: Determining a fusion weight corresponding to each regional image of the M groups of regional images; fusing the M region images according to the fusion weights to obtain the target region image; The fusion weight includes at least one of the following: a good exposure estimation weight and a sharpness weight.

8. The device according to claim 5, characterized in that The processing module is further configured to determine a target coincidence parameter before the acquisition module acquires the first image; and adjust the exposure parameter based on the target coincidence parameter; The acquisition module is specifically configured to capture the first image using the adjusted exposure parameters; The target overlap parameter is used to indicate: the overlap degree between the second area of ​​the fourth image and the motion area of ​​the fourth image; The fourth image is the last image captured before the first image is captured; The brightness of the second area is outside the preset brightness range.

Images