Image processing method, electronic device and storage medium

Abstract

Provided are an image processing method, an electronic device, and a storage medium. The image processing method includes: acquiring a first image and a second image shot for the same scene, wherein a first exposure of the first image is greater than a second exposure of the second image, and the first image and the second image have the same image area; determining a grayscale value of a first object in the scene corresponding to a pixel position of at least one first pixel in the first image based on a first response curve corresponding to the first exposure and a pixel value of at least one first pixel in the first image; determining a grayscale value of a second object in the scene corresponding to a pixel position of at least one second pixel in the second image based on a second response curve corresponding to the second exposure and a pixel value of at least one second pixel in the second image; and determining a target image based on the grayscale value of the first object and the grayscale value of the second object.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, a method for calibrating a response curve of an image sensor, an electronic device, and a storage medium. Background Art

[0002] The dynamic range (DR) of brightness in real scenes is usually greater than the dynamic range of the camera. When the camera captures an image, a single exposure can only record a portion of the dynamic range in the real scene. Therefore, the captured image often has overexposed or underexposed areas, resulting in the loss of scene detail information and poor image visual effects.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides an image processing method, an electronic device and a storage medium to achieve high-quality and efficient image fusion.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: acquiring a first image and a second image taken for the same scene, wherein a first exposure of the first image is greater than a second exposure of the second image, and the first image and the second image have the same image area; determining a grayscale value of a first object in the scene corresponding to a pixel position of at least one first pixel in the first image based on a first response curve corresponding to the first exposure and a pixel value of at least one first pixel in the first image; determining a grayscale value of a second object in the scene corresponding to a pixel position of at least one second pixel in the second image based on a second response curve corresponding to the second exposure and a pixel value of at least one second pixel in the second image; and determining a target image based on the grayscale value of the first object and the grayscale value of the second object.

[0006] According to another aspect of the present disclosure, a method for calibrating a response curve of an image sensor is provided, comprising: acquiring a first calibration image including a plurality of calibrated grayscales in a first calibrated grayscale group at a first calibrated exposure; determining a first calibration curve based on pixel values ​​at a plurality of first calibrated positions in the first calibrated image that respectively correspond to the plurality of calibrated grayscales in the first calibrated grayscale group, wherein the first calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the first calibrated exposure and grayscale values ​​of the plurality of calibrated grayscales in the first calibrated grayscale group; acquiring a second calibration image including the plurality of calibrated grayscales in a second calibrated grayscale group at a second calibrated exposure different from the first calibrated exposure; determining a first calibration curve based on pixel values ​​at a plurality of first calibrated positions in the first calibrated image that respectively correspond to the plurality of calibrated grayscales in the first calibrated grayscale group A second calibration curve is determined by pixel values ​​at multiple second calibration positions in the second calibration image that respectively correspond to multiple calibrated grayscales in the second calibrated grayscale group, wherein the second calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor under the second exposure and grayscale values ​​of multiple calibrated grayscales in the second calibrated grayscale group; and a response curve of the image sensor is determined based on the first calibration curve and the second calibration curve, wherein the response curve indicates a mapping relationship between pixel values ​​collected by the image sensor and the exposure used when collecting the image and grayscale values ​​within a grayscale range of multiple calibrated grayscales including at least one of the first calibrated grayscale group and the second calibrated grayscale group.

[0007] According to another aspect of the present disclosure, an electronic device is provided, which includes: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a program is provided. The program includes instructions, which, when executed by a processor of an electronic device, cause the electronic device to execute the above method.

[0009] According to another aspect of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0010] According to the embodiments of the present disclosure, by using the response curve to determine the grayscale value corresponding to the image pixel value in the real environment under the current exposure, the real brightness information of the object in the image can be easily obtained. By using the response curve corresponding to different exposures, all dynamic ranges in the scene can be covered, so that an image with a high dynamic range can be obtained.

[0011] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A flowchart showing an exemplary process of an image processing method according to an embodiment of the present disclosure;

[0014] Figure 2A shows an example of a first response curve and a second response curve according to an embodiment of the present disclosure;

[0015] Figure 2B shows an example of a first response curve and a second response curve and an inverse mapping curve according to an embodiment of the present disclosure;

[0016] Figure 3 A flowchart showing an exemplary process of a method for calibrating a response curve according to an embodiment of the present disclosure;

[0017] Figure 4A shows an example of a calibration curve according to an embodiment of the present disclosure;

[0018] Figure 4B shows an example of an extended calibration curve according to an embodiment of the present disclosure;

[0019] Figure 5A An example of an image processing process according to an embodiment of the present disclosure is shown;

[0020] Figure 5B An example of an image captured by an image sensor according to an embodiment of the present disclosure is shown;

[0021] Figure 5C shows an example of a target image according to an example of the present disclosure; and

[0022] Figure 6 is a block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0024] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0025] In the related art, the Debevec method can be used to recover a high dynamic range radiance map from photos taken by an image sensor based on the physical properties of the imaging system (i.e., photochemical and electronic reciprocity). In this method, multiple photos of a scene are taken with different exposures. The algorithm uses these differently exposed photos to recover the response function of the imaging process, using the assumption of reciprocity based on a scaling factor. Using the known response function, the Debevec method can fuse multiple photos into a single high dynamic range brightness map whose pixel values ​​are proportional to the true brightness radiance values ​​in the scene.

[0026] As with film, the most significant nonlinearity in the response curve is at the saturation point, where any pixel with radiance above a certain level is mapped to the same maximum image value, so the dynamic range of a single image is limited. To obtain a response curve with a high dynamic range, you can choose a range of luminance radiance values ​​of interest and determine an appropriate exposure time. Cover the entire dynamic range in the scene by taking and blending a series of photos at different exposures.

[0027] Using the Debevec method, a response curve describing the relationship between the pixel value in the image and the exposure value used when acquiring the image can be obtained through a series of photos of the scene with different exposures. Therefore, the Debevec method does not introduce the actual brightness of the scene to calibrate the response curve, and can only deduce the relative brightness between pixels. When the picture is too dark or too bright, the estimated relative brightness also has a large error compared to the actual brightness. In addition, the Debevec method does not take into account the exposure gain parameters of the current complementary metal oxide semiconductor (CMOS) sensor.

[0028] In order to solve the above problems in the related art, the present disclosure provides a new method for acquiring a target image with a high dynamic range by using a response curve. The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0029] Figure 1 A flowchart showing an exemplary process of an image processing method according to an embodiment of the present disclosure is shown.

[0030] like Figure 1As shown, in step S102, a first image and a second image shot for the same scene may be acquired, wherein the first exposure of the first image may be greater than the second exposure of the second image, and the first image and the second image may have the same image area.

[0031] In step S104, based on the first response curve corresponding to the first exposure and the pixel value of at least one first pixel in the first image, the grayscale value of the first object in the scene corresponding to the pixel position of the at least one first pixel in the first image can be determined.

[0032] In step S106, based on the second response curve corresponding to the second exposure and the pixel value of at least one second pixel in the second image, the grayscale value of the second object in the scene corresponding to the pixel position of at least one second pixel in the second image can be determined.

[0033] In step S108 , a target image may be determined based on the grayscale value of the first object and the grayscale value of the second object.

[0034] By using the image processing method provided by the embodiments of the present disclosure, it is possible to use response curves corresponding to different exposure amounts to respectively determine the true brightness of the objects in the scene corresponding to each pixel in the images acquired based on the different exposure amounts, thereby being able to further use the true brightness of the objects in the scene to determine a target image containing the objects in the scene, wherein the dynamic range of the target image is greater than that of the first image and the second image.

[0035] Each step of the method 100 is described in detail below.

[0036] In step S102, a first image and a second image captured for the same scene may be acquired, wherein a first exposure amount of the first image may be greater than a second exposure amount of the second image, and the first image and the second image may have the same image area.

[0037] In some embodiments, the first image and the second image include almost the same object. Since the first exposure used when acquiring the first image and the second exposure used when acquiring the second image are different, the first image and the second image have different dynamic ranges. When the first exposure is greater than the second exposure, for an area with a higher actual brightness in the captured scene, the area may be overexposed in the image acquired using the first exposure.

[0038] In some implementations, the first exposure amount may be determined based on a first exposure time used when acquiring the first image, and the second exposure amount may be determined based on a second exposure time used when acquiring the second image. In other implementations, the first exposure amount may be determined based on a first exposure time and a first exposure gain used when acquiring the first image, and the second exposure amount may be determined based on a second exposure time and a second exposure gain used when acquiring the second image. For example, the first exposure amount may be the product of the first exposure time and the first exposure gain, and the second exposure amount may be the product of the second exposure time and the second exposure gain. By controlling the exposure time and exposure gain of the image sensor, the exposure amount used by the image sensor when acquiring the image may be adjusted, thereby enabling the acquisition of the first image and the second image with different dynamic ranges.

[0039] In step S104, based on the first response curve corresponding to the first exposure and the pixel value of at least one first pixel in the first image, the grayscale value of the first object in the scene corresponding to the pixel position of the at least one first pixel in the first image can be determined.

[0040] In step S106, based on the second response curve corresponding to the second exposure and the pixel value of at least one second pixel in the second image, the grayscale value of the second object in the scene corresponding to the pixel position of at least one second pixel in the second image can be determined.

[0041] The first response curve used in step S104 may correspond to a first mapping relationship between a pixel value of at least one first pixel in a first image at a first exposure and a grayscale value of a first object in a scene corresponding to the first pixel, as indicated by a calibration response curve of the image sensor. The second response curve used in step S106 corresponds to a second mapping relationship between a pixel value of at least one second pixel in a second image at a second exposure and a grayscale value of a second object in a scene corresponding to the second pixel, as indicated by the calibration response curve. The grayscale value indicates the true brightness of an object in a scene.

[0042] By using the above-mentioned first response curve and second response curve, the real brightness of the object in the captured scene can be determined based on the pixel values ​​in the first image and the pixel values ​​in the second image. For any pixel in the first image, the grayscale value for the pixel can be determined using the first response curve. Similarly, for any pixel in the second image, the grayscale value for the pixel can be determined using the second response curve. Since the first response curve and the second response curve both indicate the relationship between the pixel value and the real brightness of the object in the scene under the corresponding exposure, the first response curve can be used to determine the first grayscale map corresponding to all pixels in the image area in the first image, and the second response curve can be used to determine the second grayscale map corresponding to all pixels in the image area in the second image. In the absence of overexposure, the first grayscale map and the second grayscale map should be the same, that is, both reflect the real brightness of the object in the image area in the scene.

[0043] However, if the image is overexposed, the pixels above a certain brightness level will all correspond to the same saturated pixel value in the image, so the response curve cannot be used to determine the grayscale value corresponding to the pixel in the overexposed area. In this case, the grayscale values ​​corresponding to the pixels in the image area can be obtained by fusing the grayscale images determined by images obtained at different exposure levels.

[0044] An image obtained with a smaller exposure (such as a second image) can be used to determine the grayscale values ​​corresponding to the pixels in the overexposed area. In some embodiments, if overexposure also exists in the second image, a third image with a third exposure can be used to obtain image information of the overexposed area in the second image. The third exposure can be less than the second exposure. Without departing from the principles of the embodiments of the present disclosure, the pixel value information and corresponding response curves of two or more images can be used to obtain the true brightness of objects in the scene corresponding to the pixels in the image area. In the present disclosure, the principle of the present disclosure is described by taking the example that there is no overexposed area in the second image, but it can be understood by those skilled in the art that the embodiments of the present disclosure are not limited to this.

[0045] The overexposed area in the first image can be determined using the pixel threshold. For example, a set of pixels in the first image that are higher than the pixel threshold can be determined as the overexposed area. In other words, a set of at least one first pixel in the first image that is not higher than the pixel threshold forms a non-overexposed area in the first image. The overexposed area and the non-overexposed area are located at different positions in the image area.

[0046] In some embodiments, at least one first pixel in the first image has a pixel value not higher than a pixel threshold, and at least one second pixel in the second image has a position in the image region that is different from the position of the at least one first pixel in the image region. The pixel threshold may be less than a saturated pixel value of the image sensor. The saturated pixel value of the image sensor may be determined based on the data bit width of the image sensor. For an image sensor with a data bit width of k bits, the saturated pixel value may be 2 to the kth power. For example, for an image sensor with a data bit width of 12 bits, the saturated pixel value may be 2 12 =4096 (i.e. 2 to the 12th power).

[0047] In some implementations, the above pixel threshold may be determined based on the saturated pixel value of the image sensor and an overexposure factor. In some examples, the overexposure factor may be a factor greater than 0 and less than 1. Therefore, the pixel threshold may be less than the saturated pixel value of the image sensor. In other implementations, the pixel threshold may also be any pixel value pre-specified by the user that is less than the saturated pixel value.

[0048] In step S108 , a target image may be determined based on the grayscale value of the first object and the grayscale value of the second object.

[0049] The pixel value of each pixel in the target image can be determined using the grayscale value of the first object (corresponding to the non-overexposed area in the first image) and the grayscale value of the second object (corresponding to the overexposed area in the first image) determined in step S104 and step S106.

[0050] In some embodiments, the grayscale value of the first object and the grayscale value of the second object can be inversely mapped using an inverse mapping curve to obtain target pixel values ​​at the pixel positions of the first object and the second object in the target image. The inverse mapping curve can indicate the mapping relationship between the target pixel value and the grayscale value at each pixel position in the target image. In the inverse mapping curve, the larger the grayscale value, the larger the target pixel value. The specific form of the inverse mapping curve is not limited herein. In other embodiments, the grayscale value of the first object and the grayscale value of the second object can be processed using any other data processing method to obtain image pixel values ​​corresponding to each grayscale value, as long as the constraint condition that the larger the grayscale value, the larger the target pixel value is satisfied.

[0051] For a color target image, the pixel values ​​of the images of each color channel can be processed using the method provided by the present disclosure to obtain the target pixel values ​​of each color channel. The color target image can be obtained by fusing the target pixel values ​​of each color channel.

[0052] Figure 2AExamples of first and second response curves according to embodiments of the present disclosure are shown.

[0053] like Figure 2A As shown, the first response curve 201 indicates the mapping relationship between the image pixel value and the grayscale value of the object in the scene under the first exposure, and the second response curve 202 indicates the mapping relationship between the image pixel value and the grayscale value of the object in the scene under the second exposure.

[0054] The saturation demarcation point 203 indicates the demarcation point between the overexposed area and the non-overexposed area in the first image. In some examples, for an image sensor with a 12-bit data width, the pixel value of the saturation demarcation point 203 may be a pixel threshold value slightly less than the saturated pixel value of the image sensor, 4096. For example, the pixel threshold value may be determined as 4090 or any other suitable value.

[0055] from Figure 2A It can be seen that within the range of pixel values ​​not higher than the saturation dividing point 203, the first response curve 201 can well represent the mapping relationship between the image pixel value and the grayscale value. However, within the range of pixel values ​​higher than the saturation dividing point 203, the first response curve 201 begins to enter the saturation region, and the grayscale value corresponding to the pixel value higher than the pixel threshold cannot be determined using the first response curve 201, because any grayscale value higher than the saturation dividing point 203 is mapped to the same saturated pixel value.

[0056] Therefore, for the overexposed area in the first image, the grayscale values ​​corresponding to the pixels in the overexposed area can be determined using the second response curve 202. It can be seen that since the second exposure is less than the first exposure, the second response curve enters the saturation region later than the first response curve. The pixels in the overexposed area in the first image are not overexposed in the second image. Therefore, the grayscale values ​​of the objects in the scene corresponding to the pixels in the overexposed area can be determined using the second image and the second response curve 202 for the second image.

[0057] Figure 2B Examples of first and second response curves and an inverse mapping curve according to an embodiment of the present disclosure are shown.

[0058] like Figure 2B As shown, curve 206 and curve 207 correspond to a first response curve of a first image at a first exposure and a second response curve of a second image at a second exposure, respectively. Curve 204 corresponds to a reverse mapping curve of an embodiment of the present disclosure.

[0059] After the grayscale values ​​for each pixel in the image region are determined using the first response curve 206 and the second response curve 207, the pixel value of each pixel in the target image can be determined using the inverse mapping curve 204. The higher the grayscale value, the greater the pixel value of the pixel.

[0060] In some embodiments, Figure 2B As shown, at least a portion of the inverse mapping curve 204 can be determined based on a portion of the first response curve 206. In the range where the grayscale value is less than the critical point 205, the inverse mapping curve and the first response curve 206 can be overlapped. Using this method, for objects whose grayscale values ​​are less than the critical point 205, the pixel values ​​in the target image obtained by the inverse mapping curve can be the same as the pixel values ​​actually obtained by the image sensor, so that a portion of the target image can be generated in a manner close to the real captured image. For objects whose grayscale values ​​are not less than the critical point 205, any mathematical tool can be used to generate a specific form of the inverse mapping curve, as long as the constraint condition that the higher the grayscale value, the larger the pixel value of the pixel is satisfied. The inverse mapping curve can be implemented as a linear function, a polynomial function, or any possible monotonically increasing function. In this way, in the target image, objects with higher real brightness in the environment are also reflected in the image as having higher pixel values, so that the target image can correctly reflect the brightness difference of the objects in the real environment, and thus a target image with a larger dynamic range can be generated based on the first image and the second image.

[0061] In order to get the combination Figure 1 , Figure 2A and Figure 2B The described embodiments involve a response curve that describes the mapping relationship between image pixel values ​​and grayscale values ​​of objects in the environment. The present disclosure also provides a method for calibrating the response curve of an image sensor.

[0062] Figure 3 A flow chart showing an exemplary process of a method for calibrating a response curve according to an embodiment of the present disclosure is shown.

[0063] like Figure 3 As shown, in step S302, a first calibration image including a plurality of calibrated grayscales in a first calibrated grayscale group may be acquired with a first calibrated exposure.

[0064] In step S304, a first calibration curve may be determined based on pixel values ​​at a plurality of first calibration positions in the first calibration image that respectively correspond to a plurality of calibration grayscales in the first calibration grayscale group, wherein the first calibration curve may indicate a mapping relationship between pixel values ​​collected by the image sensor at a first calibration exposure and grayscale values ​​of a plurality of calibration grayscales in the first calibration grayscale group.

[0065] In step S306 , a second calibration image including a plurality of calibration grayscales in a second calibration grayscale group may be acquired with a second calibration exposure different from the first calibration exposure.

[0066] In step S308, a second calibration curve may be determined based on the pixel values ​​at the second calibration positions corresponding to the plurality of calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve may indicate a mapping relationship between the pixel values ​​collected by the image sensor at the second exposure and the grayscale values ​​of the plurality of calibration grayscales in the second calibration grayscale group.

[0067] In step S310, fitting can be performed based on the first calibration curve and the second calibration curve to obtain a response curve of the image sensor. The response curve can indicate a mapping relationship between a pixel value collected by the image sensor and an exposure used when collecting an image and a grayscale value within a grayscale range including a plurality of calibrated grayscales. The grayscale range can include at least one of a first calibrated grayscale group and a second calibrated grayscale group.

[0068] By utilizing the method for calibrating the response curve of an image sensor provided by an embodiment of the present disclosure, the mapping relationship between the calibration grayscale (i.e., the actual brightness in the environment) and the pixel value in the captured image can be determined based on images captured under different exposure amounts, thereby being able to determine the mapping relationship between the actual brightness in the environment within a predetermined grayscale range and the pixel value in the captured image by fusing image information under different exposure amounts.

[0069] Each step of method 300 is described in detail below.

[0070] In step S302, a first calibration image including a plurality of calibration grayscales in a first calibration grayscale group is acquired with a first calibration exposure, wherein the first calibration image is original data (raw image) acquired by an image sensor.

[0071] In some embodiments, an image of a predetermined grayscale card may be obtained as a first calibration image at a first calibration exposure. The predetermined grayscale card may include 20 grayscales that vary with a predetermined grayscale variation. In some examples, the grayscales in the predetermined grayscale card may vary uniformly.

[0072] In step S304, a first calibration curve may be determined based on pixel values ​​at a plurality of first calibration positions in the first calibration image that respectively correspond to a plurality of calibration grayscales in the first calibration grayscale group.

[0073] Based on the first calibration image acquired in step S302, the pixel value of each calibration grayscale in the first calibration image can be determined. By determining each first calibration position corresponding to each calibration grayscale in the first calibration image and reading the pixel value at each first calibration position, the pixel value of each calibration grayscale in the first calibration image can be determined. A first calibration curve can be obtained by fitting the pixel values ​​corresponding to each calibration grayscale, wherein the first calibration curve indicates a mapping relationship between the pixel value collected by the image sensor under the first calibration exposure and the grayscale values ​​of multiple calibration grayscales in the first calibration grayscale group.

[0074] In step S306, a second calibration image including a plurality of calibrated grayscales in a second calibrated grayscale group may be acquired with a second calibrated exposure different from the first calibrated exposure, wherein the second calibration image is raw data acquired by the image sensor.

[0075] In some implementations, the first calibrated exposure and the second calibrated exposure used in step S304 and step S306 may be determined based on the exposure time used when acquiring the image. Taking the CMOS image as an example, which acquires a maximum of 30 frames of images per second, the exposure time may be up to 33ms. The first calibrated exposure may be greater than the second calibrated exposure, or less than the second calibrated exposure. In some exemplary calibration processes, the minimum calibrated exposure may be determined as 1ms, and the exposure time used when acquiring the image may be gradually increased in increments of 1ms. For example, the exposure time of the first calibrated exposure may be 1ms, and the exposure time of the second calibrated exposure may be 2ms. For another example, the exposure time of the first calibrated exposure may be 20ms, and the exposure time of the second calibrated exposure may be 3ms. The specific exposure times of the first calibrated exposure and the second calibrated exposure are not limited here, and those skilled in the art may select appropriate exposure times according to actual conditions.

[0076] In some other implementations, the first calibrated exposure and the second calibrated exposure used in step S304 and step S306 may be based on the exposure time and exposure gain used when acquiring the image. For example, the first calibrated exposure may be the product of the first calibrated exposure time and the first calibrated exposure, and the first calibrated exposure may be the product of the first calibrated exposure time and the first calibrated exposure gain. In some exemplary calibration processes, the exposure used when acquiring the image may be increased by first increasing the exposure time while keeping the exposure gain at 1. When the maximum exposure time is reached, the exposure time is kept unchanged while gradually changing the exposure gain to the maximum exposure gain.

[0077] In step S308, a second calibration curve may be determined based on the pixel values ​​at the second calibration positions corresponding to the plurality of calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve may indicate a mapping relationship between the pixel values ​​collected by the image sensor at the second exposure and the grayscale values ​​of the plurality of calibration grayscales in the second calibration grayscale group.

[0078] Similar to step S304, each second calibration position corresponding to each calibration grayscale in the second calibration image can be determined and the pixel value at each second calibration position can be read, and the pixel value of each calibration grayscale in the second calibration image can be determined. A second calibration curve can be obtained by fitting the pixel values ​​corresponding to each calibration grayscale, wherein the second calibration curve indicates the mapping relationship between the pixel value collected by the image sensor under the second calibration exposure and the grayscale values ​​of multiple calibration grayscales in the second calibration grayscale group.

[0079] In step S310, fitting may be performed based on the first calibration curve and the second calibration curve to obtain a response curve of the image sensor, wherein the response curve may indicate a mapping relationship between pixel values ​​collected by the image sensor and the exposure used when collecting the image and grayscale values ​​within a grayscale range including multiple calibrated grayscales.

[0080] In some embodiments, the multiple calibrated grayscales in the first calibrated grayscale group and the multiple calibrated grayscales in the second calibrated grayscale group may be the same. In some implementations, the first calibrated grayscale group may include multiple calibrated grayscales that change from a reference grayscale value by a predetermined grayscale change amount, and the second calibrated grayscale group includes the same number of calibrated grayscales that change from the reference grayscale value by the same predetermined grayscale change amount. Since the calibrated grayscales included in the first calibrated image and the second calibrated image are the same, the first calibration curve and the second calibration curve can be used to determine the different pixel values ​​corresponding to the same grayscale value in the images captured at different exposures.

[0081] In other embodiments, the multiple calibrated grayscales in the first calibrated grayscale group and the multiple calibrated grayscales in the second calibrated grayscale group may be different. For example, the first calibrated grayscale group may include a first number of multiple calibrated grayscales that change from a first reference grayscale value by a first predetermined grayscale change amount. The second calibrated grayscale group may include a second number of multiple calibrated grayscales that change from a second reference grayscale value by a second predetermined grayscale change amount. The various first reference grayscale values ​​and the second reference grayscale values ​​may be the same or different. The first predetermined grayscale change amount and the second predetermined grayscale change amount may be the same or different. The first number and the second number may be the same or different.

[0082] In some embodiments, the grayscale range corresponding to the response curve may include multiple calibrated grayscales in the first calibrated grayscale group and / or multiple calibrated grayscales in the second calibrated grayscale group. In some implementations, the grayscale range corresponding to the response curve may be greater than the grayscale range of the first calibrated grayscale group and / or the second calibrated grayscale group. That is, the maximum grayscale value of the grayscale range corresponding to the response curve may be greater than the maximum calibrated grayscale value in the first calibrated grayscale group and the maximum calibrated grayscale value in the second calibrated grayscale group. In some examples, the grayscale range corresponding to the response curve may include 60 grayscales based on a predetermined grayscale variation. In other examples, the grayscale range corresponding to the response curve also includes any other number of grayscales based on a predetermined grayscale variation, as long as the grayscale range corresponding to the response curve is not less than the grayscale range of the first calibrated grayscale group and / or the second calibrated grayscale group.

[0083] In step S310, based on the first calibration curve and the second calibration curve determined in step S306 and step S308, the grayscale range covered by the calibration curve can be extended by translating the calibration curve obtained by measurement based on the assumption of physical reciprocity of the image sensor, thereby obtaining a calibration curve covering a grayscale range larger than the predetermined calibration grayscale group without actually capturing a larger range of grayscale values.

[0084] Figure 4A An example of a calibration curve according to an embodiment of the present disclosure is shown.

[0085] like Figure 4A As shown, by continuously changing the exposure used by the image sensor when acquiring an image, a series of calibration curves indicating the mapping relationship between the image pixel value and the grayscale value can be obtained. Figure 4A The curves shown in are measured at different exposure levels. For the same grayscale value, the higher the exposure level, the higher the pixel value corresponding to the grayscale value in the image acquired by the image sensor. Figure 4A In the example shown, the exposure corresponding to calibration curve 401 is greater than the exposure corresponding to calibration curve 402 , and the exposure corresponding to calibration curve 402 is greater than the exposure corresponding to calibration curve 403 .

[0086] exist Figure 4A In the example shown, for the 20 grayscales that can be directly obtained in the calibration image, each calibration curve within the range of the 20 grayscales can be directly obtained through the image captured by the image sensor.

[0087] In some embodiments, based on the assumption of physical reciprocity of the image sensor, Figure 4ABased on the calibration curves obtained in the method, the calibration curve is theoretically extended by translating the curve, so as to obtain a calibration curve available in a larger grayscale range.

[0088] Figure 4B An example of an extended calibration curve according to an embodiment of the present disclosure is shown.

[0089] like Figure 4B As shown in the figure, for the curve with lower exposure, even when the brightest grayscale value in the calibrated grayscale is 20, the pixel value corresponding to grayscale 20 in the image is still lower and farther from the saturation region. In this case, a portion of the calibration curve with higher exposure can be translated and spliced ​​with the calibration curve with lower exposure, so as to obtain a curve from grayscale 20 to higher grayscale (until the saturation region) under lower exposure.

[0090] In some embodiments, the second calibration curve may be extended using the first calibration curve.

[0091] The maximum pixel value corresponding to the maximum calibrated grayscale value of the second calibrated grayscale group under the second calibrated exposure can be determined based on the second calibration curve. Based on the corresponding maximum pixel value on the second calibration curve, the curve portion of the first calibration curve that is higher than the maximum pixel value of the second calibration curve is spliced ​​with the second calibration curve to obtain an adjusted second calibration curve, wherein the adjusted second calibration curve indicates a mapping relationship between at least one grayscale value higher than the maximum calibrated grayscale value in the grayscale range and the pixel value collected by the image sensor under the second calibrated exposure. A response curve of the image sensor is determined based on the first calibration curve and the second calibration curve.

[0092] like Figure 4BAs shown, for calibration curve 407, the maximum pixel value of calibration curve 407 at grayscale 20 can be determined, and the curve portion of calibration curve 404 that is greater than the maximum pixel value of calibration curve 407 is translated to grayscale value 20 to form calibration curve 404', and by splicing calibration curve 404' and calibration curve 407 at grayscale value 20, calibration curve 407 can be extended to a grayscale range with grayscale values ​​greater than 20 and a saturation region of calibration curve 407 can be obtained. Similarly, calibration curve 405' obtained by translating calibration curve 405 can be spliced ​​with calibration curve 408 at grayscale value 20, and calibration curve 406' obtained by translating calibration curve 406 can be spliced ​​with calibration curve 409 at grayscale value 20. According to the physical reciprocity of the image sensor, the first calibration curve used to extend the second calibration curve can be any, as long as the first calibrated exposure corresponding to the first calibration curve is greater than the second calibrated exposure corresponding to the second calibration curve. In some cases, more than two calibration curves corresponding to different exposures can also be spliced.

[0093] After determining the calibration curves corresponding to the grayscale range including the calibration grayscale, the calibration curves in the grayscale range can be used for fitting to obtain the response curve of the image sensor. For example, the adjusted second calibration curve can be fitted with the first calibration curve to obtain the response curve of the image sensor.

[0094] In some embodiments, the response curve of the image sensor can be represented as a function of the image pixel value with respect to the exposure when the image is captured and the grayscale value of the captured object. Multiple calibration curves including the first calibration curve and the adjusted second calibration curve can be used to fit the parameters required for the response curve.

[0095] The response curve of the image sensor can be expressed by formula (1):

[0096]

[0097] Where I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure, and a, γ, b, δ, and w are constants.

[0098] Figure 5A An example of an image processing process according to an embodiment of the present disclosure is shown.

[0099] like Figure 5AAs shown, in step S501, multiple calibration images can be collected with multiple calibration exposures using an image sensor 510. According to a predetermined relationship between calibration grayscales and image pixel values ​​in the multiple calibration images, a calibration response curve 530 can be determined. The calibration response curve 530 indicates a mapping relationship between pixel values ​​collected by the image sensor and the exposure used when collecting the image and grayscale values ​​within a grayscale range including multiple calibration grayscales.

[0100] In step S502 , the image sensor 510 may respectively capture a first image 520 - 1 , a second image 520 - 2 . . . and an nth image 502 - n at different exposure levels, where n is an integer greater than 2.

[0101] In step S503, the first image 520-1, the second image 520-2 ... and the nth image 502-n can be processed respectively using the calibration response curve 530. According to the exposure corresponding to the first image 520-1, the second image 520-2 ... and the nth image 520-n and the pixel value of each pixel in the image, the first brightness mapping image 540-1, the second brightness mapping image 540-2 ... and the nth brightness mapping image 540-n corresponding to the first image 520-1, the second image 520-2 ... and the nth image 520-n can be determined using the calibration response curve 530, wherein the first brightness mapping image 540-1, the second brightness mapping image 540-2 ... and the nth brightness mapping image 540-n indicate the grayscale value of the object in the scene corresponding to each pixel point in the image area, that is, the real brightness.

[0102] Figure 5B An example of an image captured by an image sensor according to an embodiment of the present disclosure is shown. It can be seen that Figure 5B There are overexposed areas in the image, and the image details in the overexposed areas cannot be obtained through the original image.

[0103] In step S504, the first brightness mapping image 540-1, the second brightness mapping image 540-2, ... and the nth brightness mapping image 540-n may be fused based on the pixel values ​​of the first image 520-1, the second image 520-2, ... and the nth brightness mapping image 540-n to obtain a fused brightness mapping image 550. The processing may be started from the image with the highest exposure. Taking the first image with the highest exposure as an example, the grayscale value in the first brightness mapping image 540-1 corresponding to at least one first pixel (i.e., non-overexposed area) in the first image 520-1 that is not higher than the pixel threshold may be determined as the grayscale value of the corresponding pixel in the fused brightness mapping image 550. For the overexposed area in the first image 520-1, the brightness mapping image corresponding to the image with a smaller exposure (such as the second image) may be used to determine the grayscale value of at least part of the pixels in the overexposed area. And so on, until the grayscale values ​​of all pixels in the fused brightness mapping image 550 are determined.

[0104] In step S505, the target image 560 may be determined using the grayscale values ​​of the pixels indicated in the fused brightness map 550. Pixels with higher grayscale values ​​in the fused brightness map 550 have higher pixel values ​​in the target image.

[0105] Figure 5C An example of a target image according to an example of the present disclosure is shown. Figure 5C The images shown are based on Figure 5B The image shown in FIG. 4 shows a target image with a high dynamic range. Figure 5C As shown, by fusing the information of other images, Figure 5B Image details in the overexposed areas are restored. Figure 5C Relative to Figure 5B With better visual effects.

[0106] According to an embodiment of the present disclosure, an image processing device is also provided. The image processing device may include an image acquisition unit, a first grayscale determination unit, a second grayscale determination unit, and a target image determination unit, wherein the image acquisition unit may be configured to acquire a first image and a second image taken for the same scene, wherein the first exposure of the first image is greater than the second exposure of the second image, and the first image and the second image have the same image area. The first grayscale determination unit may be configured to determine the grayscale value of a first object in the scene corresponding to the pixel position of at least one first pixel in the first image based on a first response curve corresponding to the first exposure and a pixel value of at least one first pixel in the first image. The second grayscale determination unit may be configured to determine the grayscale value of a second object in the scene corresponding to the pixel position of at least one second pixel in the second image based on a second response curve corresponding to the second exposure and a pixel value of at least one second pixel in the second image. The target image determination unit may be configured to determine the target image based on the grayscale value of the first object and the grayscale value of the second object.

[0107] Here, the operations of the above-mentioned units of the image processing device are similar to the operations of steps S102 to S108 described above, and will not be repeated here.

[0108] According to an embodiment of the present disclosure, a device for calibrating a response curve of an image sensor is also provided. The device for calibrating a response curve of an image sensor may include a first calibration image acquisition unit, a first calibration curve determination unit, a second calibration image acquisition unit, a second calibration curve determination unit, and a response curve determination unit. The first calibration image acquisition unit may be configured to acquire a first calibration image including multiple calibration grayscales in a first calibration grayscale group at a first calibration exposure. The first calibration curve determination unit may be configured to determine a first calibration curve based on pixel values ​​at multiple first calibration positions in the first calibration image that respectively correspond to multiple calibration grayscales in the first calibration grayscale group, wherein the first calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor under the first calibration exposure and grayscale values ​​of multiple calibration grayscales in the first calibration grayscale group. The second calibration image acquisition unit may be configured to acquire a second calibration image including multiple calibration grayscales in a second calibration grayscale group at a second calibration exposure different from the first calibration exposure. The second calibration curve determination unit may be configured to determine a second calibration curve based on pixel values ​​at multiple second calibration positions corresponding to multiple calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor under the second exposure and grayscale values ​​of multiple calibration grayscales in the second calibration grayscale group. The response curve determination unit may be configured to determine a response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the response curve indicates a mapping relationship between pixel values ​​collected by the image sensor and the exposure used when collecting the image and grayscale values ​​within a grayscale range of multiple calibration grayscales including at least one of the first calibration grayscale group and the second calibration grayscale group.

[0109] Here, the operations of the above-mentioned units of the apparatus for calibrating the response curve of the image sensor are respectively similar to the operations of steps S302 to S310 described above, and will not be described in detail herein.

[0110] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0111] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a program is also provided. The program includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device executes the above method.

[0112] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0113] See also Figure 6 , an electronic device 600 will now be described, which is an example of a hardware device (electronic device) that can be applied to various aspects of the present disclosure. The electronic device 600 can be any machine configured to perform processing and / or calculations, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a robot, a smart phone, a vehicle-mounted computer, or any combination thereof. The above-mentioned image processing method 100 and the method 300 for calibrating a response curve can be implemented in whole or in part by the electronic device 600 or a similar device or system.

[0114] The electronic device 600 may include an element connected to or in communication with the bus 602 (possibly via one or more interfaces). For example, the electronic device 600 may include a bus 602, one or more processors 604, one or more input devices 606, and one or more output devices 608. The one or more processors 604 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 606 may be any type of device capable of inputting information to the electronic device 600, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 608 may be any type of device capable of presenting information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The electronic device 600 may also include a non-transitory storage device 610, which may be any storage device that is non-transitory and can enable data storage, including but not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory and / or any other memory chip or box, and / or any other medium from which a computer can read data, instructions and / or code. The non-transitory storage device 610 may be removable from the interface. The non-transitory storage device 610 may have data / programs (including instructions) / code for implementing the above-mentioned methods and steps. The electronic device 600 may also include a communication device 612. The communication device 612 may be any type of device or system that enables communication with an external device and / or with a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as Bluetooth TM devices, 1302.11 devices, Wi-Fi devices, Wi-Max devices, cellular communication devices, and / or the like.

[0115] The electronic device 600 may also include a working memory 614, which may be any type of working memory that may store programs (including instructions) and / or data useful for the operation of the processor 604, and may include but is not limited to random access memory and / or read-only memory devices.

[0116] The software elements (programs) may be located in the working memory 614, including but not limited to an operating system 616, one or more application programs 618, a driver and / or other data and code. Instructions for executing the above methods and steps may be included in one or more application programs 618, and the above image processing method 100 and the method 300 for calibrating a response curve may be implemented by the processor 604 reading and executing the instructions of one or more application programs 618. More specifically, in the above image processing method 100, steps S102-S108 may be implemented, for example, by the processor 604 executing an application program 618 having instructions for steps S102-S108. In the above method 300 for calibrating a response curve, steps S302-S310 may be implemented, for example, by the processor 604 executing an application program 618 having instructions for steps S302-S310. In addition, other steps in the above image processing method 100 and the method 300 for calibrating a response curve may be implemented, for example, by the processor 604 executing an application program 618 having instructions for executing the corresponding steps. The executable code or source code of the instructions of the software element (program) can be stored in a non-transitory computer-readable storage medium (such as the storage device 610 described above), and when executed, can be stored in the working memory 614 (possibly compiled and / or installed). The executable code or source code of the instructions of the software element (program) can also be downloaded from a remote location.

[0117] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0118] It should also be understood that the foregoing method can be implemented through a server-client model. For example, the client can receive data input by the user and send the data to the server. The client can also receive data input by the user, perform a portion of the processing in the foregoing method, and send the processed data to the server. The server can receive data from the client, and execute the foregoing method or another portion of the foregoing method, and return the execution result to the client. The client can receive the execution result of the method from the server, and present it to the user, for example, through an output device.

[0119] It should also be understood that the components of the electronic device 600 can be distributed over a network. For example, one processor can be used to perform some processing, while other processing can be performed by another processor remote from the one processor. Other components of the computing system 600 can also be distributed similarly. In this way, the electronic device 600 can be interpreted as a distributed computing system that performs processing at multiple locations.

[0120] Some exemplary aspects of the disclosure are described below.

[0121] Aspect 1. An image processing method, comprising:

[0122] Acquire a first image and a second image captured for the same scene, wherein a first exposure amount of the first image is greater than a second exposure amount of the second image, and the first image and the second image have the same image area;

[0123] determining a grayscale value of a first object in the scene corresponding to a pixel position of the at least one first pixel in the first image based on a first response curve corresponding to the first exposure and a pixel value of at least one first pixel in the first image;

[0124] determining a grayscale value of a second object in the scene corresponding to a pixel position of the at least one second pixel in the second image based on a second response curve corresponding to the second exposure and a pixel value of the at least one second pixel in the second image; and

[0125] A target image is determined based on the grayscale value of the first object and the grayscale value of the second object.

[0126] Aspect 2. An image processing method as described in Aspect 1, wherein at least one first pixel in the first image has a pixel value not higher than a pixel threshold, and a position of at least one second pixel in the second image in the image area is different from a position of the at least one first pixel in the image area.

[0127] Aspect 3. The image processing method as described in Aspect 2, wherein the pixel threshold is less than a saturated pixel value of the image sensor.

[0128] Aspect 4. The image processing method as described in Aspect 3, wherein the pixel threshold is determined based on a saturated pixel value and an overexposure coefficient of the image sensor, and the overexposure coefficient is a coefficient greater than 0 and less than 1.

[0129] Aspect 5. The image processing method according to aspect 1, wherein determining the target image based on the grayscale value of the first object and the grayscale value of the second object comprises:

[0130] The grayscale value of the first object and the grayscale value of the second object are inversely mapped using an inverse mapping curve to obtain target pixel values ​​of the pixel positions of the first object and the pixel positions of the second object in the target image, wherein the inverse mapping curve indicates a mapping relationship between the target pixel value and the grayscale value at each pixel position in the target image.

[0131] Aspect 6. In the image processing method as described in Aspect 5, in the inverse mapping curve, the larger the grayscale value is, the larger the target pixel value is.

[0132] Aspect 7. An image processing method as described in Aspect 1, wherein the first exposure amount is determined based on a first exposure time and a first exposure gain used when acquiring the first image, and the second exposure amount is determined based on a second exposure time and a second exposure gain used when acquiring the second image.

[0133] Aspect 8. An image processing method as described in Aspect 1, wherein the first response curve corresponds to a first mapping relationship between a pixel value of at least one first pixel in the first image at the first exposure and a grayscale value of a first object in the scene corresponding to the first pixel, as indicated by a calibrated response curve of the image sensor, and the second response curve corresponds to a second mapping relationship between a pixel value of at least one second pixel in the second image at the second exposure and a grayscale value of a second object in the scene corresponding to the second pixel, as indicated by the calibrated response curve.

[0134] Aspect 9. The image processing method according to aspect 8, wherein the calibration response curve is determined by the following steps:

[0135] Acquire a first calibration image including a plurality of calibrated grayscales in a first calibrated grayscale group with a first calibrated exposure amount;

[0136] Determining a first calibration curve based on pixel values ​​at a plurality of first calibration positions respectively corresponding to a plurality of calibration grayscales in the first calibration grayscale group in the first calibration image, wherein the first calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the first calibration exposure and grayscale values ​​of the plurality of calibration grayscales in the first calibration grayscale group;

[0137] Acquire a second calibration image including a plurality of calibrated grayscales in a second calibrated grayscale group with a second calibrated exposure amount different from the first calibrated exposure amount;

[0138] determining a second calibration curve based on pixel values ​​at a plurality of second calibration positions respectively corresponding to a plurality of calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the second exposure amount and grayscale values ​​of the plurality of calibration grayscales in the second calibration grayscale group; and

[0139] A calibration response curve of the image sensor is determined based on the first calibration curve and the second calibration curve, wherein the calibration response curve indicates a mapping relationship between pixel values ​​captured by the image sensor and the exposure used when capturing an image and grayscale values ​​within a grayscale range of multiple calibrated grayscales including at least one of the first calibrated grayscale group and the second calibrated grayscale group.

[0140] Aspect 10. The image processing method according to aspect 9, wherein determining the calibration response curve of the image sensor based on the first calibration curve and the second calibration curve comprises:

[0141] Determining, based on the second calibration curve, a maximum pixel value corresponding to a maximum calibrated grayscale value among a plurality of calibrated grayscales in the second calibrated grayscale group under a second calibrated exposure;

[0142] splicing a curve portion of the first calibration curve that is higher than the maximum pixel value of the second calibration curve with the second calibration curve based on the maximum pixel value to obtain an adjusted second calibration curve, wherein the adjusted second calibration curve indicates a mapping relationship between at least one grayscale value in the grayscale range that is higher than the maximum calibrated grayscale value and a pixel value acquired by the image sensor at a second calibrated exposure; and

[0143] The adjusted second calibration curve is fitted to the first calibration curve to obtain the calibration response curve.

[0144] Aspect 11. The image processing method as described in Aspect 9, wherein the second calibrated exposure is smaller than the first calibrated exposure.

[0145] Aspect 12. An image processing method as described in Aspect 9, wherein the first calibrated grayscale group includes multiple calibrated grayscales that change from a reference grayscale value by a predetermined grayscale change amount, and the second calibrated grayscale group includes the same number of multiple calibrated grayscales that change from the reference grayscale value by the same predetermined grayscale change amount.

[0146] Aspect 13. The image processing method according to any one of aspects 9 to 12, wherein the calibration response curve is expressed as the following formula:

[0147]

[0148] Where I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure, and a, γ, b, δ, and w are constants.

[0149] Aspect 14. A method for calibrating a response curve of an image sensor, comprising:

[0150] Acquire a first calibration image including a plurality of calibrated grayscales in a first calibrated grayscale group with a first calibrated exposure amount;

[0151] Determining a first calibration curve based on pixel values ​​at a plurality of first calibration positions respectively corresponding to a plurality of calibration grayscales in the first calibration grayscale group in the first calibration image, wherein the first calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the first calibration exposure and grayscale values ​​of the plurality of calibration grayscales in the first calibration grayscale group;

[0152] Acquire a second calibration image including a plurality of calibrated grayscales in a second calibrated grayscale group with a second calibrated exposure amount different from the first calibrated exposure amount;

[0153] determining a second calibration curve based on pixel values ​​at a plurality of second calibration positions respectively corresponding to a plurality of calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the second exposure amount and grayscale values ​​of the plurality of calibration grayscales in the second calibration grayscale group; and

[0154] A response curve of the image sensor is determined based on the first calibration curve and the second calibration curve, wherein the response curve indicates a mapping relationship between pixel values ​​captured by the image sensor and the exposure used when capturing an image and grayscale values ​​within a grayscale range of multiple calibrated grayscales including at least one of the first calibrated grayscale group and the second calibrated grayscale group.

[0155] Clause 15. The method according to clause 14, wherein the exposure amount is determined based on an exposure time and an exposure gain used when acquiring an image using the image sensor.

[0156] Aspect 16. The method of aspect 14, wherein the second calibrated exposure is less than the first calibrated exposure.

[0157] Aspect 17. A method as described in Aspect 14, wherein the first calibrated grayscale group includes multiple calibrated grayscales that change from a reference grayscale value by a predetermined grayscale change amount, and the second calibrated grayscale group includes the same number of multiple calibrated grayscales that change from the reference grayscale value by the same predetermined grayscale change amount.

[0158] Aspect 18. A method as described in Aspect 14, wherein the multiple calibrated grayscales change with a predetermined grayscale change amount, and the maximum grayscale value including the grayscale range is greater than the maximum calibrated grayscale value of the first calibrated grayscale group and the maximum calibrated grayscale value of the second calibrated grayscale group.

[0159] Aspect 19. The method of aspect 14, wherein determining a response curve of the image sensor based on the first calibration curve and the second calibration curve comprises:

[0160] Determining a maximum pixel value corresponding to a maximum calibrated grayscale value of the second calibrated grayscale group under a second calibrated exposure based on the second calibration curve;

[0161] splicing a curve portion of the first calibration curve that is higher than the maximum pixel value of the second calibration curve with the second calibration curve based on the maximum pixel value to obtain an adjusted second calibration curve, wherein the adjusted second calibration curve indicates a mapping relationship between at least one grayscale value in the grayscale range that is higher than the maximum calibrated grayscale value and a pixel value acquired by the image sensor at a second calibrated exposure; and

[0162] The adjusted second calibration curve is fitted to the first calibration curve to obtain the response curve.

[0163] Aspect 20. The method according to any one of aspects 14 to 19, wherein the response curve is expressed as follows:

[0164]

[0165] Where I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure, and a, γ, b, δ, and w are constants.

[0166] Aspect 21. An electronic device comprising:

[0167] Processor; and

[0168] A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform a method according to any one of aspects 1-20.

[0169] Aspect 22. A non-transitory computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform a method according to any one of aspects 1-20.

[0170] Aspect 23. A computer program product, comprising a computer program, wherein the computer program implements the method according to any one of aspects 1-20 when executed by a processor.

[0171] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An image processing method, comprising: Acquire a first image and a second image captured for the same scene, wherein a first exposure amount of the first image is greater than a second exposure amount of the second image, and the first image and the second image have the same image area; determining a grayscale value of a first object in a scene corresponding to a pixel position of at least one first pixel in the first image based on a first response curve corresponding to the first exposure amount and a pixel value of at least one first pixel in the first image, the first response curve corresponding to a first mapping relationship between the pixel value of at least one first pixel in the first image at the first exposure amount and the grayscale value of the first object in the scene corresponding to the first pixel indicated by a calibration response curve of the image sensor; determining a grayscale value of a second object in the scene corresponding to a pixel position of at least one second pixel in the second image based on a second response curve corresponding to the second exposure amount and a pixel value of at least one second pixel in the second image, the second response curve corresponding to a second mapping relationship between the pixel value of at least one second pixel in the second image at the second exposure amount indicated by the calibration response curve and the grayscale value of the second object in the scene corresponding to the second pixel; and determining a target image based on the grayscale value of the first object and the grayscale value of the second object, Wherein, the calibration response curve is determined by the following steps: Acquire a first calibration image including a plurality of calibrated grayscales in a first calibrated grayscale group with a first calibrated exposure amount; Determining a first calibration curve based on pixel values ​​at a plurality of first calibration positions respectively corresponding to a plurality of calibration grayscales in the first calibration grayscale group in the first calibration image, wherein the first calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the first calibration exposure and grayscale values ​​of the plurality of calibration grayscales in the first calibration grayscale group; Acquire a second calibration image including a plurality of calibrated grayscales in a second calibrated grayscale group with a second calibrated exposure amount different from the first calibrated exposure amount; determining a second calibration curve based on pixel values ​​at a plurality of second calibration positions respectively corresponding to a plurality of calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the second calibration exposure and grayscale values ​​of the plurality of calibration grayscales in the second calibration grayscale group; and Determining a calibration response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the calibration response curve indicates a mapping relationship between pixel values ​​acquired by the image sensor and an exposure used when acquiring an image and grayscale values ​​within a grayscale range of a plurality of calibration grayscales including at least one of the first calibration grayscale group and the second calibration grayscale group, wherein determining the calibration response curve of the image sensor based on the first calibration curve and the second calibration curve comprises: Determining a maximum pixel value corresponding to a maximum calibrated grayscale value of the second calibrated grayscale group under a second calibrated exposure based on the second calibration curve; splicing a curve portion of the first calibration curve that is higher than the maximum pixel value of the second calibration curve with the second calibration curve based on the maximum pixel value to obtain an adjusted second calibration curve, wherein the adjusted second calibration curve indicates a mapping relationship between at least one grayscale value in the grayscale range that is higher than the maximum calibrated grayscale value and a pixel value acquired by the image sensor at a second calibrated exposure; and The adjusted second calibration curve is fitted to the first calibration curve to obtain the calibration response curve.

2. The image processing method according to claim 1, wherein: At least one first pixel in the first image has a pixel value not higher than a pixel threshold, and at least one second pixel in the second image has a position in the image region different from the position of the at least one first pixel in the image region.

3. The image processing method according to claim 2, wherein: The pixel threshold is less than a saturated pixel value of the image sensor.

4. The image processing method according to claim 3, wherein: The pixel threshold is determined based on a saturated pixel value of the image sensor and an overexposure coefficient, where the overexposure coefficient is a coefficient greater than 0 and less than 1.

5. The image processing method according to claim 1, wherein: Determining a target image based on the grayscale value of the first object and the grayscale value of the second object includes: The grayscale value of the first object and the grayscale value of the second object are inversely mapped using an inverse mapping curve to obtain target pixel values ​​of the pixel positions of the first object and the pixel positions of the second object in the target image, wherein the inverse mapping curve indicates a mapping relationship between the target pixel value and the grayscale value at each pixel position in the target image. 6 . The image processing method according to claim 5 , wherein in the inverse mapping curve, the larger the grayscale value is, the larger the target pixel value is.

7. The image processing method according to claim 1, wherein: The first exposure amount is determined based on a first exposure time and a first exposure gain used when acquiring the first image, and the second exposure amount is determined based on a second exposure time and a second exposure gain used when acquiring the second image. 8 . The image processing method according to claim 1 , wherein the second calibrated exposure is smaller than the first calibrated exposure.

9. The image processing method of claim 1, wherein the first calibrated grayscale group comprises a plurality of calibrated grayscales that change from a reference grayscale value by a predetermined grayscale change amount, and the second calibrated grayscale group comprises a plurality of calibrated grayscales of the same number that change from the reference grayscale value by the same predetermined grayscale change amount.

10. The image processing method according to any one of claims 1 to 9, wherein: The calibration response curve is expressed as follows: Where I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure, and a, γ, b, δ, and w are constants.

11. A method for calibrating a response curve of an image sensor, comprising: Acquire a first calibration image including a plurality of calibrated grayscales in a first calibrated grayscale group with a first calibrated exposure amount; Determining a first calibration curve based on pixel values ​​at a plurality of first calibration positions respectively corresponding to a plurality of calibration grayscales in the first calibration grayscale group in the first calibration image, wherein the first calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the first calibration exposure and grayscale values ​​of the plurality of calibration grayscales in the first calibration grayscale group; Acquire a second calibration image including a plurality of calibrated grayscales in a second calibrated grayscale group with a second calibrated exposure amount different from the first calibrated exposure amount; determining a second calibration curve based on pixel values ​​at a plurality of second calibration positions respectively corresponding to a plurality of calibration grayscales in the second calibration grayscale group in the second calibration image, wherein the second calibration curve indicates a mapping relationship between pixel values ​​collected by the image sensor at the second calibration exposure and grayscale values ​​of the plurality of calibration grayscales in the second calibration grayscale group; and determining a response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the response curve indicates a mapping relationship between pixel values ​​acquired by the image sensor and an exposure used when acquiring an image and grayscale values ​​within a grayscale range of a plurality of calibrated grayscales including at least one of the first calibrated grayscale group and the second calibrated grayscale group, Wherein, determining the response curve of the image sensor based on the first calibration curve and the second calibration curve includes: Determining a maximum pixel value corresponding to a maximum calibrated grayscale value of the second calibrated grayscale group under a second calibrated exposure based on the second calibration curve; splicing a curve portion of the first calibration curve that is higher than the maximum pixel value of the second calibration curve with the second calibration curve based on the maximum pixel value to obtain an adjusted second calibration curve, wherein the adjusted second calibration curve indicates a mapping relationship between at least one grayscale value in the grayscale range that is higher than the maximum calibrated grayscale value and a pixel value acquired by the image sensor at a second calibrated exposure; and The adjusted second calibration curve is fitted to the first calibration curve to obtain the response curve.

12. The method of claim 11, wherein: The exposure amount is determined based on an exposure time and an exposure gain used when acquiring an image using the image sensor.

13. The method of claim 11, wherein the second calibrated exposure is less than the first calibrated exposure.

14. The method of claim 11, wherein the first calibrated grayscale group includes a plurality of calibrated grayscales that change from a reference grayscale value by a predetermined grayscale change amount, and the second calibrated grayscale group includes a same number of calibrated grayscales that change from the reference grayscale value by the same predetermined grayscale change amount.

15. The method of claim 11, wherein: The plurality of calibrated grayscales vary with a predetermined grayscale variation, and a maximum grayscale value of the grayscale range is greater than a maximum calibrated grayscale value of the first calibrated grayscale group and a maximum calibrated grayscale value of the second calibrated grayscale group.

16. The method according to any one of claims 11 to 15, wherein: The response curve is expressed as follows: Where I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure, and a, γ, b, δ, and w are constants.

17. An electronic device comprising: processor; as well as A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-16.

18. A non-transitory computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of claims 1-16.

19. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 16 when executed by a processor.

Images