Image processing method, apparatus, device, storage medium, and program

Abstract

The application provides an image processing method, device, equipment, storage medium and program, and relates to the technical field of computer vision. The electronic device comprises at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores a computer program, and the at least one processor executes the computer program. The technical scheme of the application can improve the quality of the fused image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image processing method, apparatus, device, storage medium, and program. Background Technology

[0002] With the development of surveillance technology, the requirements for images acquired by image acquisition equipment in the surveillance field are becoming increasingly higher.

[0003] In practical applications, the dynamic range of image acquisition devices has certain limitations. When both extremely bright and extremely dark areas exist in the shooting scene, the image output by the device may suffer from underexposure in dark areas and overexposure in bright areas, severely affecting image quality. Improving image quality in such shooting scenarios is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, storage medium, and program to improve the quality of images acquired by an image acquisition device.

[0005] In a first aspect, this application provides an image processing method, comprising:

[0006] Acquire a first image and a second image captured by an image sensor according to a shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image;

[0007] Determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel;

[0008] Based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, the first image and the second image are fused to obtain a fused image.

[0009] Secondly, this application provides an image processing apparatus, comprising:

[0010] The acquisition module is used to acquire a first image and a second image captured by the image sensor according to the shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image;

[0011] The determining module is used to determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel.

[0012] The processing module is used to perform fusion processing on the first image and the second image based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, to obtain a fused image.

[0013] Thirdly, this application provides an electronic device, comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program, and the at least one processor executes the computer program to implement the method as described in any of the first aspects.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0018] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0019] The image processing method, apparatus, device, storage medium, and program provided in this application acquire a first image and a second image captured by an image sensor according to a shooting command, wherein the exposure duration of the first image is greater than that of the second image; acquire a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel, wherein the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or the first type pixel is a moving pixel and the second type pixel is an overexposed pixel; and perform a fusion process on the first image and the second image according to the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel to obtain a fused image. In the above fusion process, since the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel are considered, the proportion of the pixel values ​​of the first image and the second image in the pixel values ​​of the fused image can be adjusted according to the first probability and the second probability during the fusion process, so that the texture information of each pixel in the fused image is richer and there is no trailing, thereby improving the quality of the fused image. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram illustrating the processing procedure of an image acquisition device provided in an embodiment of this application;

[0022] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0024] Figure 4 A schematic diagram illustrating the first and second image fusion processing steps provided in this application embodiment;

[0025] Figure 5 A schematic diagram illustrating the processing procedure of another image acquisition device provided in an embodiment of this application;

[0026] Figure 6 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0027] Figure 7A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram illustrating the process of determining the target noise distribution provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0033] The image processing method provided in this application can be applied to image acquisition devices. Image acquisition devices include, but are not limited to: cameras, video cameras, snapshot cameras, face recognition cameras, barcode scanners, vehicle-mounted cameras, panoramic detail cameras, etc.

[0034] Image acquisition devices may include an image sensor and an image processor. An image sensor generates an image through exposure. The image generated by the image sensor can be output to the image processor, which performs post-processing on the image, such as noise reduction, and outputs the processed image.

[0035] Typically, image sensors in image acquisition devices operate in linear mode. In this mode, the dynamic range of the image acquisition device is limited. Dynamic range refers to the ability of an image acquisition device to adapt to changes in light reflection from objects in the scene being captured, such as its ability to adapt to changes in brightness or color temperature within the same scene.

[0036] In practical applications, when there are both extremely bright and extremely dark areas in the shooting scene, due to the limited dynamic range of the image acquisition device, the output image may have problems such as underexposure in dark areas and overexposure in bright areas, which seriously affects the image quality.

[0037] To address the aforementioned technical issues, in this embodiment, the image acquisition device also supports a Wide Dynamic Range (WDR) mode. In WDR mode, the dynamic range of the image acquisition device is larger / wider. It is understood that a larger / wider dynamic range indicates a greater range of brightness variations in the image acquired by the device. Therefore, it can capture both bright and dark areas in the image, thereby improving image quality.

[0038] Below, in conjunction with Figure 1 The implementation principle of WDR mode will be explained.

[0039] Figure 1 This is a schematic diagram illustrating the processing procedure of an image acquisition device provided in an embodiment of this application. See also... Figure 1 In WDR mode, the image sensor in the image acquisition device acquires a first image and a second image of the same scene through two exposures. The exposure durations of the first and second images are different. For ease of description, in this embodiment, it is assumed that the exposure duration of the first image is longer than that of the second image. In this case, the first image can also be referred to as a long-frame image, and the second image can also be referred to as a short-frame image.

[0040] Further, see Figure 1 The image processor fuses the first and second images according to certain fusion principles to obtain a fused image. The image acquisition device can then output this fused image.

[0041] In this embodiment of the application, when fusing the first image and the second image, the fusion is performed at the pixel level. That is, the pixel value of the (i,j)th pixel in the first image is fused with the pixel value of the (i,j)th pixel in the second image to obtain the pixel value of the (i,j)th pixel in the fused image. The following fusion principle can be adopted:

[0042] (1) Regarding bright areas in the shooting scene, due to the longer exposure time of the first image, the pixels corresponding to the bright areas in the first image may be overexposed; while the exposure time of the second image is shorter, and the pixels corresponding to the bright areas in the second image retain more texture information. Therefore, during the fusion process, for the pixels corresponding to the bright areas, the proportion of pixel values ​​in the second image can be greater than that in the first image. This makes the texture information of the bright areas in the fused image richer.

[0043] (2) Regarding moving areas in the shooting scene, since the first image has a longer exposure time, the pixels corresponding to the moving areas in the first image may have a trailing problem; while the second image has a shorter exposure time, the pixels corresponding to the moving areas in the second image do not have a trailing problem. Therefore, during the fusion process, for the pixels corresponding to the moving areas, the proportion of pixel values ​​in the second image can be greater than the proportion of pixel values ​​in the first image. In this way, the moving areas in the fused image are free from trailing.

[0044] (3) For other areas in the shooting scene (e.g., dark areas, non-moving areas), since the exposure time of the first image is longer than that of the second image, other areas in the first image retain more texture information than other areas in the second image. Therefore, during the fusion process, for the pixels corresponding to other areas, the proportion of pixel values ​​in the first image can be greater than that in the second image. This makes the texture information of other areas in the fused image richer.

[0045] Based on the above fusion principle, in this embodiment, the first image and the second image can be fused according to the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel to obtain a fused image. Here, the first type of pixel is an overexposed pixel, and the second type of pixel is a moving pixel, or vice versa. This allows the fused image to take into account both bright and dark areas, thereby improving image quality.

[0046] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0047] Figure 2 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The method of this embodiment can be executed by an image acquisition device. For example, the image acquisition device is equipped with an image processor or an image processor chip, and the method of this embodiment is executed by the image processor or the image processor chip. Figure 2 As shown, the method in this embodiment includes:

[0048] S201: Acquire a first image and a second image captured by the image sensor according to the shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image.

[0049] In this embodiment, the first image and the second image are two frames acquired according to the same shooting command. After receiving the shooting command, the image acquisition device controls the image sensor to perform two exposures, generating the first image and the second image respectively.

[0050] In an example scenario, when a user takes a picture using an image acquisition device, in response to the image acquisition device detecting the user's input shooting operation, the image acquisition device receives a shooting instruction and controls the image sensor to perform two exposures according to the shooting instruction, thereby generating a first image and a second image respectively.

[0051] In another example scenario, during image capture, in response to the image capture device detecting a target object in the scene, the image capture device receives a shooting instruction and controls the image sensor to perform two exposures according to the shooting instruction, generating a first image and a second image respectively.

[0052] In another example scenario, during video shooting, the image acquisition device generates a shooting command every preset time interval according to a preset frame rate. Based on each shooting command, it controls the image sensor to perform two exposures, generating a first image and a second image respectively.

[0053] In this embodiment, the exposure time corresponding to the first image is greater than the exposure time corresponding to the second image. The first image can also be called a long-frame image, and the second image can also be called a short-frame image. It should be noted that this embodiment does not limit the generation order of the first and second images; the first image can be generated first, followed by the second image, or vice versa.

[0054] Optionally, the exposure start time for the first image and the second image are the same, and the exposure end time for the first image is later than the exposure end time for the second image.

[0055] S202: Determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel.

[0056] For ease of description, the following examples will use overexposed pixels as the first type of pixel and moving pixels as the second type of pixel.

[0057] In this embodiment, an overexposed pixel refers to a pixel that is overexposed. Overexposed pixels appear very bright or even washed out in an image, resulting in a loss of texture information. For example, for each pixel in the first image, the first probability of the pixel being an overexposed pixel can be determined as follows: if the pixel value is greater than or equal to a preset threshold, the first probability of the pixel being an overexposed pixel is 1. If the pixel value is less than the preset threshold, the first probability of the pixel being an overexposed pixel is 0.

[0058] It is understandable that by using the above method to judge each pixel in the first image, the probability of each pixel being an overexposed pixel is either 0 or 1.

[0059] In one possible implementation, to ensure a smoother first probability, for each pixel in the first image, the first probability of that pixel being an overexposed pixel can be determined as follows: Obtain a first difference between the pixel value of that pixel and a first preset threshold; based on the first difference, determine the first probability of that pixel being an overexposed pixel, where the first probability is directly proportional to the first difference. That is, the larger the first difference, the greater the first probability of that pixel being an overexposed pixel; the smaller the first difference, the smaller the first probability of that pixel being an overexposed pixel.

[0060] In this embodiment, a moving pixel refers to a pixel that is in motion. Moving pixels manifest in an image as phenomena such as trailing or blurring. For example, suppose that at the moment of shooting, the person being photographed is waving their hand; the pixels corresponding to the hand in the image may exhibit trailing or blurring. In this case, the pixels corresponding to the hand are moving pixels.

[0061] For example, assuming the first image and the second image each include M*N pixels, where M and N are integers greater than 1, for the (i, j)th pixel in the first image, where i is an integer less than or equal to M and j is an integer less than or equal to N, the second probability that this pixel is a moving pixel can be determined as follows: Obtain the second difference between the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image. If the second difference is greater than or equal to a second preset threshold, the second probability that the (i, j)th pixel in the first image is a moving pixel is 1; if the second difference is less than the second preset threshold, the second probability that the (i, j)th pixel in the first image is a moving pixel is 0.

[0062] It is understandable that by using the above method to judge each pixel in the first image, the second probability of each pixel being a moving pixel is determined to be 0 or 1.

[0063] In one possible implementation, to ensure a smoother second probability, for the (i, j)th pixel in the first image, the second probability of the pixel being a moving pixel can be determined as follows: Obtain the second difference between the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image. Based on this second difference, determine the second probability of the pixel being a moving pixel. The second probability is directly proportional to the second difference. That is, the larger the second difference, the greater the second probability of the pixel being a moving pixel; the smaller the second difference, the smaller the second probability of the pixel being a moving pixel.

[0064] S203: Based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, the first image and the second image are fused to obtain a fused image.

[0065] In this embodiment of the application, when fusing the first image and the second image, the fusion is performed at the pixel level. Specifically, it is assumed that the first image and the second image each include M*N pixels, where M and N are integers greater than 1. The pixel value of the (i,j)th pixel in the first image is fused with the pixel value of the (i,j)th pixel in the second image to obtain the pixel value of the (i,j)th pixel in the fused image.

[0066] For example, the pixel value of the (i,j)th pixel in the first image and the pixel value of the (i,j)th pixel in the second image can be weighted and summed to obtain the pixel value of the (i,j)th pixel in the fused image.

[0067] When performing fusion processing, one or more of the following implementation methods can be used:

[0068] In the first implementation, if the probability of the (i,j)th pixel in the first image being an overexposed pixel is relatively high, then the (i,j)th pixel corresponds to a bright area. Since the exposure time of the second image is shorter than that of the first image, the (i,j)th pixel in the second image retains more texture information than the (i,j)th pixel in the first image. Therefore, during the fusion process, the weight of the (i,j)th pixel value in the second image can be greater than the weight of the (i,j)th pixel value in the first image. This results in richer texture information for the (i,j)th pixel in the fused image.

[0069] In the second implementation, if the probability of the (i,j)th pixel in the first image being an overexposed pixel is relatively low, then the (i,j)th pixel corresponds to a dark area. Since the exposure time of the first image is longer than that of the second image, the (i,j)th pixel in the first image retains more texture information than the (i,j)th pixel in the second image. Therefore, during the fusion process, the weight of the pixel value of the (i,j)th pixel in the first image can be greater than the weight of the pixel value of the (i,j)th pixel in the second image. This results in richer texture information for the (i,j)th pixel in the fused image.

[0070] In the third implementation, if the probability that the (i,j)th pixel in the first image is a moving pixel is relatively high, then the (i,j)th pixel corresponds to a moving region. Since the exposure time of the second image is shorter than that of the first image, the (i,j)th pixel in the second image does not have a trailing effect compared to the (i,j)th pixel in the first image. Therefore, during the fusion process, the weight of the pixel value of the (i,j)th pixel in the second image can be greater than the weight of the pixel value of the (i,j)th pixel in the first image. This ensures that the (i,j)th pixel in the fused image does not have a trailing effect.

[0071] In the fourth implementation method, if the probability that the (i,j)th pixel in the first image is a moving pixel is relatively low, it indicates that the (i,j)th pixel corresponds to a non-moving region. Since the exposure time of the first image is longer than that of the second image, the (i,j)th pixel in the first image has more texture information than the (i,j)th pixel in the second image. Therefore, during the fusion process, the weight of the pixel value of the (i,j)th pixel in the first image can be greater than the weight of the pixel value of the (i,j)th pixel in the second image. This makes the texture information of the (i,j)th pixel in the fused image richer.

[0072] In this embodiment, the above-described implementation methods can enrich the texture information of each pixel in the fused image and eliminate trailing, thereby improving the quality of the fused image. It should be noted that in specific implementation processes, the above-described implementation methods can be combined with each other, and this embodiment does not limit this.

[0073] The image processing method provided in this embodiment includes: acquiring a first image and a second image captured by an image sensor according to a shooting command, wherein the exposure duration of the first image is greater than that of the second image; acquiring a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel, wherein the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or the first type pixel is a moving pixel and the second type pixel is an overexposed pixel; and performing a fusion processing on the first image and the second image according to the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel to obtain a fused image. In the above fusion process, since the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel are considered, the proportion of the pixel values ​​of the first image and the second image in the pixel values ​​of the fused image can be adjusted according to the first and second probabilities during the fusion processing, making the texture information of each pixel in the fused image richer and eliminating trailing, thereby improving the quality of the fused image.

[0074] exist Figure 2 Based on the illustrated embodiments, the image fusion process will be described in detail below with reference to a specific embodiment.

[0075] Figure 3 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 3 As shown, the method in this embodiment includes:

[0076] S301: Determine the exposure ratio based on the exposure duration of the first image and the exposure duration of the second image.

[0077] For example, the ratio between the exposure duration of the first image and the exposure duration of the second image is determined as the exposure ratio.

[0078] S302: Based on the first probability that each pixel in the first image is a first type of pixel and the exposure ratio, the first image and the second image are fused to obtain an intermediate image.

[0079] In this embodiment, assuming that the first image and the second image each include M*N pixels, the pixel value of the (i, j)th pixel in the intermediate image can be determined based on the first probability that the (i, j)th pixel in the first image is a first type pixel, the exposure ratio, the pixel value of the (i, j)th pixel in the first image, and the pixel value of the (i, j)th pixel in the second image. Here, i takes values ​​of 1, 2, ..., M, j takes values ​​of 1, 2, ..., N, and M and N are integers greater than 1.

[0080] For example, the weight corresponding to the pixel value of the (i, j)th pixel in the first image and the weight corresponding to the pixel value of the (i, j)th pixel in the second image can be determined based on the first probability that the (i, j)th pixel in the first image is a first type pixel and the exposure ratio. The pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image are then weighted and calculated to obtain the pixel value of the (i, j)th pixel in the intermediate image.

[0081] Optionally, the pixel value of the (i, j)th pixel in the intermediate image satisfies the following formula (1):

[0082]

[0083] Wherein, P′ (i,j) Let (i, j) be the pixel value of the (i, j)th pixel in the intermediate image. The first probability that the (i, j)th pixel in the first image is a pixel of the first type is given by the following formula. Let E be the pixel value of the (i, j)th pixel in the second image, and let E be the exposure ratio. Let be the pixel value of the (i, j)th pixel in the first image.

[0084] As can be seen from the above formula (1), the first probability that the (i, j)th pixel in the first image is a pixel of the first type is... The pixel value of the (i, j)th pixel in the second image The weights, when the (i, j)th pixel in the first image is a pixel of the first type, are determined by the first probability. When it is large, the pixel value of the (i, j)th pixel in the second image is... They account for a relatively high proportion in the intermediate image. Therefore, if the first type of pixel is an overexposed pixel, it makes the texture information of the (i, j)th pixel in the intermediate image richer; if the first type of pixel is a moving pixel, it makes the (i, j)th pixel in the intermediate image free of trailing.

[0085] Additionally, by comparing the exposure ratio E with the pixel value of the (i, j)th pixel in the second image... Perform the multiplication to make the pixel value of the (i, j)th pixel in the second image... The pixel value of the (i, j)th pixel in the first image The brightness of the two pixels is similar, and the brightness difference between them is reduced, so that the brightness of each pixel in the fused intermediate image is more balanced.

[0086] S303: Based on the second probability that each pixel in the first image is a second type of pixel and the exposure ratio, the second image and the intermediate image are fused to obtain a fused image.

[0087] In this embodiment, the pixel value of the (i, j)th pixel in the fused image can be determined based on the second probability that the (i, j)th pixel in the first image is a second type pixel, the exposure ratio, the pixel value of the (i, j)th pixel in the second image, and the pixel value of the (i, j)th pixel in the intermediate image. Here, i takes values ​​of 1, 2, ..., M, and j takes values ​​of 1, 2, ..., N.

[0088] For example, the weight corresponding to the pixel value of the (i, j)th pixel in the intermediate image and the weight corresponding to the pixel value of the (i, j)th pixel in the second image can be determined based on the second probability that the (i, j)th pixel in the first image is a second type pixel and the exposure ratio. The pixel value of the (i, j)th pixel in the intermediate image and the pixel value of the (i, j)th pixel in the second image are then weighted and calculated to obtain the pixel value of the (i, j)th pixel in the fused image.

[0089] Optionally, the pixel value of the (i, j)th pixel in the fused image satisfies the following formula (2):

[0090]

[0091] Wherein, P (i,j) Let (i, j) be the pixel value of the (i, j)th pixel in the fused image. The second probability is that the (i, j)th pixel in the first image is a second type pixel. Let P' be the pixel value of the (i, j)th pixel in the second image, E be the exposure ratio, and P' be the pixel value of the (i, j)th pixel in the second image. (i,j) Let be the pixel value of the (i, j)th pixel in the intermediate image.

[0092] As can be seen from the above formula (2), the second probability that the (i, j)th pixel in the first image is a second type pixel is... The pixel value of the (i, j)th pixel in the second image The weights, when the (i, j)th pixel in the first image is a second type pixel, are determined by the second probability. When it is large, the pixel value of the (i, j)th pixel in the second image is... They account for a relatively high proportion in the fused image. Therefore, if the second type of pixel is a moving pixel, the (i, j)th pixel in the fused image will not have a trailing effect; if the second type of pixel is an overexposed pixel, the (i, j)th pixel in the fused image will have richer texture information.

[0093] Additionally, by comparing the exposure ratio E with the pixel value of the (i, j)th pixel in the second image... Perform the multiplication to make the pixel value of the (i, j)th pixel in the second image... The pixel value of the (i, j)th pixel in the first image The brightness of the pixels is similar, and the brightness difference between the two is reduced, so that the brightness of each pixel in the fused image is more balanced.

[0094] As an example, Figure 4 This is a schematic diagram illustrating the fusion process of the first and second images provided in an embodiment of this application. Figure 4 As shown, the fusion process of the first image and the second image in this embodiment includes two steps. In the first step, the first image and the second image are fused using the first probability that each pixel in the first image is a first type of pixel and the exposure ratio to obtain an intermediate image. Thus, the fused intermediate image does not contain any first type of pixels. In the second step, the second image and the intermediate image are fused using the second probability that each pixel in the first image is a second type of pixel and the exposure ratio to obtain a fused image. Thus, the fused image contains neither first type of pixels nor second type of pixels, thereby improving the quality of the fused image.

[0095] It should be understood that Figure 3 and Figure 4 In the illustrated embodiment, the first type of pixel can be an overexposed pixel, and the second type of pixel can be a moving pixel; or, the first type of pixel can be a moving pixel, and the second type of pixel can be an overexposed pixel.

[0096] In practical applications, image sensors inevitably generate noise during image acquisition. Therefore, after the image sensor generates an image, an image processor is usually needed to perform noise reduction processing. One possible implementation is to pre-calibrate the image acquisition device to determine the noise model corresponding to the image acquisition process. This noise model indicates the noise distribution in the image generated by the image sensor. For example, the noise model can indicate how much effective signal and noise are contained in each pixel of the image. This noise model is then stored in the image processor. After receiving the image acquired by the image sensor, the image processor uses the noise model to perform noise reduction processing, thereby improving image quality. This implementation is typically applied in the linear mode of the image acquisition device.

[0097] In this embodiment of the application, when the image acquisition device is operating in WDR mode, noise will also exist in the fused image obtained by fusing the first image and the second image. Therefore, Figure 5This is a schematic diagram illustrating the processing procedure of another image acquisition device provided in an embodiment of this application. For example... Figure 5 As shown, after fusing the first image and the second image to obtain the fused image, the image acquisition device can also perform noise reduction processing on the fused image to obtain the target image.

[0098] Understandably, the aforementioned fusion process disrupts the noise distribution in the original image acquired by the image sensor. In other words, the noise distribution in the fused image differs from that in the original image. Therefore, if the pre-calibrated noise model is still used to denoise the fused image, the denoising effect will be compromised.

[0099] Therefore, in the technical solution provided in this application embodiment, when performing noise reduction processing on the fused image, it does not directly use a pre-calibrated noise model for noise reduction; instead, it determines the target noise distribution in the fused image based on the actual fusion processing process, and then performs noise reduction processing on the fused image according to the target noise distribution. This can further improve the quality of the fused image. The following is in conjunction with... Figure 6 and Figure 7 The noise reduction process is explained in detail.

[0100] Figure 6 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 6 As shown, the method in this embodiment includes:

[0101] S601: Acquire a first image and a second image captured by the image sensor according to the shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image.

[0102] S602: Determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel.

[0103] S603: Based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, perform a fusion process on the first image and the second image to obtain a fused image.

[0104] It should be understood that S601 to S603 can be found in [reference needed]. Figure 2 or Figure 3 The detailed description of the embodiments is omitted here.

[0105] S604: Determine the target noise distribution of the fused image based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, wherein the target noise distribution includes the noise of each pixel in the fused image.

[0106] It should be noted that in this embodiment, the execution order of S603 and S604 is not limited. S603 can be executed first and then S604, or S604 can be executed first and then S603, or S603 and S604 can be executed simultaneously.

[0107] S605: Perform noise reduction processing on the fused image according to the target noise distribution to obtain the target image.

[0108] It is understood that the first image and the second image are raw images acquired by the image sensor, and both contain noise. Furthermore, the noise distribution in the first image is consistent with a pre-calibrated noise model, and the noise distribution in the second image is also consistent with a pre-calibrated noise model. When the first image and the second image are fused in S603, the noise in both images is also fused in the same way. Therefore, in this embodiment, the target noise distribution of the fused image can be determined based on the fusion method of pixels in the image fusion process. The target noise distribution indicates the noise of each pixel in the fused image.

[0109] Since S603 performs fusion processing on the first image and the second image based on the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel, in this embodiment, the target noise distribution of the fused image can be determined based on the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel.

[0110] Furthermore, after determining the target noise distribution of the fused image, noise reduction processing can be performed on the fused image based on the target noise distribution to obtain the target image. The target image can be an image output by the image acquisition device.

[0111] In this embodiment, the first and second images are fused based on a first probability that each pixel in the first image is a first type of pixel and a second probability that each pixel in the first image is a second type of pixel to obtain a fused image. The target noise distribution of the fused image is then determined based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, making the determined target noise distribution more accurate. Furthermore, denoising the fused image based on the target noise distribution can further improve the quality of the fused image.

[0112] exist Figure 6 Based on the illustrated embodiment, the following is combined with Figure 7 The illustrated embodiment details the process of determining the target noise distribution.

[0113] Figure 7 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 7 As shown, the method in this embodiment includes:

[0114] S701: Obtain the first noise distribution of the first image according to the preset noise model, wherein the first noise distribution includes the noise of each pixel in the first image.

[0115] S702: According to the preset noise model, obtain the second noise distribution of the second image, wherein the second noise distribution includes the noise of each pixel in the second image.

[0116] The preset noise model can be a noise model determined beforehand by calibrating the image acquisition device. The preset noise model indicates the noise distribution in the original image acquired by the image sensor in the image acquisition device. For example, in this embodiment, the noise distribution in the first image is consistent with the preset noise model, and the noise distribution in the second image is also consistent with the preset noise model. Therefore, the first noise distribution of the first image can be determined using the preset noise model, and the first noise distribution includes the noise of each pixel in the first image. The second noise distribution of the second image can also be determined using the preset noise model, and the second noise distribution includes the noise of each pixel in the second image.

[0117] S703: Determine the exposure ratio based on the exposure duration of the first image and the exposure duration of the second image.

[0118] For example, the ratio between the exposure duration of the first image and the exposure duration of the second image can be determined as the exposure ratio.

[0119] It should be noted that this embodiment does not limit the execution order of S701, S702, and S703; the execution order of the three can be arbitrary. In some possible implementations, the three can also be executed simultaneously.

[0120] Furthermore, in this embodiment, the target noise distribution can be determined based on the first probability that each pixel in the first image is a first type of pixel, the second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution. It is understood that since the first noise distribution of the first image and the second noise distribution of the second image are already known, the target noise distribution can be determined based on... Figure 3The fusion method of each pixel in the first image and the second image in the illustrated embodiment performs a similar fusion process on the first noise distribution and the second noise distribution to obtain the target noise distribution.

[0121] One possible implementation can be found in the detailed descriptions of S704 and S705.

[0122] S704: Determine the intermediate noise distribution based on the first probability that each pixel in the first image is a first type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution.

[0123] In other words, based on the first probability and exposure ratio of each pixel in the first image being a first type of pixel, the first noise distribution and the second noise distribution are fused to obtain the intermediate noise distribution.

[0124] In this embodiment, it is assumed that the first image and the second image each include M*N pixels, where M and N are integers greater than 1. The noise of the (i, j)th pixel in the intermediate noise distribution can be determined based on the first probability that the (i, j)th pixel in the first image is a first type pixel, the exposure ratio, the noise of the (i, j)th pixel in the first noise distribution, and the noise of the (i, j)th pixel in the second noise distribution. Here, i takes values ​​of 1, 2, ..., M, and j takes values ​​of 1, 2, ..., N.

[0125] For example, the weights corresponding to the noise of the (i, j)th pixel in the first noise distribution and the noise of the (i, j)th pixel in the second noise distribution can be determined based on the first probability that the (i, j)th pixel in the first image is a first type pixel and the exposure ratio. The noise of the (i, j)th pixel in the first noise distribution and the noise of the (i, j)th pixel in the second noise distribution are then weighted and calculated to obtain the noise of the (i, j)th pixel in the intermediate noise distribution.

[0126] Optionally, the noise of the (i, j)th pixel in the intermediate noise distribution satisfies the following formula (3):

[0127]

[0128] Wherein, the N′ (i,j) The noise of the (i, j)th pixel in the intermediate noise distribution, the The first probability that the (i, j)th pixel in the first image is a pixel of the first type is given by the following formula. The noise of the (i, j)th pixel in the second noise distribution, where E is the exposure ratio, and... Let be the noise of the (i, j)th pixel in the first noise distribution.

[0129] In the above formula (3), the exposure ratio E is compared with the noise of the (i, j)th pixel in the second noise distribution. Multiply them to reduce the noise of the (i, j)th pixel in the second noise distribution. The noise of the (i, j)th pixel in the first noise distribution The values ​​of are similar, narrowing the gap between the two, making the noise values ​​of each pixel in the determined intermediate noise distribution more balanced, thereby improving the accuracy of the intermediate noise distribution.

[0130] It should be understood that the intermediate noise distribution determined in this embodiment represents... Figure 3 The noise of each pixel in the intermediate image shown in the embodiment.

[0131] S705: Determine the target noise distribution based on the second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the second noise distribution, and the intermediate noise distribution.

[0132] In other words, based on the second probability and exposure ratio of each pixel in the first image being a second type of pixel, the second noise distribution and the intermediate noise distribution are fused to obtain the target noise distribution.

[0133] In this embodiment, the noise of the (i, j)th pixel in the target noise distribution can be determined based on the second probability that the (i, j)th pixel in the first image is a second type pixel, the exposure ratio, the noise of the (i, j)th pixel in the second noise distribution, and the noise of the (i, j)th pixel in the intermediate noise distribution. Here, i takes values ​​of 1, 2, ..., M, and j takes values ​​of 1, 2, ..., N.

[0134] For example, the weights corresponding to the noise of the (i, j)th pixel in the intermediate noise distribution and the noise of the (i, j)th pixel in the second noise distribution can be determined based on the second probability that the (i, j)th pixel in the first image is a second type pixel and the exposure ratio. The noise of the (i, j)th pixel in the intermediate noise distribution and the noise of the (i, j)th pixel in the second noise distribution can be weighted and calculated to obtain the noise of the (i, j)th pixel in the target noise distribution.

[0135] Optionally, the noise of the (i, j)th pixel in the target noise distribution satisfies the following formula (4):

[0136]

[0137] Wherein, the N (i,j) For the noise of the (i, j)th pixel in the target noise distribution, the The second probability is that the (i, j)th pixel in the first image is a second type pixel. The noise of the (i, j)th pixel in the second noise distribution, where E is the exposure ratio, and N′ is the noise of the (i, j)th pixel. (i,j) Let be the noise of the (i, j)th pixel in the intermediate noise distribution.

[0138] In the above formula (4), the exposure ratio E is compared with the noise of the (i, j)th pixel in the second noise distribution. Multiply them to reduce the noise of the (i, j)th pixel in the second noise distribution. The noise of the (i, j)th pixel in the first noise distribution The values ​​of are similar, narrowing the gap between the two, making the noise values ​​of each pixel in the determined target noise distribution more balanced, thereby improving the accuracy of the target noise distribution.

[0139] As an example, Figure 8 This is a schematic diagram illustrating the process of determining the target noise distribution provided in an embodiment of this application. Figure 8 As shown, in this embodiment, a first noise distribution of the first image and a second noise distribution of the second image can be determined according to a preset noise model. The target noise distribution is obtained by fusing the first and second noise distributions in two steps: In the first step, the first noise distribution and the second noise distribution are fused using the first probability and exposure ratio of each pixel in the first image being a first type of pixel to obtain an intermediate noise distribution. In the second step, the second noise distribution and the intermediate noise distribution are fused using the second probability and exposure ratio of each pixel in the first image being a second type of pixel to obtain the target noise distribution.

[0140] In this embodiment, a fusion processing method based on a first image and a second image is used to fuse the first noise distribution of the first image and the second noise distribution of the second image to obtain the target noise distribution of the fused image. This allows the target noise distribution to accurately represent the noise of each pixel in the fused image, improving the accuracy of the target noise distribution. Furthermore, using the accurate target noise distribution to perform noise reduction processing on the fused image can truly improve the quality of the fused image.

[0141] It should be understood that Figure 7 and Figure 8 In the illustrated embodiment, the first type of pixel can be an overexposed pixel, and the second type of pixel can be a moving pixel; or, the first type of pixel can be a moving pixel, and the second type of pixel can be an overexposed pixel.

[0142] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 9 As shown, the image processing apparatus 900 provided in this embodiment includes: an acquisition module 901, a determination module 902, and a processing module 903. Wherein,

[0143] The acquisition module 901 is used to acquire a first image and a second image captured by the image sensor according to the shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image;

[0144] The determining module 902 is used to determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel.

[0145] The processing module 903 is used to perform fusion processing on the first image and the second image according to the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, to obtain a fused image.

[0146] In one possible implementation, the processing module 903 is specifically used for:

[0147] The exposure ratio is determined based on the exposure duration of the first image and the exposure duration of the second image;

[0148] Based on the first probability that each pixel in the first image is a first type of pixel and the exposure ratio, the first image and the second image are fused to obtain an intermediate image;

[0149] Based on the second probability that each pixel in the first image is a second type of pixel and the exposure ratio, the second image and the intermediate image are fused to obtain the fused image.

[0150] In one possible implementation, the first image and the second image each include M*N pixels; the processing module 903 is specifically used for:

[0151] The pixel value of the (i, j)th pixel in the intermediate image is determined based on the first probability that the (i, j)th pixel in the first image is a first type pixel, the exposure ratio, the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image.

[0152] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0153] In one possible implementation, the pixel value of the (i, j)th pixel in the intermediate image satisfies the following formula:

[0154]

[0155] Wherein, P′ (i,j) Let (i, j) be the pixel value of the (i, j)th pixel in the intermediate image. The first probability that the (i, j)th pixel in the first image is a pixel of the first type is given by the following formula. Let E be the pixel value of the (i, j)th pixel in the second image, and let E be the exposure ratio. Let be the pixel value of the (i, j)th pixel in the first image.

[0156] In one possible implementation, the first image and the second image each include M*N pixels; the processing module 903 is specifically used for:

[0157] The pixel value of the (i, j)th pixel in the fused image is determined based on the second probability that the (i, j)th pixel in the first image is a second type pixel, the exposure ratio, the pixel value of the (i, j)th pixel in the second image, and the pixel value of the (i, j)th pixel in the intermediate image.

[0158] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0159] In one possible implementation, the pixel value of the (i, j)th pixel in the fused image satisfies the following formula:

[0160]

[0161] Wherein, P′ (i,j) Let (i, j) be the pixel value of the (i, j)th pixel in the fused image. The second probability is that the (i, j)th pixel in the first image is a second type pixel. Let P' be the pixel value of the (i, j)th pixel in the second image, E be the exposure ratio, and P' be the pixel value of the (i, j)th pixel in the second image. (i,j) Let be the pixel value of the (i, j)th pixel in the intermediate image.

[0162] In one possible implementation, for any pixel in the first image, the determining module 902 is specifically used for:

[0163] Obtain the first difference between the pixel value of the pixel and the first preset threshold;

[0164] Based on the first difference, a first probability is determined that the pixel is a first type of pixel, and the first probability is directly proportional to the first difference.

[0165] In one possible implementation, the first image and the second image each include M*N pixels, where M and N are integers greater than 1.

[0166] For any (i, j)th pixel in the first image; the determining module 902 is specifically used for:

[0167] Obtain the second difference between the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image;

[0168] Based on the second difference, a second probability is determined that the (i, j)th pixel in the first image is a second type pixel, and the second probability is directly proportional to the second difference.

[0169] Where i is an integer less than or equal to M, and j is an integer less than or equal to N.

[0170] In one possible implementation, the processing module 903 is further used for:

[0171] Based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, the target noise distribution of the fused image is determined, wherein the target noise distribution includes the noise of each pixel in the fused image;

[0172] The fused image is denoised according to the target noise distribution to obtain the target image.

[0173] In one possible implementation, the processing module 903 is specifically used for:

[0174] According to a preset noise model, a first noise distribution of the first image is obtained, wherein the first noise distribution includes the noise of each pixel in the first image;

[0175] According to the preset noise model, a second noise distribution of the second image is obtained, wherein the second noise distribution includes the noise of each pixel in the second image;

[0176] The exposure ratio is determined based on the exposure duration of the first image and the exposure duration of the second image;

[0177] The target noise distribution is determined based on the first probability that each pixel in the first image is a first type of pixel, the second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution.

[0178] In one possible implementation, the processing module 903 is specifically used for:

[0179] Based on the first probability that each pixel in the first image is a first type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution, the intermediate noise distribution is determined.

[0180] The target noise distribution is determined based on the second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the second noise distribution, and the intermediate noise distribution.

[0181] In one possible implementation, the first image and the second image each include M*N pixels; the processing module 903 is specifically used for:

[0182] Based on the first probability that the (i, j)th pixel in the first image is a first type pixel, the exposure ratio, the noise of the (i, j)th pixel in the first noise distribution and the noise of the (i, j)th pixel in the second noise distribution, the noise of the (i, j)th pixel in the intermediate noise distribution is determined.

[0183] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0184] In one possible implementation, the noise of the (i, j)th pixel in the intermediate noise distribution satisfies the following formula:

[0185]

[0186] Wherein, the N′ (i,j) The noise of the (i, j)th pixel in the intermediate noise distribution, the The first probability that the (i, j)th pixel in the first image is a pixel of the first type is given by the following formula. The noise of the (i, j)th pixel in the second noise distribution, where E is the exposure ratio, and... Let be the noise of the (i, j)th pixel in the first noise distribution.

[0187] In one possible implementation, the first image and the second image each include M*N pixels; the processing module 903 is specifically used for:

[0188] Based on the second probability that the (i, j)th pixel in the first image is a second type pixel, the exposure ratio, the noise of the (i, j)th pixel in the second noise distribution, and the noise of the (i, j)th pixel in the intermediate noise distribution, the noise of the (i, j)th pixel in the target noise distribution is determined.

[0189] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0190] In one possible implementation, the noise of the (i, j)th pixel in the target noise distribution satisfies the following formula:

[0191]

[0192] Wherein, the N (i,j) For the noise of the (i, j)th pixel in the target noise distribution, the The second probability is that the (i, j)th pixel in the first image is a second type pixel. The noise of the (i, j)th pixel in the second noise distribution, where E is the exposure ratio, and N′ is the noise of the (i, j)th pixel. (i,j) Let be the noise of the (i, j)th pixel in the intermediate noise distribution.

[0193] The image processing apparatus provided in this embodiment can be used to execute the image processing methods in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0194] Figure 10 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. The electronic device can be an image acquisition device, or a processing chip within an image acquisition device. Figure 10 As shown, the electronic device 1000 provided in this embodiment includes at least one processor 1001 and a memory 1002. Exemplarily, the processor 1001 and the memory 1002 can be connected via a bus 1003.

[0195] Memory 1002 is used to store computer programs;

[0196] At least one processor 1001 is used to execute the computer program stored in the memory, so that the electronic device 1000 performs the image processing method provided in any of the above embodiments. The implementation principle and technical effect are similar, and will not be described in detail here.

[0197] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the image processing method provided in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be described in detail here.

[0198] This application also provides a chip, including: a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to implement the image processing method provided in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be described in detail here.

[0199] This application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the image processing method provided in any of the above method embodiments. The implementation principle and technical effects are similar, and will not be described in detail here.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0201] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0203] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0204] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU); it can be a dedicated processor, including a Graphics Processing Unit (GPU) and an Intelligence Processing Unit (IPU); it can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0205] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0206] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0207] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0208] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0209] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0210] The foregoing can be better understood in accordance with the following terms:

[0211] Clause A1. An image processing method, comprising:

[0212] Acquire a first image and a second image captured by an image sensor according to a shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image;

[0213] Determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel;

[0214] Based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, the first image and the second image are fused to obtain a fused image.

[0215] Clause A2. The method according to Clause A1, wherein fusing the first image and the second image based on a first probability that each pixel in the first image is a first type of pixel and a second probability that each pixel in the first image is a second type of pixel to obtain a fused image, includes:

[0216] The exposure ratio is determined based on the exposure duration of the first image and the exposure duration of the second image;

[0217] Based on the first probability that each pixel in the first image is a first type of pixel and the exposure ratio, the first image and the second image are fused to obtain an intermediate image;

[0218] Based on the second probability that each pixel in the first image is a second type of pixel and the exposure ratio, the second image and the intermediate image are fused to obtain the fused image.

[0219] Clause A3. According to the method described in Clause A2, the first image and the second image each include M*N pixels; the step of fusing the first image and the second image based on a first probability that each pixel in the first image is a first type of pixel and the exposure ratio to obtain an intermediate image includes:

[0220] The pixel value of the (i, j)th pixel in the intermediate image is determined based on the first probability that the (i, j)th pixel in the first image is a first type pixel, the exposure ratio, the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image.

[0221] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0222] Clause A4. According to the method described in Clause A3, the pixel value of the (i, j)th pixel in the intermediate image satisfies the following formula:

[0223]

[0224] Wherein, P′ (i,j) Let (i, j) be the pixel value of the (i, j)th pixel in the intermediate image. The first probability that the (i, j)th pixel in the first image is a pixel of the first type is given by the following formula. Let E be the pixel value of the (i, j)th pixel in the second image, and let E be the exposure ratio. Let be the pixel value of the (i, j)th pixel in the first image.

[0225] Clause A5. The method according to any one of Clauses A2 to 4, wherein the first image and the second image each comprise M*N pixels; the step of fusing the second image and the intermediate image based on a second probability that each pixel in the first image is a second type of pixel and the exposure ratio to obtain the fused image includes:

[0226] The pixel value of the (i, j)th pixel in the fused image is determined based on the second probability that the (i, j)th pixel in the first image is a second type pixel, the exposure ratio, the pixel value of the (i, j)th pixel in the second image, and the pixel value of the (i, j)th pixel in the intermediate image.

[0227] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0228] Clause A6. According to the method described in Clause A5, the pixel value of the (i, j)th pixel in the fused image satisfies the following formula:

[0229]

[0230] Wherein, P (i,j) Let (i, j) be the pixel value of the (i, j)th pixel in the fused image. The second probability is that the (i, j)th pixel in the first image is a second type pixel. Let P' be the pixel value of the (i, j)th pixel in the second image, E be the exposure ratio, and P' be the pixel value of the (i, j)th pixel in the second image. (i,j) Let be the pixel value of the (i, j)th pixel in the intermediate image.

[0231] Clause A7. The method according to any one of Clauses A1 to A6, for any pixel in the first image, determining a first probability that the pixel in the first image is a first type pixel, includes:

[0232] Obtain the first difference between the pixel value of the pixel and the first preset threshold;

[0233] Based on the first difference, a first probability is determined that the pixel is a first type of pixel, and the first probability is directly proportional to the first difference.

[0234] Clause A8. The method according to any one of Clauses A1 to 7, wherein the first image and the second image each comprise M*N pixels, where M and N are integers greater than 1;

[0235] For any (i, j)-th pixel in the first image, determining the second probability that the (i, j)-th pixel in the first image is a second type pixel includes:

[0236] Obtain the second difference between the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image;

[0237] Based on the second difference, a second probability is determined that the (i, j)th pixel in the first image is a second type pixel, and the second probability is directly proportional to the second difference.

[0238] Where i is an integer less than or equal to M, and j is an integer less than or equal to N.

[0239] Clause A9. The method according to any one of Clauses A1 to A8, wherein after fusing the first image and the second image based on a first probability that each pixel in the first image is a first type of pixel and a second probability that each pixel in the first image is a second type of pixel to obtain a fused image, the method further includes:

[0240] Based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, the target noise distribution of the fused image is determined, wherein the target noise distribution includes the noise of each pixel in the fused image;

[0241] The fused image is denoised according to the target noise distribution to obtain the target image.

[0242] Clause A10. The method according to Clause A9, wherein determining the target noise distribution of the fused image based on a first probability that each pixel in the first image is a first type of pixel and a second probability that each pixel in the first image is a second type of pixel, comprises:

[0243] According to a preset noise model, a first noise distribution of the first image is obtained, wherein the first noise distribution includes the noise of each pixel in the first image;

[0244] According to the preset noise model, a second noise distribution of the second image is obtained, wherein the second noise distribution includes the noise of each pixel in the second image;

[0245] The exposure ratio is determined based on the exposure duration of the first image and the exposure duration of the second image;

[0246] The target noise distribution is determined based on the first probability that each pixel in the first image is a first type of pixel, the second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution.

[0247] Clause A11. The method according to Clause A10, wherein determining the target noise distribution based on a first probability that each pixel in the first image is a first type of pixel, a second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution comprises:

[0248] Based on the first probability that each pixel in the first image is a first type of pixel, the exposure ratio, the first noise distribution, and the second noise distribution, the intermediate noise distribution is determined.

[0249] The target noise distribution is determined based on the second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the second noise distribution, and the intermediate noise distribution.

[0250] Clause A12. The method according to Clause A11, wherein the first image and the second image each comprise M*N pixels; the step of determining the intermediate noise distribution based on the first probability that each pixel in the first image is a first type pixel, the exposure ratio, the first noise distribution, and the second noise distribution includes:

[0251] Based on the first probability that the (i, j)th pixel in the first image is a first type pixel, the exposure ratio, the noise of the (i, j)th pixel in the first noise distribution and the noise of the (i, j)th pixel in the second noise distribution, the noise of the (i, j)th pixel in the intermediate noise distribution is determined.

[0252] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0253] Clause A13. According to the method described in Clause A12, the noise of the (i, j)th pixel in the intermediate noise distribution satisfies the following formula:

[0254]

[0255] Wherein, the N′ (i,j) The noise of the (i, j)th pixel in the intermediate noise distribution, the The first probability that the (i, j)th pixel in the first image is a pixel of the first type is given by the following formula. The noise of the (i, j)th pixel in the second noise distribution, where E is the exposure ratio, and... Let be the noise of the (i, j)th pixel in the first noise distribution.

[0256] Clause A14. The method according to any one of Clauses A11 to A13, wherein the first image and the second image each comprise M*N pixels; the step of determining the target noise distribution based on a second probability that each pixel in the first image is a second type of pixel, the exposure ratio, the second noise distribution, and the intermediate noise distribution includes:

[0257] Based on the second probability that the (i, j)th pixel in the first image is a second type pixel, the exposure ratio, the noise of the (i, j)th pixel in the second noise distribution, and the noise of the (i, j)th pixel in the intermediate noise distribution, the noise of the (i, j)th pixel in the target noise distribution is determined.

[0258] Where i takes the values ​​1, 2, ..., M in sequence, j takes the values ​​1, 2, ..., N in sequence, and M and N are integers greater than 1.

[0259] Clause A15. According to the method described in Clause A14, the noise of the (i, j)th pixel in the target noise distribution satisfies the following formula:

[0260]

[0261] Wherein, the N (i,j) For the noise of the (i, j)th pixel in the target noise distribution, the The second probability is that the (i, j)th pixel in the first image is a second type pixel. The noise of the (i, j)th pixel in the second noise distribution, where E is the exposure ratio, and N′ is the noise of the (i, j)th pixel. (i,j) Let be the noise of the (i, j)th pixel in the intermediate noise distribution.

[0262] Clause A16. An image processing apparatus comprising:

[0263] The acquisition module is used to acquire a first image and a second image captured by the image sensor according to the shooting command, wherein the exposure duration of the first image is greater than the exposure duration of the second image;

[0264] The determining module is used to determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel and the second type pixel is a moving pixel, or, the first type pixel is a moving pixel and the second type pixel is an overexposed pixel.

[0265] The processing module is used to perform fusion processing on the first image and the second image based on the first probability that each pixel in the first image is a first type of pixel and the second probability that each pixel in the first image is a second type of pixel, to obtain a fused image.

[0266] Clause A17. An electronic device comprising:

[0267] At least one processor; and

[0268] A memory communicatively connected to the at least one processor; wherein,

[0269] The memory stores a computer program, and the at least one processor executes the computer program to implement the method as described in any one of clauses A1 to 15.

[0270] Clause A18. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of Clauses A1 to A15.

[0271] Clause A19. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of Clauses A1 to A15.

[0272] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: acquiring a first image and a second image collected by an image sensor according to a shooting instruction, the exposure time of the first image being longer than the exposure time of the second image; determining a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein the first type pixel is an overexposed pixel, and the second type pixel is a motion pixel, or the first type pixel is a motion pixel, and the second type pixel is an overexposed pixel; determining an exposure ratio according to an exposure time length of the first image and an exposure time length of the second image, the first image and the second image respectively comprising M N pixels; performing fusion processing on the first image and the second image according to the first probability that each pixel in the first image is a first type pixel and the exposure ratio, to obtain an intermediate image; performing fusion processing on the second image and the intermediate image according to the second probability that each pixel in the first image is a second type pixel and the exposure ratio, to obtain a fusion image; wherein the pixel value of the (i, j)th pixel in the intermediate image satisfies the following formula: ； the pixel value of the (i, j)th pixel in the fusion image satisfies the following formula: ； wherein the is a pixel value of an (i, j)th pixel in the intermediate image, the is a first probability that the (i, j)th pixel in the first image is a first type pixel, the is a pixel value of an (i, j)th pixel in the second image, the E is the exposure ratio, the is a pixel value of an (i, j)th pixel in the first image, the is a pixel value of an (i, j)th pixel in the fused image, the is a second probability that the (i, j)th pixel in the first image is a second type pixel, the i is taken 1, 2, …, M in turn, the j is taken 1, 2, …, N in turn, and the M and the N are integers greater than 1, respectively.

2. The method of claim 1, wherein, For any pixel in the first image, determining the first probability that the pixel in the first image is a first type pixel comprises: acquiring a first difference between the pixel value of the pixel and a first preset threshold; determining the first probability that the pixel is a first type pixel according to the first difference, the first probability being in a positive proportional relationship with the first difference.

3. The method of claim 1, wherein, For any (i, j)th pixel in the first image, determining the second probability that the (i, j)th pixel in the first image is a second type pixel comprises: acquiring a second difference between the pixel value of the (i, j)th pixel in the first image and the pixel value of the (i, j)th pixel in the second image; determining the second probability that the (i, j)th pixel in the first image is a second type pixel according to the second difference, the second probability being in a positive proportional relationship with the second difference; wherein the i is an integer less than or equal to M, and the j is an integer less than or equal to N.

4. The method according to any one of claims 1 to 3, characterized in that, After the fusion image is obtained by performing fusion processing on the first image and the second image according to the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel, the method further comprises: determining a target noise distribution of the fusion image according to the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel, the target noise distribution including the noise of each pixel in the fusion image; performing noise reduction processing on the fusion image according to the target noise distribution, to obtain a target image.

5. The method of claim 4, wherein, The method of determining the target noise distribution of the fusion image according to the first probability that each pixel in the first image is a first type pixel and the second probability that each pixel in the first image is a second type pixel comprises: acquiring a first noise distribution of the first image according to a preset noise model, the first noise distribution including the noise of each pixel in the first image; obtaining a second noise distribution of the second image according to the preset noise model, the second noise distribution including noise of each pixel in the second image; determining an exposure ratio according to an exposure time length of the first image and an exposure time length of the second image; determining the target noise distribution according to the first probability that each pixel in the first image is a first type pixel, the second probability that each pixel in the first image is a second type pixel, the exposure ratio, the first noise distribution and the second noise distribution.

6. The method of claim 5, wherein, The determining the target noise distribution according to the first probability that each pixel in the first image is a first type pixel, the second probability that each pixel in the first image is a second type pixel, the exposure ratio, the first noise distribution and the second noise distribution includes: determining an intermediate noise distribution according to the first probability that each pixel in the first image is a first type pixel, the exposure ratio, the first noise distribution and the second noise distribution; determining the target noise distribution according to the second probability that each pixel in the first image is a second type pixel, the exposure ratio, the second noise distribution and the intermediate noise distribution.

7. The method of claim 6, wherein, The determining the intermediate noise distribution according to the first probability that each pixel in the first image is a first type pixel, the exposure ratio, the first noise distribution and the second noise distribution includes: determining noise of an (i, j) pixel in the intermediate noise distribution according to the first probability that the (i, j) pixel in the first image is a first type pixel, the exposure ratio, noise of the (i, j) pixel in the first noise distribution and noise of the (i, j) pixel in the second noise distribution.

8. The method of claim 7, wherein, The noise of the (i, j) pixel in the intermediate noise distribution satisfies the following formula: ； wherein the is the noise of the (i, j)th pixel in the second noise distribution, the is the noise of the (i, j)th pixel in the second noise distribution, the is the noise of the (i, j)th pixel in the first noise distribution.

9. The method according to any one of claims 6 to 8, characterized in that, M N pixels; and determining the target noise distribution according to the second probability that each pixel in the first image is a second type pixel, the exposure ratio, the second noise distribution, and the intermediate noise distribution. determining noise of an (i, j) pixel in the target noise distribution according to the second probability that the (i, j) pixel in the first image is a second type pixel, the exposure ratio, noise of the (i, j) pixel in the second noise distribution and noise of the (i, j) pixel in the intermediate noise distribution; wherein, the i takes 1, 2, …, M in turn, the j takes 1, 2, …, N in turn, and the M and the N are integers greater than 1.

10. The method of claim 9, wherein, The noise of the (i, j) pixel in the target noise distribution satisfies the following formula: ； wherein the is the noise of the (i, j)th pixel in the target noise distribution, the is the second probability that the (i, j)th pixel in the first image is of the second type, the is the noise of the (i, j)th pixel in the second noise distribution, the E is the exposure ratio, the is the noise of the (i, j)th pixel in the intermediate noise distribution.

11. An image processing apparatus characterized by comprising: including: an obtaining module, configured to obtain a first image and a second image collected by an image sensor according to a shooting instruction, an exposure time length of the first image being greater than an exposure time length of the second image; a determining module, configured to determine a first probability that each pixel in the first image is a first type pixel and a second probability that each pixel in the first image is a second type pixel; wherein, the first type pixel is an overexposed pixel, and the second type pixel is a motion pixel, or the first type pixel is a motion pixel, and the second type pixel is an overexposed pixel; a processing module configured to determine an exposure ratio according to an exposure time of the first image and an exposure time of the second image, the first image and the second image each comprising M N pixels; performing fusion processing on the first image and the second image according to the first probability that each pixel in the first image is a first type pixel and the exposure ratio, to obtain an intermediate image; According to a second probability that each pixel in the first image is a second type pixel and the exposure ratio, a fusion processing is performed on the second image and the intermediate image to obtain a fusion image; wherein a pixel value of an (i, j)th pixel in the intermediate image satisfies the following formula: ； a pixel value of an (i, j)th pixel in the fusion image satisfies the following formula: ； wherein the is a pixel value of an (i, j)th pixel in the intermediate image, the is a first probability that the (i, j)th pixel in the first image is a first type pixel, the is a pixel value of an (i, j)th pixel in the second image, the E is the exposure ratio, the is a pixel value of an (i, j)th pixel in the first image, the is a pixel value of an (i, j)th pixel in the fused image, the is a second probability that the (i, j)th pixel in the first image is a second type pixel, the i is taken 1, 2, …, M in turn, the j is taken 1, 2, …, N in turn, and the M and the N are integers greater than 1, respectively.

12. An electronic device, comprising: comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program, and the at least one processor executes the computer program to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product, characterised in that, comprising a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 10.

Images