Image processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses an image processing method and device, equipment, a storage medium and a program product, and the method is applied to an image sensing module, and comprises the steps: obtaining an overexposure frame, a reference frame and a weak exposure frame of different exposure parameters in a YUV format through the image sensing module; according to the brightness of each pixel position of the reference frame and a preset brightness threshold, obtaining exposure fusion weights of the overexposure frame, the reference frame and the weak exposure frame; according to the exposure fusion weights of the overexposure frame, the reference frame and the weak exposure frame, respectively obtaining numerical values of a brightness Y component and a chroma UV component corresponding to each pixel position of the fused image; and generating the fused image according to the numerical values of the Y component and the UV component corresponding to each pixel position of the fused image. According to the method, the exposure fusion weights of the reference frame, the overexposure frame and the weak exposure frame can be obtained according to the brightness of each pixel position in the reference frame in the YUV space, and the frames are fused, so that the image quality and the generation efficiency of the generated fused image are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and relate to, but are not limited to, an image processing method and apparatus, equipment, storage medium, and program product. Background Art

[0002] When taking photos using electronic terminal devices such as mobile phones and smart watches with cameras, some areas of the captured image may be brighter or darker, resulting in unclear details in the corresponding areas of the image, affecting the image quality and the user's viewing experience. In the related technical field, it is common to obtain multiple frames of image frames with different exposure parameters, obtain the differences in contrast and saturation between the frames, and then obtain the fusion weights of each image frame to generate a fused image. Compared with a single-frame image, the fused image generated in this way enriches the details of the brighter or darker areas, thereby improving the quality of the presented image.

[0003] In existing image fusion methods, when obtaining the fusion weights of multiple frames of images, there are usually differences between the fused image and the normal area of ​​the image frame captured under normal exposure conditions, which affects the imaging quality, has a large amount of calculation, and is difficult to implement on low-computing-power terminal devices. Summary of the Invention

[0004] In view of this, the image processing method, apparatus, device, and storage medium provided in the embodiments of the present application can obtain the exposure fusion weights of the reference frame, overexposed frame, and underexposed frame based on the brightness of each pixel position in the reference frame in YUV space and fuse each frame, thereby improving the image quality and generation efficiency of the generated fused image. The image processing method, apparatus, device, and storage medium provided in the embodiments of the present application are implemented as follows:

[0005] An image processing method provided in an embodiment of the present application is applied to an electronic terminal device provided with an image sensor module, comprising:

[0006] Acquire, through the image sensing module, an overexposed frame, a reference frame, and a weakly exposed frame having different exposure parameters, wherein the exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame, and the exposure parameter of the weakly exposed frame is less than the exposure parameter of the reference frame, and the overexposed frame, the reference frame, and the weakly exposed frame are image frames in a YUV format;

[0007] Obtaining exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame based on the brightness of each pixel position in the reference frame and a preset brightness threshold, the exposure fusion weights including the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame;

[0008] According to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, respectively obtaining the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image;

[0009] The fused image is generated according to the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

[0010] In some embodiments, the preset brightness threshold includes a preset high brightness threshold and a preset low brightness threshold;

[0011] The step of obtaining exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame according to the brightness of each pixel position of the reference frame and a preset brightness threshold comprises:

[0012] Obtaining an overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold, the overexposure fusion weight including the fusion weight of each pixel position of the overexposed frame;

[0013] Obtaining a weak exposure fusion weight of the weak exposure frame according to the brightness of each pixel position of the reference frame and the preset high brightness threshold, wherein the weak exposure fusion weight includes a fusion weight of each pixel position of the weak exposure frame;

[0014] According to the overexposure fusion weight of the overexposed frame and the underexposure fusion weight of the underexposed frame, a baseline fusion weight of the reference frame is obtained, wherein the baseline fusion weight includes the fusion weight of each pixel position of the reference frame, and the sum of the fusion weights corresponding to the same pixel position of the overexposed frame, the reference frame and the underexposed frame is 1.

[0015] In some embodiments, the low brightness threshold includes a first low brightness threshold and a second low brightness threshold, and the first low brightness threshold is less than the second low brightness threshold; the high brightness threshold includes a first high brightness threshold and a second high brightness threshold, and the first high brightness threshold is greater than the second high brightness threshold;

[0016] The acquiring the overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold comprises:

[0017] When the brightness of the target pixel position of the reference frame is less than or equal to the first low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame to be 1;

[0018] When the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining a fusion weight for the corresponding pixel position in the overexposed frame according to a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame;

[0019] When the brightness of the target pixel position in the reference frame is greater than the second low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame to be 0;

[0020] The step of obtaining the low-exposure fusion weight of the low-exposure frame according to the brightness of each pixel position of the reference frame and the preset high-brightness threshold comprises:

[0021] When the brightness of the target pixel position of the reference frame is greater than or equal to the first high brightness threshold, determining the fusion weight of the corresponding pixel position in the weakly exposed frame to be 1;

[0022] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the weakly exposed frame according to a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold;

[0023] When the brightness of the target pixel position in the reference frame is less than the second high brightness threshold, the fusion weight of the corresponding pixel position in the weakly exposed frame is determined to be 0.

[0024] In some embodiments, when the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame according to the difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame includes:

[0025] When the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining a fusion weight for the corresponding pixel position in the overexposed frame according to a preset overexposure weight formula and a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame;

[0026] The preset overexposure weight formula is:

[0027]

[0028] Wherein, W(i,j,0) is the fusion weight at the pixel position (i,j) in the overexposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, L1 is the first low brightness threshold, and L2 is the second low brightness threshold;

[0029] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight of the corresponding pixel position in the low-exposure frame according to a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold, including:

[0030] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the low-exposure frame according to a preset low-exposure weight formula and a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold;

[0031] The preset weak exposure weight formula is:

[0032]

[0033] Wherein, W(i,j,2) is the fusion weight at the pixel position (i,j) in the weakly exposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, H1 is the first high brightness threshold, and H2 is the second high brightness threshold.

[0034] In some embodiments, obtaining a value of a brightness Y component corresponding to each pixel position of the fused image according to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame includes:

[0035] performing Laplacian pyramid decomposition on the overexposed frame, the reference frame, and the underexposed frame in channels corresponding to the Y components to obtain Laplacian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, respectively;

[0036] Performing Gaussian pyramid decomposition on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame to obtain weighted Gaussian pyramid sequences corresponding to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, respectively;

[0037] According to the Laplacian pyramid sequences and weighted Gaussian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, a value of the Y component corresponding to each pixel position of the fused image is obtained.

[0038] In some embodiments, obtaining the value of the chroma UY component corresponding to each pixel position of the fused image according to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame includes:

[0039] performing weighted summation processing on the UY components at corresponding pixel positions of the overexposed frame, the reference frame, and the underexposed frame according to a preset weighting formula and the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, to obtain a value of the UY component corresponding to each pixel position of the fused image;

[0040] The preset weighting formula is:

[0041] UV result (i,j)=W(i,j,0)*UV(i,j,0)+W(i,j,1)*UV(i,j,1)+W(i,j,2)*UV(i,j,2),

[0042] Among them, UV result (i, j) is the value of the UV component at the pixel position (i, j) in the fused image, W(i, j, 0), W(i, j1) and W(i, j, 2) are the fusion weights at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively, and UV(i, j, 0), UV(i, j, 1) and UV(i, j, 2) are the values ​​of the UV component at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively.

[0043] In some embodiments, obtaining overexposed frames, reference frames, and underexposed frames with different exposure parameters through the image sensing module includes:

[0044] Acquire, by the image sensing module, the reference frame, at least one to-be-selected overexposed frame having an exposure parameter greater than that of the reference frame and different from that of the reference frame, and at least one to-be-selected underexposed frame having an exposure parameter less than that of the reference frame and different from that of the reference frame, wherein the reference frame, the at least one to-be-selected overexposed frame, and the at least one to-be-selected underexposed frame are image frames in a YUV format;

[0045] According to the exposure condition of the reference frame, the overexposed frame is selected from the at least one overexposed frame to be selected, and the underexposed frame is selected from the at least one underexposed frame to be selected.

[0046] In some embodiments, selecting the overexposed frame from the at least one to-be-selected overexposed frame and selecting the underexposed frame from the at least one to-be-selected underexposed frame according to the exposure condition of the reference frame includes:

[0047] Obtaining a first pixel ratio of pixel positions whose Y component values ​​are less than a preset too-dark threshold in the reference frame to the total pixel positions;

[0048] According to a preset correspondence between the first pixel ratio interval and the target exposure parameter, obtaining a first exposure parameter corresponding to the target first pixel ratio interval in which the first pixel ratio is located;

[0049] Selecting, from the at least one overexposed frame to be selected, the overexposed frame having an exposure parameter greater than and / or closest to the first exposure parameter;

[0050] Obtaining a second pixel ratio of pixel positions whose brightness Y component is greater than a preset overbrightness threshold to total pixel positions in the reference frame;

[0051] According to a preset correspondence between a second pixel ratio interval and a target exposure parameter, obtaining a second exposure parameter corresponding to a target second pixel ratio interval in which the second pixel ratio is located;

[0052] Among the at least one weakly exposed frame to be selected, the weakly exposed frame having an exposure parameter smaller than and / or closest to the second exposure parameter is selected.

[0053] In some embodiments, after acquiring the overexposed frame, the reference frame, and the underexposed frame with different exposure parameters through the image sensing module, the method further includes:

[0054] According to a preset image alignment algorithm, the overexposed frame, the reference frame and the underexposed frame are aligned. The alignment includes scaling and / or rotating the overexposed frame and the underexposed frame so that the image content presented at the target pixel positions of the overexposed frame and the underexposed frame is the same as the image content presented at the corresponding pixel positions of the reference frame.

[0055] In some embodiments, the image sensing module includes multiple image sensors, and the overexposed frame, the reference frame, and the underexposed frame are collected by the multiple image sensors at the same time.

[0056] The image processing device provided in the embodiment of the present application includes:

[0057] an image acquisition module, configured to acquire, through the image sensing module, overexposed frames, reference frames, and underexposed frames having different exposure parameters, wherein the exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame, the exposure parameter of the underexposed frame is less than the exposure parameter of the reference frame, and the overexposed frames, the reference frames, and the underexposed frames are image frames in YUV format;

[0058] a weight acquisition module, configured to acquire exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame based on the brightness of each pixel position in the reference frame and a preset brightness threshold, wherein the exposure fusion weights include the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame;

[0059] The image generation module is configured to obtain, based on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image; and generate the fused image based on the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

[0060] The computer device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0061] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0062] The computer program product provided in the embodiments of the present application includes a computer program, which implements the method provided in the embodiments of the present application when executed.

[0063] The image processing method, apparatus, computer device, computer-readable storage medium, and computer program product provided in the embodiments of the present application are applied to an electronic terminal device provided with an image sensing module. The image sensing module is used to obtain a reference frame, an overexposed frame with an exposure parameter greater than the reference frame, and an underexposed frame with an exposure parameter less than the reference frame. The overexposed frame, the reference frame, and the underexposed frame are all image frames in YUV format for subsequent image fusion processing. Then, based on the brightness of each pixel position in the obtained reference frame and a preset brightness threshold, the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame are obtained. The exposure fusion weights include the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame. This eliminates the need to calculate the contrast or saturation differences between the frames, reducing the amount of computation. Thereafter, based on the fusion weights of each pixel position in the obtained overexposed frame, the reference frame, and the underexposed frame, the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image are respectively obtained, and a fused image is generated, thereby improving the image quality and generation efficiency of the generated fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0065] Figure 1 A flowchart illustrating an implementation of the image processing method provided in an embodiment of the present application;

[0066] Figure 2 Schematic diagram of the storage format of YUV color encoding and RGB color encoding;

[0067] Figure 3 A flowchart illustrating an implementation of obtaining exposure fusion weights for overexposed frames, reference frames, and underexposed frames in the image processing method provided in an embodiment of the present application;

[0068] Figure 4 This is a flowchart illustrating an implementation of obtaining the value of the brightness Y component corresponding to each pixel position of the fused image in the image processing method provided in an embodiment of the present application;

[0069] Figure 5 A flowchart illustrating an implementation of obtaining underexposed frames and overexposed frames in the image processing method provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0074] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0075] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0076] When taking photos with camera-equipped electronic devices such as mobile phones and smartwatches, some areas may appear too bright or too dark. This blurs the details in these areas, affecting the overall photo quality and the user's visual experience.

[0077] In order to improve the visual experience of pictures, the dynamic range of pictures is improved by fusing multiple frames of images in related fields. The dynamic range of an image refers to the range of brightness differences between the darkest and brightest parts that can be displayed in a photo or image. The concept of dynamic range is not only related to the contrast between light and dark in an image, but also affects the details of the highlights and shadows in the photo. Furthermore, high dynamic range rendering (HDR) technology has been derived, which can enrich the details of each dark area, make dark areas darker, and enrich more detailed colors, so that movies, pictures, and game screens can all present excellent effects, allowing users to watch movies and play games closer to the visual experience of the real environment.

[0078] In the related technical field, for multiple frames of image frames with different exposure parameters shot continuously, the weight of each frame in the final fused image is calculated at the pixel granularity, and then the final photo is generated according to a specific weight fusion algorithm. The commonly used weight calculation method is to compare each frame with other image frames based on reference conditions such as contrast and / or saturation to obtain the weight of each frame. Although the synthesized fused image in this way integrates the display conditions of each frame, the need to calculate the differences between each frame results in a large amount of calculation, which is less efficient on devices such as smart watches with lower computing power or mobile phones with weaker performance.

[0079] In addition, the fusion weights for each frame are determined by calculating the differences between them. Since the weights are calculated independently for each frame, each region in each frame has a weight. This means that the image quality of the resulting fused image will be different regardless of the display quality of any region in each frame. This may result in significant subjective differences between some areas of the fused image and the normally exposed image. For example, the face area in a photo is fair and rosy with normal exposure, but after merging overexposed, normally exposed, and underexposed content, the face area will appear yellowish.

[0080] In view of this, an embodiment of the present application provides an image processing method, which is applied to an electronic terminal device, which is provided with an image sensing module. Among them, the image sensing module can be a single camera, a camera module composed of multiple cameras, or a highly integrated component including a camera, a sensor and related image processing circuits, which is not limited here. The electronic terminal device can be a mobile phone, a wearable device (such as a smart watch, a smart bracelet, smart glasses, etc.), a tablet computer, a laptop computer, etc. provided with the above-mentioned image sensing module, which is not limited here. The function implemented by the method can be implemented by calling the program code by the processor in the electronic terminal device. Of course, the program code can be saved in a computer storage medium. It can be seen that the electronic terminal device at least includes a processor and a storage medium.

[0081] See also Figure 1 , Figure 1 This is a flowchart of an implementation of the image processing method provided in the embodiment of the present application. Figure 1 As shown, the method may include the following steps 101 to 104:

[0082] Step 101: Obtain overexposed frames, reference frames, and underexposed frames with different exposure parameters through an image sensor module.

[0083] In an embodiment of the present application, overexposed frames, reference frames and underexposed frames with different exposure parameters are obtained through the image sensing module, wherein the exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame, and the exposure parameter of the underexposed frame is less than the exposure parameter of the reference frame, and the overexposed frame, the reference frame and the underexposed frame are image frames in YUV format.

[0084] It should be noted that in this application, the exposure parameter, also known as the exposure value (EV), represents all camera aperture and shutter combinations that can produce the same exposure. For example, exposure parameter 0 (EV0) corresponds to an exposure time of 1 second and an aperture of f / 1.0, or its equivalent. If the exposure value has been determined, then the exposure time and aperture combination can be selected based on it. When the shooting environment is dim and the brightness needs to be increased, but the flash is not effective, the exposure can be compensated by appropriately increasing the exposure. When performing exposure compensation, if the photo is too dark, the EV value base of the camera light meter should be corrected. Each increase of 1.0 in the EV value is equivalent to doubling the amount of light entering. If the photo is too bright, the EV value should be reduced. Each decrease of 1.0 in the EV value is equivalent to halving the amount of light entering.

[0085] In some embodiments, the exposure parameter of the reference frame is EV0. The exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame and is EV+, which may be EV+1, EV+2, etc., without limitation. The exposure parameter of the underexposed frame is less than the exposure parameter of the reference frame and is EV-, which may be EV-1, EV-2, etc., without limitation.

[0086] In this application, an overexposed frame with a higher exposure parameter than the base frame is used to fuse darker areas of the base frame, enhancing dark detail. Furthermore, an underexposed frame with a lower exposure parameter than the base frame enhances image detail in brighter areas of the base frame, thereby improving the quality of the fused image. Furthermore, by introducing the base frame, the fused image appears closer to the normally exposed base frame.

[0087] It is understandable that when using the same device to shoot the same scene, selecting different exposure parameters can produce different imaging effects. By adjusting the exposure parameters, you can control the brightness, contrast and detail of the photo.

[0088] In the embodiment of the present application, the overexposed frame, the reference frame, and the underexposed frame are image frames in YUV format.

[0089] It should be noted that YUV is a common color encoding method widely used in video processing, image processing, and television signal transmission. YUV stands for luminance (Y) and chrominance (U, V), respectively. The Y component represents brightness information, while the U and V components represent chrominance information, namely the hue and saturation of the color. In image processing, using YUV encoding can effectively reduce data bandwidth because the human visual system is much more sensitive to luminance than to chrominance. Therefore, by appropriately reducing the amount of chrominance data, the data processing burden can be reduced while maintaining image quality.

[0090] For easier understanding, see Figure 2 , Figure 2 Schematic diagram of the storage format of YUV color encoding and RGB color encoding.

[0091] It should be noted that RGB is also a common color encoding method. It produces a variety of colors by varying the three color channels of red (R), green (G), and blue (B) and superimposing them. Unlike RGB, where brightness is obtained by superimposing the R, G, and B components, RGB encoding and YUV encoding can be converted to each other using corresponding conversion formulas.

[0092] There are many YUV color encoding storage formats, the most common of which are YUV4:4:4, YUV4:2:2, and YUV4:2:0. In YUV4:4:4 sampling, each Y corresponds to a set of UV components. In YUV4:2:2 sampling, every two Ys share a set of UV components. In YUV4:2:0 sampling, every four Ys share a set of UV components.

[0093] See also Figure 2 (a) is a schematic diagram of the NV21 format for YUV4:2:0 sampling. Figure 2 As shown in (a), every four Y components share a set of U and V components, with Y sorted continuously and U and V sorted alternately. Among them, y1, y2, y5, and y6 share v1 and u1, and y3, y4, y7, and y8 share u2 and v2.

[0094] RGB color encoding storage formats include RGB24, RGB565, and RGB555. In RGB24, each pixel consists of 24 bits, of which the R component (red) occupies 8 bits, the G component (green) occupies 8 bits, and the B component (blue) occupies 8 bits. RGB565 is a 16-bit color encoding scheme. In this format, the R component (red) occupies 5 bits, the G component (green) occupies 6 bits, and the B component (blue) occupies 5 bits. Since green is usually the color most sensitive to the human eye, more bits are allocated to green to provide finer color changes. RGB555 is also a 16-bit color encoding scheme. Unlike RGB565, R, G, and B in RGB555 each occupy 5 bits, and the highest bit is not used.

[0095] See also Figure 2 (b) is a schematic diagram of the RGB24 format. Figure 2 (b) In the RGB24 format, the R component, G component, and B component are arranged in the order of R, G, B, and each component occupies 8 bits.

[0096] It's understandable that a 1280*720 image has 1280*720 pixels. If each pixel's color is displayed using RGB24 encoding, different RGB values ​​will display different colors. The data size is: length * width * 3. If the NV21 format with YUV color encoding is used, the image's 2*2 matrix is ​​extracted with one U and one V per matrix. The data size is: length * width * 3 / 2, which is half the data size of the RGB24 format.

[0097] In some embodiments, overexposed frames, reference frames, and underexposed frames are stored in YUV4:2:2 or YUV4:2:0 sampling format, and the amount of data is reduced compared to RGB. When the above three frames are fused, the amount of computation of the fusion part is reduced, which is more suitable for electronic terminal devices with relatively reduced computing power, thereby improving the generation efficiency of the generated fused image.

[0098] In some embodiments, after acquiring the overexposed frame, the reference frame, and the underexposed frame with different exposure parameters through the image sensing module, the method further includes:

[0099] According to a preset image alignment algorithm, the overexposed frame, the reference frame and the underexposed frame are aligned. The alignment includes scaling and / or rotating the overexposed frame and the underexposed frame so that the image content presented at the target pixel positions of the overexposed frame and the underexposed frame is the same as the image content presented at the corresponding pixel positions of the reference frame.

[0100] It should be noted that in actual applications, the time required for electronic terminal devices to obtain overexposed frames, reference frames and underexposed frames is relatively short, but there may still be displacement between the overexposed frames, reference frames and underexposed frames due to reasons such as user hand shaking. That is, when the three frames overlap, the outlines of some displayed content in the image cannot be aligned.

[0101] The image processing method provided by the present application is to perform pixel-dimensional fusion processing on each frame image. If the deviation of each frame is large, it will affect the imaging effect of the obtained fused image. Therefore, the overexposed frame, the reference frame and the underexposed frame are aligned by a preset image alignment algorithm. With the reference frame as the alignment template, the overexposed frame and the underexposed frame are rotated and / or rotated so that the image content presented at the target pixel position of the overexposed frame and the underexposed frame is the same as the image content presented at the corresponding pixel position of the reference frame, that is, the outlines of the picture content of the three frames coincide after the alignment processing, thereby improving the image quality after subsequent exposure fusion, avoiding the deviation of each frame, resulting in the mismatch of the content presented at the corresponding pixel position during the pixel-dimensional fusion process, and causing the edges of the brighter or darker areas of the fused image to appear disconnected, so that the final fused image is more natural in visual effect.

[0102] In some embodiments, the image sensing module includes multiple image sensors, and the overexposed frame, the reference frame, and the underexposed frame are collected by the multiple image sensors at the same time.

[0103] It should be noted that the image sensing module uses multiple image sensors to simultaneously capture overexposed frames, reference frames, and underexposed frames, ensuring temporal consistency among these three frames, thereby avoiding image content discrepancies caused by time differences. Furthermore, displacement deviations among these three frames can be aligned using the camera intrinsics or extrinsic parameters of the multiple image sensors. This ensures spatial consistency among the fused overexposed, reference, and underexposed frames, improving the quality of the fused image.

[0104] Step 102 : Obtain exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame based on the brightness of each pixel position of the reference frame and a preset brightness threshold.

[0105] In an embodiment of the present application, the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame are obtained based on the brightness of each pixel position in the reference frame and a preset brightness threshold. The exposure fusion weights include the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame.

[0106] It should be noted that since the reference frame, overexposed frame and underexposed frame are all image frames in YUV format, and the Y component in the YUV format is the brightness component, in this application, the value of the Y component corresponding to each pixel position of the reference frame is the brightness of each pixel position. Compared with the RGB format, no additional calculations are required, which improves the image fusion efficiency.

[0107] In addition, in the present application, the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame participating in the image fusion can be obtained based on the brightness of each pixel position of the reference frame and the preset brightness threshold. In the process of obtaining the exposure fusion weights, there is no need to obtain the brightness of the pixel positions in the overexposed frame and the underexposed frame, nor is there any need to compare the saturation or contrast and other parameters of the overexposed frame, the reference frame, and the underexposed frame with each other. The weight calculation amount of the present application is smaller than that of the commonly used methods in this field, which can improve the generation efficiency of the fused image and be implemented on devices with lower computing power. Moreover, by using the reference frame as a reference, the final fused image can be made closer to the reference frame in appearance, and the image quality of the fused image is still guaranteed while reducing the computing power requirements.

[0108] Step 103 : According to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image are respectively obtained.

[0109] In the embodiment of the present application, the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image are respectively obtained according to the exposure fusion weights of the overexposed frame, the reference frame and the underexposed frame.

[0110] It can be understood that after obtaining the fusion weights of each pixel position of the overexposed frame, the reference frame and the underexposed frame, the values ​​of the Y component and UV component of each pixel position of the overexposed frame, the reference frame and the underexposed frame can be combined according to the preset image fusion algorithm to obtain the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image.

[0111] The preset image fusion algorithm can simply perform a weighted summation of the Y and UV components at each pixel position in the overexposed, reference, and underexposed frames to obtain the corresponding Y and UV components of the luminance in the fused image. Alternatively, a more complex algorithm can be employed, such as incorporating a neural network model to segment the image frames and employ different fusion strategies for different regions based on the exposure fusion weights of each frame. This is not a limitation here.

[0112] In the embodiment of the present application, the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image are obtained separately. The method for obtaining the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image can be the same or different, and is not limited here.

[0113] It should be noted that in the YUV color encoding and the corresponding storage format, the Y component is the luminance component providing brightness information, and the UV component is the chrominance component providing color information.

[0114] In some embodiments, obtaining the value of the chroma UY component corresponding to each pixel position of the fused image according to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame includes:

[0115] performing weighted summation processing on the UY components at corresponding pixel positions of the overexposed frame, the reference frame, and the underexposed frame according to a preset weighting formula and the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, to obtain a value of the UY component corresponding to each pixel position of the fused image;

[0116] The preset weighting formula is:

[0117] UV result (i,j)=W(i,j,0)*UV(i,j,0)+W(i,j,1)*UV(i,j,1)+W(i,j,2)*UV(i,j,2),

[0118] Among them, UV result(i, j) is the value of the UV component at the pixel position (i, j) in the fused image, W(i, j, 0), W(i, j1) and W(i, j, 2) are the fusion weights at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively, and UV(i, j, 0), UV(i, j, 1) and UV(i, j, 2) are the values ​​of the UV component at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively.

[0119] In this application, if the image resolution of each frame involved in exposure fusion is 1280*720, the image resolution of the final fused image is also 1280*720. The target pixel position in each frame can be uniquely determined by the pixel position (i, j) and the preset pixel coordinate system, which facilitates subsequent weight acquisition and image fusion.

[0120] It can be understood that the value of the UV component of each pixel position in the fused image can be obtained through the preset weighting formula. The value of the above-mentioned UV component includes the value of the U component and the value of the V component. The acquisition method of the two is the same and will not be repeated here.

[0121] It should be noted that for YUV format images, if the Y, U, and V components are separated, the image will be output as three-channel images: Y, U, and V. The Y channel image represents the grayscale image of the original image. In the grayscale image, the image's light and dark distribution and contour details can be clearly seen. The U and V channel images represent the blue and red components of the image's chromaticity information, and more importantly, they show the color distribution and changes in the image.

[0122] In this application, the overexposed, baseline, and underexposed frames are images of the same scene captured with different exposure parameters. The brightness, or Y component, of each image varies significantly, while the color, or UV component, varies less. Therefore, a simple weighted summation method is used to obtain the UV component values ​​at each pixel position in the fused image. This minimizes the impact on the quality of the fused image, ensuring image quality. Furthermore, the computational complexity is lower than that of complex algorithms such as deep learning, improving the efficiency of fused image generation.

[0123] Step 104 : generating a fused image according to the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

[0124] In the embodiment of the present application, the fused image is generated according to the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

[0125] It should be noted that in the above steps, the values ​​of the Y component and UV component corresponding to each pixel position of the fused image are obtained, which can generate a fused image with better image quality than the normally exposed reference frame.

[0126] By implementing the above technical solution, it is possible to obtain the exposure fusion weights of the reference frame, overexposed frame and underexposed frame according to the brightness of each pixel position in the reference frame in the YUV space and fuse each frame, thereby improving the image quality and generation efficiency of the generated fused image.

[0127] The following will describe a method for obtaining exposure fusion weights of overexposed frames, reference frames, and underexposed frames based on the brightness of each pixel position of the reference frame and a preset brightness threshold in an embodiment of the present application, in conjunction with the drawings in the specification.

[0128] See also Figure 3 , Figure 3 This is a flowchart illustrating an implementation of obtaining the exposure fusion weights of overexposed frames, reference frames, and underexposed frames in the image processing method provided in an embodiment of the present application. Figure 3 As shown, the following steps 301 to 303 may be included:

[0129] Step 301: Obtain an overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold.

[0130] In some embodiments, the preset brightness threshold includes a preset high brightness threshold and a preset low brightness threshold. The exposure fusion weight includes an overexposure fusion weight for an overexposed frame, an underexposure fusion weight for an underexposed frame, and a reference fusion weight for a reference frame.

[0131] In some embodiments, the overexposure fusion weight of the overexposed frame is obtained based on the brightness of each pixel position of the reference frame and the preset low brightness threshold, and the overexposure fusion weight includes the fusion weight of each pixel position of the overexposed frame.

[0132] It should be noted that the preset low-brightness threshold can be derived from expert experience or dynamically adjusted based on specific application scenarios and image characteristics. If the brightness of any pixel position in the reference frame is less than or equal to the preset low-brightness threshold, the brightness of that pixel position is low, and there is a possibility that image details cannot be displayed. Therefore, the reference frame can be adjusted based on the brightness of the corresponding pixel position in the overexposed frame. In other words, a corresponding overexposure fusion weight can be set for the overexposed frame to improve the image quality of the final fused image, enabling it to present more image details in the dark areas of the original reference frame.

[0133] In some embodiments, the low brightness threshold includes a first low brightness threshold and a second low brightness threshold, and the first low brightness threshold is smaller than the second low brightness threshold;

[0134] The acquiring the overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold comprises:

[0135] When the brightness of the target pixel position of the reference frame is less than or equal to the first low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame to be 1;

[0136] When the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining a fusion weight for the corresponding pixel position in the overexposed frame according to a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame;

[0137] When the brightness of the target pixel position in the reference frame is greater than the second low brightness threshold, the fusion weight of the corresponding pixel position in the overexposed frame is determined to be 0.

[0138] It is understood that to achieve better imaging effects, the low brightness threshold is further divided into a first low brightness threshold and a second low brightness threshold, and the first low brightness threshold is smaller than the second low brightness threshold. When the brightness of any target pixel position in the reference frame, that is, the value of the Y component, is smaller than the first low brightness threshold, indicating that the pixel position is too dark to present image details, the fusion weight of the corresponding pixel position in the overexposed frame is set to 1. The corresponding pixel position in the overexposed frame refers to the pixel position with the same pixel position coordinates (i, j) as the target pixel position in the reference frame.

[0139] In the case where the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold. Although the target pixel position in the reference frame is darker than the corresponding pixel position in the overexposed frame, it is greater than the first low brightness threshold. At this time, if the fusion weight of the corresponding pixel position in the overexposed frame is still set to 1, the corresponding pixel position in the fused image may become brighter, and the image details will also not be presented. Therefore, according to the difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame, the fusion weight of the corresponding pixel position in the overexposed frame is dynamically determined to achieve a natural image fusion effect. When the brightness of the target pixel position in the reference frame is greater than the second low brightness threshold, there is no problem of being unable to display details due to being too dark or too dark, and there is no need to adjust through the exposed frame. Therefore, the fusion weight of the corresponding pixel position in the overexposed frame is set to 0.

[0140] In some embodiments, when the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame according to the difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame includes:

[0141] When the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining a fusion weight for the corresponding pixel position in the overexposed frame according to a preset overexposure weight formula and a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame;

[0142] The preset overexposure weight formula is:

[0143]

[0144] Wherein, W(i,j,0) is the fusion weight at the pixel position (i,j) in the overexposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, L1 is the first low brightness threshold, and L2 is the second low brightness threshold.

[0145] It should be noted that the fusion weight of the pixel position area in the reference frame whose brightness is greater than the first low brightness threshold and less than or equal to the second low brightness threshold is obtained through the preset overexposure weight calculation formula. The closer the brightness at the target pixel position (i, j) in the reference frame is to the first low brightness threshold, the closer the fusion weight of the target pixel position is to 1. In other words, the darker the target pixel position, the larger the fusion weight of the corresponding pixel position in the overexposed frame will be, until the fusion weight reaches 1. When the final fused image is merged, the overexposed frame can participate more in the fusion of the target pixel position, achieving the effect of brightening the dark parts and improving image quality. The formula is simple to operate, reducing the amount of calculation, and is suitable for some electronic terminal devices with low computing power.

[0146] Step 302 : Obtain the low-exposure fusion weight of the low-exposure frame according to the brightness of each pixel position of the reference frame and a preset high-brightness threshold.

[0147] In some embodiments, the second high brightness threshold is greater than the second low brightness threshold. When the brightness of the target pixel position in the reference frame is greater than the second low brightness threshold, the fusion weight of the corresponding pixel position in the overexposed frame is set to 0, and the fusion weight of the corresponding pixel position in the underexposed frame needs to be further determined. It should be noted that the fusion weights of each pixel position in the underexposed frame and the overexposed frame can be obtained sequentially or simultaneously according to the actual code implementation, and are not limited here.

[0148] The step of obtaining the low-exposure fusion weight of the low-exposure frame according to the brightness of each pixel position of the reference frame and the preset high-brightness threshold comprises:

[0149] When the brightness of the target pixel position of the reference frame is greater than or equal to the first high brightness threshold, determining the fusion weight of the corresponding pixel position in the weakly exposed frame to be 1;

[0150] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the weakly exposed frame according to a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold;

[0151] When the brightness of the target pixel position in the reference frame is less than the second high brightness threshold, the fusion weight of the corresponding pixel position in the weakly exposed frame is determined to be 0.

[0152] Similar to the method for obtaining the overexposure fusion weight in the overexposed frame, the underexposure fusion weight for the underexposure frame is as follows. If the brightness of the target pixel position in the reference frame is greater than or equal to the first high brightness threshold, it is confirmed that the target pixel position is too bright, and the fusion weight of the corresponding pixel position in the underexposure frame is set to 1. When the brightness of the target pixel position in the reference frame is less than the second high brightness threshold, it is confirmed that the target pixel position does not need to be adjusted in combination with the underexposure frame, and the fusion weight of the corresponding pixel position in the underexposure frame is determined to be 0. When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, the fusion weight of the corresponding pixel position in the underexposure frame is determined based on the difference between the brightness of the target pixel position and the second high brightness threshold to achieve a natural image fusion effect.

[0153] It should be noted that the overly bright or overly dark areas in the image frame are usually composed of multiple pixels and include multiple pixel positions. Except for the edges of shadows or objects, the brightness of each pixel position usually changes gradually. Therefore, dynamically determining the fusion weight can make the brightness changes of the fused image more natural and improve the image quality. In addition, for the overexposed frames, reference frames and weakly exposed frames obtained from the same scene, the brightness of the same pixel position is decreasing. From the perspective of the human eye, the overly dark display area in the reference frame may be displayed normally in the overexposed frame, while the darker display area in the reference frame may be brighter in the overexposed frame. Therefore, it is necessary to select appropriate exposure parameters and set appropriate low brightness thresholds including a first low brightness threshold and a second low brightness threshold, and high brightness thresholds including a first high brightness threshold and a second high brightness threshold.

[0154] In some embodiments, when the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining the fusion weight of the corresponding pixel position in the low-exposure frame according to the difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold includes:

[0155] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the low-exposure frame according to a preset low-exposure weight formula and a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold;

[0156] The preset weak exposure weight formula is:

[0157]

[0158] Wherein, W(i,j,2) is the fusion weight at the pixel position (i,j) in the weakly exposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, H1 is the first high brightness threshold, and H2 is the second high brightness threshold.

[0159] It should be noted that a preset low-exposure weight calculation formula is used to obtain the fusion weights for pixel locations in the reference frame whose brightness is less than a first high-brightness threshold and greater than or equal to a second high-brightness threshold. The closer the brightness at the target pixel position (i, j) in the reference frame is to the first high-brightness threshold, the closer the fusion weight for that target pixel position is to 1. In other words, the brighter the target pixel position, the larger the fusion weight for the corresponding pixel position in the low-exposure frame, until the fusion weight reaches 1. This allows the low-exposure frame to participate more in the fusion of the target pixel position when the final fused image is merged, achieving the effect of darkening bright areas, revealing more image detail, and improving image quality. Furthermore, compared to the method of comparing differences between frames in related technical fields, this low-exposure weight formula is simpler and reduces the amount of computation required, making it suitable for some low-computing electronic terminal devices.

[0160] In the YUV color space, values ​​typically range from 0 to 255. Within this range, 0 represents the darkest black, while 255 represents the brightest white. In some embodiments, based on expert experience or historical evaluation of multiple images, 5 can be selected as the first low brightness threshold, 37 as the second low brightness threshold, 250 as the first high brightness threshold, and 218 as the second high brightness threshold. The thresholds selected here are merely examples and are not intended to be limiting in actual applications.

[0161] Step 303 : Obtain a reference fusion weight of the reference frame according to the overexposure fusion weight of the overexposed frame and the underexposure fusion weight of the underexposed frame.

[0162] According to the overexposure fusion weight of the overexposed frame and the underexposure fusion weight of the underexposed frame, a baseline fusion weight of the reference frame is obtained, wherein the baseline fusion weight includes the fusion weight of each pixel position of the reference frame, and the sum of the fusion weights corresponding to the same pixel position of the overexposed frame, the reference frame and the underexposed frame is 1.

[0163] It can be understood that after obtaining the overexposed fusion weight of the overexposed frame and the underexposed fusion weight of the underexposed frame, the sum of the fusion weights of the reference frame, the overexposed frame, and the underexposed frame is 1 according to the corresponding pixel positions, and the baseline fusion weight of the reference frame can be obtained. The baseline fusion weight includes the fusion weight of each pixel position in the reference frame.

[0164] By implementing the above technical solution, the exposure fusion weights of the overexposed frame, the reference frame and the underexposed frame can be obtained based on the brightness of each pixel position of the reference frame, so as to perform subsequent image fusion processing.

[0165] After obtaining the exposure fusion weights for each frame, it is necessary to obtain the values ​​of the luminance Y component and chrominance UV component corresponding to each pixel position in the fused image to generate the fused image. The above content describes how to obtain the values ​​of the UV component at each pixel position in the fused image through weighted summation. The following describes how to obtain the Y component, with reference to the accompanying drawings.

[0166] See also Figure 4 , Figure 4 This is a flowchart illustrating an implementation of obtaining the value of the brightness Y component corresponding to each pixel position of the fused image in the image processing method provided in the embodiment of the present application. Figure 4 As shown, the following steps 401 to 403 may be included:

[0167] In some embodiments, obtaining a value of a brightness Y component corresponding to each pixel position of the fused image according to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame includes:

[0168] performing Laplacian pyramid decomposition on the overexposed frame, the reference frame, and the underexposed frame in channels corresponding to the Y components to obtain Laplacian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, respectively;

[0169] Performing Gaussian pyramid decomposition on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame to obtain weighted Gaussian pyramid sequences corresponding to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, respectively;

[0170] According to the Laplacian pyramid sequences and weighted Gaussian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, a value of the Y component corresponding to each pixel position of the fused image is obtained.

[0171] In step 401 , Laplacian pyramid decomposition is performed on the overexposed frame, the reference frame, and the underexposed frame in the Y component corresponding channel to obtain Laplacian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, respectively.

[0172] It should be noted that the Laplacian Pyramid and the Gaussian Pyramid are two different types of image pyramids used for multi-scale image processing and feature extraction.

[0173] A Gaussian pyramid involves continuously reducing the size of an image through Gaussian blur filtering and downsampling, thereby containing a set of images at multiple resolutions. Typically, the bottom layer of a Gaussian pyramid contains the original image, and each layer upwards uses Gaussian blur filtering and downsampling to reduce the image size.

[0174] The Laplacian pyramid is constructed by taking the difference between the image at each level in the Gaussian pyramid and the image at the previous level. The Laplacian pyramid is used together with the Gaussian pyramid to obtain a series of high-frequency detail images by performing multiple downsampling and upsampling operations on the image and then subtracting it from the original image. The Laplacian pyramid can be used in image processing for image compression, edge detection, and texture analysis.

[0175] In some embodiments, by performing a Laplace pyramid transform on the Y-component corresponding channels of the overexposed frames, the reference frames, and the underexposed frames, i.e., the corresponding grayscale images, brightness distribution information of each image frame participating in exposure fusion can be obtained, such as the contours and edge information of brighter and darker areas. This facilitates making the edges of brighter or darker areas more natural during the subsequent fusion process according to the exposure fusion weight.

[0176] In some embodiments, the Y component of each pixel position in the fused image is obtained by weighted summing the frames involved in the exposure fusion, which reduces computational complexity and improves the efficiency of generating the fused image. However, unlike the fusion of the chrominance UV components, when fusing the Y component, in areas where the Y component's value jumps, i.e., the boundary between light and dark in the image or areas with large brightness changes, the original brightness differences between the image frames are too large, and the fusion weights of the pixel positions in the corresponding areas of some image frames approach 1, resulting in the fused image showing many black and white spots similar to noise, or black or white linear noise appearing at the boundary between light and dark, affecting the image quality of the fused image.

[0177] In step 402 , Gaussian pyramid decomposition is performed on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame to obtain weighted Gaussian pyramid sequences corresponding to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame.

[0178] In some embodiments, Gaussian pyramid decomposition is performed on the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame to obtain a weighted Gaussian pyramid sequence corresponding to the exposure fusion weights of each frame.

[0179] It's important to note that the Gaussian pyramid can be used for image smoothing, primarily because it employs a Gaussian filter. A Gaussian filter is a linear filter that maintains image smoothness and continuity at all scales. When an image passes through a Gaussian filter, the value of each pixel is calculated as a weighted average of its own value and the values ​​of its neighboring pixels, thereby achieving image smoothing.

[0180] In some embodiments, the exposure fusion weights of the overexposed frames, reference frames, and underexposed frames participating in exposure fusion are smoothed by a Gaussian pyramid decomposition algorithm to ensure that the frames are more naturally fused during the fusion process.

[0181] In step 403 , the value of the Y component corresponding to each pixel position of the fused image is obtained according to the Laplacian pyramid sequence and the weighted Gaussian pyramid sequence corresponding to the overexposed frame, the reference frame, and the underexposed frame.

[0182] In some embodiments, after performing Laplacian pyramid decomposition on the overexposed frame, the reference frame, and the underexposed frame, and performing Gaussian pyramid decomposition on the exposure fusion weights corresponding to the overexposed frame, the reference frame, and the underexposed frame, the Gaussian pyramid is reconstructed according to a preset reconstruction formula. The reconstruction formula is as follows:

[0183]

[0184] Where L represents the Laplacian pyramid, G represents the Gaussian pyramid, R represents the brightness of the fused image at (i, j), k represents the selected image frame, k = 0 represents the overexposed frame, k = 1 represents the reference frame, and k = 2 represents the underexposed frame. (i, j) represents the pixel position (i, j) in the corresponding image, m represents the number of layers of the Laplacian pyramid or weighted Gaussian pyramid image in the pyramid sequence, W represents the fusion weight of the pixel position (i, j), and Y represents the brightness of the pixel position (i, j).

[0185] According to the reconstructed Laplacian pyramid sequence, the value of the Y component of each pixel position of the fused image can be obtained, and then combined with the value of the UV component corresponding to each pixel position of the fused image obtained in the above content to generate a fused image.

[0186] In some embodiments, based on the display of the reference frame, multiple frames of overexposed frames and multiple frames of underexposed frames to be selected are obtained for selection to obtain overexposed frames and underexposed frames for exposure fusion, ensuring that their exposure parameters can more accurately adjust the reference frame, thereby obtaining a more ideal fusion effect.

[0187] See also Figure 5 , Figure 5 This is a flowchart of obtaining underexposed frames and overexposed frames in the image processing method provided in the embodiment of the present application. Figure 5 As shown, the following steps 501 to 502 may be included:

[0188] In step 501 , a reference frame, at least one overexposed frame to be selected with an exposure parameter greater than that of the reference frame and different exposure parameters, and at least one underexposed frame to be selected with an exposure parameter less than that of the reference frame and different exposure parameters are obtained through an image sensor module.

[0189] In some embodiments, the image sensing module is used to obtain the base frame, at least one frame of overexposed frame to be selected whose exposure parameters are greater than the base frame and whose exposure parameters are different, and at least one frame of underexposed frame to be selected whose exposure parameters are less than the base frame and whose exposure parameters are different. The base frame, the at least one frame of overexposed frame to be selected, and the at least one frame of underexposed frame to be selected are image frames in YUV format.

[0190] The image sensor module acquires at least one candidate overexposed frame with an exposure parameter greater than that of the reference frame, and at least one candidate underexposed frame with an exposure parameter less than that of the reference frame. This acquisition method enables selection of overexposed or underexposed frames with appropriate exposure parameters based on the display of the reference frame, thereby improving the quality of the fused image.

[0191] Step 502 : According to the exposure condition of the reference frame, an overexposed frame is selected from at least one overexposed frame to be selected, and an underexposed frame is selected from at least one underexposed frame to be selected.

[0192] In some embodiments, selecting the overexposed frame from the at least one to-be-selected overexposed frame and selecting the underexposed frame from the at least one to-be-selected underexposed frame according to the exposure condition of the reference frame includes:

[0193] Obtaining a first pixel ratio of pixel positions whose Y component values ​​are less than a preset too-dark threshold in the reference frame to the total pixel positions;

[0194] According to a preset correspondence between the first pixel ratio interval and the target exposure parameter, obtaining a first exposure parameter corresponding to the target first pixel ratio interval in which the first pixel ratio is located;

[0195] Selecting, from the at least one overexposed frame to be selected, the overexposed frame having an exposure parameter greater than and / or closest to the first exposure parameter;

[0196] Obtaining a second pixel ratio of pixel positions whose brightness Y component is greater than a preset overbrightness threshold to total pixel positions in the reference frame;

[0197] According to a preset correspondence between a second pixel ratio interval and a target exposure parameter, obtaining a second exposure parameter corresponding to a target second pixel ratio interval in which the second pixel ratio is located;

[0198] Among the at least one weakly exposed frame to be selected, the weakly exposed frame having an exposure parameter smaller than and / or closest to the second exposure parameter is selected.

[0199] It should be noted that by implementing the above method, the candidate overexposed and underexposed frames can be selected based on the ratio of low-brightness pixels and high-brightness pixels in the base frame. For example, if the ratio of low-brightness pixels in the base frame, i.e., the first pixel ratio, is high, it indicates that the base frame is overall dark. Therefore, the corresponding overexposed frame's exposure parameters need to be higher to achieve the effect of brightening the dark areas. Designing a correspondence between the preset first pixel ratio range or the second pixel ratio range and the target exposure parameters based on the above principle can better meet the shooting requirements under different lighting conditions and improve image quality and detail expression.

[0200] It should be understood that, although the steps in the above-mentioned flowcharts are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0201] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the modules included and the units included in each module, and can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0202] Figure 6 A schematic diagram of the structure of the image processing device provided in the embodiment of the present application is shown in FIG. Figure 6As shown, the image processing device 600 includes an image acquisition module 601, a weight acquisition module 602 and an image generation module 603, wherein:

[0203] An image acquisition module 601 is configured to acquire, through the image sensing module, overexposed frames, reference frames, and underexposed frames having different exposure parameters, wherein the exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame, and the exposure parameter of the underexposed frame is less than the exposure parameter of the reference frame, and the overexposed frames, the reference frames, and the underexposed frames are image frames in YUV format;

[0204] A weight acquisition module 602 is configured to acquire exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame based on the brightness of each pixel position in the reference frame and a preset brightness threshold, wherein the exposure fusion weights include the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame;

[0205] The image generation module 603 is used to obtain the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image based on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame; and generate the fused image based on the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

[0206] In some embodiments, the preset brightness threshold includes a preset high brightness threshold and a preset low brightness threshold; the weight acquisition module 602 is further used to obtain the overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold, and the overexposure fusion weight includes the fusion weight of each pixel position of the overexposed frame; obtain the weak exposure fusion weight of the underexposed frame according to the brightness of each pixel position of the reference frame and the preset high brightness threshold, and the weak exposure fusion weight includes the fusion weight of each pixel position of the weak exposure frame; obtain the baseline fusion weight of the reference frame according to the overexposure fusion weight of the overexposed frame and the weak exposure fusion weight of the underexposed frame, and the baseline fusion weight includes the fusion weight of each pixel position of the reference frame, and the sum of the fusion weights corresponding to the same pixel position of the overexposed frame, the reference frame and the underexposed frame is 1.

[0207] In some embodiments, the low brightness threshold includes a first low brightness threshold and a second low brightness threshold, and the first low brightness threshold is less than the second low brightness threshold; the high brightness threshold includes a first high brightness threshold and a second high brightness threshold, and the first high brightness threshold is greater than the second high brightness threshold; the weight acquisition module 602 is also used to determine that the fusion weight of the corresponding pixel position in the overexposed frame is 1 when the brightness of the target pixel position of the reference frame is less than or equal to the first low brightness threshold; when the brightness of the target pixel position of the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, the fusion weight of the corresponding pixel position in the overexposed frame is determined according to the difference between the second low brightness threshold and the brightness of the target pixel position of the reference frame. When the brightness of the target pixel position of the reference frame is greater than the second low brightness threshold, the fusion weight of the corresponding pixel position in the overexposed frame is determined to be 0; the weak exposure fusion weight of the weak exposure frame is obtained according to the brightness of each pixel position of the reference frame and the preset high brightness threshold, including: when the brightness of the target pixel position of the reference frame is greater than or equal to the first high brightness threshold, the fusion weight of the corresponding pixel position in the weak exposure frame is determined to be 1; when the brightness of the target pixel position of the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, the fusion weight of the corresponding pixel position in the weak exposure frame is determined according to the difference between the brightness of the target pixel position of the reference frame and the second high brightness threshold; when the brightness of the target pixel position of the reference frame is less than the second high brightness threshold, the fusion weight of the corresponding pixel position in the weak exposure frame is determined to be 0.

[0208] In some embodiments, the weight acquisition module 602 is further configured to determine, when the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, a fusion weight for the corresponding pixel position in the overexposed frame based on a preset overexposure weight formula and a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame;

[0209] The preset overexposure weight formula is:

[0210]

[0211] Wherein, W(i,j,0) is the fusion weight at the pixel position (i,j) in the overexposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, L1 is the first low brightness threshold, and L2 is the second low brightness threshold;

[0212] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight of the corresponding pixel position in the low-exposure frame according to a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold, including:

[0213] When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the low-exposure frame according to a preset low-exposure weight formula and a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold;

[0214] The preset weak exposure weight formula is:

[0215]

[0216] Wherein, W(i,j,2) is the fusion weight at the pixel position (i,j) in the weakly exposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, H1 is the first high brightness threshold, and H2 is the second high brightness threshold.

[0217] In some embodiments, the image generation module 603 is further configured to perform Laplacian pyramid decomposition on the Y component channels of the overexposed frame, the reference frame, and the underexposed frame to obtain Laplacian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, respectively; perform Gaussian pyramid decomposition on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame to obtain weighted Gaussian pyramid sequences corresponding to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, respectively; and obtain the Y component value corresponding to each pixel position of the fused image based on the Laplacian pyramid sequences and weighted Gaussian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, respectively.

[0218] In some embodiments, the image generation module 603 is further configured to perform weighted summation processing on the UY components at corresponding pixel positions of the overexposed frame, the reference frame, and the underexposed frame according to a preset weighting formula and the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, to obtain a value of the UY component corresponding to each pixel position of the fused image;

[0219] The preset weighting formula is:

[0220] UV result (i,j)=W(i,j,0)*UV(i,j,0)+W(i,j,1)*UV(i,j,1)+W(i,j,2)*UV(i,j,2),

[0221] Among them, UV result (i, j) is the value of the UV component at the pixel position (i, j) in the fused image, W(i, j, 0), W(i, j1) and W(i, j, 2) are the fusion weights at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively, and UV(i, j, 0), UV(i, j, 1) and UV(i, j, 2) are the values ​​of the UV component at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively.

[0222] In some embodiments, the image acquisition module 601 is further used to obtain, through the image sensing module, the base frame, at least one frame of overexposed frame to be selected whose exposure parameters are greater than the base frame and whose exposure parameters are different, and at least one frame of underexposed frame to be selected whose exposure parameters are less than the base frame and whose exposure parameters are different, wherein the base frame, the at least one frame of overexposed frame to be selected and the at least one frame of underexposed frame to be selected are image frames in YUV format; according to the exposure condition of the base frame, the overexposed frame is selected from the at least one frame of overexposed frame to be selected, and the underexposed frame is selected from the at least one frame of underexposed frame to be selected.

[0223] In some embodiments, the image acquisition module 601 is further used to obtain a first pixel ratio of pixel positions in the reference frame whose Y component values ​​are less than a preset over-dark threshold value to the total pixel positions; obtain a first exposure parameter corresponding to the target first pixel ratio interval in which the first pixel ratio is located based on a correspondence between a preset first pixel ratio interval and a target exposure parameter; select the overexposed frame whose exposure parameter is greater than and / or closest to the first exposure parameter from the at least one frame of overexposed frames to be selected; obtain a second pixel ratio of pixel positions in the reference frame whose brightness Y component is greater than a preset over-bright threshold value to the total pixel positions; obtain a second exposure parameter corresponding to the target second pixel ratio interval in which the second pixel ratio is located based on a correspondence between a preset second pixel ratio interval and the target exposure parameter; and select the underexposed frame whose exposure parameter is less than and / or closest to the second exposure parameter from the at least one frame of underexposed frames to be selected.

[0224] In some embodiments, after obtaining the overexposed frames, reference frames and underexposed frames with different exposure parameters through the image sensing module, the image acquisition module 601 is further used to align the overexposed frames, the reference frames and the underexposed frames according to a preset image alignment algorithm. The alignment includes scaling and / or rotating the overexposed frames and the underexposed frames so that the image content presented at the target pixel positions of the overexposed frames and the underexposed frames is the same as the image content presented at the corresponding pixel positions of the reference frame.

[0225] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0226] It should be noted that in the embodiments of this application Figure 6 The division of modules in the image processing device shown is schematic and is only a logical functional division. In actual implementation, other division methods may be used. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. It can also be implemented in the form of a combination of software and hardware.

[0227] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0228] The embodiment of the present application provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0229] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method provided in the above embodiment are implemented.

[0230] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0231] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0232] In one embodiment, the image processing apparatus provided by the present application can be implemented in the form of a computer program. The computer program can be used in Figure 7 The computer device is operated on the computer device shown. The memory of the computer device can store various program modules that constitute the above-mentioned device. The computer program composed of each program module enables the processor to execute the steps of the method of each embodiment of the present application described in this specification.

[0233] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0234] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0235] The term "and / or" in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, object A and / or object B can mean: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0236] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0238] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

[0239] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0240] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0241] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0242] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0243] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0244] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0245] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic terminal device provided with an image sensor module, and the method includes: Acquire, through the image sensing module, an overexposed frame, a reference frame, and a weakly exposed frame having different exposure parameters, wherein the exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame, and the exposure parameter of the weakly exposed frame is less than the exposure parameter of the reference frame, and the overexposed frame, the reference frame, and the weakly exposed frame are image frames in a YUV format; Obtaining exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame based on the brightness of each pixel position in the reference frame and a preset brightness threshold, the exposure fusion weights including the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame; According to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, respectively obtaining the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image; The fused image is generated according to the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

2. The method according to claim 1, characterized in that The preset brightness threshold includes a preset high brightness threshold and a preset low brightness threshold; The step of obtaining exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame according to the brightness of each pixel position of the reference frame and a preset brightness threshold comprises: Obtaining an overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold, the overexposure fusion weight including the fusion weight of each pixel position of the overexposed frame; Obtaining a weak exposure fusion weight of the weak exposure frame according to the brightness of each pixel position of the reference frame and the preset high brightness threshold, wherein the weak exposure fusion weight includes a fusion weight of each pixel position of the weak exposure frame; According to the overexposure fusion weight of the overexposed frame and the underexposure fusion weight of the underexposed frame, a baseline fusion weight of the reference frame is obtained, wherein the baseline fusion weight includes the fusion weight of each pixel position of the reference frame, and the sum of the fusion weights corresponding to the same pixel position of the overexposed frame, the reference frame and the underexposed frame is 1.

3. The method according to claim 2, characterized in that The low brightness threshold includes a first low brightness threshold and a second low brightness threshold, and the first low brightness threshold is smaller than the second low brightness threshold; the high brightness threshold includes a first high brightness threshold and a second high brightness threshold, and the first high brightness threshold is larger than the second high brightness threshold; The acquiring the overexposure fusion weight of the overexposed frame according to the brightness of each pixel position of the reference frame and the preset low brightness threshold comprises: When the brightness of the target pixel position of the reference frame is less than or equal to the first low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame to be 1; When the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining a fusion weight for the corresponding pixel position in the overexposed frame according to a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame; When the brightness of the target pixel position in the reference frame is greater than the second low brightness threshold, determining the fusion weight of the corresponding pixel position in the overexposed frame to be 0; The step of obtaining the low-exposure fusion weight of the low-exposure frame according to the brightness of each pixel position of the reference frame and the preset high-brightness threshold comprises: When the brightness of the target pixel position of the reference frame is greater than or equal to the first high brightness threshold, determining the fusion weight of the corresponding pixel position in the weakly exposed frame to be 1; When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the weakly exposed frame according to a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold; When the brightness of the target pixel position in the reference frame is less than the second high brightness threshold, the fusion weight of the corresponding pixel position in the weakly exposed frame is determined to be 0.

4. The method according to claim 3, characterized in that The method further comprises: determining, when the brightness of the target pixel position of the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, a fusion weight of the corresponding pixel position in the overexposed frame according to a difference between the second low brightness threshold and the brightness of the target pixel position of the reference frame, including: When the brightness of the target pixel position in the reference frame is greater than the first low brightness threshold and less than or equal to the second low brightness threshold, determining a fusion weight for the corresponding pixel position in the overexposed frame according to a preset overexposure weight formula and a difference between the second low brightness threshold and the brightness of the target pixel position in the reference frame; The preset overexposure weight formula is: Wherein, W(i,j,0) is the fusion weight at the pixel position (i,j) in the overexposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, L1 is the first low brightness threshold, and L2 is the second low brightness threshold; When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight of the corresponding pixel position in the low-exposure frame according to a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold, including: When the brightness of the target pixel position in the reference frame is less than the first high brightness threshold and greater than or equal to the second high brightness threshold, determining a fusion weight for the corresponding pixel position in the low-exposure frame according to a preset low-exposure weight formula and a difference between the brightness of the target pixel position in the reference frame and the second high brightness threshold; The preset weak exposure weight formula is: Wherein, W(i,j,2) is the fusion weight at the pixel position (i,j) in the weakly exposed frame, Y(i,j,1) is the brightness at the pixel position (i,j) in the reference frame, H1 is the first high brightness threshold, and H2 is the second high brightness threshold.

5. The method according to claim 1, characterized in that The obtaining, according to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, a value of the brightness Y component corresponding to each pixel position of the fused image includes: performing Laplacian pyramid decomposition on the overexposed frame, the reference frame, and the underexposed frame in channels corresponding to the Y components to obtain Laplacian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, respectively; Performing Gaussian pyramid decomposition on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame to obtain weighted Gaussian pyramid sequences corresponding to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, respectively; According to the Laplacian pyramid sequences and weighted Gaussian pyramid sequences corresponding to the overexposed frame, the reference frame, and the underexposed frame, a value of the Y component corresponding to each pixel position of the fused image is obtained.

6. The method according to claim 1, wherein The obtaining, according to the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, a value of the chromaticity UY component corresponding to each pixel position of the fused image includes: performing weighted summation processing on the UY components at corresponding pixel positions of the overexposed frame, the reference frame, and the underexposed frame according to a preset weighting formula and the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, to obtain a value of the UY component corresponding to each pixel position of the fused image; The preset weighting formula is: UV result (i,j)=W(i,j,0)*UV(i,j,0)+W(i,j,1)*UV(i,j,1)+W(i,j,2)*UV(i,j,2), Among them, UV result (i, j) is the value of the UV component at the pixel position (i, j) in the fused image, W(i, j, 0), W(i, j1) and W(i, j, 2) are the fusion weights at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively, and UV(i, j, 0), UV(i, j, 1) and UV(i, j, 2) are the values ​​of the UV component at the pixel position (i, j) in the overexposed frame, the reference frame and the underexposed frame, respectively.

7. The method according to any one of claims 1 to 6, characterized in that: The method of obtaining overexposed frames, reference frames, and underexposed frames with different exposure parameters through the image sensing module includes: Acquire, by the image sensing module, the reference frame, at least one to-be-selected overexposed frame having an exposure parameter greater than that of the reference frame and different from that of the reference frame, and at least one to-be-selected underexposed frame having an exposure parameter less than that of the reference frame and different from that of the reference frame, wherein the reference frame, the at least one to-be-selected overexposed frame, and the at least one to-be-selected underexposed frame are image frames in a YUV format; According to the exposure condition of the reference frame, the overexposed frame is selected from the at least one overexposed frame to be selected, and the underexposed frame is selected from the at least one underexposed frame to be selected.

8. The method according to claim 7, characterized in that The selecting, according to the exposure condition of the reference frame, the overexposed frame from the at least one frame of overexposed frames to be selected, and the selecting, according to the at least one frame of underexposed frames to be selected, the underexposed frame, comprises: Obtaining a first pixel ratio of pixel positions whose Y component values ​​are less than a preset too-dark threshold in the reference frame to the total pixel positions; According to a preset correspondence between the first pixel ratio interval and the target exposure parameter, obtaining a first exposure parameter corresponding to the target first pixel ratio interval in which the first pixel ratio is located; Selecting, from the at least one overexposed frame to be selected, the overexposed frame having an exposure parameter greater than and / or closest to the first exposure parameter; Obtaining a second pixel ratio of pixel positions whose brightness Y component is greater than a preset overbrightness threshold to total pixel positions in the reference frame; According to a preset correspondence between a second pixel ratio interval and a target exposure parameter, obtaining a second exposure parameter corresponding to a target second pixel ratio interval in which the second pixel ratio is located; Among the at least one weakly exposed frame to be selected, the weakly exposed frame having an exposure parameter smaller than and / or closest to the second exposure parameter is selected.

9. The method according to any one of claims 1 to 6, characterized in that: After acquiring the overexposed frame, the reference frame, and the underexposed frame with different exposure parameters through the image sensing module, the method further includes: According to a preset image alignment algorithm, the overexposed frame, the reference frame and the underexposed frame are aligned. The alignment includes scaling and / or rotating the overexposed frame and the underexposed frame so that the image content presented at the target pixel positions of the overexposed frame and the underexposed frame is the same as the image content presented at the corresponding pixel positions of the reference frame.

10. The method according to any one of claims 1 to 6, characterized in that: The image sensing module includes multiple image sensors, and the overexposed frame, the reference frame, and the underexposed frame are collected by the multiple image sensors at the same time.

11. An image processing device, the device being provided with an image sensor module, characterized in that: include: an image acquisition module, configured to acquire, through the image sensing module, overexposed frames, reference frames, and underexposed frames having different exposure parameters, wherein the exposure parameter of the overexposed frame is greater than the exposure parameter of the reference frame, the exposure parameter of the underexposed frame is less than the exposure parameter of the reference frame, and the overexposed frames, the reference frames, and the underexposed frames are image frames in YUV format; a weight acquisition module, configured to acquire exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame based on the brightness of each pixel position in the reference frame and a preset brightness threshold, wherein the exposure fusion weights include the fusion weights of each pixel position in the overexposed frame, the reference frame, and the underexposed frame; The image generation module is configured to obtain, based on the exposure fusion weights of the overexposed frame, the reference frame, and the underexposed frame, the values ​​of the luminance Y component and the chrominance UV component corresponding to each pixel position of the fused image; and generate the fused image based on the values ​​of the Y component and the UV component corresponding to each pixel position of the fused image.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed, the method according to any one of claims 1 to 10 is implemented.

Images