Image fusion method, device and electronic system

Abstract

The present invention provides an image fusion method, device and electronic system. The method includes: registering and aligning multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the exposure value EV corresponding to the reference brightness image is different from the EV of the underexposed image; determining the effective pixel points in the highlight area based on the color channel parameters corresponding to each pixel point in the highlight area of ​​the underexposed image and the corresponding relationship; determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixel points in the highlight area, and applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposed optimized image; fusing the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene, thereby alleviating the problem of color cast in the highlight area of ​​the fused image in the existing method, ensuring the realistic effect of the fused image, and improving the practical value of the fused image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image fusion method, device and electronic system. Background Art

[0002] With the continuous development of digital imaging technology, high-definition and realistic images are becoming more and more popular. At present, HDR (High Dynamic Range) imaging technology is widely used in consumer electronics, remote sensing, security monitoring and digital television.

[0003] Among them, HDR images are obtained by fusing images of different EVs (Exposure Values) based on the HDR algorithm. Although it can well reproduce the rich brightness levels in real life and produce realistic effects, in the fusion process, due to the direct fusion of images of different EVs, the color of the highlight area in the fused image becomes abnormal, which cannot meet the actual application needs of users. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an image fusion method, device and electronic system to alleviate the above-mentioned problem, so as to ensure that the highlight area of ​​the fused image will not have color cast, thereby improving the practical value of the fused image.

[0005] In a first aspect, an embodiment of the present invention provides an image fusion method, which includes: registering and aligning multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the exposure value EV corresponding to the reference brightness image is different from the EV of the underexposed image; obtaining the correspondence between multiple standard color temperature intervals and color channel parameters, and determining the effective pixels in the highlight area based on the color channel parameters corresponding to each pixel in the highlight area of ​​the underexposed image and the correspondence; determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, and applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposed optimized image; fusing the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene.

[0006] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein the step of determining the valid pixels in the highlight area based on the color channel parameters corresponding to each pixel in the highlight area of ​​the underexposed image and the corresponding relationship includes: determining the highlight area of ​​the underexposed image based on the overexposed area of ​​the reference brightness image; checking the color channel value of each pixel in the highlight area; marking the pixels in the highlight area whose color channel values ​​are greater than or equal to a preset maximum value as invalid pixels; calculating the color channel parameters of the pixels whose color channel values ​​are less than the preset maximum value; and determining the pixels whose color channel parameters fall between the minimum color channel parameter and the maximum color channel parameter in the corresponding relationship as valid pixels in the highlight area.

[0007] In combination with the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein, before the above-mentioned step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, the method also includes: determining a first target standard color temperature range corresponding to the reference brightness image and a second target standard color temperature range corresponding to the highlight area of ​​the underexposed image according to the corresponding relationship; comparing whether the first target standard color temperature range is the same as the second target standard color temperature range; if different, executing the step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area.

[0008] In combination with the second possible implementation of the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the above-mentioned step of determining the first target standard color temperature range corresponding to the baseline brightness image and the second target standard color temperature range corresponding to the highlight area of ​​the underexposed image according to the corresponding relationship includes: determining the first target standard color temperature range corresponding to the baseline brightness image based on the color channel parameters and the corresponding relationship of each pixel point in the baseline brightness image; determining the second target standard color temperature range corresponding to the highlight area based on the color channel parameters and the corresponding relationship of the effective pixel points in the highlight area of ​​the underexposed image.

[0009] In combination with the third possible implementation of the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the step of determining the second target standard color temperature range corresponding to the highlight area based on the color channel parameters and correspondence of the effective pixels in the highlight area of ​​the underexposed image includes: determining the second target standard color temperature range corresponding to the highlight area according to the proportion of the effective pixels in the highlight area corresponding to multiple standard color temperature ranges.

[0010] In combination with the fourth possible implementation of the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein there are multiple highlight areas; the above-mentioned step of determining the second target standard color temperature range corresponding to the highlight area based on the proportion of valid pixels in the highlight area corresponding to multiple standard color temperature ranges includes one of the following: taking the area with the most pixels in the multiple highlight areas as the effective area, and determining the second target standard color temperature range corresponding to the highlight area based on the proportion of valid pixels in the effective area corresponding to multiple standard color temperature ranges; determining the second target standard color temperature range corresponding to each highlight area according to the proportion of valid pixels in each highlight area corresponding to multiple standard color temperature ranges; counting the proportion of valid pixels in all highlight areas corresponding to multiple standard color temperature ranges, and determining the second target standard color temperature range corresponding to all highlight areas according to the counted proportions.

[0011] In combination with the fourth possible implementation of the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the multiple standard color temperature intervals are respectively a high color temperature interval, a medium color temperature interval and a low color temperature interval; the above-mentioned step of determining the second target standard color temperature interval corresponding to the highlight area according to the proportion of the valid pixels in the highlight area corresponding to the multiple standard color temperature intervals includes: counting a first proportion value of the first type of valid pixels in the highlight area belonging to the high color temperature interval, a second proportion value of the second type of valid pixels belonging to the medium color temperature interval, and a third proportion value of the third type of valid pixels belonging to the low color temperature interval; comparing the first proportion value, the second proportion value and the third proportion value; if the first proportion value is the largest and is greater than a preset proportion threshold, determining that the highlight area corresponds to the high color temperature interval; if the first proportion value is less than or equal to the preset proportion threshold, setting the valid pixel corresponding to the smallest proportion value among the first proportion value, the second proportion value and the third proportion value to an invalid pixel point, and determining the second target standard color temperature interval of the highlight area based on the color channel parameters corresponding to the current valid pixels in the highlight area.

[0012] In combination with the first aspect, an embodiment of the present invention provides a seventh possible implementation of the first aspect, wherein the step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the valid pixels in the highlight area includes: calculating the mean of the color channel parameters corresponding to all valid pixels in the highlight area; and determining the calibration parameters of the highlight area based on the calculated mean.

[0013] In combination with the second possible implementation of the first aspect, an embodiment of the present invention provides an eighth possible implementation of the first aspect, wherein the step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area includes: calculating the mean of the color channel parameters corresponding to the effective pixels in the highlight area belonging to the second target standard color temperature range; determining the calibration parameters of the highlight area based on the calculated mean.

[0014] In combination with the first aspect, an embodiment of the present invention provides a ninth possible implementation of the first aspect, wherein there are two underexposed images, the EV corresponding to the first underexposed image is smaller than the EV corresponding to the second underexposed image, and the EVs corresponding to the first underexposed image and the second underexposed image are both greater than 0; the step of fusing the baseline brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene includes: performing high dynamic range HDR fusion processing on the baseline brightness image, the underexposed optimized image corresponding to the first underexposed image, and the underexposed optimized image corresponding to the second underexposed image to obtain a fused image corresponding to the same scene.

[0015] In combination with the first aspect, an embodiment of the present invention provides a tenth possible implementation of the first aspect, wherein the EV corresponding to the above-mentioned reference brightness image is equal to 0.

[0016] In combination with the first aspect, an embodiment of the present invention provides an eleventh possible implementation of the first aspect, wherein the EV corresponding to the baseline brightness image is not equal to 0; the above-mentioned step of fusing the baseline brightness image and the underexposure optimized image to obtain a fused image corresponding to the same scene includes: performing a brightness transformation on the baseline brightness image according to the EV transformation rule corresponding to the baseline brightness image to obtain a baseline optimized image, and fusing the baseline optimized image and the underexposure optimized image to obtain a fused image corresponding to the same scene; or, fusing the baseline brightness image and the underexposure optimized image to obtain a preliminary fused image corresponding to the same scene, and performing a brightness transformation on the preliminary fused image according to the EV transformation rule corresponding to the baseline brightness image to obtain a fused image corresponding to the same scene.

[0017] In combination with the first aspect, an embodiment of the present invention provides a twelfth possible implementation manner of the first aspect, wherein the above-mentioned acquisition of the correspondence between multiple standard color temperature intervals and color channel parameters includes: acquiring photographed images under multiple preset standard color temperature light sources; statistically determining the color channel parameters of the photographed images corresponding to each preset standard color temperature light source; and generating the correspondence between multiple standard color temperature intervals and color channel parameters based on the color channel parameters corresponding to each preset standard color temperature light source.

[0018] In a second aspect, an embodiment of the present invention further provides an image fusion device, which includes: a registration and alignment module, which is used to register and align multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the exposure value EV corresponding to the reference brightness image is different from the EV of the underexposed image; an effective pixel determination module, which is used to obtain the correspondence between multiple standard color temperature intervals and color channel parameters, and determine the effective pixels in the highlight area based on the color channel parameters corresponding to each pixel in the highlight area of ​​the underexposed image and the corresponding relationship; an underexposed optimized image acquisition module, which is used to determine the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, and apply the calibration parameters of the highlight area to calibrate the highlight area to obtain the underexposed optimized image; a fusion processing module, which is used to fuse the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene.

[0019] In a third aspect, an embodiment of the present invention further provides an electronic system, which includes: an image acquisition device, a processing device and a storage device; wherein the image acquisition device is used to obtain multiple images of the same scene; a computer program is stored on the storage device, and the computer program executes the image fusion method of the first aspect when the processed device is running.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image fusion method of the first aspect are executed.

[0021] The embodiments of the present invention bring the following beneficial effects:

[0022] The embodiments of the present invention provide an image fusion method, device and electronic system, which determine the calibration parameters of the highlight area through the effective pixel points in the highlight area of ​​the underexposed image, and apply the calibration parameters to calibrate the highlight area to obtain an underexposed optimized image, and then fuse the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene. Compared with the existing method of directly fusing the reference brightness image and the underexposed image, the method alleviates the problem of color cast in the highlight area of ​​the fused image caused by directly fusing the reference brightness image and the underexposed image in the existing method, thereby ensuring the realistic effect of the fused image and improving the practical value of the fused image.

[0023] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0024] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of the structure of an electronic system provided by an embodiment of the present invention;

[0027] Figure 2 A flowchart of an image fusion method provided by an embodiment of the present invention;

[0028] Figure 3 A flowchart of another image fusion method provided by an embodiment of the present invention;

[0029] Figure 4 A flowchart of another image fusion method provided by an embodiment of the present invention;

[0030] Figure 5 A flow chart of a method for generating a correspondence between a standard color temperature range and a color channel parameter provided by an embodiment of the present invention;

[0031] Figure 6 A schematic diagram of an image fusion method provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of an image fusion device provided by an embodiment of the present invention;

[0033] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In response to the problem of abnormal color in the highlight area of ​​the fused image in the existing HDR algorithm, the embodiments of the present invention provide an image fusion method, device and electronic system, which alleviate the problem of color cast in the highlight area of ​​the fused image in the existing method, ensure the realistic effect of the fused image, and enhance the practical value of the fused image.

[0036] To facilitate understanding of this embodiment, an image fusion method provided by an embodiment of the present invention is first introduced in detail below.

[0037] First, refer to Figure 1 The electronic system shown is a schematic diagram of the structure of the electronic system. The electronic system can be used to implement the image fusion method, device and electronic system of the embodiment of the present invention.

[0038] like Figure 1 The electronic system 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and one or more image acquisition devices 110. These components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structure of the electronic system 100 shown are merely exemplary and not limiting. The electronic system may also have other components and structures as required.

[0039] The processing device 102 can be a server, an intelligent terminal, or a device including a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic system 100 and can also control other components in the electronic system 100 to perform image fusion functions.

[0040] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processing device 102 may run the program instructions to implement the image fusion function in the following embodiment of the present invention (implemented by the processing device). Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0041] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0042] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0043] The image acquisition device 110 can acquire multiple images of the same scene, and store the acquired multiple images in the storage device 104 for use by other components.

[0044] Exemplarily, the devices used to implement the image fusion method, device, and electronic system according to the embodiment of the present invention can be integrated or dispersed, such as integrating the processing device 102, the storage device 104, the input device 106, and the output device 108 into one body, and setting the image acquisition device 110 at a designated position where the image can be acquired. When the devices in the above electronic system are integrated, the electronic system can be implemented as a smart terminal such as a camera, a smart phone, a tablet computer, a computer, a vehicle-mounted terminal, etc.

[0045] The embodiment of the present invention provides an image fusion method, the execution subject of the method is an electronic device, the electronic device may pre-store a plurality of standard color temperature intervals and color channel parameters, respectively, or may obtain a plurality of standard color temperature intervals and color channel parameters from other storage devices. The corresponding relationship can be set according to the actual situation. Among them, the plurality of standard color temperature intervals are respectively a high color temperature interval, a medium color temperature interval and a low color temperature interval, and the color channels include an R channel (i.e., a red channel), a B channel (i.e., a blue channel) and a G channel (i.e., a green channel). In practical applications, the G channel is generally used as the denominator to count the R / G value and the B / G value, or other channel ratios, such as the B channel as the denominator, the R / B value and the G / B value, etc., which can be set according to the actual situation. For ease of understanding, the color channel parameters include the red-green channel ratio (i.e., the R / G value) and the blue-green channel ratio (i.e., the B / G value) as an example. Among them, Table 1 illustrates a corresponding relationship between a standard color temperature interval and a color channel, and each color temperature interval may have a superimposed color temperature interval or may not have a superimposed color temperature interval:

[0046] Table 1

[0047]

[0048] Based on the above corresponding relationship, the flowchart of the image fusion method provided by the embodiment of the present invention is as follows: Figure 2 As shown, the method comprises the following steps:

[0049] Step S202, registering and aligning multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the EV corresponding to the reference brightness image is different from the EV corresponding to the underexposed image.

[0050] Specifically, multiple images of the same scene are acquired through a camera, and the multiple images are registered and aligned so that the frames of each image are aligned; wherein the registration and alignment can be performed by homography transformation, mesh flow algorithm, optical flow algorithm, etc. A reference brightness image and an underexposed image are determined based on the aligned multiple images, wherein the EV of the reference brightness image may be 0 or may not be 0, such as by transforming the brightness of an EV-2 image into the brightness of an EV0 image through array gain, then EV-2 may be used as the reference brightness image; and, the underexposed image may be one or a collection of multiple underexposed images, as long as the EV corresponding to the reference brightness image is different from the EV of the underexposed image, and the number of reference brightness images and underexposed images may be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0051] Step S204, obtaining the correspondence between a plurality of standard color temperature intervals and color channel parameters, and determining the effective pixel points in the highlight area based on the color channel parameters corresponding to each pixel point in the highlight area of ​​the underexposed image and the correspondence relationship.

[0052] In practical applications, the highlight area of ​​each of the above-mentioned underexposed images contains multiple pixels, among which the pixels include overexposed pixels and valid pixels. The overexposed pixels will cause the highlight area to be overexposed, thereby causing serious distortion of the underexposed image. Therefore, it is necessary to determine the valid pixels in the highlight area and discard the overexposed pixels.

[0053] Since each pixel has a corresponding R / G value and a B / G value, it is possible to determine whether the pixel is within the standard color temperature range through the color channel parameters and corresponding relationships of each pixel. If so, the pixel is determined to be a valid pixel in the highlight area, so that the underexposed image can be optimized based on the valid pixels, thereby ensuring the realistic effect of the fused image.

[0054] Step S206 , determining calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, and applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image.

[0055] In practical applications, for underexposed images, a common practice is to restore the color of the underexposed image through the AWB (Automatic whitebalance) algorithm, that is, to correct the pixels of the underexposed image by calculating the AWB correction parameters to obtain a corrected image. Therefore, based on the effective pixels of the highlight area determined above, the calibration parameters corresponding to the highlight area, namely, AWB Gain, are determined, wherein, since the color channel parameters of each pixel include R / G value and B / G value, the calibration parameter AWB Gain of the highlight area includes R Gain corresponding to the R / G value and B Gain corresponding to the B / G value, and each effective pixel in the highlight area is corrected based on the calibration parameters to obtain a corrected underexposed image, namely, an underexposed optimized image.

[0056] In addition, since the calibration parameters of the above-mentioned highlight area are determined based on the effective pixels of the highlight area, compared with the existing method of directly determining the calibration parameters through all the pixels of the highlight area, the accuracy of the calibration parameters is improved, thereby improving the calibration accuracy of the highlight area and ensuring the color restoration of the underexposure optimized image.

[0057] Step S208, fusing the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene.

[0058] Specifically, if there is only one underexposed image, the HDR fusion processing is directly performed on the reference brightness image and the underexposed optimized image corresponding to the underexposed image to obtain a fused image corresponding to the same scene. In addition, if there are two underexposed images, the EV corresponding to the first underexposed image is smaller than the EV corresponding to the second underexposed image, and the EVs corresponding to the first underexposed image and the second underexposed image are both greater than 0; for example, the EV corresponding to the first underexposed image is 2, and the EV corresponding to the second underexposed image is 4, then the HDR fusion processing is performed on the reference brightness image, the underexposed optimized image corresponding to the first underexposed image, and the underexposed optimized image corresponding to the second underexposed image to obtain a fused image corresponding to the same scene.

[0059] And, if the EV corresponding to the baseline brightness image is not equal to 0; at this time, the baseline brightness image is transformed in brightness according to the EV transformation rule corresponding to the baseline brightness image to obtain a baseline optimized image, and the baseline optimized image and the underexposure optimized image are fused to obtain a fused image corresponding to the same scene; or, the baseline brightness image and the underexposure optimized image are fused to obtain a preliminary fused image corresponding to the same scene, and the preliminary fused image is transformed in brightness according to the EV transformation rule corresponding to the baseline brightness image to obtain a fused image corresponding to the same scene. For example, here n EV-2 images are taken as the reference brightness images, and the underexposed images include a first underexposed image corresponding to EV-4 and a second underexposed image corresponding to EV-6. At this time, the reference brightness image can be transformed according to the EV transformation rule corresponding to the reference brightness image, such as EV0=EV-2*4, so that the EV corresponding to the obtained reference optimized image is 0, and HDR fusion processing is performed based on the reference optimized image and the underexposed optimized image to obtain a fused image corresponding to the same scene; or, first, the reference brightness image corresponding to EV-2, the underexposed optimized image corresponding to EV-4, and the underexposed optimized image corresponding to EV-6 are HDR fused to obtain a preliminary fused image, and the preliminary fused image is transformed according to the EV transformation rule corresponding to the reference brightness image to obtain a fused image corresponding to the same scene. It should be noted that the EV transformation rule corresponding to the reference brightness image here can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0060] The image fusion method provided by the embodiment of the present invention determines the calibration parameters of the highlight area through the effective pixel points in the highlight area of ​​the underexposed image, and applies the calibration parameters to calibrate the highlight area to obtain the underexposed optimized image, and then fuses the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene. Compared with the existing method of directly fusing the reference brightness image and the underexposed image, the image fusion method alleviates the problem of color cast in the highlight area of ​​the fused image caused by directly fusing the reference brightness image and the underexposed image in the existing method, thereby ensuring the realistic effect of the fused image and improving the practical value of the fused image.

[0061] exist Figure 2 Based on the above, an embodiment of the present invention further provides another image fusion method, the execution subject is the above electronic device, wherein the electronic device pre-stores a correspondence between multiple standard color temperature intervals and color channel parameters, and the color channel parameters include: a red-green channel ratio and a blue-green channel ratio. The method focuses on describing a process of determining effective pixels in the highlight area based on the color channel parameters corresponding to each pixel in the highlight area of ​​the underexposed image and the correspondence relationship, such as Figure 3 As shown, the method includes the following processes:

[0062] Step S302, registering and aligning multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the EV corresponding to the reference brightness image is different from the EV corresponding to the underexposed image.

[0063] Step S304: determining a highlight area of ​​the underexposed image based on the overexposed area of ​​the reference brightness image.

[0064] Specifically, after the baseline brightness image and the underexposed image are aligned, the overexposed area of ​​the baseline brightness image is generally slightly larger than the highlight area of ​​the underexposed image. Therefore, for the overexposed area of ​​the baseline brightness image, the three channel values ​​of the R channel, B channel and G channel of each pixel in the overexposed area are generally counted. If the three channel values ​​are all saturated values, the area where the pixel is located is considered to be the highlight area, and thus the highlight area of ​​the underexposed image can be determined by judging the channel values ​​of the pixels in the overexposed area.

[0065] Step S306, checking the color channel value of each pixel in the highlight area.

[0066] Step S308: Mark the pixels in the highlight area whose color channel values ​​are greater than or equal to the preset maximum value as invalid pixels.

[0067] Specifically, for the three channel values ​​(i.e., color channel values) of the R channel, B channel, and G channel of each pixel in the highlight area, it is determined whether each color channel value is greater than or equal to a preset maximum value, such as the preset maximum value of 250 (8 bits). If it is greater than or equal to the preset maximum value, the pixel corresponding to the channel value is considered to be an overexposed pixel, i.e., an invalid pixel. In addition, the R / G value and B / G value of each pixel can be calculated based on the three color channel values ​​of the R channel, B channel, and G channel of each pixel in the highlight area. If one of the R / G value or the B / G value is 255 (8 bits) / 1024 (10 bits), the pixel corresponding to the R / G value or the B / G value is also determined to be an invalid pixel. The invalid pixel will cause the highlight area to be overexposed, thereby causing the underexposed image to be severely distorted. Therefore, when counting the pixels in the highlight area, invalid pixels need to be discarded.

[0068] Step S310, calculating the color channel parameters of the pixels whose color channel values ​​are less than a preset maximum value.

[0069] Step S312: Determine the pixel points whose color channel parameters fall between the minimum color channel parameter and the maximum color channel parameter in the corresponding relationship as valid pixel points in the highlight area.

[0070] Specifically, after obtaining the color channel parameters of the pixel whose color channel value is less than the preset maximum value, based on the correspondence between the standard color temperature interval and the color channel parameter, determine whether the color channel parameter of each pixel is between the minimum color channel parameter and the maximum color channel parameter in the correspondence list. If it is, that is, the color channel parameter falls within the color channel parameter range corresponding to the high color temperature interval, the medium color temperature interval, or the low color temperature interval, then the pixel corresponding to the color channel parameter is a valid pixel in the highlight area. If the color channel parameter falls outside the color channel parameter range corresponding to the high color temperature interval, the medium color temperature interval, and the low color temperature interval, then the pixel corresponding to the color channel parameter is still determined to be an invalid pixel. At the same time, for the statistically valid pixel, the color temperature interval corresponding to the valid pixel can also be determined based on the corresponding color channel parameter.

[0071] Step S314, determining calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, and applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image.

[0072] Step S316, fusing the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene.

[0073] The above steps S314 to S316 are described in detail in the above embodiment, and the embodiment of the present invention will not be described in detail here. Therefore, through the above effective pixel determination process, invalid pixels in the highlight area can be discarded, so as to optimize the underexposed image according to the effective pixels, thereby ensuring the realistic effect of the fused image.

[0074] exist Figure 2 Based on the above, an embodiment of the present invention further provides another image fusion method, the execution subject is an electronic device, wherein the electronic device pre-stores a correspondence between multiple standard color temperature intervals and color channel parameters, and the color channel parameters include: a red-green channel ratio and a blue-green channel ratio. The method focuses on how to determine the process of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, such as Figure 4 As shown, the method includes the following processes:

[0075] Step S402, registering and aligning multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the EV corresponding to the reference brightness image is different from the EV corresponding to the underexposed image.

[0076] Step S404 , determining effective pixels in the highlight area based on the color channel parameters and corresponding relationships corresponding to each pixel in the highlight area of ​​the underexposed image.

[0077] The above steps may refer to the aforementioned embodiment, and the embodiment of the present invention will not be described in detail here.

[0078] Step S406 , determining a first target standard color temperature interval corresponding to the reference brightness image and a second target standard color temperature interval corresponding to the highlight area of ​​the underexposed image according to the corresponding relationship.

[0079] Specifically, according to the color channel parameters and corresponding relationship of each pixel in the reference brightness image, the color temperature range of each pixel in the reference brightness image can be determined, and then the first target standard color temperature range corresponding to the reference brightness image can be determined. In particular, the reference brightness image mostly uses the reference brightness image corresponding to EV0. Since the main light source environment in the shooting scene is mostly a low color temperature ambient light, the first target standard color temperature range corresponding to the reference brightness image is mostly a low color temperature range. The remaining color temperature ranges can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0080] At the same time, based on the color channel parameters and corresponding relationship of the effective pixels in the highlight area of ​​the underexposed image, the second target standard color temperature range corresponding to the highlight area can be determined. Since the effective pixels in the highlight area may be in different color temperature ranges, in actual applications, it is also necessary to determine the second target standard color temperature range corresponding to the highlight area based on the proportion of the effective pixels in the highlight area corresponding to multiple standard color temperature ranges.

[0081] Specifically, there may be multiple highlight areas in an underexposed image. In practical applications, valid pixels in the underexposed image can be extracted and the underexposed image can be binarized, where valid pixels are 1 and invalid pixels are 0, thereby forming a mask of the highlight area. The mask is then subjected to certain expansion and corrosion operations, so that it can be found that there are many overexposed areas of 1 in the image after the operation.

[0082] Optionally, the area with the most pixels in multiple highlight areas can be taken as the effective area, and based on the proportion of the effective pixels in the effective area corresponding to the multiple standard color temperature intervals, the second target standard color temperature interval corresponding to the highlight area is determined. For example, if there are three highlight areas in an underexposed image, after counting the effective pixels in each highlight area, the number of effective pixels in highlight area 1 is A1, the number of effective pixels in highlight area 2 is A2, and the number of effective pixels in highlight area 3 is A3, and A1<A2<A3, then highlight area 3 can be taken as the effective area, and based on the proportion of the multiple standard color temperature intervals corresponding to the effective pixels in highlight area 3, the second target standard color temperature interval corresponding to highlight area 3 is determined, and the second target standard color temperature interval corresponding to highlight area 3 is determined as the second target standard color temperature interval corresponding to the highlight area of ​​the underexposed image.

[0083] Optionally, the second target standard color temperature interval corresponding to each highlight area can be determined according to the proportion of multiple standard color temperature intervals corresponding to the effective pixels in each highlight area in the underexposed image, such as determining the second target standard color temperature intervals corresponding to highlight area 1, highlight area 2 and highlight area 3 in the underexposed image according to the proportion of multiple standard color temperature intervals corresponding to the effective pixels in highlight area 1, highlight area 2 and highlight area 3. Alternatively, the proportion of multiple standard color temperature intervals corresponding to the effective pixels in all highlight areas is counted, and the second target standard color temperature intervals corresponding to all highlight areas are determined according to the counted proportions. For example, the proportion of multiple standard color temperature intervals corresponding to all effective pixels in highlight area 1, highlight area 2 and highlight area 3 is counted, and the second target standard color temperature intervals corresponding to highlight area 1, highlight area 2 and highlight area 3 are determined according to the proportions. Specifically, the method for determining the second target standard color temperature interval corresponding to the highlight area can be set according to actual conditions, and the embodiment of the present invention does not limit this description.

[0084] In addition, the standard color temperature range includes a high color temperature range, a medium color temperature range, and a low color temperature range. The above process of determining the second target standard color temperature range corresponding to the highlight area according to the proportion of the effective pixels in the highlight area corresponding to the multiple standard color temperature ranges is as follows:

[0085] (1) counting a first proportion of first-category effective pixels belonging to the high color temperature range, a second proportion of second-category effective pixels belonging to the medium color temperature range, and a third proportion of third-category effective pixels belonging to the low color temperature range in the highlight region;

[0086] (2) comparing the first proportion value, the second proportion value, and the third proportion value;

[0087] (3) If the first proportion value is the largest and is greater than the preset proportion threshold, it is determined that the highlight area corresponds to the high color temperature interval;

[0088] (4) If the first proportion value is less than or equal to the preset proportion threshold value, the valid pixel point corresponding to the minimum proportion value among the first proportion value, the second proportion value and the third proportion value is set as an invalid pixel point, and the second target standard color temperature range of the highlight area is determined based on the color channel parameters corresponding to the current valid pixel point in the highlight area.

[0089] For example, for a highlight area, the preset proportion threshold is 50%. If the first proportion value of the first type of valid pixels belonging to the high color temperature interval is the largest and exceeds 50%, the highlight area is determined to be the high color temperature interval, and the pixels of other color temperature intervals in the highlight area are discarded; if the first proportion value of the first type of valid pixels belonging to the high color temperature interval and the second proportion value of the second type of valid pixels belonging to the medium color temperature interval are both less than 50%, and are both greater than the third proportion value of the third type of valid pixels belonging to the low color temperature interval, the third type of valid pixels are considered to be invalid pixels, and the color channel values ​​of the first type of valid pixels and the second type of valid pixels are calculated to determine the second target standard color temperature interval of the highlight area.

[0090] Step S408, comparing whether the first target standard color temperature interval is the same as the second target standard color temperature interval; if they are different, executing step S410, if they are the same, executing step S412.

[0091] In practical applications, the highlight area will only be color-casted when the calibration parameter AWB Gain of the highlight area is significantly different from the calibration parameter AWB Gain of the reference brightness image. For example, the calibration parameter AWB Gain of the reference brightness image corresponding to EV0 is mainly affected by the dominant light source. Assuming that the main light source environment in the current shooting scene is a low color temperature ambient light, but the highlight area is a high color temperature light, the calibration parameter AWB Gain calculated by the reference brightness image is determined based on the main ambient light. Therefore, the calibration parameter AWB Gain of the dominant light source in the reference brightness image is close to the calibration parameter AWB Gain of the ambient light, and the calibration parameter AWB Gain of the dominant light source is applied to the low color temperature scene, while the highlight area of ​​the reference brightness image is a high color temperature light. Even if the calibration parameter AWB Gain of the dominant light source with a low color temperature is used, it is still a highlight area. Therefore, the highlight area of ​​the reference brightness image corresponding to EV0 is not affected, that is, the dominant light source does not need to be AWB processed.

[0092] The highlight area of ​​the underexposed image corresponding to EV-2 is in an underexposed state. When the calibration parameter AWB Gain calculated based on the low color temperature ambient light is applied to the high color temperature range, since the Blue Gain in the AWB Gain is larger, the high color temperature range of the underexposed image corresponding to EV-2 will be blue. Therefore, it is necessary to determine which standard color temperature range the calibration parameter AWB Gain of the reference brightness image corresponding to EV0 belongs to. Only when the determined color temperature range of the highlight area of ​​the underexposed image is inconsistent with the color temperature range of the reference brightness image, the calibration parameter of the highlight area is determined. Otherwise, the calibration parameter AWB Gain of the reference brightness image can be directly used to calibrate the highlight area. Therefore, it is necessary to compare whether the first target standard color temperature range is the same as the second target standard color temperature range. If they are different, execute step S410; if they are the same, execute step S412.

[0093] Step S410, determining calibration parameters of the highlight area based on color channel parameters corresponding to effective pixels in the highlight area, and applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image.

[0094] Specifically, the mean of the color channel parameters corresponding to all valid pixels in the highlight area can be calculated; and the calibration parameters of the highlight area are determined based on the calculated mean, that is, R Gain=1 / (R / G_mean) and B Gain=1 / (B / G_mean) in the calibration parameters of the highlight area. Alternatively, the mean of the color channel parameters corresponding to the valid pixels in the highlight area belonging to the second target standard color temperature interval is calculated; and the calibration parameters of the highlight area are determined based on the calculated mean. For example, if the second target standard color temperature interval of the highlight area is the high color temperature interval, then R Gain=1 / (R / G_high color temperature mean) and B Gain=1 / (B / G_high color temperature mean) in the calibration parameters of the highlight area are eliminated, thereby eliminating the influence of the small area corresponding to the invalid pixels in the highlight area, and only retaining the area corresponding to the valid pixels with a relatively large weight on the highlight area, thereby improving the accuracy of the calibration parameters, thereby improving the calibration accuracy of the highlight area, and ensuring the color restoration of the underexposure optimized image.

[0095] Step S412, calibrating the highlight area based on the calibration parameters of the reference brightness image to obtain an underexposure optimized image.

[0096] Specifically, the calibration parameters of the reference brightness image are directly used to calibrate the highlight area to obtain an underexposed optimized image. For example, for a dim street light scene, the overall tone of the entire environment is yellow, and the calibration parameter AWBGain of the ambient light is mainly used to compensate for the yellow environment, that is, the calibration parameter AWB Gain of the reference brightness image is mainly used to compensate for the yellow environment; while the calibration parameter calculated by the yellow street light, that is, the calibration parameter AWB Gain of the underexposed image, still needs to compensate for the yellow color temperature environment. Therefore, the two are relatively close. In order to ensure the uniformity of the picture style, the calibration parameter AWB Gain of the reference brightness image can be used as the calibration parameter of the underexposed image, and correction can be performed to obtain an underexposed optimized image.

[0097] Step S414: fusing the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene.

[0098] Therefore, the calibration parameters of the highlight area in the underexposed image are determined by comparing whether the first target standard color temperature interval of the benchmark brightness image is the same as the second target standard color temperature interval of the underexposed image. That is, the calibration parameters of the benchmark brightness image are used as the calibration parameters of the highlight area when they are the same, and the calibration parameters of the highlight area are determined according to the color channel parameters corresponding to the effective pixels in the highlight area when they are different. In this way, the calibration parameters of the highlight area are adaptively selected, which improves the accuracy of the calibration parameters, ensures the color restoration of the underexposed optimized image, and further ensures the realistic effect of the fused image and enhances the practical value of the fused image.

[0099] In another possible embodiment, the embodiment of the present invention further provides a method for generating a corresponding relationship between a standard color temperature range and a color channel parameter, such as Figure 5 As shown, the method comprises the following steps:

[0100] Step S502: acquiring photographic images of the electronic device under a plurality of preset standard color temperature light sources.

[0101] Specifically, a camera is used to take photos under standard color temperature light sources of 7500K, 6500K, 5000K, 4000K, 3000K and 2300K to obtain corresponding photographic images. It should be noted that the standard color temperature light source can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0102] Step S504, statistically determining the color channel parameters of the photographed image corresponding to each preset standard color temperature light source;

[0103] For the photographed images under the above-mentioned multiple preset standard color temperature light sources, the red-green channel ratio, i.e., the R / G value, and the blue-green channel ratio, i.e., the B / G value, of each pixel point under each standard color temperature light source are respectively counted, so as to divide the color temperature intervals of the preset standard color temperature light source according to the corresponding color channel parameters.

[0104] Step S506: Generate corresponding relationships between a plurality of standard color temperature intervals and color channel parameters based on the color channel parameters corresponding to each preset standard color temperature light source.

[0105] Specifically, according to the R / G value and B / G value of each pixel under the above-mentioned standard color temperature light sources, the color temperature range is divided into three color temperature ranges, namely, a high color temperature range of 5000K-7500K, a medium color temperature range of 3000K-5000K and a low color temperature range of 2300K-3000K, and a correspondence between multiple standard color temperature ranges and color channel parameters is established, so as to distinguish the color temperature ranges according to the color channel parameters.

[0106] In addition, the color temperature range can be expanded by setting a reasonable error range, such as setting the error range to ±15%, that is, when the R / G value and B / G value of a certain pixel in the image are within the ±15% range of this color temperature range, the pixel is also considered to be a valid pixel. For example, the R / G value corresponding to D75 (7500K) is 0.3, and the B / G value is 0.6. After error calculation, the color temperature range of D75 is R / G = [0.255, 0.345], B / G = [0.51, 0.69]. When the R / G value and B / G value of a certain pixel fall within the above range, the pixel is considered to be a valid pixel. Since D75 belongs to the high color temperature range, the pixel is a valid pixel in the high color temperature range.

[0107] It should be noted that when a pixel point in a highlight area is within the color temperature range of D75, this pixel point is considered to be a pixel point of D75; when a pixel point is in a transitional color temperature range, such as this pixel point belongs to both the high color temperature range and the medium color temperature range, then this pixel point is included in both the high color temperature range and the medium color temperature range for calculation; and, when a pixel point is at the junction of D75 and D65 (6500K), this pixel point is considered to be a pixel point in the high color temperature range.

[0108] For ease of understanding, an EV value corresponding to the reference brightness image is 0, an EV corresponding to the first underexposed image is 2, and an EV corresponding to the second underexposed image is 4 as an example. In the existing method, the highlight area of ​​the reference brightness image EV0 and the highlight area of ​​the first underexposed image EV2 are first fused, and then the highlight area of ​​the second underexposed image EV4 is fused. Since the calibration parameters of the reference brightness image EV0 are applied to the highlight area of ​​the first underexposed image EV2 and the highlight area of ​​the second underexposed image EV4, and since the R channel component is relatively large and the B channel component is relatively small in the low color temperature light source, in order to achieve normal white balance of the low color temperature light source, more B Gain needs to be compensated, so that the values ​​of the three channels of R channel, G channel and B channel are relatively close; however, the highlight area of ​​the high color temperature is just the opposite, that is, the B channel component is relatively large and the R channel component is relatively small, and theoretically more R Gain needs to be compensated. When the calibration parameter AWB Gain of the low color temperature fused reference brightness image EV0 is applied to the high color temperature area of ​​the highlight area of ​​the first underexposed image EV2 and the highlight area of ​​the second underexposed image EV4, the color of the high color temperature area will be bluish, thereby causing the highlight area in the fused image to be bluish, that is, the color of the highlight area in the fused image is abnormal, which cannot meet the needs of actual applications.

[0109] like Figure 6 As shown in the schematic diagram, the image fusion method provided by the embodiment of the present invention includes the following process:

[0110] (61) First, the camera is calibrated. Specifically, the camera takes photos of the same scene under a 2300K-7500K light source, and the color channel parameters of each image are counted, that is, the R / G value and the B / G value are counted, so as to establish the corresponding relationship between multiple standard color temperature intervals and the color channel parameters; and a reasonable error range of ±15% is set, that is, when the R / G value and the B / G value of each pixel point in the highlight area are within the range of ±15% of the standard color temperature interval, that is, the calibration data range corresponding to the standard color temperature interval, it is considered to be a valid pixel point;

[0111] (62) aligning the reference brightness image EV0, the first underexposed image EV2, and the second underexposed image EV4;

[0112] (63) performing highlight overexposure area detection on the reference brightness image EV0 to determine the highlight area of ​​the reference brightness image EV0;

[0113] (64) determining the highlight region of the first underexposed image EV2 based on the highlight region of the reference brightness image EV0, specifically, extracting the highlight region boundary of the reference brightness image EV0, and determining the highlight region of the first underexposed image EV2 according to the highlight region boundary; and executing steps (641)-(647);

[0114] (641) calculating the R / G value and the B / G value of each pixel in the highlight area of ​​the first underexposed image EV2;

[0115] (642) Determine whether each pixel is an overexposed pixel (i.e., an invalid pixel). If yes, discard the pixel. If no, proceed to step (643).

[0116] (643) Determine whether the R / G value and B / G value of the pixel point are within the calibration data range. If not, discard the pixel point. If yes, determine it as a valid pixel point in the highlight area and execute step (644);

[0117] (644) Calculate the R / G value and B / G value of each effective pixel in the highlight area, and determine the standard color temperature range corresponding to the effective pixel;

[0118] (645) Based on the standard color temperature range corresponding to each effective pixel, determine the standard color temperature range corresponding to the highlight area. For details, please refer to the above embodiment, and the embodiment of the present invention will not be described in detail here.

[0119] (646) determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the valid pixels in the highlight area, that is, calculating R Gain=1 / (R / G_mean), B Gain=1 / (B / G_mean);

[0120] (647) applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image corresponding to the first underexposed image EV2; specifically, multiplying each effective pixel in the highlight area of ​​the first underexposed image EV2 by R Gain and B Gain to perform color restoration on the first underexposed image EV2 to obtain an underexposure optimized image corresponding to the first underexposed image EV2;

[0121] (65) Determine the highlight area of ​​the second underexposed image EV4 based on the highlight area of ​​the first underexposed image EV2, and execute steps (651) to (657). The specific method can refer to step (64). In addition, the highlight area of ​​the second underexposed image EV4 can also be determined based on the highlight area of ​​the reference brightness image EV0. The embodiment of the present invention will not be described in detail here.

[0122] (651) calculating the R / G value and the B / G value of each pixel in the highlight area of ​​the second underexposed image EV4;

[0123] (652) Determine whether each pixel is an overexposed pixel (i.e., an invalid pixel). If yes, discard the pixel. If no, proceed to step (653).

[0124] (653) Determine whether the R / G value and B / G value of the pixel point are within the calibration data range. If not, discard the pixel point. If yes, determine it as a valid pixel point in the highlight area and execute step (654);

[0125] (654) Calculate the R / G value and B / G value of each effective pixel in the highlight area, and determine the standard color temperature range corresponding to the effective pixel;

[0126] (655) determining the standard color temperature interval corresponding to the highlight area based on the standard color temperature interval corresponding to each effective pixel;

[0127] (656) determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, that is, calculating R Gain=1 / (R / G_mean), B Gain=1 / (B / G_mean);

[0128] (657) applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image corresponding to the second underexposed image EV4; specifically, multiplying each effective pixel in the highlight area of ​​the second underexposed image EV4 by R Gain and B Gain to perform color restoration on the second underexposed image EV4 to obtain an underexposure optimized image corresponding to the second underexposed image EV4;

[0129] (66) The reference brightness image EV0, the underexposure optimization image corresponding to the first underexposure image EV2, and the underexposure optimization image corresponding to the second underexposure image EV4 are fused to obtain a fused image corresponding to the same scene.

[0130] Therefore, the above-mentioned image fusion method determines the calibration parameters of the highlight area through the effective pixel points in the highlight area of ​​the underexposed image, and applies the calibration parameters to calibrate the highlight area to obtain the underexposed optimized image, and then fuses the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene. Compared with the existing method of directly fusing the reference brightness image and the underexposed image calibrated with the calibration parameters of the reference brightness image, the problem of color cast in the highlight area of ​​the fused image is alleviated, the realistic effect of the fused image is ensured, and the practical value of the fused image is improved.

[0131] Based on the above embodiments, the present invention further provides an image fusion device, which is applied to electronic devices such as Figure 7 As shown, the device includes a registration and alignment module 71, an effective pixel point determination module 72, an underexposure optimization image acquisition module 73 and a fusion processing module 74 which are connected in sequence, wherein the functions of each module are as follows:

[0132] A registration and alignment module 71 is used to register and align multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the exposure value EV corresponding to the reference brightness image is different from the EV of the underexposed image;

[0133] The effective pixel determination module 72 is used to obtain the corresponding relationship between the plurality of standard color temperature intervals and the color channel parameters, and determine the effective pixel of the highlight area based on the color channel parameter corresponding to each pixel in the highlight area of ​​the underexposed image and the corresponding relationship;

[0134] An underexposure optimized image acquisition module 73 is used to determine the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, and to calibrate the highlight area using the calibration parameters of the highlight area to obtain an underexposure optimized image;

[0135] The fusion processing module 74 is used to perform fusion processing on the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene.

[0136] The image fusion device provided by the embodiment of the present invention determines the calibration parameters of the highlight area through the effective pixel points in the highlight area of ​​the underexposed image, and applies the calibration parameters to calibrate the highlight area to obtain the underexposed optimized image, and then fuses the reference brightness image and the underexposed optimized image to obtain a fused image corresponding to the same scene. Compared with the existing method of directly fusing the reference brightness image and the underexposed image, the image fusion device alleviates the problem of color cast in the highlight area of ​​the fused image caused by directly fusing the reference brightness image and the underexposed image in the existing method, thereby ensuring the realistic effect of the fused image and improving the practical value of the fused image.

[0137] In one possible embodiment, the effective pixel determination module 72 is further used to: determine the highlight area of ​​the underexposed image based on the overexposed area of ​​the reference brightness image; check the color channel value of each pixel in the highlight area; mark the pixels in the highlight area whose color channel values ​​are greater than or equal to a preset maximum value as invalid pixels; calculate the color channel parameters of the pixels whose color channel values ​​are less than the preset maximum value; and determine the pixels whose color channel parameters fall between the minimum color channel parameter and the maximum color channel parameter in the corresponding relationship as valid pixels in the highlight area.

[0138] In another possible embodiment, before the above-mentioned underexposure optimization image acquisition module 73, the device is also used to: determine a first target standard color temperature range corresponding to the reference brightness image and a second target standard color temperature range corresponding to the highlight area of ​​the underexposed image according to a corresponding relationship; compare whether the first target standard color temperature range is the same as the second target standard color temperature range; if different, determine the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area.

[0139] In another possible embodiment, the above-mentioned determining the first target standard color temperature range corresponding to the reference brightness image and the second target standard color temperature range corresponding to the highlight area of ​​the underexposed image according to the correspondence includes: determining the first target standard color temperature range corresponding to the reference brightness image based on the color channel parameters and the correspondence of each pixel in the reference brightness image; and determining the second target standard color temperature range corresponding to the highlight area based on the color channel parameters and the correspondence of the effective pixels in the highlight area of ​​the underexposed image.

[0140] In another possible embodiment, the above-mentioned determination of the second target standard color temperature range corresponding to the highlight area based on the color channel parameters and the corresponding relationship of the effective pixels in the highlight area of ​​the underexposed image includes: determining the second target standard color temperature range corresponding to the highlight area according to the proportion of the effective pixels in the highlight area corresponding to multiple standard color temperature ranges.

[0141] In another possible embodiment, there are multiple highlight areas; the above-mentioned determining the second target standard color temperature range corresponding to the highlight area based on the proportion of valid pixels in the highlight area corresponding to multiple standard color temperature ranges includes one of the following: taking the area with the most pixels in the multiple highlight areas as the effective area, and determining the second target standard color temperature range corresponding to the highlight area based on the proportion of valid pixels in the effective area corresponding to multiple standard color temperature ranges; determining the second target standard color temperature range corresponding to each highlight area according to the proportion of valid pixels in each highlight area corresponding to multiple standard color temperature ranges; counting the proportion of valid pixels in all highlight areas corresponding to multiple standard color temperature ranges, and determining the second target standard color temperature range corresponding to all highlight areas according to the counted proportions.

[0142] In another possible embodiment, the multiple standard color temperature intervals are respectively a high color temperature interval, a medium color temperature interval and a low color temperature interval; the above-mentioned determination of the second target standard color temperature interval corresponding to the highlight area according to the proportion of the valid pixels in the highlight area corresponding to the multiple standard color temperature intervals includes: counting a first proportion of the first type of valid pixels in the highlight area belonging to the high color temperature interval, a second proportion of the second type of valid pixels belonging to the medium color temperature interval, and a third proportion of the third type of valid pixels belonging to the low color temperature interval; comparing the first proportion, the second proportion and the third proportion; if the first proportion is the largest and is greater than a preset proportion threshold, determining that the highlight area corresponds to the high color temperature interval; if the first proportion is less than or equal to the preset proportion threshold, setting the valid pixel corresponding to the smallest proportion among the first proportion, the second proportion and the third proportion to an invalid pixel, and determining the second target standard color temperature interval of the highlight area based on the color channel parameters corresponding to the current valid pixels in the highlight area.

[0143] In another possible embodiment, the underexposure optimization image acquisition module 73 is further used to: calculate the mean value of the color channel parameters corresponding to all valid pixels in the highlight area; and determine the calibration parameters of the highlight area based on the calculated mean value.

[0144] In another possible embodiment, the underexposure optimization image acquisition module 73 is further used to: calculate the mean of the color channel parameters corresponding to the effective pixel points in the highlight area belonging to the second target standard color temperature range; and determine the calibration parameters of the highlight area based on the calculated mean.

[0145] In another possible embodiment, there are two underexposed images, the EV corresponding to the first underexposed image is smaller than the EV corresponding to the second underexposed image, and the EVs corresponding to the first underexposed image and the second underexposed image are both greater than 0; the fusion processing module 74 is also used to perform high dynamic range HDR fusion processing on the reference brightness image, the underexposed optimized image corresponding to the first underexposed image, and the underexposed optimized image corresponding to the second underexposed image to obtain a fused image corresponding to the same scene.

[0146] In another possible embodiment, the EV corresponding to the above-mentioned reference brightness image is equal to 0.

[0147] In another possible embodiment, the EV corresponding to the baseline brightness image is not equal to 0; the above-mentioned fusion processing module 74 is also used to: perform brightness transformation on the baseline brightness image according to the EV transformation rule corresponding to the baseline brightness image to obtain a baseline optimized image, fuse the baseline optimized image and the underexposure optimized image to obtain a fused image corresponding to the same scene; or, fuse the baseline brightness image and the underexposure optimized image to obtain a preliminary fused image corresponding to the same scene, perform brightness transformation on the preliminary fused image according to the EV transformation rule corresponding to the baseline brightness image to obtain a fused image corresponding to the same scene.

[0148] In another possible embodiment, the device is also used to: obtain photographic images under multiple preset standard color temperature light sources; statistically determine the color channel parameters of the photographic images corresponding to each preset standard color temperature light source; and generate a correspondence between multiple standard color temperature intervals and the color channel parameters based on the color channel parameters corresponding to each preset standard color temperature light source.

[0149] The image fusion device provided in the embodiment of the present invention has the same technical features as the image fusion method provided in the above embodiment, and can therefore solve the same technical problems and achieve the same technical effects.

[0150] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image fusion method.

[0151] See also Figure 8 As shown, the electronic device includes a processor 80 and a memory 81. The memory 81 stores machine executable instructions that can be executed by the processor 80. The processor 80 executes the machine executable instructions to implement the above-mentioned image fusion method.

[0152] Further, Figure 8 The electronic device shown further includes a bus 82 and a communication interface 83 , and the processor 80 , the communication interface 83 and the memory 81 are connected via the bus 82 .

[0153] Among them, the memory 81 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 83 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 82 can be an ISA (Industrial Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Enhanced Industry Standard Architecture) bus, etc. The above-mentioned bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] The processor 80 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 80. The above processor 80 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 81, and the processor 80 reads the information in the memory 81 and completes the steps of the method of the above embodiment in combination with its hardware.

[0155] This embodiment further provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image fusion method.

[0156] The computer program products of the image fusion method, device, and electronic system provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0158] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0159] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0160] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0161] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image fusion method, characterized in that: The method comprises: Performing registration and alignment on multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the exposure value EV corresponding to the reference brightness image is different from the EV of the underexposed image; Obtaining corresponding relationships between a plurality of standard color temperature intervals and color channel parameters, and determining whether each pixel in the highlight area is within the standard color temperature interval based on the color channel parameter corresponding to each pixel in the highlight area of ​​the underexposed image and the corresponding relationship, so as to determine the pixel in the standard color temperature interval as a valid pixel in the highlight area; Calculate the mean value of the color channel parameters corresponding to all valid pixels in the highlight area; Determining calibration parameters of the highlight area based on the calculated mean value, and applying the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image; The reference brightness image and the underexposure optimization image are fused to obtain a fused image corresponding to the same scene.

2. The image fusion method according to claim 1, characterized in that: The step of determining effective pixels in the highlight area based on the color channel parameters corresponding to each pixel in the highlight area of ​​the underexposed image and the corresponding relationship comprises: Determine a highlight area of ​​the underexposed image based on an overexposed area of ​​the reference brightness image; Checking the color channel value of each pixel in the highlight area; Marking pixels in the highlight area whose color channel values ​​are greater than or equal to a preset maximum value as invalid pixels; Calculate the color channel parameters of the pixels whose color channel values ​​are less than the preset maximum value; Pixels whose color channel parameters fall between the minimum color channel parameter and the maximum color channel parameter in the corresponding relationship are determined as valid pixels in the highlight area.

3. The image fusion method according to claim 1, characterized in that: Before the step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area, the method further includes: Determine, according to the corresponding relationship, a first target standard color temperature interval corresponding to the reference brightness image and a second target standard color temperature interval corresponding to the highlight area of ​​the underexposed image; comparing whether the first target standard color temperature range is the same as the second target standard color temperature range; If they are different, the step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the valid pixels in the highlight area is performed.

4. The image fusion method according to claim 3, characterized in that: The step of determining a first target standard color temperature interval corresponding to the reference brightness image and a second target standard color temperature interval corresponding to the highlight area of ​​the underexposed image according to the corresponding relationship comprises: Determining a first target standard color temperature range corresponding to the reference brightness image based on the color channel parameters of each pixel in the reference brightness image and the corresponding relationship; Based on the color channel parameters of the effective pixels in the highlight area of ​​the underexposed image and the corresponding relationship, a second target standard color temperature range corresponding to the highlight area is determined.

5. The image fusion method according to claim 4, characterized in that: The step of determining a second target standard color temperature range corresponding to the highlight area based on the color channel parameters of the effective pixels in the highlight area of ​​the underexposed image and the corresponding relationship comprises: According to the proportion of the valid pixels in the highlight area corresponding to the plurality of standard color temperature intervals, a second target standard color temperature interval corresponding to the highlight area is determined.

6. The image fusion method according to claim 5, characterized in that: There are multiple highlight areas; and according to the proportion of valid pixels in the highlight area corresponding to the multiple standard color temperature intervals, the step of determining the second target standard color temperature interval corresponding to the highlight area comprises one of the following: Taking the area with the most pixels in the multiple highlight areas as a valid area, and determining a second target standard color temperature interval corresponding to the highlight area based on the proportion of the valid pixels in the valid area corresponding to the multiple standard color temperature intervals; Determine the second target standard color temperature interval corresponding to each of the highlight areas according to the proportion of the effective pixels in each of the highlight areas corresponding to the multiple standard color temperature intervals; The proportions of the valid pixels in all the highlight areas corresponding to the plurality of the standard color temperature intervals are counted, and the second target standard color temperature interval corresponding to all the highlight areas is determined according to the counted proportions.

7. The image fusion method according to claim 5, characterized in that: The multiple standard color temperature intervals are respectively a high color temperature interval, a medium color temperature interval and a low color temperature interval; and according to the proportion of the effective pixels in the highlight area corresponding to the multiple standard color temperature intervals, the step of determining the second target standard color temperature interval corresponding to the highlight area comprises: Counting a first proportion of first-category effective pixels belonging to a high color temperature range, a second proportion of second-category effective pixels belonging to a medium color temperature range, and a third proportion of third-category effective pixels belonging to a low color temperature range in the highlight area; Comparing the first proportion value, the second proportion value and the third proportion value; If the first proportion value is the largest and is greater than a preset proportion threshold, it is determined that the highlight area corresponds to the high color temperature interval; If the first proportion value is less than or equal to the preset proportion threshold, the valid pixel corresponding to the minimum proportion value among the first proportion value, the second proportion value and the third proportion value is set as an invalid pixel, and the second target standard color temperature range of the highlight area is determined based on the color channel parameters corresponding to the current valid pixel in the highlight area.

8. The image fusion method according to claim 3, characterized in that: The step of determining the calibration parameters of the highlight area based on the color channel parameters corresponding to the effective pixels in the highlight area comprises: Calculating the mean value of the color channel parameters corresponding to the valid pixel points in the highlight area belonging to the second target standard color temperature range; A calibration parameter of the highlight area is determined based on the calculated mean value.

9. The image fusion method according to any one of claims 1 to 8, characterized in that: There are two underexposed images, the EV corresponding to the first underexposed image is smaller than the EV corresponding to the second underexposed image, and the EVs corresponding to the first underexposed image and the second underexposed image are both greater than 0; The step of fusing the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene comprises: A high dynamic range (HDR) fusion process is performed on the reference brightness image, the underexposure optimization image corresponding to the first underexposed image, and the underexposure optimization image corresponding to the second underexposed image to obtain a fused image corresponding to the same scene.

10. The image fusion method according to claim 1, characterized in that: The EV corresponding to the reference brightness image is equal to 0.

11. The image fusion method according to claim 1, characterized in that: The EV corresponding to the reference brightness image is not equal to 0; The step of fusing the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene comprises: According to the EV transformation rule corresponding to the reference brightness image, the reference brightness image is transformed to obtain a reference optimized image, and the reference optimized image is fused with the underexposure optimized image to obtain a fused image corresponding to the same scene; or, The reference brightness image and the underexposure optimized image are fused to obtain a preliminary fused image corresponding to the same scene, and the preliminary fused image is subjected to brightness transformation according to an EV transformation rule corresponding to the reference brightness image to obtain a fused image corresponding to the same scene.

12. The image fusion method according to claim 1, characterized in that: The obtaining of the corresponding relationships between the plurality of standard color temperature intervals and the color channel parameters includes: Acquire photographic images under multiple preset standard color temperature light sources; Statistically determining the color channel parameters of the photographed image corresponding to each of the preset standard color temperature light sources; Based on the color channel parameters corresponding to each of the preset standard color temperature light sources, a corresponding relationship between a plurality of standard color temperature intervals and the color channel parameters is generated.

13. An image fusion device, characterized in that: The device comprises: A registration and alignment module, used to register and align multiple images of the same scene to obtain a spatially aligned reference brightness image and at least one underexposed image; wherein the exposure value EV corresponding to the reference brightness image is different from the EV of the underexposed image; an effective pixel determination module, configured to obtain a correspondence between a plurality of standard color temperature intervals and color channel parameters, and determine whether each pixel in the highlight area is within the standard color temperature interval based on the color channel parameter corresponding to each pixel in the highlight area of ​​the underexposed image and the correspondence, so as to determine the pixel in the standard color temperature interval as an effective pixel in the highlight area; An underexposure optimized image acquisition module is used to: calculate the mean value of the color channel parameters corresponding to all valid pixels in the highlight area; determine the calibration parameters of the highlight area based on the calculated mean value, and apply the calibration parameters of the highlight area to calibrate the highlight area to obtain an underexposure optimized image; The fusion processing module is used to perform fusion processing on the reference brightness image and the underexposure optimization image to obtain a fused image corresponding to the same scene.

14. An electronic system, characterized in that: The electronic system includes: an image acquisition device, a processing device and a storage device; The image acquisition device is used to acquire multiple images of the same scene; The storage device stores a computer program, and when the computer program is executed by the processing device, the image fusion method according to any one of claims 1 to 12 is executed.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 12 are executed.

Images