Image fusion method and apparatus thereof

Abstract

The application provides an image fusion method and device, the method comprising: converting a visible light image collected by an image collection device into an RGB image, and converting a near-infrared image collected by the image collection device into a first luminance image; converting the RGB luminance image into a second luminance image, and performing near-infrared image fusion weight calculation according to the first luminance image to obtain a fusion weight map of the near-infrared image; performing luminance fusion on the first luminance image and the second luminance image according to the fusion weight map to obtain a luminance fusion image; and performing RGB fusion according to the second luminance image, the luminance fusion image and the RGB image to obtain a fused RGB image. The method can improve the luminance of an image in a low-illumination environment, maintain the color information of the image and improve the quality of the fused image.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image fusion method and apparatus thereof. Background Technology

[0002] In video surveillance, when the ambient light level is low (such as at night), high-energy-density visible light supplementary lighting is often required to obtain good image quality. However, this supplementary lighting can be very irritating to the human eye and can easily create temporary blind spots for pedestrians and drivers, leading to serious light pollution and even traffic accidents.

[0003] The human eye has weak or no perception of infrared light, while the imaging system (including lenses and sensors) in surveillance equipment has excellent imaging capabilities for near-infrared light. Therefore, using infrared illumination and imaging can solve the light pollution problem. However, infrared images suffer from a lack of color and poor detail. Summary of the Invention

[0004] In view of this, this application provides an image fusion method and apparatus.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of the embodiments of this application, an image fusion method is provided, applied to an image acquisition device, the method comprising:

[0007] The visible light image acquired by the image acquisition device is converted into a red-green-blue RGB image, and the near-infrared image acquired by the image acquisition device is converted into a first brightness image;

[0008] The RGB brightness image is converted into a second brightness image, and near-infrared image fusion weights are calculated based on the first brightness image to obtain a fusion weight map of the near-infrared image.

[0009] The first brightness image and the second brightness image are fused according to the fusion weight map to obtain a fused brightness image;

[0010] RGB fusion is performed based on the second brightness image, the brightness fusion image, and the RGB image to obtain the fused RGB image.

[0011] Optionally, before performing brightness fusion on the first brightness image and the second brightness image according to the fusion weight, the method further includes:

[0012] Perform detail calculations on the first brightness image to obtain a first detail image, and perform detail calculations on the second brightness image to obtain a second detail image;

[0013] The step of performing brightness fusion on the first brightness image and the second brightness image according to the fusion weight includes:

[0014] The first luminance image and the second luminance image are fused according to the fusion weight, the first detail image, and the second detail image.

[0015] Optionally, performing detail calculations on the first brightness image to obtain a first detail image includes:

[0016] The first brightness image is subjected to mean filtering to obtain a first mean image;

[0017] The difference between the first brightness image and the first mean image is calculated to obtain the first difference image;

[0018] A cutoff operation is performed on the first difference image to obtain the first detail image.

[0019] Optionally, performing detail calculations on the second brightness image to obtain a second detail image includes:

[0020] The second brightness image is subjected to mean filtering to obtain a second mean image;

[0021] The difference between the second brightness image and the second mean image is calculated to obtain the second difference image;

[0022] A cutoff operation is performed on the second difference image to obtain the second detail image.

[0023] Optionally, the cutoff operation on the difference image can be achieved using the following formula:

[0024] Detail p = CLIP(Diff p *str / 128, deNirMin, deNirMax)

[0025] Among them, Detail p To capture the brightness detail of pixel P in a detailed image, Diff p Here, P is the value of pixel P in the difference image, str is the detail intensity control parameter, [deNirMin, deNirMax] is the cutoff interval, and CLIP() is the cutoff calculation.

[0026] Optionally, the step of performing brightness fusion on the first brightness image and the second brightness image based on the fusion weight, the first detail image, and the second detail image includes:

[0027] For any pixel location, the blended brightness Y of that pixel is determined by the following formula:

[0028] Y = CLIP(((Y nir *wt+Y LL *(256-wt)) / 256+Detail nir +Detail LL ), 0, 255)

[0029] Among them, Y nir Y represents the brightness value of the pixel at that location in the first brightness image, wt represents the fusion weight of the pixel at that location in the fusion weight image, and Y represents the fusion weight of the pixel at that location. LL The brightness value of the pixel at that location in the second brightness image, Detail nir For the brightness detail of the pixel at that location in the first detail image, Detail LL This refers to the brightness detail of the pixel at that location in the second detail image.

[0030] Optionally, the near-infrared image fusion weight calculation based on the first brightness image includes:

[0031] For any pixel in the first brightness image, a preset brightness mapping model is queried based on the brightness value of the pixel to determine the fusion weight corresponding to the brightness value; wherein, the brightness mapping model records the correspondence between brightness values ​​and fusion weights.

[0032] Optionally, the step of performing RGB fusion based on the second luminance image, the luminance fusion image, and the RGB image to obtain the fused RGB image includes:

[0033] For any pixel location, the R, G, and B channel values ​​V of that pixel are determined using the following formula. out :

[0034] When Y LL When V > 0, out = CLIP(V in *Y / Y LL (0, 255)

[0035] When Y LL When V = 0, out = CLIP(Y, 0, 255)

[0036] Where Vin represents the R, G, and B channel values ​​of the pixel at that location in the RGB image, and Y... LL Y is the brightness value of the pixel at that position in the second brightness image, and Y is the brightness value of the pixel at that position in the brightness fusion image.

[0037] According to a second aspect of the embodiments of this application, an image fusion apparatus is provided, applied to an image acquisition device, the apparatus comprising:

[0038] The first visible light processing unit is used to convert the visible light image acquired by the image acquisition device into a red-green-blue RGB image;

[0039] The first infrared processing unit is used to convert the near-infrared image acquired by the image acquisition device into a first brightness image;

[0040] The second visible light processing unit is used to convert the RGB brightness image into a second brightness image;

[0041] The second infrared processing unit is used to calculate the near-infrared image fusion weight based on the first brightness image to obtain the fusion weight map of the near-infrared image.

[0042] A brightness fusion unit is used to perform brightness fusion on the first brightness image and the second brightness image according to the fusion weight map to obtain a brightness fused image;

[0043] The RGB fusion unit is used to perform RGB fusion based on the second luminance image, the luminance fusion image, and the RGB image to obtain a fused RGB image.

[0044] Optionally, the second visible light processing unit is further configured to perform detail calculations on the first brightness image to obtain a first detail image;

[0045] The second infrared processing unit is further configured to perform detail calculations on the second brightness map to obtain a second detail image;

[0046] The brightness fusion unit is specifically used to perform brightness fusion on the first brightness image and the second brightness image according to the fusion weight, the first detail image, and the second detail image.

[0047] Optionally, the second visible light processing unit is specifically configured to perform mean filtering on the first brightness image to obtain a first mean image; calculate the difference between the first brightness image and the first mean image to obtain a first difference image; and perform a cutoff operation on the first difference image to obtain the first detail image.

[0048] Optionally, the second infrared processing unit is specifically used to perform mean filtering on the second brightness image to obtain a second mean image; to calculate the difference between the second brightness image and the second mean image to obtain a second difference image; and to perform a cutoff operation on the second difference image to obtain the second detail image.

[0049] Optionally, the cutoff operation on the difference image can be achieved using the following formula:

[0050] Detail p = CLIP(Diff p *str / 128, deNirMin, deNirMax)

[0051] Among them, Detail p To capture the brightness detail of pixel P in a detailed image, Diff p Here, P is the value of pixel P in the difference image, str is the detail intensity control parameter, [deNirMin, deNirMax] is the cutoff interval, and CLIP() is the cutoff calculation.

[0052] Optionally, the brightness fusion unit is specifically used to determine the fused brightness Y of a pixel at any given location using the following formula:

[0053] Y = CLIP(((Y nir *wt+Y LL *(256-wt)) / 256+Detail nir +Detail LL ), 0, 255)

[0054] Among them, Y nir Y represents the brightness value of the pixel at that location in the first brightness image, wt represents the fusion weight of the pixel at that location in the fusion weight image, and Y represents the fusion weight of the pixel at that location. LL The brightness value of the pixel at that location in the second brightness image, Detail nir For the brightness detail of the pixel at that location in the first detail image, Detail LL This refers to the brightness detail of the pixel at that location in the second detail image.

[0055] Optionally, the second infrared processing unit is specifically used to query a preset brightness mapping model based on the brightness value of any pixel in the first brightness image to determine the fusion weight corresponding to the brightness value; wherein, the brightness mapping model records the correspondence between brightness values ​​and fusion weights.

[0056] Optionally, the RGB fusion unit is specifically used to determine the R, G, and B channel values ​​V of a pixel at any given location using the following formula. out :

[0057] When Y LL When V > 0, out = CLIP(V in *Y / Y LL (0, 255)

[0058] When Y LL When V = 0, out = CLIP(Y, 0, 255)

[0059] Where Vin represents the R, G, and B channel values ​​of the pixel at that location in the RGB image, and Y... LL Y is the brightness value of the pixel at that position in the second brightness image, and Y is the brightness value of the pixel at that position in the brightness fusion image.

[0060] According to a third aspect of the embodiments of this application, an image fusion apparatus is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being prompted by the machine-executable instructions to:

[0061] The visible light image acquired by the image acquisition device is converted into a red-green-blue RGB image, and the near-infrared image acquired by the image acquisition device is converted into a first brightness image;

[0062] The RGB brightness image is converted into a second brightness image, and near-infrared image fusion weights are calculated based on the first brightness image to obtain a fusion weight map of the near-infrared image.

[0063] The first brightness image and the second brightness image are fused according to the fusion weight map to obtain a fused brightness image;

[0064] RGB fusion is performed based on the second brightness image, the brightness fusion image, and the RGB image to obtain the fused RGB image.

[0065] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to:

[0066] The visible light image acquired by the image acquisition device is converted into a red-green-blue RGB image, and the near-infrared image acquired by the image acquisition device is converted into a first brightness image;

[0067] The RGB brightness image is converted into a second brightness image, and near-infrared image fusion weights are calculated based on the first brightness image to obtain a fusion weight map of the near-infrared image.

[0068] The first brightness image and the second brightness image are fused according to the fusion weight map to obtain a fused brightness image;

[0069] RGB fusion is performed based on the second brightness image, the brightness fusion image, and the RGB image to obtain the fused RGB image.

[0070] The image fusion method of this application embodiment converts a visible light image acquired by an image acquisition device into an RGB image, and converts a near-infrared image acquired by the image acquisition device into a first brightness image. Then, it converts the RGB brightness image into a second brightness image, and calculates the near-infrared image fusion weight based on the first brightness image to obtain a near-infrared image fusion weight map. Then, it performs brightness fusion on the first brightness image and the second brightness image based on the fusion weight map to obtain a brightness fusion image. Finally, it performs RGB fusion based on the second brightness image, the brightness fusion image, and the RGB image to obtain a fused RGB image. This method achieves the fusion of visible light images and near-infrared images, improves the brightness of the image in low-light environments while maintaining the color information of the image, and improves the quality of the fused image. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application of an image fusion method;

[0072] Figure 2 This is a schematic diagram illustrating a brightness mapping model according to an exemplary embodiment of this application;

[0073] Figure 3 This is a flowchart illustrating an image fusion method in yet another exemplary embodiment of this application;

[0074] Figure 4 This is a schematic diagram illustrating a detailed calculation process according to an exemplary embodiment of this application;

[0075] Figure 5 This is a schematic diagram illustrating a fusion weight calculation process according to an exemplary embodiment of this application;

[0076] Figure 6 This is a flowchart illustrating an image fusion method in yet another exemplary embodiment of this application;

[0077] Figure 7 This is a schematic diagram of the structure of an image fusion apparatus shown in an exemplary embodiment of this application;

[0078] Figure 8 This is a schematic diagram of the hardware structure of an image fusion device according to an exemplary embodiment of this application. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0080] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0081] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0082] Please see Figure 1 This is a flowchart illustrating an image fusion method provided in an embodiment of this application. The image fusion method can be applied to image acquisition devices, such as surveillance cameras in video surveillance scenarios. Figure 1 As shown, the image fusion method may include the following steps:

[0083] Step S100: Convert the visible light image acquired by the image acquisition device into an RGB image.

[0084] Step S110: Convert the near-infrared image acquired by the image acquisition device into a first brightness image.

[0085] In this embodiment of the application, when the external ambient brightness of the area where the image acquisition device is located is low, the image acquisition device needs to use infrared supplementary lighting. At this time, the image acquisition device will simultaneously acquire visible light images and near-infrared images, and improve the image quality by fusing the visible light images and near-infrared images.

[0086] In this embodiment of the application, after the image acquisition device acquires visible light images and near-infrared images, it can convert the visible light images into RGB (Red, Green, Blue) images and the near-infrared images into infrared brightness images (also known as infrared Y-channel images, referred to as the first brightness image in this document).

[0087] For example, for visible light images, the image acquisition device can restore the color of the visible light image through AWB (Automatic White Balance) correction and perform DENOISE (noise reduction) processing. Then, it can interpolate the visible light image to an RGB image through DEMOISC (de-mosaic) processing and perform GAMMA (gamma) correction on the RGB image to improve the image brightness. For near-infrared images, the image acquisition device can interpolate the near-infrared image to an RGB image through DEMOISC processing and perform GAMMA correction on the RGB image to improve the image brightness. Then, it can convert the RGB image into a luminance image and align the near-infrared image and the corresponding visible light image pixel by pixel through Y channel registration processing. For specific implementation details, please refer to the relevant descriptions in existing related technologies. The embodiments of this application will not be elaborated here.

[0088] It should be noted that there is no necessary temporal relationship between steps S100 and S110. That is, the operation in step S100 can be executed first and then the operation in step S110; or the operation in step S110 can be executed first and then the operation in step S100; or the operations in steps S100 and S110 can be executed concurrently.

[0089] Step S120: Convert the RGB image into a second brightness image.

[0090] In this embodiment of the application, after the image acquisition device converts the visible light image into an RGB image, it can convert the RGB image into a visible light brightness image (also known as a visible light Y channel image, referred to as the second brightness image in this document).

[0091] For example, an image acquisition device can convert an RGB image into a second-luminance image using the following formula:

[0092] y p =(R p *77+G p *150+B p *29) / 256

[0093] Among them, R p G p B p These are the three channel values ​​of pixel P in the RGB image, y p Let P be the luminance channel value at pixel P in the second luminance image. Pixel P is any pixel in the RGB image, and the position of pixel P in the RGB image is the same as that of pixel P in the second luminance image.

[0094] Step S130: Calculate the near-infrared image fusion weights based on the first brightness image to obtain the near-infrared image fusion weights.

[0095] In this embodiment of the application, after the image acquisition module converts the near-infrared image into a first brightness image, it can also perform near-infrared image fusion weight calculation based on the first brightness image to determine the weight of the brightness value of each pixel in the near-infrared image when performing brightness fusion of the visible light image and the near-infrared image, thereby obtaining the fusion weight of the near-infrared image; the weight of the near-infrared image fusion weight records the weight of the brightness value of each pixel in the near-infrared image when performing brightness fusion of the visible light image and the near-infrared image.

[0096] Specifically, in the darker parts of the image (i.e., pixels with low brightness values ​​in the first brightness image (e.g., below a preset threshold, which can be set according to the actual scene), the weight of the brightness values ​​of the near-infrared image pixels in the fused image can be increased to improve the brightness of the fused image. Conversely, in the brighter parts of the image (i.e., pixels with high brightness values ​​in the first brightness image (e.g., above a preset threshold, which can be set according to the actual scene), the weight of the brightness values ​​of the near-infrared image pixels can be decreased so that more details from the visible light image can be retained in the fused image.

[0097] In one embodiment of this application, the above-mentioned near-infrared image fusion weight calculation based on the first brightness image may include:

[0098] For any pixel in the first brightness image, the preset brightness mapping model can be queried based on the brightness value of that pixel to determine the fusion weight corresponding to that brightness value.

[0099] In this embodiment, in order to determine the near-infrared image fusion weights, a brightness mapping model can be preset, which can record the correspondence between brightness values ​​and fusion weights.

[0100] Accordingly, after the image acquisition device converts the near-infrared image into a first brightness image, it can query the brightness mapping model based on the brightness value of each pixel to obtain the fusion weight of each pixel.

[0101] For example, please see Figure 2 This is a schematic diagram of a brightness mapping model provided in an embodiment of this application, as shown below. Figure 2 As shown, this brightness mapping model is controlled by three parameters: min_wt, min_limit, and max_limit. The horizontal axis of the brightness mapping model represents the brightness value, and the vertical axis represents the fusion weight. Wherein:

[0102] For any pixel in the first brightness image, when its brightness value is less than min_limit, its fusion weight is 255; when its brightness value is greater than max_limit, its fusion weight is min_wt; when its brightness value is in [min_limit, max_limit], its fusion weight gradually decreases as the brightness value increases. The specific mapping relationship between its brightness value and fusion weight can be determined according to the actual brightness mapping model.

[0103] The values ​​of the parameters min_wt, min_limit, and max_limit can be empirical values, for example, 180, 200, and 250 respectively.

[0104] It should be noted that, in this embodiment, in order to optimize the image fusion effect, after determining the fusion weight map of the near-infrared image based on the brightness mapping model, the fusion weight map can also be subjected to mean filtering. The specific implementation is not described here.

[0105] It should be noted that there is no necessary temporal relationship between steps S120 and S130. That is, the operation in step S120 can be executed first and then the operation in step S130; or the operation in step S130 can be executed first and then the operation in step S120; or the operations in steps S120 and S130 can be executed concurrently.

[0106] Step S140: Perform brightness fusion on the first brightness image and the second brightness image according to the fusion weight map to obtain a brightness fused image.

[0107] In this embodiment of the application, after obtaining the fusion weight map of the near-infrared image, the image acquisition device can perform brightness fusion processing on the first brightness image obtained in step S110 and the second brightness image obtained in step S120 according to the fusion weight map to obtain a brightness fused image.

[0108] Step S150: Perform RGB fusion based on the second brightness image, the brightness fusion image, and the RGB image to obtain the fused RGB image.

[0109] In this embodiment of the application, after the image acquisition device obtains the brightness fusion image, it can perform RGB fusion processing based on the brightness fusion image, the second brightness image obtained in step S120, and the RGB image obtained in step S100 to obtain the fused RGB image.

[0110] In one embodiment of this application, the above-mentioned RGB fusion based on the second luminance image, the luminance fusion image, and the RGB image to obtain the fused RGB image may include:

[0111] For any pixel location, the R, G, and B channel values ​​V of that pixel are determined using the following formula. out :

[0112] When Y LL When V > 0, out = CLIP(V in *Y / Y LL (0, 255)

[0113] When Y LL When V = 0, out = CLIP(Y, 0, 255)

[0114] Where Vin represents the R, G, and B channel values ​​of the pixel at that location in the RGB image (the RGB image before fusion), and Y represents the Y channel values. LL Y is the brightness value of the pixel at that location in the second brightness image, and Y is the brightness value of the pixel at that location in the brightness fusion image.

[0115] Among them, when Y LL When V > 0, if in When V is the R channel value of the pixel at that location in the RGB image, then out The R channel value of the pixel at that location in the merged RGB image; if V in When V is the G channel value of the pixel at that location in the RGB image, then... out The value of the G channel of the pixel at that location in the merged RGB image; if V in When V is the B channel value of the pixel at that location in the RGB image, then... out This is the B channel value of the pixel at this location in the merged RGB image.

[0116] When Y LL When = 0, the R, G, and B channel values ​​of the same pixel in the fused RGB image are equal.

[0117] It should be understood that the RGB fusion method described in this embodiment is merely a specific example of RGB fusion in this application embodiment, and is not a limitation on the scope of protection of this application. Based on the embodiments of this application, any adjustments and modifications made by those skilled in the art to the RGB fusion method in the embodiments of this application without creative effort shall fall within the scope of protection of this application.

[0118] It can be seen that, in Figure 1In the method flow shown, when fusing visible light images and near-infrared images, the ideal brightness information of the scene is obtained through the near-infrared image, and the color information is obtained from the visible light image. Based on this, the brightness and color of the visible light image and near-infrared image are fused. While improving the brightness of the image in low-light environments (i.e., in scenes with low external ambient brightness), the color information of the image is maintained, thus improving the quality of the fused image.

[0119] Please see Figure 3 This is a flowchart illustrating another image fusion method provided in an embodiment of this application. This image fusion method can be applied to image acquisition devices, such as surveillance cameras in video surveillance scenarios. Figure 3 As shown, the image fusion method may include the following steps:

[0120] Step S300: Convert the visible light image acquired by the image acquisition device into an RGB image.

[0121] Step S310: Convert the near-infrared image acquired by the image acquisition device into a first brightness image.

[0122] Step S320: Convert the RGB image into a second brightness image.

[0123] Step S330: Calculate the near-infrared image fusion weights based on the first brightness image to obtain the near-infrared image fusion weights.

[0124] In this embodiment of the application, the specific implementation of steps S300 to S330 can be found in the relevant descriptions in steps S100 to S130, and will not be repeated here.

[0125] Step S340: Perform detail calculations on the first brightness image to obtain the first detail image.

[0126] Step S350: Perform detail calculations on the second brightness image to obtain a second detail image.

[0127] In this embodiment of the application, in order to improve the detail information of the fused image, after the image acquisition device obtains the first brightness image (second brightness image), it can also perform detail calculation on the first brightness image (second brightness image) to obtain the corresponding detail image (in this document, the detail image obtained by detail calculation of the first brightness image is called the first detail image, and the detail image obtained by detail calculation of the second brightness image is called the second detail image).

[0128] In one embodiment of this application, the above-mentioned detailed calculation of the first brightness image to obtain the first detail image may include:

[0129] The first brightness image is subjected to mean filtering to obtain the first mean image;

[0130] The difference between the first brightness image and the first average finger image is calculated to obtain the first difference image;

[0131] A cutoff operation is performed on the first difference image to obtain the first detail image.

[0132] In this embodiment, after the image acquisition device obtains the first brightness image, it can perform mean filtering on the first brightness image to obtain the corresponding mean image (referred to as the first mean image in this document).

[0133] For example, an image acquisition device can perform mean filtering on a first brightness image with a radius of r (r is an empirical value that can be set according to the actual scene) to obtain a first mean image.

[0134] In this embodiment, after the image acquisition device obtains the first mean image, it can calculate the difference between the first brightness image and the first mean image to obtain a signed difference image (referred to as the first difference image in this document).

[0135] Considering that the difference between the first brightness image and the first mean image will be magnified. For example, assuming that both the first brightness image and the first mean image are 8-bit images, the difference between the first brightness image and the first mean image will result in a 9-bit difference image. Therefore, after the image acquisition device obtains the first difference image, it also needs to perform a cutoff operation on the first difference image to cut it off to a specified interval in order to obtain the corresponding detail image (referred to as the first detail image in this article).

[0136] In one embodiment of this example, the cutoff operation on the first difference image described above can be achieved using the following formula:

[0137] Detail p = CLIP(Diff p *str / 128, deNirMin, deNirMax)

[0138] Among them, Detail p For the brightness detail of pixel P in the first detail image, Diff p Here, is the value of pixel P in the first difference image, str is the detail intensity control parameter, [deNirMin, deNirMax] is the cutoff interval, and CLIP() is the cutoff calculation.

[0139] Among them, str, deNirMin, and deNirMax are all empirical values. For example, str can be 64, deNirMin can be -64, and deNirMax can be 32.

[0140] In this embodiment of the application, the image acquisition device performs detail calculations on the second brightness image to obtain the second detail image. For the specific implementation of this, please refer to the above description of performing detail calculations on the first brightness image to obtain the first detail image. This embodiment of the application will not repeat the details here.

[0141] It should be noted that there is no necessary temporal relationship between steps S340 and S350. That is, the operation in step S340 can be executed first and the operation in step S350 can be executed later; the operation in step S350 can be executed first and the operation in step S340 can be executed later; or the operations in steps S340 and S350 can be executed concurrently.

[0142] Step S360: Perform brightness fusion on the first brightness image and the second brightness image according to the fusion weight, the first detail image, and the second detail image.

[0143] In this embodiment of the application, after the image acquisition device obtains the first detail image and the second detail image, it can perform brightness fusion on the first brightness image obtained in step S310 and the second brightness image obtained in step S320 based on the first detail image, the second detail image and the fusion weight obtained in step S330.

[0144] In one embodiment of this application, the above-described brightness fusion of the first brightness image and the second brightness image based on the fusion weight, the first detail image, and the second detail image may include:

[0145] For any pixel location, the blended brightness Y of that pixel is determined by the following formula:

[0146] Y = CLIP(((Y nir *wt+Y LL *(256-wt)) / 256+Detail nir +Detail LL ), 0, 255)

[0147] Among them, Y nir Y represents the brightness value of the pixel at that location in the first brightness image, wt represents the fusion weight of the pixel at that location in the fusion weight image, and Y represents the fusion weight of the pixel at that location in the fusion weight image. LL The luminance value of the pixel at that location in the second luminance image, Detail nir For the brightness detail of the pixel at this location in the first detail image, Detail LL This refers to the brightness detail of the pixel at that location in the second detail image.

[0148] Step S370: Perform RGB fusion based on the second luminance image, the luminance fusion image, and the RGB image to obtain the fused RGB image.

[0149] In this embodiment of the application, the specific implementation of step S370 can be found in the relevant description in step S150, and will not be repeated here.

[0150] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0151] In this embodiment, the visible light image and near-infrared image acquired by the image acquisition device are both 8-bit images, as an example.

[0152] However, it should be recognized that the visible light and near-infrared images acquired by image acquisition devices are not limited to 8-bit images; they can also be 12-bit or 16-bit images.

[0153] In this embodiment, after the image acquisition device acquires visible light images and near-infrared images, the visible light images can be subjected to AWB correction, DENOISE processing, DEMOSIC processing, and GAMMA correction to obtain an 8-bit visible light RGB image; the near-infrared images can be subjected to DEMOSIC processing, GAMMA correction, RGB2Y (RGB to Y, converting the RGB image to a Y channel image, i.e., a luminance image) processing, and Y channel registration to obtain an 8-bit infrared luminance image.

[0154] For an 8-bit visible light image, the image acquisition device can convert it into an 8-bit luminance image through RGB2Y processing, and then perform detail calculations on the 8-bit luminance image to obtain an 8-bit detail image.

[0155] The flowchart illustrating the process of detail calculation for an 8-bit visible light brightness image by the image acquisition device can be found in [link to relevant documentation]. Figure 4 The image acquisition device can perform mean filtering with a radius of r on the 8-bit visible light brightness image to obtain an 8-bit mean image. Then, the difference between the 8-bit visible light brightness image and the 8-bit mean image is calculated to obtain a 9-bit signed difference image. Then, through a cutoff operation, the 9-bit signed difference image is cut off to a specified interval ([deNirMin,deNirMax]) to obtain an 8-bit visible light detail image.

[0156] For 8-bit infrared brightness images, the image acquisition device can perform fusion weight calculations.

[0157] The flowchart for calculating the fusion weights of an 8-bit infrared brightness image by the image acquisition device can be as follows: Figure 5 As shown, the image acquisition device can query a preset brightness mapping model based on the brightness value of each pixel in the 8-bit infrared brightness image (such as...). Figure 2 As shown in the figure, the fusion weight of each pixel is obtained to obtain an 8-bit fusion weight map, and mean filtering is performed on the 8-bit fusion weight map.

[0158] On the other hand, the image acquisition device can perform detail calculations on the 8-bit infrared brightness image to obtain an 8-bit infrared detail image. For the specific implementation, please refer to the relevant description of the image device for the 8-bit visible light brightness image mentioned above. The embodiments of this application will not be repeated here.

[0159] After the image acquisition device obtains an 8-bit visible light detail image and an 8-bit infrared detail image, it can perform brightness fusion on the 8-bit visible light brightness image and the 8-bit infrared brightness image based on the 8-bit visible light detail image, the 8-bit infrared detail image, and the 8-bit fusion weight map. Then, it performs RGB fusion processing based on the fused brightness map, the 8-bit visible light brightness map, and the 8-bit visible light RGB image to obtain the fused RGB image, thus realizing the fusion of visible light and near-infrared images. The flowchart is as follows: Figure 6 As shown.

[0160] In this embodiment, the visible light image acquired by the image acquisition device is converted into an RGB image, and the near-infrared image acquired by the image acquisition device is converted into a first brightness image. Then, the RGB brightness image is converted into a second brightness image, and a near-infrared image fusion weight is calculated based on the first brightness image to obtain a near-infrared image fusion weight map. Then, the first brightness image and the second brightness image are fused according to the fusion weight map to obtain a brightness fused image. Finally, RGB fusion is performed based on the second brightness image, the brightness fused image, and the RGB image to obtain a fused RGB image. This achieves the fusion of visible light and near-infrared images, improves the brightness of the image in low-light environments while maintaining the color information of the image, and improves the quality of the fused image.

[0161] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0162] Please see Figure 7 This is a schematic diagram of an image fusion device provided in an embodiment of this application. The image fusion device can be applied to the image acquisition device described in the above method embodiments, such as... Figure 7 As shown, the image fusion apparatus may include:

[0163] The first visible light processing unit 710 is used to convert the visible light image acquired by the image acquisition device into a red-green-blue RGB image;

[0164] The first infrared processing unit 720 is used to convert the near-infrared image acquired by the image acquisition device into a first brightness image;

[0165] The second visible light processing unit 730 is used to convert the RGB brightness image into a second brightness image;

[0166] The second infrared processing unit 740 is used to perform near-infrared image fusion weight calculation based on the first brightness image to obtain the fusion weight map of the near-infrared image.

[0167] A brightness fusion unit 750 is used to perform brightness fusion on the first brightness image and the second brightness image according to the fusion weight map to obtain a brightness fused image;

[0168] The RGB fusion unit 760 is used to perform RGB fusion based on the second brightness image, the brightness fusion image and the RGB image to obtain a fused RGB image.

[0169] In an optional embodiment, the second visible light processing unit 730 is further configured to perform detail calculations on the first brightness image to obtain a first detail image;

[0170] The second infrared processing unit 740 is further configured to perform detail calculations on the second brightness map to obtain a second detail image;

[0171] The brightness fusion unit 750 is specifically used to perform brightness fusion on the first brightness image and the second brightness image according to the fusion weight, the first detail image, and the second detail image.

[0172] In one optional implementation, the second visible light processing unit 730 is specifically configured to perform mean filtering on the first brightness image to obtain a first mean image; calculate the difference between the first brightness image and the first mean image to obtain a first difference image; and perform a cutoff operation on the first difference image to obtain the first detail image.

[0173] In one optional embodiment, the second infrared processing unit 740 is specifically configured to perform mean filtering on the second brightness image to obtain a second mean image; calculate the difference between the second brightness image and the second mean image to obtain a second difference image; and perform a cutoff operation on the second difference image to obtain the second detail image.

[0174] In one alternative implementation, the cutoff operation on the difference image is achieved using the following formula:

[0175] Detail p = CLIP(Diff p *str / 128, deNirMin, deNirMax)

[0176] Among them, Detail p To capture the brightness detail of pixel P in a detailed image, Diff p Here, P is the value of pixel P in the difference image, str is the detail intensity control parameter, [deNirMin, deNirMax] is the cutoff interval, and CLIP() is the cutoff calculation.

[0177] In one optional implementation, the brightness fusion unit 750 is specifically used to determine the fused brightness Y of a pixel at any given location using the following formula:

[0178] Y = CLIP(((Y nir *wt+Y LL *(256-wt)) / 256+Detail nir +Detail LL ), 0, 255)

[0179] Among them, Y nir Y represents the brightness value of the pixel at that location in the first brightness image, wt represents the fusion weight of the pixel at that location in the fusion weight image, and Y represents the fusion weight of the pixel at that location. LL The brightness value of the pixel at that location in the second brightness image, Detail nir For the brightness detail of the pixel at that location in the first detail image, Detail LL This refers to the brightness detail of the pixel at that location in the second detail image.

[0180] In one optional implementation, the second infrared processing unit 740 is specifically used to query a preset brightness mapping model based on the brightness value of any pixel in the first brightness image to determine the fusion weight corresponding to the brightness value; wherein, the brightness mapping model records the correspondence between brightness values ​​and fusion weights.

[0181] In one optional implementation, the RGB fusion unit 760 is specifically used to determine the R, G, and B channel values ​​V of a pixel at any given location using the following formula. out :

[0182] When Y LL When V > 0, out = CLIP(Vin *Y / Y LL (0, 255)

[0183] When Y LL When V = 0, out = CLIP(Y, 0, 255)

[0184] Where Vin represents the R, G, and B channel values ​​of the pixel at that location in the RGB image, and Y... LL Y is the brightness value of the pixel at that position in the second brightness image, and Y is the brightness value of the pixel at that position in the brightness fusion image.

[0185] Please see Figure 8 This is a schematic diagram of the hardware structure of an image fusion apparatus provided in an embodiment of this application. The image fusion apparatus may include a processor 801 and a machine-readable storage medium 802 storing machine-executable instructions. The processor 801 and the machine-readable storage medium 802 can communicate via a system bus 803. Furthermore, by reading and executing the machine-executable instructions corresponding to the image fusion logic in the machine-readable storage medium 802, the processor 801 can execute the image fusion method described above.

[0186] The machine-readable storage medium 802 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, a machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof.

[0187] This application also provides a machine-readable storage medium including machine-executable instructions, such as... Figure 8 The machine-readable storage medium 802 in the image fusion apparatus contains machine-executable instructions that can be executed by the processor 801 in the image fusion apparatus to implement the image fusion method described above.

[0188] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0189] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image fusion method applied to an image acquisition device, characterized in that, The method comprises: Converting the visible light image collected by the image collection device into a red-green-blue (RGB) image, and converting the near-infrared image collected by the image collection device into a first luminance image; wherein the visible light image and the near-infrared image are collected by the image collection device under the condition of infrared light supplement; Converting the RGB image into a second luminance image, and performing near-infrared image fusion weight calculation according to the first luminance image to obtain a fusion weight map of the near-infrared image; wherein the fusion weight map of the near-infrared image records the weight of the luminance value of each pixel point in the near-infrared image when luminance fusion is performed on the visible light image and the near-infrared image; Performing luminance fusion on the first luminance image and the second luminance image according to the fusion weight map to obtain a luminance fusion image; Performing RGB fusion according to the second luminance image, the luminance fusion image, and the RGB image to obtain a fused RGB image; The near-infrared image fusion weight calculation according to the first luminance image comprises: For any pixel point in the first luminance image, a preset luminance mapping model is queried according to the luminance value of the pixel point to determine the fusion weight corresponding to the luminance value; wherein the luminance mapping model records the correspondence between the luminance value and the fusion weight; the weight of the luminance value of the pixel point in the near-infrared image in the fusion image is increased for part of the pixel points in the first luminance image whose luminance value is lower than a preset threshold; the weight of the luminance value of the pixel point in the near-infrared image in the fusion image is reduced for part of the pixel points in the first luminance image whose luminance value is higher than a preset threshold; The RGB fusion according to the second luminance image, the luminance fusion image, and the RGB image to obtain a fused RGB image comprises: For any pixel point in a position, the R, G, B three channel values V of the pixel point are determined by the following formula out : When Y LL > 0, V out = CLIP(V in * Y / Y LL , 0, 255) When Y LL = 0, V out = CLIP(Y, 0, 255) wherein, Vin is the R, G, B three channel values of the pixel point at the position in the RGB image, Y LL is the luminance value of the pixel point at the position in the second luminance image, and Y is the luminance value of the pixel point at the position in the luminance fusion image.

2. The method of claim 1, wherein, Before the luminance fusion of the first luminance image and the second luminance image according to the fusion weight map, the method further comprises: Performing detail calculation on the first luminance image to obtain a first detail image, and performing detail calculation on the second luminance image to obtain a second detail image; The luminance fusion of the first luminance image and the second luminance image according to the fusion weight map comprises: Performing luminance fusion on the first luminance image and the second luminance image according to the fusion weight map, the first detail image, and the second detail image.

3. The method of claim 2, wherein, The detail calculation on the first luminance image to obtain a first detail image comprises: Performing mean value filtering on the first luminance image to obtain a first mean value image; Performing difference calculation on the first luminance image and the first mean value image to obtain a first difference value image; Performing cutoff operation on the first difference value image to obtain the first detail image.

4. The method of claim 2, wherein, The detail calculation on the second luminance image to obtain a second detail image comprises: Performing mean value filtering on the second luminance image to obtain a second mean value image; Performing difference calculation on the second luminance image and the second mean value image to obtain a second difference value image; performing a clipping operation on the second difference image to obtain the second detail image.

5. The method according to claim 3 or 4, characterized in that, The clipping operation on the difference image is implemented by the following formula: Detail p = CLIP ( Diff p * str / 128, deNirMin, deNirMax) wherein Detail p is the luminance detail of the pixel P in the detail image, Diff p is the value of the pixel P in the difference image, str is the detail intensity control parameter, [deNirMin, deNirMax] is the clipping interval, and CLIP() is the clipping calculation.

6. The method of claim 2, wherein, The luminance fusing of the first luminance image and the second luminance image according to the fusion weight map, the first detail image and the second detail image comprises: For any pixel point at a position, the fusion luminance Y of the pixel point is determined by the following formula: Y = CLIP ( ( (Y nir * wt + Y LL * (256 - wt) ) / 256 + Detail nir + Detail LL ), 0, 255) wherein Y nir is the luminance value of the pixel point at the position in the first luminance image, wt is the fusion weight of the pixel point at the position in the fusion weight image, Y LL is the luminance value of the pixel point at the position in the second luminance image, Detail nir is the luminance detail of the pixel point at the position in the first detail image, Detail LL is the luminance detail of the pixel point at the position in the second detail image.

7. An image fusion device applied to an image acquisition device, characterized in that, The device comprises: A first visible light processing unit, configured to convert a visible light image collected by the image collection device into an RGB image; A first infrared processing unit, configured to convert a near-infrared image collected by the image collection device into a first luminance image; wherein the visible light image and the near-infrared image are collected by the image collection device under the condition of infrared light supplement; A second visible light processing unit, configured to convert the RGB image into a second luminance image; A second infrared processing unit, configured to perform fusion weight calculation on the near-infrared image according to the first luminance image to obtain a fusion weight map of the near-infrared image; wherein the fusion weight map of the near-infrared image records the weight of the luminance value of each pixel point in the near-infrared image when luminance fusing is performed on the visible light image and the near-infrared image; A luminance fusing unit, configured to perform luminance fusing on the first luminance image and the second luminance image according to the fusion weight map to obtain a luminance fused image; An RGB fusing unit, configured to perform RGB fusing on the second luminance image, the luminance fused image and the RGB image to obtain a fused RGB image; The second infrared processing unit is specifically configured to, for any pixel point in the first luminance image, query a preset luminance mapping model according to the luminance value of the pixel point to determine the fusion weight corresponding to the luminance value; wherein the luminance mapping model records the corresponding relationship between the luminance value and the fusion weight; the weight of the luminance value of the pixel point in the near-infrared image in the fusion image is increased for the part of the pixel points in the first luminance image whose luminance value is lower than a preset threshold; the weight of the luminance value of the pixel point in the near-infrared image in the fusion image is decreased for the part of the pixel points in the first luminance image whose luminance value is higher than the preset threshold. The RGB fusion unit is specifically configured to determine R, G and B three channel values V of a pixel point at any position according to the following formula out : When Y LL > 0, V out = CLIP(V in * Y / Y LL , 0, 255) When Y LL = 0, V out = CLIP(Y, 0, 255) wherein, Vin is the R, G, B three channel values of the pixel point in the RGB image at the position, Y LL is the luminance value of the pixel point in the second luminance image at the position, and Y is the luminance value of the pixel point in the luminance fusion image at the position.

8. The device of claim 7, wherein The second visible light processing unit is further configured to perform detail calculation on the first luminance image to obtain a first detail image; The second infrared processing unit is further configured to perform detail calculation on the second luminance image to obtain a second detail image; The luminance fusing unit is specifically configured to perform luminance fusing on the first luminance image and the second luminance image according to the fusion weight map, the first detail image and the second detail image.

9. The device of claim 8, wherein The second visible light processing unit is specifically configured to perform mean value filtering on the first luminance image to obtain a first mean value image; and perform difference calculation on the first luminance image and the first mean value image to obtain a first difference image; performing a cut-off operation on the first difference image to obtain the first detail image.

10. The apparatus of claim 8, wherein, the second infrared processing unit is specifically configured to perform mean filtering on the second luminance image to obtain a second mean image; performing a difference operation on the second luminance image and the second mean image to obtain a second difference image; performing a cut-off operation on the second difference image to obtain the second detail image.

11. The apparatus of claim 9 or 10, wherein, The cut-off operation on the difference image is implemented by the following formula: Detail p = CLIP ( Diff p * str / 128, deNirMin, deNirMax) wherein Detail p is the luminance detail of the pixel P in the detail image, Diff p is the value of the pixel P in the difference image, str is the detail intensity control parameter, [deNirMin, deNirMax] is the clipping interval, and CLIP() is the clipping calculation.

12. The apparatus of claim 8, wherein, the luminance fusion unit is specifically configured to determine the fusion luminance Y of any pixel point at a position by the following formula: Y = CLIP ( ( (Y nir * wt + Y LL * (256 - wt) ) / 256 + Detail nir + Detail LL ), 0, 255) wherein Y nir is the luminance value of the pixel point at the position in the first luminance image, wt is the fusion weight of the pixel point at the position in the fusion weight image, Y LL is the luminance value of the pixel point at the position in the second luminance image, Detail nir is the luminance detail of the pixel point at the position in the first detail image, Detail LL is the luminance detail of the pixel point at the position in the second detail image.

13. An image fusion apparatus characterized by comprising: A processor and a machine readable storage medium are included, the machine readable storage medium stores machine executable instructions which can be executed by the processor, and the processor is prompted by the machine executable instructions: convert a visible light image collected by an image collection device into a red green blue (RGB) image, and convert a near infrared image collected by the image collection device into a first luminance image; wherein the visible light image and the near infrared image are collected by the image collection device under the condition of infrared light supplement; convert the RGB image into a second luminance image, and perform near infrared image fusion weight calculation according to the first luminance image to obtain a fusion weight map of the near infrared image; wherein the fusion weight map of the near infrared image records the weight of the luminance value of each pixel point in the near infrared image when performing luminance fusion on the visible light image and the near infrared image; perform luminance fusion on the first luminance image and the second luminance image according to the fusion weight map to obtain a luminance fusion image; perform RGB fusion according to the second luminance image, the luminance fusion image and the RGB image to obtain a fused RGB image; wherein the near infrared image fusion weight calculation according to the first luminance image comprises: for any pixel point in the first luminance image, query a preset luminance mapping model according to the luminance value of the pixel point to determine the fusion weight corresponding to the luminance value; wherein the luminance mapping model records the corresponding relationship between the luminance value and the fusion weight; increase the weight of the luminance value of the pixel point in the near infrared image in the fusion image for the part of the pixel points in the first luminance image whose luminance value is lower than a preset threshold; and decrease the weight of the luminance value of the pixel point in the near infrared image in the fusion image for the part of the pixel points in the first luminance image whose luminance value is higher than the preset threshold; the RGB fusion according to the second luminance image, the luminance fusion image and the RGB image to obtain the fused RGB image comprises: For any pixel point in a position, the R, G, B three channel values V of the pixel point are determined by the following formula out : When Y LL > 0, V out = CLIP(V in * Y / Y LL , 0, 255) When Y LL = 0, V out = CLIP(Y, 0, 255) wherein, Vin is the R, G, B three channel values of the pixel point at the position in the RGB image, Y LL is the luminance value of the pixel point at the position in the second luminance image, and Y is the luminance value of the pixel point at the position in the luminance fusion image.

14. A machine-readable storage medium, characterized in that, the machine executable instructions stored in the machine readable storage medium prompt the processor to: Converting a visible light image collected by an image collection device into a red-green-blue (RGB) image, and converting a near-infrared image collected by the image collection device into a first luminance image; wherein the visible light image and the near-infrared image are collected by the image collection device under the condition of infrared light supplement; Converting the RGB image into a second luminance image, and performing near-infrared image fusion weight calculation according to the first luminance image to obtain a fusion weight map of the near-infrared image; wherein the fusion weight map of the near-infrared image records the weight of the luminance value of each pixel point in the near-infrared image when luminance fusion is performed on the visible light image and the near-infrared image; Performing luminance fusion on the first luminance image and the second luminance image according to the fusion weight map to obtain a luminance fusion image; Performing RGB fusion according to the second luminance image, the luminance fusion image, and the RGB image to obtain a fused RGB image; The near-infrared image fusion weight calculation according to the first luminance image includes: For any pixel point in the first luminance image, a preset luminance mapping model is queried according to the luminance value of the pixel point to determine the fusion weight corresponding to the luminance value; wherein the luminance mapping model records the corresponding relationship between the luminance value and the fusion weight; the weight of the luminance value of the pixel point of the near-infrared image in the fusion image is increased for part of the pixel points in the first luminance image whose luminance value is lower than a preset threshold; the weight of the luminance value of the pixel point of the near-infrared image in the fusion image is reduced for part of the pixel points in the first luminance image whose luminance value is higher than the preset threshold; The RGB fusion according to the second luminance image, the luminance fusion image, and the RGB image to obtain a fused RGB image includes: For any pixel point in a position, the R, G, B three channel values V of the pixel point are determined by the following formula out : When Y LL > 0, V out = CLIP(V in * Y / Y LL , 0, 255) When Y LL = 0, V out = CLIP(Y, 0, 255) wherein, Vin is the R, G, B three channel values of the pixel point in the RGB image at the position, Y LL is the luminance value of the pixel point in the second luminance image at the position, and Y is the luminance value of the pixel point in the luminance fusion image at the position.

Images