Methods, apparatus and electronic devices for fusing visible light images and infrared light images

Abstract

This application provides a method, apparatus, and electronic device for fusing visible light images and infrared light images. The method involves acquiring infrared light images and visible light images; determining the reflectance parameter corresponding to each visible light pixel in the visible light image based on its brightness value; performing simulated reflectance mapping on the brightness values ​​of each infrared light pixel in the infrared light image according to the reflectance parameter corresponding to each visible light pixel to obtain an infrared brightness mapping image; and fusing the infrared brightness mapping image, the infrared light image, and the visible light image to obtain a fused image. By performing simulated reflectance mapping on the brightness values ​​of each infrared light pixel in the infrared light image according to the reflectance parameter corresponding to each visible light pixel, the infrared light image can more closely approximate the brightness distribution of the visible light image, better capturing texture areas with indistinct differences in the infrared light image. Therefore, the obtained fused image is more consistent with human vision and can reduce the distortion rate of the fused image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to methods, apparatus and electronic devices for fusing visible light images and infrared light images. Background Technology

[0002] With the booming development of the surveillance industry, users are becoming increasingly demanding in terms of the effectiveness of low-light images. Low-light environment image enhancement is a key research focus for various manufacturers, and the fusion of visible light images and infrared light images is an important research direction for low-light image enhancement.

[0003] In existing technologies, infrared and visible light images acquired in low-light environments are directly fused using the weighting coefficients of the infrared image and the weighting system of the visible light image to obtain a fused image. Infrared images are formed when infrared light emitted by a device illuminates an object's surface, and the image is created based on the object's reflectivity to infrared light. Visible light images are formed primarily based on the reflection of visible light from the object's surface. In low-light environments, visible light images have poor signal-to-noise ratios. Directly fusing the two images using the weighting coefficients of the infrared and visible light images without considering the difference in reflectivity between visible and infrared light results in a fused image with significant noise and high distortion. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, and electronic device for fusing visible light images and infrared light images, so as to reduce the distortion rate of the fused image. The specific technical solution is as follows:

[0005] According to a first aspect of the embodiments of this application, a method for fusing visible light images and infrared light images is provided, the method comprising:

[0006] Acquire infrared and visible light images;

[0007] Based on the brightness value of each visible light pixel in the visible light image, the reflectivity parameter corresponding to each visible light pixel is determined respectively;

[0008] Based on the reflectance parameters corresponding to each visible light pixel, the brightness values ​​of each infrared light pixel in the infrared light image are simulated and reflected to obtain an infrared brightness mapping image.

[0009] The infrared brightness mapping image, the infrared light image, and the visible light image are fused to obtain a fused image.

[0010] In one possible implementation, prior to the steps of acquiring the infrared and visible light images, the method further includes:

[0011] Acquire infrared light images and initial visible light images collected at the same time;

[0012] The visible light image is obtained by superimposing the pixel values ​​of the same type of sub-pixels in adjacent channels in the initial visible light image.

[0013] In one possible implementation, the step of superimposing the pixel values ​​of similar sub-pixels in adjacent channels of the initial visible light image to obtain the visible light image includes:

[0014] For each pixel region in the initial visible light image, the pixel values ​​of each sub-pixel of the same type in the pixel region are fused to obtain the pixel corresponding to the pixel region in the visible light image, wherein each pixel region includes n×m adjacent pixels.

[0015] In one possible implementation, the initial visible light image is in Bayer format; the step of fusing the pixel values ​​of all sub-pixels of the same type in each pixel region of the initial visible light image to obtain the pixel corresponding to that pixel region in the visible light image includes:

[0016] Based on the initial visible light image, the visible light image is obtained using the following formula:

[0017] R' a，b =w1R 2a-1，2b-1 +w2R 2a-1，2b+1 +w3R 2a+1，2b-1 +w4R 2a+1，2b+1

[0018] G' a，c =w1G 2a-1，2c-2 +w2G 2a-1，2c +w3G 2a+1，2c +w4G 2a+1，2c

[0019] G' c，a =w1G 2c-2，2a-1 +w2G 2c-2，2a+1 +w3G 2c，2a-1 +w4G 2c，2a+1

[0020] B' c，d =w1B 2c-2，2d-2 +W2B 2c-2，2d +w3B 2c，2d-2 +w4B 2c，2d

[0021] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，cB' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. 2a-1，2b-1 G represents the value of pixel R in row 2a-1 and column 2b-1 of the initial visible light image. 2a-1，2c B represents the value of pixel G in row 2a-1, column 2c of the initial visible light image. 2c，2d The value of pixel B in row 2c and column 2d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

[0022] In one possible implementation, the step of superimposing the pixel values ​​of similar sub-pixels in adjacent channels of the initial visible light image to obtain the visible light image includes:

[0023] For each sub-pixel in the initial visible light image, the pixel value of the sub-pixel is fused with the pixel value of its adjacent sub-pixels of the same type to obtain the corresponding sub-pixel in the visible light image. For any sub-pixel, the adjacent sub-pixels of the same type are sub-pixels of the same type and adjacent in position as the sub-pixel.

[0024] In one possible implementation, the initial visible light image is in Bayer format; the step of fusing the pixel value of each sub-pixel in the initial visible light image with the pixel value of its adjacent sub-pixels of the same type to obtain the corresponding sub-pixel in the visible light image includes:

[0025] Based on the initial visible light image, the visible light image is obtained using the following formula:

[0026] R' a，b =w1R a，b +w2R a+2，b +w3R a，b+2 +w4R a+2，b+2

[0027] G' a，c =w1G a，c +w2G a+2，c +w3G a，c+2 +w4G a+2，c+2

[0028] G' c，a =w1G c，a +w2G c+2，a +w3G c，a+2 +w4G c+2，a+2

[0029] B' c，d=w1B c，d +w2B c+2，d +w3B c，d+2 +w4B c+2，d+2

[0030] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. a，b G represents the value of pixel R in row a and column b of the initial visible light image. a，c B represents the value of pixel G in row a and column c of the initial visible light image. c，d The value of pixel B in row c and column d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

[0031] In one possible implementation, prior to the steps of acquiring the infrared and visible light images, the method further includes:

[0032] Acquire the infrared light image and the first initial visible light image collected at the same time;

[0033] Among the initial visible light images whose acquisition time is within a preset time difference range from the acquisition time of the first initial visible light image, a preset number of initial visible light images are selected to obtain a preset number of second initial visible light images;

[0034] The visible light image is obtained by superimposing the pixel values ​​of the sub-pixels at the same position in each of the first initial visible light images and each of the second initial visible light images.

[0035] In one possible implementation, determining the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image includes:

[0036] Obtain the brightness value of the pixel with the highest brightness in the visible light image to obtain the maximum brightness value;

[0037] For each visible light pixel in the visible light image, the reflectivity parameter corresponding to that visible light pixel is obtained by calculating the ratio of the brightness value of that visible light pixel to the maximum brightness value.

[0038] In one possible implementation, the step of performing simulated reflectance mapping on the brightness values ​​of each infrared pixel in the infrared image according to the reflectance parameters corresponding to each visible light pixel to obtain an infrared brightness mapping image includes:

[0039] For each infrared pixel in the infrared light image, the brightness value of the infrared pixel in the infrared brightness mapping image is obtained by multiplying the reflectance parameter corresponding to the infrared pixel with the brightness value of the infrared pixel in the infrared light image; wherein, the reflectance parameter corresponding to the infrared pixel is the reflectance parameter corresponding to the visible light pixel at the same position as the infrared pixel.

[0040] In one possible implementation, after performing simulated reflectance mapping on the brightness values ​​of each infrared pixel in the infrared image according to the reflectance parameters corresponding to each visible light pixel to obtain an infrared brightness mapping image, the method further includes:

[0041] The brightness value range of pixels in the current infrared brightness mapping image is divided into multiple brightness intervals. The brightness value range and brightness enhancement rate of each brightness interval are adjusted so that the brightness value range of each adjusted brightness interval is within the preset brightness interval range.

[0042] For each pixel in the current infrared brightness mapping image, the brightness value of the pixel is increased by using the brightness increase rate of the brightness range to which the brightness value of the pixel belongs.

[0043] In one possible implementation, fusing the infrared brightness mapping image, the infrared light image, and the visible light image to obtain a fused image includes:

[0044] The brightness values ​​of the infrared brightness mapping image, the infrared light image, and the visible light image are fused according to the following formula, and the color components of the visible light image are superimposed to obtain a fused image;

[0045] Fusion(i,j)=w5IR(i,j)+w6IRmappingChange(i,j)+w7Color(i,j)

[0046] Wherein, Color(i,j) is the brightness value of the pixel in the i-th row and j-th column of the visible light image, IR(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared light image, IRmappingChange(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared brightness mapping image, and w5, w6, and w7 are preset parameters.

[0047] In one possible implementation, the method further includes:

[0048] Obtain the image quality parameters of the visible light image;

[0049] If the image quality parameters are not less than the first preset quality threshold, let w5 = w6 = 0 and w7 = 1;

[0050] If the image quality parameters are not greater than the second preset quality threshold, let w5 = 1, w6 = w7 = 0;

[0051] When the image quality parameter is less than the first preset quality threshold and greater than the second preset quality threshold, w6 > w5 and w6 > w7.

[0052] According to a second aspect of the embodiments of this application, an apparatus for fusing visible light images and infrared light images is provided, the apparatus comprising:

[0053] The first acquisition module is used to acquire infrared light images and visible light images;

[0054] The determining module is used to determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image;

[0055] The mapping module is used to simulate reflectance mapping of the brightness value of each infrared pixel in the infrared light image according to the reflectance parameter corresponding to each visible light pixel, so as to obtain an infrared brightness mapping image.

[0056] The fusion module is used to fuse the infrared brightness mapping image, the infrared light image, and the visible light image to obtain a fused image.

[0057] In one possible implementation, the device further includes:

[0058] The second acquisition module is used to acquire the infrared light image and the initial visible light image collected at the same time.

[0059] The first overlay module is used to overlay the pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image to obtain the visible light image.

[0060] In one possible implementation, the first overlay module is specifically used to fuse the pixel values ​​of each sub-pixel of the same type in each pixel region of the initial visible light image to obtain the pixel corresponding to the pixel region in the visible light image, wherein each pixel region includes n×m adjacent pixels.

[0061] In one possible implementation, the initial visible light image is in Bayer format; the first overlay module is specifically used for:

[0062] Based on the initial visible light image, the visible light image is obtained using the following formula:

[0063] R' a，b =w1R 2a-1，2b-1 +w2R 2a-1，2b+1 +w3R 2a+1，2b-1 +w4R 2a+1，2b+1

[0064] G' a，c =w1G 2a-1，2c-2 +w2G 2a-1，2c +w3G 2a+1，2c +w4G 2a+1，2c

[0065] G' c，a =w1G 2c-2，2a-1 +w2G 2c-2，2a+1 +w3G 2c，2a-1 +w4G 2c，2a+1

[0066] B' c，d =w1B 2c-2，2d-2 +w2B 2c-2，2d +w3B 2c，2d-2 +w4B 2c，2d

[0067] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. 2a-1，2b-1 G represents the value of pixel R in row 2a-1 and column 2b-1 of the initial visible light image. 2a-1，2c B represents the value of pixel G in row 2a-1, column 2c of the initial visible light image. 2c，2d The value of pixel B in row 2c and column 2d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

[0068] In one possible implementation, the first overlay module is specifically used to fuse the pixel value of each sub-pixel in the initial visible light image with the pixel value of its adjacent sub-pixels of the same type to obtain the sub-pixel in the visible light image corresponding to the sub-pixel, wherein, for any sub-pixel, the adjacent sub-pixels of the same type are sub-pixels of the same type and adjacent in position as the sub-pixel.

[0069] In one possible implementation, the initial visible light image is in Bayer format; the first overlay module is specifically used for:

[0070] Based on the initial visible light image, the visible light image is obtained using the following formula:

[0071] R' a，b =w1R a，b +w2R a+2，b +w3R a，b+2 +w4R a+2，b+2

[0072] G' a，c =w1G a，c +w2G a+2，c +w3G a，c+2 +w4G a+2，c+2

[0073] G' c，a =w1G c，a +w2G c+2，a +w3G c，a+2 +w4G c+2，a+2

[0074] B' c，d =w1B c，d +w2B c+2，d +w3B c，d+2 +w4B c+2，d+2

[0075] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. a，b G represents the value of pixel R in row a and column b of the initial visible light image. a，c B represents the value of pixel G in row a and column c of the initial visible light image. c，d The value of pixel B in row c and column d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

[0076] In one possible implementation, the device further includes:

[0077] The third acquisition module is used to acquire the infrared light image and the first initial visible light image collected at the same time.

[0078] The selection module is used to select a preset number of initial visible light images from among the initial visible light images whose acquisition time is within a preset time difference range from the acquisition time of the first initial visible light image, so as to obtain a preset number of second initial visible light images.

[0079] The second overlay module is used to overlay the pixel values ​​of the first initial visible light image with the pixel values ​​of the sub-pixels at the same position in each of the second initial visible light images to obtain the visible light image.

[0080] In one possible implementation, the determining module is specifically used to obtain the brightness value of the pixel with the highest brightness in the visible light image, and obtain the maximum brightness value; for each visible light pixel in the visible light image, the reflectivity parameter corresponding to the visible light pixel is obtained by calculating the ratio of the brightness value of the visible light pixel to the maximum brightness value.

[0081] In one possible implementation, the mapping module is specifically used to calculate the brightness value of each infrared pixel in the infrared light image by multiplying the reflectance parameter corresponding to the infrared pixel with the brightness value of the infrared pixel in the infrared light image; wherein, the reflectance parameter corresponding to the infrared pixel is the reflectance parameter corresponding to the visible light pixel at the same position as the infrared pixel.

[0082] In one possible implementation, the device further includes:

[0083] The adjustment module is used to divide the brightness value range of pixels in the current infrared brightness mapping image into multiple brightness intervals, and adjust the brightness value range and brightness enhancement rate of each brightness interval so that the brightness value range of each adjusted brightness interval is within the preset brightness interval range.

[0084] The brightness enhancement module is used to enhance the brightness of each pixel in the current infrared brightness mapping image by using the brightness enhancement rate of the brightness range to which the pixel's brightness value belongs.

[0085] In one possible implementation, the fusion module is specifically used to fuse the brightness values ​​of the infrared brightness mapping image, the infrared light image, and the visible light image according to the following formula, and to superimpose the color components of the visible light image to obtain a fused image:

[0086] Fusion(i,j)=w5IR(i,j)+w6IRmappingChange(i,j)+w7Color(i,j)

[0087] Wherein, Color(i,j) is the brightness value of the pixel in the i-th row and j-th column of the visible light image, IR(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared light image, IRmappingChange(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared brightness mapping image, and w5, w6, and w7 are preset parameters.

[0088] In one possible implementation, the device further includes:

[0089] The fourth acquisition module is used to acquire the image quality parameters of the visible light image;

[0090] The judgment module is configured to: when the image quality parameter is not less than a first preset quality threshold, set w5 = w6 = 0 and w7 = 1; when the image quality parameter is not greater than a second preset quality threshold, set w5 = 1 and w6 = w7 = 0; when the image quality parameter is less than the first preset quality threshold and greater than the second preset quality threshold, set w6 > w5 and w6 > w7.

[0091] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0092] Memory, used to store computer programs;

[0093] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0094] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the methods described in the first aspect.

[0095] Beneficial effects of the embodiments in this application:

[0096] This application provides a method for fusing visible light images and infrared images. The method involves acquiring an infrared image and a visible light image; determining the reflectance parameter corresponding to each visible light pixel in the visible light image based on its brightness value; performing simulated reflectance mapping on the brightness values ​​of each infrared light pixel in the infrared image according to the reflectance parameter, resulting in an infrared brightness mapping image; and fusing the infrared brightness mapping image, the infrared image, and the visible light image to obtain a fused image. This application achieves the fusion of visible light and infrared images. By determining the reflectance parameter corresponding to each visible light pixel in the visible light image and then performing simulated reflectance mapping on the brightness values ​​of each infrared light pixel in the infrared image according to the reflectance parameter, the infrared brightness mapping image can more closely approximate the brightness distribution of the visible light image. This better captures texture areas with indistinct differences in the infrared image and effectively reduces the appearance of bright gray areas on black objects after fusion. Therefore, the fused image is more consistent with human vision, reducing noise and distortion.

[0097] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0098] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0099] Figure 1 This is a flowchart of a method for fusing visible light images and infrared light images provided in an embodiment of this application.

[0100] Figure 2a Another flowchart of the method for fusing visible light images and infrared light images provided in the embodiments of this application.

[0101] Figure 2b This is a schematic diagram illustrating the superposition of pixel values ​​of the same type of sub-pixels in adjacent channels of an initial visible light image, as provided in an embodiment of this application.

[0102] Figure 2c This is another schematic diagram illustrating the superposition of pixel values ​​of the same type of sub-pixels in adjacent channels of an initial visible light image, as provided in an embodiment of this application.

[0103] Figure 3a This is another flowchart illustrating the method for fusing visible light images and infrared light images provided in the embodiments of this application.

[0104] Figure 3b This is a schematic diagram illustrating the superposition of pixel values ​​of sub-pixels at the same position in the first initial visible light image and each of the second initial visible light images, as provided in an embodiment of this application.

[0105] Figure 4 This is another flowchart illustrating the method for fusing visible light images and infrared light images provided in the embodiments of this application.

[0106] Figure 5 A schematic diagram of the structure of the device for fusing visible light images and infrared light images provided in the embodiments of this application.

[0107] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0108] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0109] Because infrared images cannot capture textured areas with indistinct differences in infrared images well, and visible light images have poor signal-to-noise ratio and low brightness under low light conditions, the current technology of fusing visible light images and infrared images results in a high distortion rate of the fused image.

[0110] To address the aforementioned issues, this application provides a method, apparatus, and electronic device for fusing visible light images and infrared light images, which will be described in detail below.

[0111] The method for fusing visible light images and infrared light images in this application embodiment can be implemented by an electronic device with image processing capabilities, such as a camera, a hard disk recorder, or a peephole device that includes a camera function.

[0112] Figure 1 This is a flowchart illustrating a method for fusing visible light and infrared light images according to an embodiment of this application. Figure 1 As shown, it includes the following steps:

[0113] Step S101: Acquire infrared light image and visible light image.

[0114] The infrared and visible light images here can be registered infrared and visible light images acquired simultaneously by a binocular camera, or images obtained by a monocular camera using a beam splitter module on the visible light sensor and the infrared light sensor respectively. Furthermore, the infrared and visible light images can be processed using relevant image processing techniques, such as image signal processing and image noise reduction.

[0115] In one example, initial infrared light image data and initial visible light image data output by an infrared light sensor and a visible light sensor are acquired respectively. The initial infrared light image data and initial visible light image data are input into an image signal processor (ISP) for processing. Then, the infrared light image and visible light image processed by the image signal processor are subjected to noise reduction processing to obtain the infrared light image and visible light image in step S101.

[0116] To facilitate subsequent fusion processing of visible light and infrared images, the visible light and infrared images here are registered images, meaning that the visible light and infrared images have the same resolution and the same objects are in the same position in both images. The image registration process can refer to the registration process in the prior art, and is not limited in this application.

[0117] Step S102: Determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image.

[0118] The reflectance parameter corresponding to a visible light pixel indicates the degree of reflection of visible light by the object surface corresponding to that visible light pixel. The reflectance parameter corresponding to a visible light pixel is positively correlated with the brightness of that visible light pixel. The greater the brightness of the visible light pixel, the greater the reflectance parameter corresponding to that visible light pixel.

[0119] In one example, a preset brightness threshold can be obtained. For each visible light pixel in a visible light image, the reflectance parameter corresponding to that visible light pixel is obtained by calculating the ratio of the brightness value of that visible light pixel to the preset brightness threshold.

[0120] In one example, the brightness value of the pixel with the highest brightness in a visible light image can be obtained to obtain the maximum brightness value; for each visible light pixel in the visible light image, the reflectivity parameter corresponding to the visible light pixel can be obtained by calculating the ratio of the brightness value of the visible light pixel to the maximum brightness value.

[0121] Specifically, the reflectance parameter corresponding to each visible light pixel can be determined according to the formula Colormapping(i,j)=Color(i,j) / color_max, where Colormapping(i,j) is the reflectance parameter corresponding to the pixel in the i-th row and j-th column of the visible light image, Color(i,j) is the brightness value of the pixel in the i-th row and j-th column of the visible light image, and color_max is the brightness value of the pixel with the highest brightness in the visible light image.

[0122] Step S103: According to the reflectance parameters corresponding to each visible light pixel, simulate reflectance mapping is performed on the brightness values ​​of each infrared light pixel in the infrared light image to obtain an infrared brightness mapping image.

[0123] Infrared images differ significantly from visible light images due to their inherent photosensitive characteristics. Their brightness distributions are inconsistent, and the brightness of an infrared image is not directly related to the brightness of visible light reflected from the object's surface. Since the human eye directly observes visible light, not infrared light, an infrared brightness mapping image can be obtained by simulating reflectance mapping of the brightness values ​​of each infrared pixel in the infrared image based on the reflectance parameters corresponding to each visible light pixel. The infrared brightness mapping image more closely approximates the brightness distribution of the visible light image. In one example, for each infrared pixel in the infrared image, the brightness value of that infrared pixel in the infrared brightness mapping image is obtained by multiplying its corresponding reflectance parameter by its brightness value in the infrared image. Here, the reflectance parameter corresponding to the infrared pixel is the same as the reflectance parameter of the visible light pixel at the same location as the infrared pixel.

[0124] Specifically, the infrared brightness mapping image can be obtained according to the formula IRmapping(i,j)=Colormapping(i,j)×IR(i,j), where IRmapping(i,j) is the brightness value of the i-th row and j-th column in the infrared brightness mapping image, Colormapping(i,j) is the reflectance parameter corresponding to the pixel in the i-th row and j-th column in the visible light image, and IR(i,j) is the brightness value of the pixel in the i-th row and j-th column in the infrared light image.

[0125] Step S104: The infrared brightness mapping image, the infrared light image, and the visible light image are fused to obtain a fused image.

[0126] A fused image is obtained by fusing an infrared brightness mapping image, an infrared light image, and a visible light image. In one example, the brightness of the infrared brightness mapping image, the infrared light image, and the visible light image can be weighted and fused according to preset fusion weights, and then the color information of the infrared light image is superimposed to obtain the fused image. In another example, taking a visible light image in YUV format, the Y component of the visible light image, the brightness of the infrared brightness mapping image, and the brightness of the infrared light image are weighted and fused to obtain a fused brightness component Y. The U and V components of the visible image are then superimposed on the fused brightness component Y to obtain the fused image.

[0127] In one example, the brightness values ​​of an infrared brightness mapping image, an infrared light image, and a visible light image can be fused according to the following formula, and the color components of the visible light image can be superimposed to obtain a fused image:

[0128] Fusion(i,j) = w5IR(i,j) + w6IRmappingChange(i,j) + w7Color(i,j), where Color(i,j) is the brightness value of the pixel in the i-th row and j-th column of the visible light image, IR(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared light image, IRmappingChange(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared brightness mapping image, and w5, w6, and w7 are preset parameters. The preset parameters w5, w6, and w7 are the fusion weights for each brightness value, where w5 is the fusion weight for the brightness values ​​of the pixel in the infrared light image, w6 is the fusion weight for the brightness values ​​of the pixel in the infrared brightness mapping image, and w7 is the fusion weight for the brightness values ​​of the pixel in the visible light image. The values ​​of w5, w6, and w7 can be customized according to the actual situation. In one example, under low light conditions, to better match the visual effect of the human eye, w6 is set to a value close to 1, while the values ​​of w5 and w7 are minimized and kept below 0.15. This is because if w5 is greater than 0.15, it will increase the overall brightness of the infrared image, which may cause black objects to appear as light gray after fusion. While a w7 value greater than 0.15 will make the visible light image more consistent with human perception, it will also result in a worse signal-to-noise ratio in the fused image, failing to achieve the purpose of fusion. After fusing the brightness values ​​of the three images, the color components U and V of the visible light image are superimposed on the fused brightness component Y to obtain the fused image.

[0129] In one possible implementation, the preset parameters w5, w6, and w7 in the above formula can be adjusted according to the quality of the current visible light image to select the final output image. In one example, the image quality parameters of the visible light image are obtained; if the image quality parameters are not less than a first preset quality threshold, w5 = w6 = 0 and w7 = 1; if the image quality parameters are not greater than a second preset quality threshold, w5 = 1 and w6 = w7 = 0; if the image quality parameters are less than the first preset quality threshold and greater than the second preset quality threshold, w6 > w5 and w6 > w7.

[0130] Image quality parameters for visible light images represent the quality of the visible light image. In one example, these parameters can be determined based on the image brightness and exposure parameters. When the exposure parameters are constant, the image quality parameters are positively correlated with the image brightness. The first and second preset quality thresholds can be set based on the visible light image's quality parameters and can be customized according to actual needs. For example, based on the image brightness, the first preset quality threshold can be set to 90, and the second preset quality threshold to 10. If the image quality parameters are not less than the first preset quality threshold, the visible light image quality is considered good. Therefore, w5 = w6 = 0, w7 = 1, and the visible light image brightness value is output, with U and V components superimposed to obtain the final visible light image. If the image quality parameters are not greater than the second preset quality threshold, the visible light image quality is considered poor, and color information cannot be obtained. Therefore, w5 = 1, w6 = w7 = 0, and the infrared image brightness value is output, with the U and V components set to 0 to obtain the final infrared image. If the image quality parameter is less than the first preset quality threshold and greater than the second preset quality threshold, it indicates that the resolution and signal-to-noise ratio of the visible light image are poor, but the color information is good. Let w6 > w5 and w6 > w7, output the brightness value of the fused image, and superimpose the U and V components of the visible light image to finally output the fused image.

[0131] In this embodiment, an infrared image and a visible light image are acquired. Based on the brightness value of each visible light pixel in the visible light image, a reflectance parameter corresponding to each visible light pixel is determined. According to the reflectance parameter corresponding to each visible light pixel, a simulated reflectance mapping is performed on the brightness value of each infrared light pixel in the infrared light image to obtain an infrared brightness mapping image. The infrared brightness mapping image, the infrared light image, and the visible light image are then fused to obtain a fused image. This achieves the fusion of the visible light image and the infrared light image. By determining the reflectance parameter corresponding to each visible light pixel in the visible light image and then performing a simulated reflectance mapping on the brightness value of each infrared light pixel in the infrared light image according to the reflectance parameter, the brightness distribution of the infrared light image can be made closer to that of the visible light image. This better captures texture areas with indistinct differences in the infrared light image and effectively reduces the appearance of bright gray on black objects after fusion. Therefore, the fused image obtained by fusing the infrared brightness mapping image, the infrared light image, and the visible light image is more consistent with human vision and can reduce the distortion rate of the fused image. Furthermore, it can adjust the preset parameters in the fusion formula based on the quality parameters of the visible light image, thereby selecting the final output image and maximizing the acquisition of the best image presentation effect.

[0132] Infrared images, under infrared illumination and without filters, offer good signal-to-noise ratio and sharpness, but lack color information. Visible images, on the other hand, suffer from poor signal-to-noise ratio and low brightness. Therefore, improving the quality of the visible image before fusion is a crucial factor in determining the success of the fusion process. Thus, before acquiring both infrared and visible images, the initial visible image can be processed to obtain the final visible image. (See [link to documentation]). Figure 2a , Figure 2a Another flowchart of the method for fusing visible light images and infrared light images provided in the embodiments of this application includes the following steps:

[0133] Step S201: Acquire the infrared light image and the initial visible light image collected at the same time.

[0134] The initial visible light image can be a visible light image directly acquired by an image acquisition device.

[0135] Step S202: The pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image are superimposed to obtain the visible light image.

[0136] The initial visible light image output by the image acquisition device can be stored in a memory. Within the memory, pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image are superimposed using a preset superposition method to obtain the final visible light image. The memory can be a read-only memory, a random access memory, etc. For example, it could be a double-data-rate synchronous dynamic random access memory (DDR). Pixel values ​​represent a pixel's brightness information, reflectance (transmission) density information, etc. Various preset superposition methods can be used, selected according to actual needs. For example, 1×2, 2×2, or 2×3 superposition methods can be used to superimpose pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image. A 1×2 superposition method can be understood as selecting one row and two columns of the same type of sub-pixels in the initial visible light image and superimposing their pixel values, where the channels of the two columns of the same type of sub-pixels are adjacent. The format of the initial visible light image can be various and is not limited here.

[0137] In one example, for each pixel region in the initial visible light image, the pixel values ​​of all sub-pixels of the same type in that pixel region are fused to obtain the corresponding pixel in the visible light image. Each pixel region includes n×m adjacent pixels. In a possible embodiment, the initial visible light image is in Bayer format, and the brightness values ​​of the initial visible light image are superimposed using a 2×2 superposition method. The visible light image can be obtained using the following formula:

[0138] R' a，b =w1R 2a-1，2b-1 +w2R 2a-1，2b+1 +w3R 2a+1，2b-1 +w4R 2a+1，2b+1

[0139] G' a，c =w1G 2a-1，2c-2 +w2G 2a-1，2c +w3G 2a+1，2c +w4G 2a+1，2c

[0140] G' c，a =w1G 2c-2，2a-1 +w2G 2c-2，2a+1 +w3G 2c，2a-1 +w4G 2c，2a+1

[0141] B' c，d =w1B 2c-2，2d-2 +w2B 2c-2，2d +w3B 2c，2d-2 +w4B 2c，2d

[0142] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R'a，b G' represents the value of pixel R in row a and column b of a visible light image. a，c B' represents the value of pixel G in row a, column c of a visible light image. c，d R represents the value of pixel B in row c and column d of a visible light image. 2a-1，2b-1 G represents the value of pixel R in row 2a-1 and column 2b-1 of the initial visible light image. 2a-1，2c B represents the value of pixel G in row 2a-1, column 2c of the initial visible light image. 2c，2d This represents the value of pixel B in row 2c and column 2d of the initial visible light image. w1, w2, w3, and w4 are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1. Specifically, as shown... Figure 2b The diagram shows the superposition of pixel values ​​of the same type of sub-pixels in adjacent channels of an initial visible light image. Using the formula above, the visible light image can be represented as:

[0143] R' 11 =w1R 11 +w2R 13 +w3R 31 +w4R 33

[0144] G' 12 =w1G 12 +w2G 14 +w3G 32 +w4G 34

[0145] G' 21 =w1G 21 +w2G 23 +w3G 41 +w4G 43

[0146] B' 21 =w1B 22 +w2B 24 +w3B 42 +w4B 44

[0147] Where w1, w2, w3, and w4 represent the weights of each superimposed component, which can be adjusted according to actual needs. The weights of each channel in the superimposed image must be consistent; generally, w1 = w2 = w3 = w4. During superimposition, the brightness value of each pixel in each channel of the initial visible light image is used only once. Figure 2b As shown, this overlay method reduces the resolution of the initial visible light image.

[0148] In another example, for each sub-pixel in the initial visible light image, the pixel value of that sub-pixel is fused with the pixel value of its adjacent sub-pixels of the same type to obtain the corresponding sub-pixel in the visible light image. Here, for any given sub-pixel, its adjacent sub-pixels of the same type are those sub-pixels that are adjacent in position to the given sub-pixel. In one possible embodiment, the initial visible light image is in Bayer format, and the brightness values ​​of the initial visible light image are superimposed using a 2×2 superposition method. The visible light image can be obtained using the following formula:

[0149] R' a，b =w1R a，b +w2R a+2，b +w3R a，b+2 +w4R a+2，b+2

[0150] G' a，c =w1G a，c +w2G a+2，c +w3G a，c+2 +w4G a+2，c+2

[0151] G' c，a =w1G c，a +w2G c+2，a +w3G c，a+2 +w4G c+2，a+2

[0152] B' c，d =w1B c，d +w2B c+2，d +w3B c，d+2 +w4B c+2，d+2

[0153] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of a visible light image. a，c B' represents the value of pixel G in row a, column c of a visible light image. c，d R represents the value of pixel B in row c and column d of a visible light image. a，b G represents the value of pixel R in row a and column b of the initial visible light image. a，c B represents the value of pixel G in row a and column c of the initial visible light image. c，d This represents the value of pixel B in the c-th row and d-th column of the initial visible light image. w1, w2, w3, and w4 are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1. Specifically, as shown... Figure 2cThe diagram shows another illustration of superimposing pixel values ​​of the same type of sub-pixels in adjacent channels of an initial visible light image. Using the above formula, sub-pixels in the visible light image can be obtained. The representation of sub-pixels in a partial visible light image is as follows:

[0154] R' 11 =w1R 11 +w2R 13 +w3R 31 +w4R 33

[0155] G' 12 =w1G 12 +w2G 14 +w3G 32 +w4G 34

[0156] G' 21 =w1G 21 +w2G 23 +w3G 41 +w4G 43

[0157] B' 22 =w1B 22 +w2B 24 +w3B 42 +w4B 44

[0158] R' 13 =w1R 13 +w2R 15 +w3R 33 +w4R 35

[0159] G' 14 =w1G 14 +w2G 16 +w3G 34 +w4G 36

[0160] G' 23 =w1G 23 +w2G 25 +w3G 43 +w4G 45

[0161] B' 24 =w1B 24 +w2B 26 +w3B 44 +w4B 46

[0162] During overlay, the brightness value of each pixel in each channel of the initial visible light image is used multiple times, such as R' 11 and R' 13 R was used during the stacking. 13 ,like Figure 2c As shown, this overlay method does not reduce the resolution of the original visible light image.

[0163] When superimposing the brightness values ​​of the initial visible light image, brightness value overflow may occur. Therefore, the corresponding data bits of the memory need to be expanded to ensure that the brightness does not overflow. For example, in an RGB image, the brightness value range of each color is (0, 255). After superimposing the brightness values ​​of the initial visible light image, the value of a certain pixel may exceed 255, such as 300. The corresponding data bits of the memory storing the initial visible light image need to be expanded so that it can store an image with a brightness value of 300.

[0164] Step S101: Acquire infrared light image and visible light image.

[0165] Step S102: Determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image.

[0166] Step S103: According to the reflectance parameters corresponding to each visible light pixel, simulate reflectance mapping is performed on the brightness values ​​of each infrared light pixel in the infrared light image to obtain an infrared brightness mapping image.

[0167] Step S104: The infrared brightness mapping image, the infrared light image, and the visible light image are fused to obtain a fused image.

[0168] The relevant explanations for steps S101 to S104 are as follows: Figure 1 The embodiments shown have been described in detail and will not be repeated here.

[0169] In this embodiment, the pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image are superimposed, which can improve the brightness and signal-to-noise ratio of the initial visible light image. The superposition method is diversified and can be selected according to actual needs. There are no fixed requirements for the format of the initial visible light image. Visible light images of various formats can use the above superposition method. This superposition method is more flexible, ensures universality, improves the quality of the visible light image, and enables the subsequent fusion of visible light image and infrared light image, resulting in a better fused image effect.

[0170] Figure 3a This is another flowchart illustrating a method for fusing visible light and infrared light images provided in an embodiment of this application. Figure 3a As shown, it includes the following steps:

[0171] Step S301: Acquire the infrared light image and the first initial visible light image collected at the same time.

[0172] Step S302: Among the initial visible light images whose acquisition time is within a preset time difference range from the acquisition time of the first initial visible light image, a preset number of initial visible light images are selected to obtain a preset number of second initial visible light images.

[0173] The preset time difference range can be customized according to actual needs, such as setting it to 0.01 seconds. The preset number of initial visible light images can also be customized according to actual needs, such as selecting 5, 6, 7, etc. The preset number of second initial visible light images can be selected according to resolution, brightness, etc. The purpose is to obtain multiple frames of initial visible light images within the preset time difference range.

[0174] Step S303: The pixel values ​​of the sub-pixels at the same position in the first initial visible light image and each of the second initial visible light images are superimposed to obtain a visible light image.

[0175] For example, the pixel values ​​in the first row and first column of the initial visible light image from the first frame, second frame, third frame up to the Nth frame are superimposed to obtain the pixel values ​​in the first row and first column of the visible light image, where N is a positive integer greater than 1. Figure 3b This is a schematic diagram illustrating the superposition of pixel values ​​of sub-pixels at the same position in the first initial visible light image and each of the second initial visible light images, as shown below. Figure 3b As shown, the pixel values ​​of some visible light images can be represented by the following formula:

[0176] R' 11 =w1R1 11 +…+w n Rn 11

[0177] G' 12 =w1G1 12 ++…+w n Gn 12

[0178] G' 21 =w1G1 21 +…+w n Gn 21

[0179] B' 22 =w1B1 22 +…+w n Bn 22

[0180] Among them, w1…w nThe weights representing each superimposed component can be adjusted according to actual needs. When superimposing pixel values ​​from multiple initial visible light images cannot meet the actual requirements, a weight can be assigned to each initial visible light image. Figure 2a The superposition method shown combines the 1×2, 2×2, 2×3, and other superposition methods with a multi-frame superposition method. For example, the initial visible light images of the first frame, second frame, third frame, up to the Nth frame are superimposed using... Figure 2a The embodiments shown employ 1×2, 2×2, 2×3, and other superposition methods. Then, the pixel values ​​of sub-pixels at the same position in the initial visible light images of the first, second, third, and Nth frames after single-frame superposition are superimposed. Alternatively, the pixel values ​​of sub-pixels at the same position in the initial visible light images of the first, second, third, and Nth frames are first superimposed, and then the resulting visible light image is superimposed. Figure 2a The 1×2, 2×2, 2×3 and other stacking methods shown in the embodiments are used for stacking.

[0181] Step S101: Acquire infrared light image and visible light image.

[0182] Step S102: Determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image.

[0183] Step S103: According to the reflectance parameters corresponding to each visible light pixel, simulate reflectance mapping is performed on the brightness values ​​of each infrared light pixel in the infrared light image to obtain an infrared brightness mapping image.

[0184] Step S104: The infrared brightness mapping image, the infrared light image, and the visible light image are fused to obtain a fused image.

[0185] The relevant explanations for steps S101 to S104 are as follows: Figure 1 The embodiments shown have been described in detail and will not be repeated here.

[0186] In this embodiment, a preset number of initial visible light images are selected from among those whose acquisition time is within a preset time difference range from the acquisition time of the first initial visible light image, to obtain a preset number of second initial visible light images. The pixel values ​​of sub-pixels at the same position in the first initial visible light image and each of the second initial visible light images are superimposed to obtain the visible light image. A combination of single-frame superposition and multi-frame superposition can be used, which is more flexible and can maximize the pixel value of the initial visible light image, enabling the subsequent fusion of visible light and infrared light images to achieve a better fused image effect.

[0187] Figure 4Another flowchart of the method for fusing visible light images and infrared light images provided in the embodiments of this application is shown below. Figure 4 As shown, it includes the following steps:

[0188] Step S101: Acquire infrared light image and visible light image.

[0189] Step S102: Determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image.

[0190] Step S103: According to the reflectance parameters corresponding to each visible light pixel, simulate reflectance mapping is performed on the brightness values ​​of each infrared light pixel in the infrared light image to obtain an infrared brightness mapping image.

[0191] Step S401: Divide the brightness value range of pixels in the current infrared brightness mapping image into multiple brightness intervals, and adjust the brightness value range and brightness enhancement rate of each brightness interval so that the adjusted brightness value range of each brightness interval is within the preset brightness interval range.

[0192] Based on the reflectivity parameters corresponding to visible light pixels, a simulated reflectivity mapping is performed on the brightness values ​​of each infrared pixel in the infrared image. The resulting infrared brightness mapping image may have a slight decrease in brightness. Considering the characteristics of human vision, brightness enhancement is performed according to different brightness ranges to increase the brightness values ​​within each brightness range of the infrared brightness mapping image, thus ensuring the fusion effect. The preset brightness range can be customized according to actual needs. For example, setting it to (30, 200) will adjust the brightness value of each pixel in the current infrared brightness mapping image to meet the brightness value range of (30, 200).

[0193] Step S402: For each pixel in the current infrared brightness mapping image, the brightness value of the pixel is increased by using the brightness increase rate of the brightness range to which the brightness value of the pixel belongs.

[0194] For example, if the brightness value of a pixel is 25 and the brightness enhancement rate of the brightness range to which the brightness value of the pixel belongs is 20%, then the brightness value of the pixel after brightness enhancement is 30, so that it can meet the preset brightness value range.

[0195] Step S104: The infrared brightness mapping image, the infrared light image, and the visible light image are fused to obtain a fused image.

[0196] In this embodiment, the brightness value range of pixels in the current infrared brightness mapping image is divided into multiple brightness intervals. The brightness value range and brightness enhancement rate of each brightness interval are adjusted so that the adjusted brightness value range of each brightness interval is within a preset brightness range. For each pixel in the current infrared brightness mapping image, the brightness value of the pixel is enhanced using the brightness enhancement rate of the brightness interval to which the pixel's brightness value belongs. This enhancement of the brightness of the infrared brightness mapping image ensures the effectiveness of the fused image when fusing the infrared brightness mapping image, the infrared light image, and the visible light image, and reduces the distortion rate of the fused image.

[0197] Figure 5 A schematic diagram of the structure of the apparatus for fusing visible light images and infrared images provided in an embodiment of this application. (Refer to...) Figure 5 The device includes: a first acquisition module 501, a determination module 502, a mapping module 503, and a fusion module 504, wherein,

[0198] The first acquisition module 501 is used to acquire infrared light images and visible light images;

[0199] The determining module 502 is used to determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image;

[0200] The mapping module 503 is used to simulate reflectance mapping of the brightness value of each infrared pixel in the infrared light image according to the reflectance parameter corresponding to each visible light pixel, so as to obtain an infrared brightness mapping image.

[0201] The fusion module 504 is used to fuse the infrared brightness mapping image, the infrared light image and the visible light image to obtain a fused image.

[0202] In this embodiment, an infrared image and a visible light image are acquired. Based on the brightness value of each visible light pixel in the visible light image, a reflectance parameter corresponding to each visible light pixel is determined. According to the reflectance parameter corresponding to each visible light pixel, a simulated reflectance mapping is performed on the brightness value of each infrared light pixel in the infrared light image to obtain an infrared brightness mapping image. The infrared brightness mapping image, the infrared light image, and the visible light image are then fused to obtain a fused image. By determining the reflectance parameter corresponding to each visible light pixel in the visible light image and then performing a simulated reflectance mapping on the brightness value of each infrared light pixel in the infrared light image according to the reflectance parameter, the infrared light image can be made closer to the visible light image in terms of brightness distribution. This better captures texture areas with indistinct differences in the infrared light image and effectively reduces the appearance of light gray on black objects after fusion. Therefore, the fused image obtained by fusing the infrared brightness mapping image, the infrared light image, and the visible light image is more consistent with human vision and can reduce the distortion rate of the fused image.

[0203] In one possible implementation, the device further includes:

[0204] The second acquisition module is used to acquire the infrared light image and the initial visible light image collected at the same time.

[0205] The first overlay module is used to overlay the pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image to obtain the visible light image.

[0206] In one possible implementation, the first overlay module is specifically used to fuse the pixel values ​​of each sub-pixel of the same type in each pixel region of the initial visible light image to obtain the pixel corresponding to the pixel region in the visible light image, wherein each pixel region includes n×m adjacent pixels.

[0207] In one possible implementation, the initial visible light image is in Bayer format; the first overlay module is specifically used for:

[0208] Based on the initial visible light image, the visible light image is obtained using the following formula:

[0209] R' a，b =w1R 2a-1，2b-1 +w2R 2a-1，2b+1 +w3R 2a+1，2b-1 +w4R 2a+1，2b+1

[0210] G' a，c =w1G 2a-1，2c-2 +w2G 2a-1，2c +w3G 2a+1，2c +w4G2a+1，2c

[0211] G' c，a =w1G 2c-2，2a-1 +w2G 2c-2，2a+1 +w3G 2c，2a-1 +w4G 2c，2a+1

[0212] B' c，d =w1B 2c-2，2d-2 +w2B 2c-2，2d +w3B 2c，2d-2 +w4B 2c，2d

[0213] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. 2a-1，2b-1 G represents the value of pixel R in row 2a-1 and column 2b-1 of the initial visible light image. 2a-1，2c B represents the value of pixel G in row 2a-1, column 2c of the initial visible light image. 2c，2d The value of pixel B in row 2c and column 2d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

[0214] In one possible implementation, the first overlay module is specifically used to fuse the pixel value of each sub-pixel in the initial visible light image with the pixel value of its adjacent sub-pixels of the same type to obtain the sub-pixel in the visible light image corresponding to the sub-pixel, wherein, for any sub-pixel, the adjacent sub-pixels of the same type are sub-pixels of the same type and adjacent in position as the sub-pixel.

[0215] In one possible implementation, the initial visible light image is in Bayer format; the first overlay module is specifically used for:

[0216] Based on the initial visible light image, the visible light image is obtained using the following formula:

[0217] R' a，b =w1R a，b +w2R a+2，b +w3R a，b+2 +w4R a+2，b+2

[0218] G' a，c =w1Ga，c +w2G a+2，c +w3G a，c+2 +w4G a+2，c+2

[0219] G' c，a =w1G c，a +w2G c+2，a +w3G c，a+2 +w4G c+2，a+2

[0220] B' c，d =w1B c，d +w2B c+2，d +w3B c，d+2 +w4B c+2，d+2

[0221] Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. a，b G represents the value of pixel R in row a and column b of the initial visible light image. a，c B represents the value of pixel G in row a and column c of the initial visible light image. c，d The value of pixel B in row c and column d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

[0222] In one possible implementation, the device further includes:

[0223] The third acquisition module is used to acquire the infrared light image and the first initial visible light image collected at the same time.

[0224] The selection module is used to select a preset number of initial visible light images from among the initial visible light images whose acquisition time is within a preset time difference range from the acquisition time of the first initial visible light image, so as to obtain a preset number of second initial visible light images.

[0225] The second overlay module is used to overlay the pixel values ​​of the first initial visible light image with the pixel values ​​of the sub-pixels at the same position in each of the second initial visible light images to obtain the visible light image.

[0226] In one possible implementation, the determining module is specifically used to obtain the brightness value of the pixel with the highest brightness in the visible light image, and obtain the maximum brightness value; for each visible light pixel in the visible light image, the reflectivity parameter corresponding to the visible light pixel is obtained by calculating the ratio of the brightness value of the visible light pixel to the maximum brightness value.

[0227] In one possible implementation, the mapping module is specifically used to calculate the brightness value of each infrared pixel in the infrared light image by multiplying the reflectance parameter corresponding to the infrared pixel with the brightness value of the infrared pixel in the infrared light image; wherein, the reflectance parameter corresponding to the infrared pixel is the reflectance parameter corresponding to the visible light pixel at the same position as the infrared pixel.

[0228] In one possible implementation, the device further includes:

[0229] The adjustment module is used to divide the brightness value range of pixels in the current infrared brightness mapping image into multiple brightness intervals, and adjust the brightness value range and brightness enhancement rate of each brightness interval so that the brightness value range of each adjusted brightness interval is within the preset brightness interval range.

[0230] The brightness enhancement module is used to enhance the brightness of each pixel in the current infrared brightness mapping image by using the brightness enhancement rate of the brightness range to which the pixel's brightness value belongs.

[0231] In one possible implementation, the fusion module is specifically used to fuse the brightness values ​​of the infrared brightness mapping image, the infrared light image, and the visible light image according to the following formula, and to superimpose the color components of the visible light image to obtain a fused image:

[0232] Fusion(i,j)=w5IR(i,j)+w6IRmappingChange(i,j)+w7Color(i,j)

[0233] Wherein, Color(i,j) is the brightness value of the pixel in the i-th row and j-th column of the visible light image, IR(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared light image, IRmappingChange(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared brightness mapping image, and w5, w6, and w7 are preset parameters.

[0234] In one possible implementation, the device further includes:

[0235] The fourth acquisition module is used to acquire the image quality parameters of the visible light image;

[0236] The judgment module is configured to: when the image quality parameter is not less than a first preset quality threshold, set w5 = w6 = 0 and w7 = 1; when the image quality parameter is not greater than a second preset quality threshold, set w5 = 1 and w6 = w7 = 0; when the image quality parameter is less than the first preset quality threshold and greater than the second preset quality threshold, set w6 > w5 and w6 > w7.

[0237] This application also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0238] Memory 603 is used to store computer programs;

[0239] When the processor 601 executes the program stored in the memory 603, it implements the above-described method steps for fusing visible light images and infrared light images.

[0240] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0241] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0242] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0243] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0244] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the method steps for fusing any of the above-described visible light images and infrared light images.

[0245] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the methods for fusing visible light images and infrared light images in the above embodiments.

[0246] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0247] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0248] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and electronic device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0249] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for fusing visible light images and infrared light images, characterized in that, The method includes: Acquire infrared light images and initial visible light images collected at the same time; For each pixel region in the initial visible light image, the pixel values ​​of each sub-pixel of the same type in the pixel region are fused to obtain the pixel corresponding to the pixel region in the visible light image, wherein each pixel region includes n×m adjacent pixels; Based on the brightness value of each visible light pixel in the visible light image, the reflectivity parameter corresponding to each visible light pixel is determined respectively; Based on the reflectance parameters corresponding to each visible light pixel, the brightness values ​​of each infrared light pixel in the infrared light image are simulated and reflected to obtain an infrared brightness mapping image. The infrared brightness mapping image, the infrared light image, and the visible light image are fused to obtain a fused image.

2. The method according to claim 1, characterized in that, The initial visible light image is in Bayer format; for each pixel region in the initial visible light image, the pixel values ​​of all sub-pixels of the same type in that pixel region are fused to obtain the pixel corresponding to that pixel region in the visible light image, including: Based on the initial visible light image, the visible light image is obtained using the following formula: R' a，b =w1R 2a-1，2b-1 +w2R 2a-1，2b+1 +w3R 2a+1，2b-1 +w4R 2a+1，2b+1 G’ a，c =w1G 2a-1，2c-2 +w2G 2a-1，2c +w3G 2a+1，2c +w4G 2a+1，2c G’ c，a =w1G 2c-2，2s-1 +w2G 2c-2，2a+1 +w3G 2c，2a-1 +w4G 2c，2a+1 B’ c，d =w1B 2v-2，2d-2 +w2B 2c-2，2d +w3B 2c，2d-2 +w4B 2c，2d Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. 2a-1，2b-1 G represents the value of pixel R in row 2a-1 and column 2b-1 of the initial visible light image. 2a-1，2c B represents the value of pixel G in row 2a-1, column 2c of the initial visible light image. 2c，2d The value of pixel B in row 2c and column 2d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

3. The method according to claim 1, characterized in that, The step of superimposing the pixel values ​​of the same type of sub-pixels in adjacent channels of the initial visible light image to obtain the visible light image includes: For each sub-pixel in the initial visible light image, the pixel value of the sub-pixel is fused with the pixel value of its adjacent sub-pixels of the same type to obtain the corresponding sub-pixel in the visible light image. For any sub-pixel, the adjacent sub-pixels of the same type are sub-pixels of the same type and adjacent in position as the sub-pixel.

4. The method according to claim 3, characterized in that The initial visible light image is in Bayer format; the step of fusing the pixel value of each sub-pixel in the initial visible light image with the pixel value of its adjacent sub-pixels of the same type to obtain the corresponding sub-pixel in the visible light image includes: Based on the initial visible light image, the visible light image is obtained using the following formula: R' a，b =w1R a，b +w2R a+2，b +w3R a，b+2 +W4R a+2，b+2 G’ a，c =W1G a，c +W2G a+2，c +W3G a，c+2 +w4G a+2，c+2 G’ c，a =w1G c，a +w2G c+2，a +w3G c，a+2 +w4G c+2，a+2 B’ c，d =w1B c，d +w2B c+2，d +w3B c，d+2 +w4B c+2，d+2 Where a is an odd number, b is an odd number, c is an even number, d is an even number, and R' a，b G' represents the value of pixel R in row a and column b of the visible light image. a，c B' represents the value of pixel G in the a-th row and c-th column of the visible light image. c，d R represents the value of pixel B in the c-th row and d-th column of the visible light image. a，b G represents the value of pixel R in row a and column b of the initial visible light image. a，c B represents the value of pixel G in row a and column c of the initial visible light image. c，d The value of pixel B in row c and column d of the initial visible light image is represented by w1, w2, w3, and w4, which are preset coefficients, and the sum of w1, w2, w3, and w4 is greater than 1.

5. The method according to claim 1, characterized in that, Prior to the steps of acquiring infrared and visible light images, the method further includes: Acquire the infrared light image and the first initial visible light image collected at the same time; Among the initial visible light images whose acquisition time is within a preset time difference range from the acquisition time of the first initial visible light image, a preset number of initial visible light images are selected to obtain a preset number of second initial visible light images; The visible light image is obtained by superimposing the pixel values ​​of the sub-pixels at the same position in each of the first initial visible light images and each of the second initial visible light images.

6. The method according to claim 1, characterized in that, The step of determining the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image includes: Obtain the brightness value of the pixel with the highest brightness in the visible light image to obtain the maximum brightness value; For each visible light pixel in the visible light image, the reflectivity parameter corresponding to that visible light pixel is obtained by calculating the ratio of the brightness value of that visible light pixel to the maximum brightness value.

7. The method according to claim 1, characterized in that, The step of performing simulated reflectance mapping on the brightness values ​​of each infrared pixel in the infrared image according to the reflectance parameters corresponding to each visible light pixel to obtain an infrared brightness mapping image includes: For each infrared pixel in the infrared light image, the brightness value of the infrared pixel in the infrared brightness mapping image is obtained by multiplying the reflectance parameter corresponding to the infrared pixel with the brightness value of the infrared pixel in the infrared light image; wherein, the reflectance parameter corresponding to the infrared pixel is the reflectance parameter corresponding to the visible light pixel at the same position as the infrared pixel.

8. The method according to claim 1, characterized in that, After performing simulated reflectance mapping on the brightness values ​​of each infrared pixel in the infrared image according to the reflectance parameters corresponding to each visible light pixel to obtain an infrared brightness mapping image, the method further includes: The brightness value range of pixels in the current infrared brightness mapping image is divided into multiple brightness intervals. The brightness value range and brightness enhancement rate of each brightness interval are adjusted so that the brightness value range of each adjusted brightness interval is within the preset brightness interval range. For each pixel in the current infrared brightness mapping image, the brightness value of the pixel is increased by using the brightness increase rate of the brightness range to which the brightness value of the pixel belongs.

9. The method according to claim 1, characterized in that, The step of fusing the infrared brightness mapping image, the infrared light image, and the visible light image to obtain a fused image includes: The brightness values ​​of the infrared brightness mapping image, the infrared light image, and the visible light image are fused according to the following formula, and the color components of the visible light image are superimposed to obtain a fused image; Fusion(i,j)=w5IR(i,j)+w6IRmappingChange(i,j)+w7Color(i,j) Wherein, Color(i,j) is the brightness value of the pixel in the i-th row and j-th column of the visible light image, IR(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared light image, IRmappingChange(i,j) is the brightness value of the pixel in the i-th row and j-th column of the infrared brightness mapping image, and w5, w6, and w7 are preset parameters.

10. The method according to claim 9, characterized in that, The method further includes: Obtain the image quality parameters of the visible light image; If the image quality parameters are not less than the first preset quality threshold, let w5 = w6 = 0 and w7 = 1; If the image quality parameters are not greater than the second preset quality threshold, let w5 = 1, w6 = w7 = 0; When the image quality parameter is less than the first preset quality threshold and greater than the second preset quality threshold, w6 > w5 and w6 > w7.

11. An apparatus for fusing visible light images and infrared light images, characterized in that, The device includes: The second acquisition module is used to acquire the infrared light image and the initial visible light image collected at the same time. The first overlay module is used to fuse the pixel values ​​of each sub-pixel of the same type in each pixel region in the initial visible light image to obtain the pixel corresponding to the pixel region in the visible light image, wherein each pixel region includes n×m adjacent pixels. The determining module is used to determine the reflectance parameter corresponding to each visible light pixel based on the brightness value of each visible light pixel in the visible light image; The mapping module is used to simulate reflectance mapping of the brightness value of each infrared pixel in the infrared light image according to the reflectance parameter corresponding to each visible light pixel, so as to obtain an infrared brightness mapping image. The fusion module is used to fuse the infrared brightness mapping image, the infrared light image, and the visible light image to obtain a fused image.

12. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-10.

Images