An image fusion method, apparatus, device and medium

Abstract

Embodiments of the present application provide an image fusion method, device, equipment and medium, the method comprises: acquiring infrared thermal imaging image and visible light image at the same time, respectively determining the corresponding gray information of two images; determining the edge pixel point elimination threshold corresponding to two images according to the gray information; respectively performing edge detection on two images to determine the corresponding edge pixel points; screening the edge pixel points corresponding to two images according to the edge pixel point elimination threshold corresponding to two images, obtaining the screened edge pixel points corresponding to two images; image fusion is carried out after two images are registered according to the screened edge pixel points. Through the gray information of infrared thermal imaging image and visible light image, the edge pixel point elimination threshold of two images is determined respectively, and then the edge pixel points are screened, and image registration and fusion are carried out based on the screened edge pixel points. The imaging effect and accuracy of image fusion are improved, and the quality of the final image is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion, in particular to an image fusion method and an image fusion device. BACKGROUND

[0002] Infrared and visible light fusion technology is to fuse infrared images and visible light images, which overcomes the shortcomings of a single sensor. Infrared images can perceive the heat radiation of an object, while visible light images can perceive the shape, color and other information of an object. Fusing the two can make up for their respective shortcomings and improve the recognition and detection capabilities of images. However, on an embedded platform, in order to reduce product costs, a high-performance computing platform is not used, and both infrared and visible light use low-cost cameras, which can result in poor thermal imaging effects, blurred contours of people / objects in infrared thermal images, and sometimes even the inability to distinguish people from other heat sources. SUMMARY

[0003] In view of the above problems, the embodiments of the present application are proposed to provide an image fusion method, device, electronic equipment and storage medium which overcome the above problems or at least partially solve the above problems.

[0004] According to a first aspect of the embodiments of the present application, an image fusion method is provided, and the method comprises:

[0005] Obtaining an infrared thermal imaging image and a visible light image at the same time, and determining the gray scale information of the infrared thermal imaging image and the visible light image, respectively;

[0006] Determining a first edge pixel point rejection threshold of the infrared thermal imaging image according to the gray scale information of the infrared thermal imaging image, and determining a second edge pixel point rejection threshold of the visible light image according to the gray scale information of the visible light image;

[0007] Respectively performing edge detection on the infrared thermal imaging image and the visible light image, and respectively determining the first edge pixel point of the infrared thermal imaging image and the second edge pixel point in the visible light image according to the edge detection results;

[0008] According to the first edge pixel point rejection threshold, screening the first edge pixel point of the infrared thermal imaging image to obtain a third edge pixel point of the infrared thermal imaging image; and according to the second edge pixel point rejection threshold, screening the second edge pixel point of the visible light image to obtain a fourth edge pixel point of the visible light image;

[0009] According to the third edge pixel point of the infrared thermal imaging image and the fourth edge pixel point of the visible light image, the infrared thermal imaging image and the visible light image are registered, respectively obtaining a registered image of the infrared thermal imaging image and a registered image of the visible light image;

[0010] According to the registered image of the infrared thermal imaging image and the registered image of the visible light image, fusion is performed.

[0011] Optionally, the determination of the gray scale information of the infrared thermal imaging image and the visible light image comprises:

[0012] The gray scale information of the infrared thermal imaging image and the visible light image under different edge pixel point rejection thresholds is determined; the gray scale information comprises a foreground layer gray scale average value, a background layer gray scale average value, and a gray scale average value of the whole image.

[0013] Optionally, the determination of the first edge pixel point rejection threshold of the infrared thermal imaging image according to the gray scale information of the infrared thermal imaging image, and the determination of the second edge pixel point rejection threshold of the visible light image according to the gray scale information of the visible light image, comprises:

[0014] According to the gray scale information of the infrared thermal imaging image under different edge pixel point rejection thresholds, a first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image is determined;

[0015] The edge pixel point rejection threshold when the first inter-class variance is maximized is taken as the first edge pixel point rejection threshold by solving the first inter-class variance;

[0016] According to the gray scale information of the visible light image under different edge pixel point rejection thresholds, a second inter-class variance between the foreground layer and the background layer of the visible light image is determined;

[0017] The edge pixel point rejection threshold when the second inter-class variance is maximized is taken as the second edge pixel point rejection threshold by solving the second inter-class variance.

[0018] Optionally, the determination of the first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image according to the gray scale information of the infrared thermal imaging image under different edge pixel point rejection thresholds comprises:

[0019] According to the foreground layer gray scale average value, the background layer gray scale average value, and the gray scale average value of the whole image of the infrared thermal imaging image, the first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image is determined according to the following formula:

[0020] σ 2 = Pf x (M f -M) 2 +P b x (M b -M) 2

[0021] wherein σ is the inter-class variance; P f is the probability of the total number of foreground pixels occupying the total number of pixels; P b is the probability of the total number of background pixels occupying the total number of pixels; M f is the average value of the foreground layer grayscale; M b is the average value of the background layer grayscale; and M is the average value of the entire image.

[0022] Optionally, the screening of the first edge pixel points of the infrared thermal imaging image according to the first edge pixel point rejection threshold and the screening of the second edge pixel points of the visible light image according to the second edge pixel point rejection threshold, to obtain the third edge pixel points of the infrared thermal imaging image and the fourth edge pixel points of the visible light image, comprises:

[0023] rejection of the first edge pixel points of the infrared thermal imaging image that are less than the first edge pixel point rejection threshold, to obtain the third edge pixel points of the infrared thermal imaging image;

[0024] rejection of the second edge pixel points of the visible light image that are less than the second edge pixel point rejection threshold, to obtain the fourth edge pixel points of the infrared thermal imaging image.

[0025] Optionally, the fusing of the registered image of the infrared thermal imaging image and the registered image of the visible light image comprises:

[0026] edge detection of the registered image of the infrared thermal imaging image and the registered image of the visible light image, respectively;

[0027] determination of fifth edge pixel points of the registered image of the infrared thermal imaging image and sixth edge pixel points of the registered image of the visible light image according to the edge detection results;

[0028] determination of the Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the fifth edge pixel points of the registered image of the infrared thermal imaging image and the sixth edge pixel points of the registered image of the visible light image;

[0029] fusing of the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the Euclidean distance.

[0030] Optionally, the Euclidean distance comprises a first Euclidean distance and a second Euclidean distance; the determining the Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image comprises:

[0031] determining the zero-order matrix and the first-order matrix of the registered image of the infrared thermal imaging image according to the fifth edge pixel point, and determining the zero-order matrix and the first-order matrix of the registered image of the visible light image according to the sixth edge pixel point;

[0032] determining the first Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the zero-order matrix of the registered image of the infrared thermal imaging image and the zero-order matrix of the registered image of the visible light image;

[0033] determining the second Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the first-order matrix of the registered image of the infrared thermal imaging image and the first-order matrix of the registered image of the visible light image;

[0034] the fusing the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the Euclidean distance comprises:

[0035] fusing the registered image of the infrared thermal imaging image and the registered image of the visible light image when the first Euclidean distance and the second Euclidean distance simultaneously satisfy a preset threshold condition.

[0036] According to a first aspect of an embodiment of the present application, an image fusion device is provided, the device comprising:

[0037] an acquisition module configured to acquire an infrared thermal imaging image and a visible light image at the same time, and determine gray scale information of the infrared thermal imaging image and the visible light image, respectively;

[0038] a determination module configured to determine a first edge pixel point rejection threshold of the infrared thermal imaging image according to the gray scale information of the infrared thermal imaging image, and determine a second edge pixel point rejection threshold of the visible light image according to the gray scale information of the visible light image;

[0039] a detection module configured to perform edge detection on the infrared thermal imaging image and the visible light image, respectively, and determine a first edge pixel point of the infrared thermal imaging image and a second edge pixel point of the visible light image according to the edge detection results, respectively;

[0040] The screening module is configured to screen the first edge pixel points of the infrared thermal imaging image according to a first edge pixel point elimination threshold to obtain third edge pixel points of the infrared thermal imaging image, and screen the second edge pixel points of the visible light image according to a second edge pixel point elimination threshold to obtain fourth edge pixel points of the visible light image.

[0041] The registration module is configured to register the infrared thermal imaging image and the visible light image according to the third edge pixel points of the infrared thermal imaging image and the fourth edge pixel points of the visible light image to obtain a registered image of the infrared thermal imaging image and a registered image of the visible light image, respectively.

[0042] The fusion module is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image.

[0043] Optionally, the acquisition module comprises:

[0044] The first determination sub-module is configured to determine corresponding gray information of the infrared thermal imaging image and the visible light image under different edge pixel point elimination thresholds; the gray information comprises a foreground layer gray average value, a background layer gray average value, and a gray average value of the whole image.

[0045] Optionally, the determination module comprises:

[0046] The second determination sub-module is configured to determine a first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image according to the corresponding gray information of the infrared thermal imaging image under different edge pixel point elimination thresholds.

[0047] The first solving sub-module is configured to solve the first inter-class variance, and take the edge pixel point elimination threshold when the first inter-class variance is maximum as the first edge pixel point elimination threshold.

[0048] The third determination sub-module is configured to determine a second inter-class variance between the foreground layer and the background layer of the visible light image according to the corresponding gray information of the visible light image under different edge pixel point elimination thresholds.

[0049] The second solving sub-module is configured to solve the second inter-class variance, and take the edge pixel point elimination threshold when the second inter-class variance is maximum as the second edge pixel point elimination threshold.

[0050] Optionally, the second determination sub-module comprises:

[0051] The first determination unit is configured to determine a first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image according to a foreground layer average gray value, a background layer average gray value and an average gray value of the whole image of the infrared thermal imaging image, and according to the following formula:

[0052] σ 2 = P f × (M f - M) 2 + P b × (M b - M) 2

[0053] wherein σ is the inter-class variance; P f is a probability of a total number of foreground pixels accounting for a total number of pixels; P b is a probability of a total number of background pixels accounting for the total number of pixels; M f is the foreground layer average gray value; M b is the background layer average gray value; and M is the average value of the whole image.

[0054] Optionally, the screening module comprises:

[0055] The first screening submodule is configured to eliminate the first edge pixel points of the infrared thermal imaging image that are less than the first edge pixel elimination threshold value, to obtain third edge pixel points of the infrared thermal imaging image.

[0056] The second screening submodule is configured to eliminate the second edge pixel points of the visible light image that are less than the second edge pixel elimination threshold value, to obtain fourth edge pixel points of the infrared thermal imaging image.

[0057] Optionally, the fusion module comprises:

[0058] The first detection submodule is configured to perform edge detection on the registered image of the infrared thermal imaging image and the registered image of the visible light image respectively.

[0059] The fourth determination submodule is configured to determine fifth edge pixel points of the registered image of the infrared thermal imaging image and sixth edge pixel points of the registered image of the visible light image according to the edge detection results.

[0060] The fifth determination submodule is configured to determine an Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the fifth edge pixel points of the registered image of the infrared thermal imaging image and the sixth edge pixel points of the registered image of the visible light image.

[0061] The first fusion sub-module is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the Euclidean distance.

[0062] Optionally, the Euclidean distance includes a first Euclidean distance and a second Euclidean distance; and the fifth determination sub-module includes:

[0063] The second determination unit is configured to determine a zero-order matrix and a first-order matrix of the registered image of the infrared thermal imaging image according to the fifth edge pixel point, and determine a zero-order matrix and a first-order matrix of the registered image of the visible light image according to the sixth edge pixel point.

[0064] The third determination unit is configured to determine a first Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the zero-order matrix of the registered image of the infrared thermal imaging image and the zero-order matrix of the registered image of the visible light image.

[0065] The fourth determination unit is configured to determine a second Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the first-order matrix of the registered image of the infrared thermal imaging image and the first-order matrix of the registered image of the visible light image.

[0066] The first fusion sub-module includes:

[0067] The first fusion unit is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image when the first Euclidean distance and the second Euclidean distance both satisfy a preset threshold condition.

[0068] According to a third aspect of the present application, an electronic device is provided, which includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor, and when the computer program is executed by the processor, the steps of the image fusion method according to any one of the above aspects are implemented.

[0069] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the image fusion method according to any one of the above aspects are implemented.

[0070] The technical scheme provided by the embodiments of the present application can have the following beneficial effects:

[0071] The embodiment of the present application provides an image fusion method, by acquiring infrared thermal imaging images and visible light images at the same time, determining corresponding gray information of the two images respectively; determining corresponding edge pixel point elimination thresholds of the two images according to the gray information; respectively performing edge detection on the two images to determine corresponding edge pixel points; screening the corresponding edge pixel points of the two images according to the corresponding edge pixel point elimination thresholds of the two images, to obtain screened edge pixel points corresponding to the two images; and performing image registration and fusion on the two images according to the screened edge pixel points. The embodiment of the present application determines the edge pixel point elimination thresholds of the infrared thermal imaging images and the visible light images respectively, and then screens the edge pixel points, and performs image registration and fusion based on the screened edge pixel points. The imaging effect and accuracy of image fusion are improved, and the quality of the final image is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is a step flow chart of an image fusion method provided by the embodiment of the present application;

[0073] Figure 2 is a structural block diagram of an image fusion device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0075] One of the core ideas of the embodiment of the present application is that the edge pixel point elimination thresholds of the infrared thermal imaging images and the visible light images are determined respectively according to the gray information of the two images, and then the edge pixel points are screened, and the image registration and fusion are performed based on the screened edge pixel points. The imaging effect and accuracy of image fusion are improved, and the quality of the final image is ensured.

[0076] Referring to Figure 1 , a step flow chart of an image fusion method provided by the embodiment of the present application is shown, and the method can specifically include the following steps:

[0077] Step 101, acquiring infrared thermal imaging images and visible light images at the same time, and determining the gray information of the infrared thermal imaging images and the visible light images respectively;

[0078] After the infrared thermal imaging image and the visible light image at the same time are acquired, the resolution of the infrared thermal imaging image and the visible light image is unified, and the infrared thermal imaging image is denoised. Specifically, the resolution of the infrared thermal imaging image and the visible light image is 320*240, the original data resolution of the infrared thermal imaging image is 80*62, the infrared thermal imaging image is enlarged to 320*240 by using a bilinear interpolation, but the noise in the infrared thermal imaging image will affect the final fusion effect, so it is necessary to smooth the image by using a Gaussian filter to remove the noise, and a 3*3 convolution kernel is used, and the convolution kernel is set as:

[0079]

[0080] When the Gaussian filter is implemented by using a software convolution operator, the time consumption is serious, about 180 ms, so a convolutional neural network is constructed to implement the Gaussian filter, the method does not need to be trained, a 3*3 convolutional layer is constructed in the Keras framework, the weight is the above k, the weight is directly loaded into the convolutional layer by setting the weight in the Keras framework model, and finally the whole model is saved. Then, the model file is compiled into an operator stream supported by a chip for calculation, and the time consumption is about 13 ms, so that the time consumption of the algorithm is greatly reduced.

[0081] The visible light image and the denoised infrared thermal imaging image are converted into a gray image, and the gray information of the infrared thermal imaging image and the visible light image is determined according to the gray image. The gray image reduces the complexity of color information, reduces the consumption of computing resources, and is especially suitable for the limited processing capacity of an embedded platform. Through the gray image, more attention can be paid to brightness and edge features, which is crucial for subsequent edge detection, registration and fusion.

[0082] In step 102, a first edge pixel point rejection threshold of the infrared thermal imaging image is determined according to the gray information of the infrared thermal imaging image, and a second edge pixel point rejection threshold of the visible light image is determined according to the gray information of the visible light image.

[0083] In order to optimize the quality of the two images before fusion, the inaccurate edge pixel points of the infrared thermal imaging image and the visible light image are respectively determined according to the gray information of the infrared thermal imaging image and the visible light image, so as to ensure that only high-quality edge information participates in subsequent processing. In this way, the accuracy of image registration is improved, and the clarity and accuracy of the fused image are enhanced, so as to improve the effect of the final multi-modal image fusion.

[0084] In step 103, edge detection is respectively performed on the infrared thermal imaging image and the visible light image, and a first edge pixel point of the infrared thermal imaging image and a second edge pixel point of the visible light image are respectively determined according to the edge detection results.

[0085] The infrared thermal imaging image and the visible light image are subjected to edge detection through a Canny edge detection algorithm, and the Canny edge detection algorithm is specifically as follows:

[0086] First, the gradient of each pixel point is calculated using a first-order finite difference, so that two matrices of partial derivatives of the image in x and y directions can use a Sobel operator as a gradient operator, that is, the convolution kernels in x and y directions are respectively:

[0087]

[0088] Suppose that a preset image is:

[0089]

[0090] Take A 11 for example, the gradients of A 11 in x and y directions are respectively:

[0091] G x = 2A 12 - 2A 10 + A 22 + A 02 - A 20 - A 00

[0092] G y = 2A 21 - 2A 01 - A 00 - A 02 + A 20 + A 22

[0093] Then the gradient amplitude and direction of A 11 are respectively:

[0094]

[0095] θ = arctan(G y / G x )

[0096] The greater the element value in the image gradient amplitude matrix in the 8-neighborhood of each pixel point, the greater the gradient value of the point in the image, so that the point is an edge pixel point, and if the gradient value of a certain pixel point is not the maximum compared with the gradient values of the front and rear pixel points in the gradient direction, then the pixel point is not an edge pixel point. The first edge pixel point of the infrared thermal imaging image and the second edge pixel point in the visible light image are respectively determined through the above method.

[0097] Step 104, according to the first edge pixel point rejection threshold, the first edge pixel point of the infrared thermal imaging image is screened to obtain the third edge pixel point of the infrared thermal imaging image; and according to the second edge pixel point rejection threshold, the second edge pixel point of the visible light image is screened to obtain the fourth edge pixel point of the visible light image;

[0098] Setting the non-edge pixel gray value to 0 will obtain a binary image. Such a detection result still contains many false edges caused by noise and other reasons. The original Canny edge detection algorithm uses a double threshold screening process, and the determination of the threshold value needs to be constantly tried. Therefore, the threshold setting algorithm is improved in the present application. The first edge pixel point rejection threshold of the infrared thermal imaging image and the second edge pixel point rejection threshold of the visible light image are determined through the gray information of the infrared thermal imaging image and the visible light image. According to the corresponding edge pixel point rejection threshold, the false edges are removed to obtain accurate edge pixel points.

[0099] Step 105, according to the third edge pixel point of the infrared thermal imaging image and the fourth edge pixel point of the visible light image, the infrared thermal imaging image and the visible light image are registered to obtain the registered image of the infrared thermal imaging image and the registered image of the visible light image respectively;

[0100] The image registration according to the third edge pixel point of the infrared thermal imaging image and the fourth edge pixel point of the visible light image is to ensure that the two images are aligned in space, so as to realize accurate multi-modal information fusion. By selecting high-quality edge pixel points, the registration accuracy can be improved to ensure that the thermal features in the infrared image accurately correspond to the visual features in the visible light image. This step solves the image misalignment problem caused by different sensor viewing angles, resolutions or shooting time differences, so that the subsequent fusion processing can fully utilize the advantages of the two modalities to provide more rich and accurate information.

[0101] Step 106, according to the registered image of the infrared thermal imaging image and the registered image of the visible light image, the fusion is performed.

[0102] The infrared image can perceive the thermal radiation of objects and is suitable for low-light or night-time environments; while the visible light image captures color and shape information of objects and performs well in daytime or good lighting conditions. Through registration, the two images are accurately aligned in space, solving the problem of viewing angle, resolution or time difference between different sensors.

[0103] In one embodiment, the step 101 can include the following sub-steps:

[0104] In sub-step S11, the corresponding gray information of the infrared thermal imaging image and the visible light image under different edge pixel elimination thresholds is determined; the gray information includes a foreground layer gray average value, a background layer gray average value, and a gray average value of the entire image.

[0105] The gray average values of the foreground layer and the background layer are calculated, which can better distinguish the target objects and the background in the image and improve the contrast of the image. The gray average value of the entire image reflects the overall brightness distribution, which helps to adjust the parameters of the fusion algorithm, so that the fused image is more natural and harmonious, and the overexposure or underexposure phenomenon is avoided. By analyzing the gray information under different thresholds, noise or inaccurate edge pixels can be more accurately identified and eliminated. This ensures that high-quality edge information is involved in subsequent registration and fusion, thereby improving the quality of the final image.

[0106] In an embodiment, the step 102 can include the following sub-steps:

[0107] In sub-step S21, a first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image is determined according to the corresponding gray information of the infrared thermal imaging image under different edge pixel elimination thresholds.

[0108] In an embodiment, the sub-step S21 can include the following sub-steps:

[0109] In sub-step S211, the first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image is determined according to the foreground layer gray average value, the background layer gray average value, and the gray average value of the entire image of the infrared thermal imaging image according to the following formula:

[0110] σ 2 =P f ×(M f -M) 2 +P b ×(M b -M) 2

[0111] Where σ is the inter-class variance; P f is the probability that the total number of foreground pixels accounts for the total number of pixels; P b is the probability that the total number of background pixels accounts for the total number of pixels; M f is the foreground layer gray average value; M b is the background layer gray average value; and M is the average value of the entire image.

[0112] P f and P b are determined according to the following formula:

[0113]

[0114] Wherein, T0 is the initial edge pixel elimination threshold, the image is divided into foreground layer f and background layer b. The total number of pixels is N, the number of foreground pixels is N f , the number of background pixels is N b , the total number of image gray levels is L, and the number of pixels of each gray level is N i ,

[0115] M f , M b and M are determined according to the following formula:

[0116]

[0117] M = P f ×M f + P b ×M b

[0118] Wherein, P i is the probability of the number of pixels of each gray level accounting for the total number of pixels, and with different values of T0, the values of P f , P b , M f , M b and M will be different, so the value of inter-class variance σ will also be different. The value of T0 when σ is maximum is taken as the edge pixel elimination threshold;

[0119] According to the above method, the first inter-class variance of the infrared thermal imaging image is determined to determine the first edge pixel elimination threshold, and the second inter-class variance of the visible light image is determined to determine the second edge pixel elimination threshold.

[0120] Sub-step S22, solving the first inter-class variance, the edge pixel elimination threshold when the first inter-class variance is maximum is taken as the first edge pixel elimination threshold;

[0121] Sub-step S23, according to the corresponding gray information of the visible light image under different edge pixel elimination thresholds, the second inter-class variance between the foreground layer and the background layer of the visible light image is determined;

[0122] Sub-step S24, solving the second inter-class variance, the edge pixel elimination threshold when the second inter-class variance is maximum is taken as the second edge pixel elimination threshold.

[0123] The gray scale information of the infrared image under different edge pixel elimination thresholds is analyzed to determine the first inter-class variance between the foreground layer and the background layer. The inter-class variance measures the difference between the foreground and the background, and a larger inter-class variance indicates a better classification result. By solving the first inter-class variance and selecting the threshold that maximizes the variance as the first edge pixel elimination threshold, the most significant edge features can be ensured to be retained, while noise and inaccurate edge information are removed. This method improves the accuracy of edge detection and provides high-quality input for subsequent registration and fusion. Similarly, the second inter-class variance between the foreground layer and the background layer of the visible light image under different edge pixel elimination thresholds is calculated. Similarly, the second inter-class variance is solved and the threshold that maximizes the variance is selected as the second edge pixel elimination threshold. This step ensures that important edges in the visible light image are accurately identified and background noise is effectively suppressed.

[0124] In an embodiment, the step 104 can include the following sub-steps:

[0125] Sub-step S31, eliminating the first edge pixels of the infrared thermal imaging image that are less than the first edge pixel elimination threshold to obtain third edge pixels of the infrared thermal imaging image.

[0126] Sub-step S32, eliminating the second edge pixels of the visible light image that are less than the second edge pixel elimination threshold to obtain fourth edge pixels of the infrared thermal imaging image.

[0127] The edge pixels of the infrared image that are less than the first edge pixel elimination threshold are eliminated to obtain third edge pixels. This step ensures that only significant and reliable edges are retained, reducing false detection caused by noise or low contrast. Similarly, the edge pixels of the visible light image that are less than the second edge pixel elimination threshold are eliminated to obtain fourth edge pixels. Through this screening process, background noise and other interference can be effectively removed, and clear and accurate edge information is retained. The third and fourth edge pixels after screening provide a more reliable basis for subsequent image registration and fusion. High-quality edge information helps to improve registration accuracy and ensure the spatial consistency of infrared and visible light images, thereby improving the quality and accuracy of the fused image.

[0128] In an embodiment, the step 106 can include the following sub-steps:

[0129] Sub-step S41, performing edge detection on the registered image of the infrared thermal imaging image and the registered image of the visible light image, respectively;

[0130] Sub-step S42, according to the edge detection result, determine the fifth edge pixel point of the registered image of the infrared thermal imaging image and the sixth edge pixel point of the registered image of the visible light image;

[0131] Sub-step S43, according to the fifth edge pixel point of the registered image of the infrared thermal imaging image and the sixth edge pixel point of the registered image of the visible light image, determine the Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image;

[0132] Based on the edge detection result, the Euclidean distance between the fifth edge pixel point and the sixth edge pixel point is calculated. The Euclidean distance is used to measure the similarity of two images in space, helping to evaluate their matching degree. According to the calculated Euclidean distance, it is decided whether and how to fuse the two images. Smaller Euclidean distance indicates that the image features are similar and suitable for fusion; larger distance may indicate that the feature difference is too large and not suitable for direct fusion. In this way, the system can maximize the use of information of two modalities under the premise of ensuring image quality. This method not only improves the accuracy of image registration, but also enhances the robustness and adaptability of the system. High-quality edge information and reasonable fusion strategy ensure the clarity and accuracy of the final fused image, which is particularly suitable for multi-modal image processing tasks in complex environments such as security monitoring and intelligent driving.

[0133] In an embodiment, the sub-step S43 can include the following sub-steps:

[0134] Sub-step S431, according to the fifth edge pixel point, determine the zero-order matrix and the first-order matrix of the registered image of the infrared thermal imaging image; according to the sixth edge pixel point, determine the zero-order matrix and the first-order matrix of the registered image of the visible light image;

[0135] Sub-step S432, according to the zero-order matrix of the registered image of the infrared thermal imaging image and the zero-order matrix of the registered image of the visible light image, determine the first Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image;

[0136] Sub-step S433, according to the first-order matrix of the registered image of the infrared thermal imaging image and the first-order matrix of the registered image of the visible light image, determine the second Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image;

[0137] Specifically, edge detection is performed on the registered image of the infrared thermal imaging image and the registered image of the visible light image by a Canny edge detection algorithm to obtain fifth edge pixel points and sixth edge pixel points. Then, the contour moments, i.e., zero-order moments and first-order moments, are calculated from the fifth edge pixel points and the sixth edge pixel points for the registered image of the infrared thermal imaging image and the registered image of the visible light image respectively, in the following specific manner:

[0138] The p+q order geometric moment matrix m of the registered image f(i,j) of MxN is determined pq

[0139]

[0140] Wherein, f(i,j) is the gray value of the image at (i,j), and from the above formula, the zero-order moment matrix, i.e., p=q=0, is:

[0141]

[0142] The first-order moment matrix is:

[0143]

[0144] When the image is a binary image, when the image is a binary image, m 00 is the total sum of white areas on the image; m 10 is the cumulative sum of white pixels on the image with respect to the x coordinate, and m 01 is the cumulative sum of y coordinates;

[0145] The first Euclidean distance is:

[0146]

[0147] The second Euclidean distance includes a second Euclidean distance in the x direction and a second Euclidean distance in the y direction, which are respectively:

[0148] Sub-step S44, according to the Euclidean distance, the registered image of the infrared thermal imaging image and the registered image of the visible light image are fused.

[0149] In an embodiment, the sub-step S44 can include the following sub-steps:

[0150] Sub-step S441, when the first Euclidean distance and the second Euclidean distance simultaneously satisfy a preset threshold condition, the registered image of the infrared thermal imaging image and the registered image of the visible light image are fused.

[0151] ​When the first Euclidean distance and the second Euclidean distance are both less than the preset threshold condition, the registered image of the infrared thermal imaging image and the registered image of the visible light image are fused. And since the infrared thermal imaging image is shot by a low-resolution camera, the imaging effect is poor, and the edge detection effect is poor, therefore, when the set threshold is met, the edge of the visible light is set as the edge after fusion, and the gray-white pixels are mainly visible light.

[0152] The embodiment of the present application provides an image fusion method, by acquiring infrared thermal imaging image and visible light image at the same time, determining the corresponding gray information of the two images respectively; determining the corresponding edge pixel point elimination threshold of the two images according to the gray information; respectively performing edge detection on the two images to determine the corresponding edge pixel points; screening the corresponding edge pixel points of the two images according to the corresponding edge pixel point elimination threshold of the two images, to obtain the screened edge pixel points of the two images; registering the two images according to the screened edge pixel points and then performing image fusion. The embodiment of the present application determines the edge pixel point elimination threshold of the infrared thermal imaging image and the visible light image respectively according to the gray information of the two images, and then screens the edge pixel points, and performs image registration and fusion based on the screened edge pixel points. The imaging effect and accuracy of image fusion are improved, and the quality of the final image is ensured.

[0153] It should be noted that, for the method embodiment, in order to simply describe, it is expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the present application is not limited by the action sequence described, because according to the embodiment of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiment of the present application.

[0154] Reference Figure 2 , a structure block diagram of an image fusion device provided by the embodiment of the present application is shown, which can specifically include the following modules:

[0155] The acquisition module 201 is used for acquiring infrared thermal imaging image and visible light image at the same time, and determining the gray information of the infrared thermal imaging image and the visible light image respectively;

[0156] The determination module 202 is used for determining the first edge pixel point elimination threshold of the infrared thermal imaging image according to the gray information of the infrared thermal imaging image, and determining the second edge pixel point elimination threshold of the visible light image according to the gray information of the visible light image;

[0157] The detection module 203 is configured to perform edge detection on the infrared thermal imaging image and the visible light image respectively, and determine first edge pixel points of the infrared thermal imaging image and second edge pixel points in the visible light image according to the edge detection results respectively.

[0158] The screening module 204 is configured to screen the first edge pixel points of the infrared thermal imaging image according to a first edge pixel point elimination threshold to obtain third edge pixel points of the infrared thermal imaging image, and screen the second edge pixel points of the visible light image according to a second edge pixel point elimination threshold to obtain fourth edge pixel points of the visible light image.

[0159] The registration module 205 is configured to register the infrared thermal imaging image and the visible light image according to the third edge pixel points of the infrared thermal imaging image and the fourth edge pixel points of the visible light image to obtain a registered image of the infrared thermal imaging image and a registered image of the visible light image respectively.

[0160] The fusion module 206 is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image.

[0161] In an embodiment, the acquisition module comprises:

[0162] The first determination sub-module is configured to determine corresponding gray information of the infrared thermal imaging image and the visible light image under different edge pixel point elimination thresholds; the gray information comprises a foreground layer gray average value, a background layer gray average value and a gray average value of the whole image.

[0163] In an embodiment, the determination module comprises:

[0164] The second determination sub-module is configured to determine a first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image according to the corresponding gray information of the infrared thermal imaging image under different edge pixel point elimination thresholds.

[0165] The first solving sub-module is configured to solve the first inter-class variance, and take the edge pixel point elimination threshold when the first inter-class variance is maximum as the first edge pixel point elimination threshold.

[0166] The third determination sub-module is configured to determine a second inter-class variance between the foreground layer and the background layer of the visible light image according to the corresponding gray information of the visible light image under different edge pixel point elimination thresholds.

[0167] The second solving sub-module is configured to solve the second inter-class variance, and take the edge pixel point elimination threshold when the second inter-class variance is maximum as the second edge pixel point elimination threshold.

[0168] In an embodiment, the second determining sub-module comprises:

[0169] The first determining unit is configured to determine the first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image according to the foreground layer average gray value, the background layer average gray value and the average gray value of the whole image of the infrared thermal imaging image, and according to the following formula:

[0170] σ 2 = P f × (M f - M) 2 + P b × (M b - M) 2

[0171] wherein σ is the inter-class variance; P f is the probability of the total number of foreground pixels accounting for the total number of pixels; P b is the probability of the total number of background pixels accounting for the total number of pixels; M f is the foreground layer average gray value; M b is the background layer average gray value; and M is the average value of the whole image.

[0172] In an embodiment, the screening module comprises:

[0173] The first screening sub-module is configured to remove the first edge pixel points of the infrared thermal imaging image which are less than the first edge pixel point removal threshold value, to obtain third edge pixel points of the infrared thermal imaging image.

[0174] The second screening sub-module is configured to remove the second edge pixel points of the visible light image which are less than the second edge pixel point removal threshold value, to obtain fourth edge pixel points of the infrared thermal imaging image.

[0175] In an embodiment, the fusion module comprises:

[0176] The first detecting sub-module is configured to perform edge detection on the registered image of the infrared thermal imaging image and the registered image of the visible light image respectively.

[0177] The fourth determining sub-module is configured to determine the fifth edge pixel points of the registered image of the infrared thermal imaging image and the sixth edge pixel points of the registered image of the visible light image according to the edge detection result.

[0178] a fifth determining sub-module, configured to determine an Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to a fifth edge pixel point of the registered image of the infrared thermal imaging image and a sixth edge pixel point of the registered image of the visible light image;

[0179] a first fusing sub-module, configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the Euclidean distance.

[0180] In an embodiment, the Euclidean distance includes a first Euclidean distance and a second Euclidean distance; and the fifth determining sub-module includes:

[0181] a second determining unit, configured to determine a zero-order matrix and a first-order matrix of the registered image of the infrared thermal imaging image according to the fifth edge pixel point, and determine a zero-order matrix and a first-order matrix of the registered image of the visible light image according to the sixth edge pixel point;

[0182] a third determining unit, configured to determine a first Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the zero-order matrix of the registered image of the infrared thermal imaging image and the zero-order matrix of the registered image of the visible light image;

[0183] a fourth determining unit, configured to determine a second Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the first-order matrix of the registered image of the infrared thermal imaging image and the first-order matrix of the registered image of the visible light image;

[0184] the first fusing sub-module includes:

[0185] a first fusing unit, configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image when the first Euclidean distance and the second Euclidean distance both satisfy a preset threshold condition.

[0186] For the device embodiment, it is basically similar to the method embodiment, so the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0187] The embodiment of the present application provides an image fusion method, by acquiring infrared thermal imaging images and visible light images at the same time, determining corresponding gray information of the two images respectively; determining corresponding edge pixel point elimination thresholds of the two images according to the gray information; performing edge detection on the two images respectively to determine corresponding edge pixel points; screening the corresponding edge pixel points of the two images according to the corresponding edge pixel point elimination thresholds of the two images, to obtain screened edge pixel points corresponding to the two images; and performing image registration and fusion on the two images according to the screened edge pixel points. The embodiment of the present application determines the edge pixel point elimination thresholds of the infrared thermal imaging images and the visible light images respectively, and then screens the edge pixel points based on the screened edge pixel points, to perform image registration and fusion. The imaging effect and accuracy of image fusion are improved, and the quality of the final image is ensured.

[0188] The embodiment of the present application further provides an electronic device, comprising:

[0189] The electronic device comprises a processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein the computer program is executed by the processor to realize each process of the image fusion method embodiment and achieve the same technical effect, and details are not repeated here to avoid repetition.

[0190] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize each process of the image fusion method embodiment and achieve the same technical effect, and details are not repeated here to avoid repetition.

[0191] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between each embodiment are referred to each other.

[0192] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, an apparatus, or a computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0193] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more of the flowcharts and / or block diagrams. Figure 1 one or more of the flowcharts and / or block diagrams.

[0194] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more of the flowcharts and / or block diagrams. Figure 1 one or more of the flowcharts and / or block diagrams.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices to cause a series of operational steps to be performed on the computer or other programmable terminal devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more of the flowcharts and / or block diagrams. Figure 1 one or more of the flowcharts and / or block diagrams.

[0196] Although preferred embodiments of the present application have been described, those skilled in the art will be able to make additional modifications and variations to the described embodiments without departing from the inventive concepts disclosed in the present application. Accordingly, the appended claims are intended to cover all such modifications and variations as falling within the scope of the present application.

[0197] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that there is any such actual relationship or order between these entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover non-exclusive inclusions, so that a process, method, article, or terminal device including a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article, or terminal device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or terminal device including the element.

[0198] The above describes in detail the image fusion method and the image fusion device provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. An image fusion method, characterized by, The method comprises: acquiring an infrared thermal imaging image and a visible light image at the same time, and determining gray scale information of the infrared thermal imaging image and the visible light image respectively; determining a first edge pixel point elimination threshold of the infrared thermal imaging image according to the gray scale information of the infrared thermal imaging image, and determining a second edge pixel point elimination threshold of the visible light image according to the gray scale information of the visible light image; respectively performing edge detection on the infrared thermal imaging image and the visible light image, and respectively determining first edge pixel points of the infrared thermal imaging image and second edge pixel points in the visible light image according to the edge detection results; screening the first edge pixel points of the infrared thermal imaging image according to the first edge pixel point elimination threshold to obtain third edge pixel points of the infrared thermal imaging image, and screening the second edge pixel points of the visible light image according to the second edge pixel point elimination threshold to obtain fourth edge pixel points of the visible light image; registering the infrared thermal imaging image and the visible light image according to the third edge pixel points of the infrared thermal imaging image and the fourth edge pixel points of the visible light image to respectively obtain a registered image of the infrared thermal imaging image and a registered image of the visible light image; respectively performing edge detection on the registered image of the infrared thermal imaging image and the registered image of the visible light image; determining fifth edge pixel points of the registered image of the infrared thermal imaging image and sixth edge pixel points of the registered image of the visible light image according to the edge detection results; determining a zero-order matrix and a first-order matrix of the registered image of the infrared thermal imaging image according to the fifth edge pixel points, and determining a zero-order matrix and a first-order matrix of the registered image of the visible light image according to the sixth edge pixel points; determining a first Euclidean distance of the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the zero-order matrix of the registered image of the infrared thermal imaging image and the zero-order matrix of the registered image of the visible light image; determining a second Euclidean distance of the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the first-order matrix of the registered image of the infrared thermal imaging image and the first-order matrix of the registered image of the visible light image; fusing the registered image of the infrared thermal imaging image and the registered image of the visible light image when the first Euclidean distance and the second Euclidean distance simultaneously satisfy a preset threshold condition.

2. The method of claim 1, wherein, The determination of the gray scale information of the infrared thermal imaging image and the visible light image comprises: determining corresponding gray scale information of the infrared thermal imaging image and the visible light image under different edge pixel point elimination thresholds; the gray scale information comprises a foreground layer gray scale average value, a background layer gray scale average value, and a gray scale average value of the whole image.

3. The method of claim 2, wherein, The method comprises the following steps: According to the gray information of the infrared thermal imaging image under different edge pixel elimination thresholds, the first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image is determined. Solving the first inter-class variance, the edge pixel elimination threshold that makes the first inter-class variance maximum is taken as the first edge pixel elimination threshold. According to the gray information of the visible light image under different edge pixel elimination thresholds, the second inter-class variance between the foreground layer and the background layer of the visible light image is determined. Solving the second inter-class variance, the edge pixel elimination threshold that makes the second inter-class variance maximum is taken as the second edge pixel elimination threshold.

4. The method of claim 3, wherein, The method comprises the following steps: According to the foreground layer gray average value, the background layer gray average value and the gray average value of the whole image of the infrared thermal imaging image, the first inter-class variance between the foreground layer and the background layer of the infrared thermal imaging image is determined according to the following formula: wherein is the inter-class variance; is the probability of the total number of foreground pixels occupying the total number of pixels; is the probability of the total number of background pixels occupying the total number of pixels; is the average value of the foreground layer gray scale; is the average value of the background layer gray scale; is the average value of the whole image.

5. The method of claim 1, wherein, The method comprises the following steps: The method comprises the following steps: The first edge pixels of the infrared thermal imaging image are eliminated if they are less than the first edge pixel elimination threshold, and the third edge pixels of the infrared thermal imaging image are obtained.

6. An image fusion apparatus characterized by comprising: The second edge pixels of the visible light image are eliminated if they are less than the second edge pixel elimination threshold, and the fourth edge pixels of the visible light image are obtained. The device comprises: An acquisition module is configured to acquire an infrared thermal imaging image and a visible light image at the same time, and determine the gray information of the infrared thermal imaging image and the visible light image respectively; A determination module is configured to determine a first edge pixel elimination threshold of the infrared thermal imaging image according to the gray information of the infrared thermal imaging image, and determine a second edge pixel elimination threshold of the visible light image according to the gray information of the visible light image; A detection module is configured to perform edge detection on the infrared thermal imaging image and the visible light image respectively, and determine the first edge pixel of the infrared thermal imaging image and the second edge pixel of the visible light image according to the edge detection results respectively. The screening module is configured to screen the first edge pixel points of the infrared thermal imaging image according to a first edge pixel point rejection threshold, to obtain third edge pixel points of the infrared thermal imaging image; and screen the second edge pixel points of the visible light image according to a second edge pixel point rejection threshold, to obtain fourth edge pixel points of the visible light image; The registration module is configured to register the infrared thermal imaging image and the visible light image according to the third edge pixel points of the infrared thermal imaging image and the fourth edge pixel points of the visible light image, to obtain a registered image of the infrared thermal imaging image and a registered image of the visible light image, respectively. The fusion module is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image. The fusion module includes: The first detection submodule is configured to perform edge detection on the registered image of the infrared thermal imaging image and the registered image of the visible light image, respectively. The fourth determination submodule is configured to determine fifth edge pixel points of the registered image of the infrared thermal imaging image and sixth edge pixel points of the registered image of the visible light image according to the edge detection results. The fifth determination submodule is configured to determine an Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the fifth edge pixel points of the registered image of the infrared thermal imaging image and the sixth edge pixel points of the registered image of the visible light image. The first fusion submodule is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the Euclidean distance. The Euclidean distance includes a first Euclidean distance and a second Euclidean distance; and the fifth determination submodule includes: The second determination unit is configured to determine a zero-order matrix and a first-order matrix of the registered image of the infrared thermal imaging image according to the fifth edge pixel points, and determine a zero-order matrix and a first-order matrix of the registered image of the visible light image according to the sixth edge pixel points. The third determination unit is configured to determine the first Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the zero-order matrix of the registered image of the infrared thermal imaging image and the zero-order matrix of the registered image of the visible light image. The fourth determination unit is configured to determine the second Euclidean distance between the registered image of the infrared thermal imaging image and the registered image of the visible light image according to the first-order matrix of the registered image of the infrared thermal imaging image and the first-order matrix of the registered image of the visible light image. The first fusion submodule includes: The first fusion unit is configured to fuse the registered image of the infrared thermal imaging image and the registered image of the visible light image when the first Euclidean distance and the second Euclidean distance both satisfy a preset threshold condition.

7. An electronic device, comprising: The computer program is executed by the processor to implement the steps of the image fusion method according to any one of claims 1-5. ​ 8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the image fusion method in any one of claims 1 to 5.

Images