Method and system for fusing visible and infrared light images

Abstract

The invention discloses a visible and infrared light image fusion method and system, and an image fusion model in the fusion method is obtained through training by the following steps: carrying out the preprocessing of visible and infrared light collection images, and obtaining visible and infrared light training images; performing feature extraction on the visible and infrared light training images to obtain visible light features and infrared light features; according to the visible light features, attention confrontation fusion is carried out on the infrared light features, and a confrontation fusion image is obtained; and according to the confrontation fusion image, performing parameter updating on the initialized image fusion model to obtain a trained image fusion model. The fusion method can effectively reduce computing resources and consumed time required by image fusion, improve the image fusion efficiency, and improve the quality and practicability of the generated fused image. The invention relates to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for fusing visible and infrared light images. Background Art

[0002] As an observation device installed on carriers such as drones and unmanned vehicles, the smart pod often needs to fuse infrared images and visible light images in specific application scenarios, and then realize target search, recognition, positioning, tracking and other functions based on the fused images.

[0003] At present, the traditional visible and infrared image fusion methods used in smart pods can be roughly divided into two categories. The first category is mainly based on image registration, which is easily affected by the actual environment of the smart pod. Image registration is difficult and it is easy to ignore the target edge detail information in the image. The quality of visible and infrared image fusion is unsatisfactory. In addition, the second category is mainly based on multi-scale analysis, such as wavelet transform, which can theoretically fuse information from different frequency bands, but this method requires more computing resources, image fusion takes a long time, and the practicality of the generated fused image is unsatisfactory.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide a method for fusing visible and infrared light images, which can effectively reduce the computing resources and time required for image fusion, improve image fusion efficiency, and improve the quality and practicality of the generated fused image.

[0007] Another object of an embodiment of the present application is to provide a visible and infrared light image fusion system.

[0008] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0009] In a first aspect, an embodiment of the present application provides a method for fusing visible and infrared light images, comprising:

[0010] Acquire a visible light image and an infrared light image to be fused;

[0011] Inputting the visible light image and the infrared light image into a trained image fusion model to obtain a fused image;

[0012] The trained image fusion model is obtained by training through the following steps:

[0013] Acquire original visible light collection images and infrared light collection images;

[0014] Performing a first preprocessing on the visible light collected image to obtain a visible light training image, and performing a second preprocessing on the infrared light collected image to obtain an infrared light training image;

[0015] Performing a first feature extraction on the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize an image feature of the visible light training image at a high frequency layer, and the second visible light feature is used to characterize an image feature of the visible light training image at a low frequency layer;

[0016] Performing a second feature extraction on the infrared light training image to obtain a first infrared light feature and a second infrared light feature, wherein the first infrared light feature is used to characterize an image feature of the infrared light training image at a high frequency layer, and the second infrared light feature is used to characterize an image feature of the infrared light training image at a low frequency layer;

[0017] According to the first visible light feature and the second visible light feature, performing attention adversarial fusion on the first infrared light feature and the second visible light feature to obtain an adversarial fusion image;

[0018] According to the adversarial fusion image, the parameters of the initialized image fusion model are updated to obtain the trained image fusion model.

[0019] In addition, the fusion method according to the above embodiment of the present application may also have the following additional technical features:

[0020] Furthermore, in one embodiment of the present application, performing a first preprocessing on the visible light acquisition image to obtain a visible light training image includes:

[0021] Performing image transformation on the visible light collected image to obtain a first visible light image;

[0022] performing image normalization on the first visible light image to obtain a second visible light image;

[0023] Perform image denoising on the second visible light image to obtain the visible light training image.

[0024] Furthermore, in one embodiment of the present application, performing a second preprocessing on the infrared light acquisition image to obtain an infrared light training image includes:

[0025] Obtain preset infrared pixel blocks and infrared templates;

[0026] According to the infrared pixel blocks, grayscale mean optimization is performed on the infrared light collected image to obtain an infrared light optimized image;

[0027] According to the infrared template, local information extraction is performed on the infrared light optimized image to obtain infrared light gradient information and local information entropy corresponding to the infrared light optimized image;

[0028] According to the infrared light gradient information, the image feature value fusion is performed on the local information entropy to obtain the infrared light training image.

[0029] Furthermore, in one embodiment of the present application, the image fusion model includes a feature extraction framework, and the feature extraction framework is used to perform the following steps:

[0030] Performing image decomposition on a target image to obtain a target high-frequency layer image and a target low-frequency layer image, wherein the target image is the visible light training image or the infrared light training image;

[0031] A high-frequency feature extraction is performed on the target high-frequency layer image to obtain a first target feature, and a low-frequency feature extraction is performed on the target low-frequency layer image to obtain a second target feature.

[0032] Furthermore, in one embodiment of the present application, performing image decomposition on the target image to obtain a target high-frequency layer image and a target low-frequency layer image includes:

[0033] Performing pixel position convolution on the target image to obtain the target high-frequency layer image;

[0034] According to the target high-frequency layer image, image difference calculation is performed on the target image to obtain the target low-frequency layer image.

[0035] Further, in the embodiment of the present application, the first infrared light feature and the second visible light feature are subjected to attention adversarial fusion according to the first visible light feature and the second visible light feature to obtain an adversarial fusion image, including:

[0036] According to the first visible light feature, high-frequency attention fusion is performed on the first infrared light feature to obtain a high-frequency fused feature image;

[0037] According to the second visible light feature, performing low-frequency attention fusion on the second visible light feature to obtain a low-frequency fusion feature image;

[0038] According to the low-frequency fusion feature image, the high-frequency fusion feature image is generatively fused to obtain the adversarial fusion image.

[0039] Further, in the embodiment of the present application, the parameter updating of the initialized image fusion model according to the adversarial fusion image to obtain the trained image fusion model includes:

[0040] Obtaining a recognition probability value corresponding to the adversarial fusion image;

[0041] According to the discrimination probability value, performing adversarial loss calculation on the adversarial fusion image to obtain a first sub-loss value;

[0042] According to the high-frequency fusion feature image and the low-frequency fusion feature image, performing content loss calculation on the adversarial fusion image to obtain a second sub-loss value;

[0043] According to the visible light training image and the infrared light training image, a fusion loss is calculated for the adversarial fusion image to obtain a third sub-loss value;

[0044] Determining a target loss value based on the first sub-loss value, the second sub-loss value, and the third sub-loss value;

[0045] The initialized image fusion model is trained according to the target loss value to obtain the trained image fusion model.

[0046] In a second aspect, an embodiment of the present application provides a visible and infrared image fusion system, including:

[0047] An acquisition unit, used for acquiring a visible light image and an infrared light image to be fused;

[0048] A processing unit, used for inputting the visible light image and the infrared light image into a trained image fusion model to obtain a fused image;

[0049] The trained image fusion model is obtained by training through the following steps:

[0050] Acquire original visible light collection images and infrared light collection images;

[0051] Performing a first preprocessing on the visible light collected image to obtain a visible light training image, and performing a second preprocessing on the infrared light collected image to obtain an infrared light training image;

[0052] Performing a first feature extraction on the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize an image feature of the visible light training image at a high frequency layer, and the second visible light feature is used to characterize an image feature of the visible light training image at a low frequency layer;

[0053] Performing a second feature extraction on the infrared light training image to obtain a first infrared light feature and a second infrared light feature, wherein the first infrared light feature is used to characterize an image feature of the infrared light training image at a high frequency layer, and the second infrared light feature is used to characterize an image feature of the infrared light training image at a low frequency layer;

[0054] According to the first visible light feature and the second visible light feature, performing attention adversarial fusion on the first infrared light feature and the second visible light feature to obtain an adversarial fusion image;

[0055] According to the adversarial fusion image, the parameters of the initialized image fusion model are updated to obtain the trained image fusion model.

[0056] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0057] at least one processor;

[0058] at least one memory for storing at least one program;

[0059] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0060] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor.

[0061] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0062] A method and system for fusing visible and infrared images disclosed in an embodiment of the present application, the fusion method comprises the following steps: obtaining a visible light image and an infrared image to be fused; inputting the visible light image and the infrared image into a trained image fusion model to obtain a fused image; wherein the trained image fusion model is trained by the following steps: obtaining an original visible light acquisition image and an infrared light acquisition image; performing a first preprocessing on the visible light acquisition image to obtain a visible light training image, and performing a second preprocessing on the infrared light acquisition image to obtain an infrared light training image; performing a first feature extraction on the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize the visible light The image features of the training image at the high frequency layer, the second visible light feature is used to characterize the image features of the visible light training image at the low frequency layer; the second feature extraction is performed on the infrared light training image to obtain the first infrared light feature and the second infrared light feature, the first infrared light feature is used to characterize the image features of the infrared light training image at the high frequency layer, and the second infrared light feature is used to characterize the image features of the infrared light training image at the low frequency layer; according to the first visible light feature and the second visible light feature, the first infrared light feature and the second visible light feature are subjected to attention adversarial fusion to obtain an adversarial fusion image; according to the adversarial fusion image, the parameters of the initialized image fusion model are updated to obtain the trained image fusion model. The fusion method is based on the attention mechanism, and the first visible light feature, the second visible light feature, the first infrared light feature and the second visible light feature are subjected to adversarial fusion, which can effectively reduce the computing resources and time consumption required for image fusion, improve the image fusion efficiency, and improve the quality and practicality of the generated fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A schematic diagram of a flow chart of a method for fusing visible and infrared light images provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a training process of an image fusion model provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of the structure of a visible and infrared image fusion system provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] At present, the traditional visible and infrared image fusion methods used in smart pods can be roughly divided into two categories. The first category is mainly based on image registration, which is achieved by extracting common features from the two images to establish a correspondence between the images. This method is easily affected by the actual environment in which the smart pod is located, such as the vibration of the smart pod, unstable light conditions, etc., which makes the image registration accuracy low, the image registration is difficult, and it is easy to ignore the target edge detail information in the image, which makes the subsequent visible and infrared image fusion quality unsatisfactory.

[0071] The second method is mainly based on multi-scale analysis, such as wavelet transform, which can theoretically fuse information from different frequency bands, but this method requires more computing resources and takes a long time to image fusion. In real-time positioning and tracking of smart pods, the fused image provided by this method does not meet the real-time requirements of target tracking of smart pods, and the practicality of the generated fused image is not satisfactory.

[0072] In view of this, an embodiment of the present invention provides a method for fusing visible and infrared light images. The fusion method is based on an attention mechanism and performs adversarial fusion on a first visible light feature, a second visible light feature, a first infrared light feature and a second visible light feature. This can effectively reduce the computing resources and time required for image fusion, improve image fusion efficiency, and increase the target edge detail information in the fused image, thereby improving the quality and practicality of the generated fused image.

[0073] Reference Figure 1 and Figure 2 In an embodiment of the present application, a method for fusing visible and infrared light images includes:

[0074] Step 110: Obtaining a visible light image and an infrared light image to be fused;

[0075] Step 120: input the visible light image and the infrared light image into a trained image fusion model to obtain a fused image;

[0076] In an embodiment of the present application, an infrared camera and a visible light camera configured on the smart pod can be used to obtain fused visible light images and infrared light images respectively, and then the obtained visible light and infrared light images are input into a trained image fusion model. The trained image fusion model can be deployed in the edge device of the smart pod to obtain a fused image with less computing resources required for fusion and higher fusion efficiency.

[0077] The trained image fusion model is obtained by training through the following steps:

[0078] Step 130: acquiring original visible light collected images and infrared light collected images;

[0079] In an embodiment of the present application, the original visible light collected images and infrared light collected images may be image data about one or more scene applications of the smart pod that are publicly available on the Internet; or, the original visible light collected images and infrared light collected images may be historical image data collected by the smart pod. There are many specific ways to obtain visible light collected images and infrared light collected images, which will not be elaborated in this application.

[0080] Step 140: performing a first preprocessing on the visible light collected image to obtain a visible light training image, and performing a second preprocessing on the infrared light collected image to obtain an infrared light training image;

[0081] In some embodiments, the step 140 of performing a first preprocessing on the visible light collected image to obtain a visible light training image includes:

[0082] A1. performing image transformation on the visible light collected image to obtain a first visible light image;

[0083] A2. Normalizing the first visible light image to obtain a second visible light image;

[0084] A3. Perform image denoising on the second visible light image to obtain the visible light training image.

[0085] In an embodiment of the present application, step A1 may be to adjust the image resolution size of the visible light acquisition image so that the visible light training image and the infrared light training image are the same size. Specifically, step A1 may be based on the image resolution size of the infrared light acquisition image, and scale the image resolution of the visible light acquisition image to the image resolution corresponding to the infrared light acquisition image, thereby completing the image transformation and obtaining the first visible light image.

[0086] It can be understood that step A2 is used to convert the pixel value of each pixel point of the first visible light image into a standard form to obtain the second visible light image, which can specifically be any one of linear normalization, zero mean normalization, etc. In the embodiment of the present application, linear normalization is taken as an example. For a certain pixel point of the second visible light image, the equivalent expression of image normalization can be:

[0087]

[0088] Among them, X i_new is the pixel value of the i-th pixel in the second visible light image; X i is the pixel value of the i-th pixel in the first visible light image; Min(X) is the minimum pixel value in the first visible light image; Max(X) is the maximum pixel value in the first visible light image.

[0089] It should be noted that for the noise present in the second visible light image, step A3 can be based on a Gaussian filter kernel to perform weighted averaging of each pixel value in the second visible light image and the surrounding pixel values, where the specific weights obey a Gaussian distribution, to achieve smoothing and denoising of the second visible light image, thereby obtaining a visible light training image.

[0090] In some embodiments, the step 140 of performing a second preprocessing on the infrared light collection image to obtain an infrared light training image includes:

[0091] B1. Obtaining a preset infrared pixel block and infrared template;

[0092] B2. Optimizing the grayscale mean of the infrared light collected image according to the infrared pixel block to obtain an infrared light optimized image;

[0093] B3. Extracting local information from the infrared light optimized image according to the infrared template to obtain infrared light gradient information and local information entropy corresponding to the infrared light optimized image;

[0094] B4. Perform image feature value fusion on the local information entropy according to the infrared light gradient information to obtain the infrared light training image.

[0095] In the embodiment of the present application, the block size of the infrared pixel block can be preset, and the specific block size can be any one of 2×2, 3×3, 5×5, etc. The embodiment of the present application takes the block size of the infrared pixel block as 3×3 as an example. The grayscale mean optimization in step B2 can first be to fill and update the preset blank pixel blocks according to several local images in the infrared light acquisition image to obtain an infrared pixel block with a local image; then, for a certain infrared pixel block, it can be to calculate the similarity between each target pixel point in the infrared pixel block and the pixel points in other blocks, the target pixel point can be any pixel point in the infrared pixel block, and the pixel points in the block can be the remaining pixel points in the infrared pixel block except the target pixel point.

[0096] For example, the similarity between a pixel in a certain block and a target pixel can be measured according to the gray value difference, and its equivalent expression can be:

[0097]

[0098] Where Similarity(p,q) is the similarity between the pixel p in the block and the target pixel q; I(p) is the gray value of the pixel in the block; I(q) is the gray value of the target pixel; σ is the standard deviation threshold of the similarity.

[0099] It should be noted that the standard deviation threshold σ is a parameter indicating the sensitivity of the similarity measurement, and its specific size can be flexibly set according to actual conditions. Specifically, when the standard deviation threshold σ is large, the similarity corresponding to the gray value difference between the pixel point p in the block and the target similarity q is large, which is also a relatively loose measurement of the similarity between the pixel point p in the block and the target similarity q; and when the standard deviation threshold σ is small, the similarity corresponding to the gray value difference between the pixel point p in the block and the target similarity q is small, which is also a relatively strict measurement of the similarity between the pixel point p in the block and the target similarity q. In addition, after obtaining the similarity between the target pixel point and all the pixels in the block, all the similarities corresponding to the target pixel point can be screened based on the preset similarity threshold, so as to screen out a set of similar pixels corresponding to the target pixel point, and each pixel in the similar pixel point set has a similarity with the target pixel point greater than or equal to the preset similarity threshold, and the similarity threshold can be any one of 0.8, 0.85, 0.93, 0.98, etc.

[0100] It should be mentioned that after obtaining a set of similar pixels corresponding to the target pixel, the grayscale average value of all similar pixels in the set of similar pixels can be calculated, and the calculated grayscale average value can be assigned to the target pixel, so as to optimize the grayscale value of the target pixel. For the remaining target pixels of the infrared pixel block, and other infrared pixel blocks corresponding to the infrared light acquisition image, they can be derived by analogy with the content of a certain target pixel mentioned above, and this application will not elaborate on them here.

[0101] It is worth noting that after all target pixels in all infrared pixel blocks are optimized for grayscale value, an infrared light optimized image can be obtained. In addition, the traditional method of searching for other similar pixels in the entire image for each pixel in the image is relatively inefficient and is more likely to add new noise to the image when the image resolution is large; the embodiment of the present application optimizes the grayscale mean of the infrared light acquisition image based on the infrared pixel block, and can optimize the grayscale value of the target pixel within the local range selected by the infrared pixel block, which can effectively improve the execution efficiency and avoid adding noise to the image.

[0102] For step B3, it can be based on the infrared light template to obtain the infrared light gradient information and local information entropy corresponding to each pixel of the infrared light optimized image. Specifically, the template size of the infrared light template in the embodiment of the present application can be 3×3. At this time, for the middle pixel in a certain infrared light template, the gradient features in 8 directions of the middle pixel can be determined by gradient feature calculation, which are specifically left gradient features, right gradient features, upper gradient features, lower gradient features, upper left gradient features, lower left gradient features, upper right gradient features and lower right gradient features, wherein the left gradient features and the right gradient features can be summarized as horizontal gradient features, the upper gradient features and the lower gradient features can be summarized as vertical gradient features, the upper left gradient features and the lower right gradient features can be summarized as first diagonal gradient features, and the lower left gradient features and the upper right gradient features can be summarized as second diagonal gradient features. Then, based on the horizontal gradient features, the vertical gradient features, the first diagonal gradient features and the second diagonal gradient features, the infrared light gradient information corresponding to the middle pixel is determined, and the equivalent expression of the infrared light gradient information can be:

[0103]

[0104] Among them, G(a) is the infrared light gradient information corresponding to the a-th middle pixel; G(a) is the horizontal gradient feature corresponding to the a-th middle pixel; G v (a) is the vertical gradient feature corresponding to the a-th middle pixel; G m (a) is the first diagonal gradient feature corresponding to the a-th middle pixel; G s(a) is the second diagonal gradient feature corresponding to the a-th middle pixel.

[0105] The equivalent expression of the local information entropy of the middle pixel can be:

[0106]

[0107] Here, E(a) is the local information entropy corresponding to the a-th intermediate pixel; j is the pixel value corresponding to the a-th intermediate pixel; Hj is the number of pixels with pixel value j in the a-th intermediate pixel in the infrared template; and N is the size of the infrared template.

[0108] It should be added that after obtaining the infrared light gradient information and local information entropy corresponding to each pixel of the infrared light optimized image, the pixels can be fused one by one according to the sum and average method to obtain the fused infrared light optimized image, and then a convolution filtering operation is performed on each pixel of the fused infrared light optimized image to obtain an infrared light training image, in which the target edge detail information is more prominent and the contrast between the target and the background is more obvious.

[0109] Step 150: extract a first feature from the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize an image feature of the visible light training image at a high frequency layer, and the second visible light feature is used to characterize an image feature of the visible light training image at a low frequency layer;

[0110] Step 160: extract a second feature from the infrared light training image to obtain a first infrared light feature and a second infrared light feature, wherein the first infrared light feature is used to characterize an image feature of the infrared light training image at a high frequency layer, and the second infrared light feature is used to characterize an image feature of the infrared light training image at a low frequency layer;

[0111] In some embodiments, the image fusion model includes a feature extraction framework, the feature extraction framework is used to perform the following steps:

[0112] C1. Decomposing a target image to obtain a target high-frequency layer image and a target low-frequency layer image, wherein the target image is the visible light training image or the infrared light training image;

[0113] Furthermore, the step C1, performing image decomposition on the target image to obtain a target high-frequency layer image and a target low-frequency layer image, includes:

[0114] C11, performing pixel position convolution on the target image to obtain the target high-frequency layer image;

[0115] C12. Perform image difference calculation on the target image according to the target high-frequency layer image to obtain the target low-frequency layer image.

[0116] In an embodiment of the present application, the target image may be a visible light training image or an infrared light training image, the target high-frequency layer image may be a visible light high-frequency layer image or an infrared light high-frequency layer image; the target low-frequency layer image may be a visible light low-frequency layer image or an infrared light low-frequency layer image. Specifically, in an embodiment of the present application, the target image is a visible light training image, then the corresponding target high-frequency layer image is a visible light high-frequency layer image, and the corresponding target low-frequency layer image is a visible light low-frequency layer image. Step C11 may be based on a high-pass filter convolution kernel, the size of which is 3×3, and convolution is performed on each position of the visible light training image to achieve high-pass filtering, thereby obtaining a visible light high-frequency layer image corresponding to the first visible light feature; step C12 may be performed by calculating the image difference between the visible light training image and the visible light high-frequency layer image, thereby obtaining a visible light low-frequency layer image.

[0117] It is understandable that if the target image is an infrared light training image, the content is similar to the aforementioned target image being a visible light training image, and can be simply deduced. It is worth mentioning that the high-frequency layer image in the embodiment of the present application is used to characterize the image content with drastic changes in image signal intensity (such as brightness, grayscale, etc.), and the low-frequency layer image is used to characterize the image content with smooth changes in image signal intensity. Specifically, the image information output by the high-pass filter convolution kernel can be used as the high-frequency layer image, which will not be elaborated in this application.

[0118] C2. Extract high-frequency features from the target high-frequency layer image to obtain a first target feature, and extract low-frequency features from the target low-frequency layer image to obtain a second target feature.

[0119] In an embodiment of the present application, the first target feature may be a first visible light feature or a first infrared light feature, and the second target feature may be a second visible light feature or a second infrared light feature. If the target high-frequency layer image is a visible light high-frequency layer image, the corresponding target low-frequency layer image is a visible light low-frequency layer image. Specifically, step C2 can be implemented based on the first extraction network and the second extraction network in the feature extraction framework. The first extraction network has the same network structure as the second extraction network, which may specifically include five cascaded convolutional layers, each convolutional layer is connected to a batch normalization layer (BatchNormalization, BN) and an activation function layer (ReLU). The network parameters between the first extraction network and the second extraction network are different. The same network structure between the first extraction network and the second extraction network can effectively reduce the complexity of the image fusion model and reduce the computing resources and time required for image fusion; and the network parameters between the first extraction network and the second extraction network are different, which can effectively improve the quality of the extracted feature information, thereby improving the quality of the fused image.

[0120] It can be understood that in the embodiment of the present application, the visible light high-frequency layer image can be input into the first extraction network to realize feature extraction on the high-frequency layer, and the visible light low-frequency layer image can be input into the second extraction network to realize feature extraction on the low-frequency layer, so as to obtain the first visible light feature and the second visible light feature. In addition, if the target high-frequency layer image is an infrared light high-frequency layer image, and the corresponding target low-frequency layer image is an infrared light low-frequency layer image, it can be simply deduced by analogy to obtain the first infrared light feature and the second infrared light feature, which will not be elaborated in this application.

[0121] Step 170: Perform attention adversarial fusion on the first infrared light feature and the second visible light feature according to the first visible light feature and the second visible light feature to obtain an adversarial fusion image;

[0122] In some embodiments, the step 170 of performing attention adversarial fusion on the first infrared light feature and the second visible light feature according to the first visible light feature and the second visible light feature to obtain an adversarial fusion image includes:

[0123] D1. Based on the first visible light feature, perform high-frequency attention fusion on the first infrared light feature to obtain a high-frequency fused feature image;

[0124] D2. According to the second visible light feature, perform low-frequency attention fusion on the second visible light feature to obtain a low-frequency fusion feature image;

[0125] D3. Based on the low-frequency fusion feature image, generate adversarial fusion on the high-frequency fusion feature image to obtain the adversarial fusion image.

[0126] In an embodiment of the present application, the correlation between the high-frequency layer features of visible light and infrared light, and the correlation between the low-frequency layer features of visible light and infrared light can be captured based on the attention mechanism, which can be specifically captured through attention weights. Specifically, for the first eigenvector in the first visible light feature, and the second eigenvector corresponding to the first eigenvector, the second eigenvector is the eigenvector in the first infrared light feature, step D1 can add the first eigenvector and the second eigenvector element-wise to obtain the attention scores corresponding to the first eigenvector and the second eigenvector, and then normalize the attention scores based on the Softmax function to obtain the attention weights corresponding to the first eigenvector and the second eigenvector.

[0127] It can be understood that the attention weights corresponding to the feature vectors of the remaining first visible light features and the first infrared light features on the high-frequency layer are similar. After obtaining all the attention weights, an attention weight matrix corresponding to the first visible light feature and the first infrared light feature can be constructed. Based on the attention weight matrix, high-frequency fusion between the first visible light feature and the second visible light feature can be achieved, thereby obtaining a high-frequency fused feature image. The equivalent expression of the high-frequency fused feature image can be:

[0128] F DW =P(F i_DW +F b_DW )

[0129] Among them, F DW is the high-frequency fusion feature image; P is the attention weight matrix corresponding to the first visible light feature and the first infrared light feature; F i_DW is the first infrared light feature; F v_DW It is the first visible light feature.

[0130] It should be noted that step D2 is similar to the content of the aforementioned step D1 and can be simply deduced, so this application will not elaborate on it here. Step D3 can be based on an adversarial learning method to generate an adversarial fusion image with rich information of a low-frequency fusion feature image and rich information of a high-frequency fusion feature image. Specifically, the low-frequency fusion feature image and the high-frequency fusion feature image can be input into a generator in a generative adversarial network (GAN), and the output of the generator is determined as an adversarial fusion image.

[0131] Step 180: Update the parameters of the initialized image fusion model according to the adversarial fusion image to obtain the trained image fusion model.

[0132] In some embodiments, the step 180 of updating parameters of the initialized image fusion model according to the adversarial fusion image to obtain the trained image fusion model includes:

[0133] E1. Obtaining a recognition probability value corresponding to the adversarial fusion image;

[0134] E2. According to the discrimination probability value, perform adversarial loss calculation on the adversarial fusion image to obtain a first sub-loss value;

[0135] E3. Calculate the content loss of the adversarial fusion image according to the high-frequency fusion feature image and the low-frequency fusion feature image to obtain a second sub-loss value;

[0136] E4. Calculate the fusion loss of the adversarial fusion image according to the visible light training image and the infrared light training image to obtain a third sub-loss value;

[0137] E5. Determine a target loss value based on the first sub-loss value, the second sub-loss value, and the third sub-loss value;

[0138] E6. Train the initialized image fusion model according to the target loss value to obtain the trained image fusion model.

[0139] In the embodiment of the present application, the discrimination probability value may be the probability value output by the discriminator in the generative adversarial network, the discriminator being used to distinguish between a real image and an adversarial fusion image, the real image being a visible light training image, an infrared light training image, or a sample fusion image corresponding to a visible light training image and an infrared light training image; the first sub-loss value is used to measure the degree of difference between the adversarial fusion image and the discrimination probability value, the first sub-loss value may be calculated by the adversarial loss function of the generator in the generative adversarial network. Specifically, the equivalent expression of the adversarial loss function corresponding to the first sub-loss value in step E2 may be:

[0140] L adv = -log(D(x))

[0141] Among them, L adv is the adversarial loss function corresponding to the first sub-loss value; D(x) is the discrimination probability value corresponding to the adversarial fusion image x.

[0142] It can be understood that the second sub-loss value is used to measure the difference in content between the adversarial fusion image and the high-frequency fusion feature image, and the difference in content between the adversarial fusion image and the low-frequency fusion feature image. The second sub-loss value can be calculated by the content loss function of the generator in the generative adversarial network. Specifically, the equivalent expression of the content loss function corresponding to the second sub-loss value in step E3 can be:

[0143] L content =MSE(x,I BW)+MSE(x,I DW )

[0144] Among them, L content is the content loss function corresponding to the second sub-loss value; MSE() is the mean square error function, x is the adversarial fusion image; I BW is the low-frequency fusion feature image; I DW It is the high-frequency fusion feature image.

[0145] It should be noted that the third sub-loss value is used to measure the degree of retention of high-frequency information and low-frequency information in the adversarial fusion image, that is, the degree of retention of high-frequency information and low-frequency information in the visible light training image and the degree of retention of high-frequency information and low-frequency information in the infrared light training image by the adversarial fusion image. The third sub-loss value can be calculated by proposing a new fusion loss function in the generative adversarial network generator. Specifically, the equivalent expression of the fusion loss function corresponding to the third sub-loss value in step E4 can be:

[0146] L fusion =MSE(x feature ,F i_DW )+MSE(x feature ,F i_BW )+MSE(x feature ,F v_DW )+MSE(x feature ,F v_BW )

[0147] Among them, L fusiom is the fusion loss function corresponding to the third sub-loss value; F i_DW is the first infrared light feature in the infrared light training image; F v_DW is the first visible light feature in the visible light training image; F i_BW is the second infrared light feature in the infrared light training image; F v_BW is the second visible light feature in the visible light training image.

[0148] Step E5 may be to assign a corresponding loss weight to each of the first sub-loss value, the second sub-loss value, and the third sub-loss value, and then perform weighted summation on the first sub-loss value, the second sub-loss value, and the third sub-loss value according to the three corresponding loss weights, so as to calculate the target loss value. After obtaining the target loss value, step E6 may be to update the parameters of the model using a back propagation algorithm, and a trained image fusion model may be obtained after several rounds of iteration. The specific number of iterations may be preset, or the training may be considered completed when the test set meets the accuracy requirements.

[0149] It is worth mentioning that during the back propagation of the model, the parameters of the discriminator in the generative adversarial network will also be updated. During each back propagation of the model, the discriminator can be updated based on the discriminant loss value of the discriminator. Specifically, the equivalent expression of the discriminant loss function of the discriminator can be:

[0150] L D =-[y*log(D(c))+(1-y)*log(1-D(c))]

[0151] Among them, L D is the discrimination loss function of the discriminator; D(c) is the discrimination probability value, c is the discriminator input image, which can be an adversarial fusion image or a real image; y is the label of the discriminator input image.

[0152] It should be added that, in the embodiment of the present application, by proposing a third sub-loss value, which is used to measure the similarity of high-frequency and low-frequency features between the adversarial fusion image and the real image, the image fusion model can be constrained to completely retain the key high-frequency and low-frequency features in the visible and infrared images during the image fusion process. While pursuing realism, it can maintain a close connection with the real image, so that the generated fused adversarial image is always highly consistent with the real image. As a result, when the smart pod faces complex actual scenes, such as light changes, target occlusion, smart pod vibration, etc., the smart pod can still obtain more accurate and realistic image output, thereby improving the accuracy of decision-making.

[0153] A visible and infrared image fusion system proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0154] Reference Figure 3 A visible and infrared image fusion system proposed in an embodiment of the present application includes:

[0155] An acquisition unit 101 is used to acquire a visible light image and an infrared light image to be fused;

[0156] A processing unit 102, configured to input the visible light image and the infrared light image into a trained image fusion model to obtain a fused image;

[0157] The trained image fusion model is obtained by training through the following steps:

[0158] Acquire original visible light collection images and infrared light collection images;

[0159] Performing a first preprocessing on the visible light collected image to obtain a visible light training image, and performing a second preprocessing on the infrared light collected image to obtain an infrared light training image;

[0160] Performing a first feature extraction on the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize an image feature of the visible light training image at a high frequency layer, and the second visible light feature is used to characterize an image feature of the visible light training image at a low frequency layer;

[0161] Performing a second feature extraction on the infrared light training image to obtain a first infrared light feature and a second infrared light feature, wherein the first infrared light feature is used to characterize an image feature of the infrared light training image at a high frequency layer, and the second infrared light feature is used to characterize an image feature of the infrared light training image at a low frequency layer;

[0162] According to the first visible light feature and the second visible light feature, performing attention adversarial fusion on the first infrared light feature and the second visible light feature to obtain an adversarial fusion image;

[0163] According to the adversarial fusion image, the parameters of the initialized image fusion model are updated to obtain the trained image fusion model.

[0164] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] Reference Figure 4 , the embodiment of the present application further provides an electronic device, including:

[0166] at least one processor 201;

[0167] At least one memory 202, used to store at least one program;

[0168] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0169] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0171] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0172] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0173] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0174] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0175] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0176] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0177] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0178] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0179] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0180] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for fusing visible and infrared images, It is characterized in that include: Acquire a visible light image and an infrared light image to be fused; Inputting the visible light image and the infrared light image into a trained image fusion model to obtain a fused image; The trained image fusion model is obtained by training through the following steps: Acquire original visible light collection images and infrared light collection images; Performing a first preprocessing on the visible light collected image to obtain a visible light training image, and performing a second preprocessing on the infrared light collected image to obtain an infrared light training image; Performing a first feature extraction on the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize an image feature of the visible light training image at a high frequency layer, and the second visible light feature is used to characterize an image feature of the visible light training image at a low frequency layer; Performing a second feature extraction on the infrared light training image to obtain a first infrared light feature and a second infrared light feature, wherein the first infrared light feature is used to characterize an image feature of the infrared light training image at a high frequency layer, and the second infrared light feature is used to characterize an image feature of the infrared light training image at a low frequency layer; According to the first visible light feature and the second visible light feature, performing attention adversarial fusion on the first infrared light feature and the second visible light feature to obtain an adversarial fusion image; According to the adversarial fusion image, the parameters of the initialized image fusion model are updated to obtain the trained image fusion model.

2. The fusion method according to claim 1, It is characterized in that The first preprocessing of the visible light collected image to obtain a visible light training image includes: Performing image transformation on the visible light collected image to obtain a first visible light image; performing image normalization on the first visible light image to obtain a second visible light image; Perform image denoising on the second visible light image to obtain the visible light training image.

3. The fusion method according to claim 1, It is characterized in that The performing second preprocessing on the infrared light acquisition image to obtain an infrared light training image includes: Obtain preset infrared pixel blocks and infrared templates; According to the infrared pixel blocks, grayscale mean optimization is performed on the infrared light collected image to obtain an infrared light optimized image; According to the infrared template, local information extraction is performed on the infrared light optimized image to obtain infrared light gradient information and local information entropy corresponding to the infrared light optimized image; According to the infrared light gradient information, the image feature value fusion is performed on the local information entropy to obtain the infrared light training image.

4. The fusion method according to claim 1, It is characterized in that The image fusion model includes a feature extraction framework, which is used to perform the following steps: Performing image decomposition on a target image to obtain a target high-frequency layer image and a target low-frequency layer image, wherein the target image is the visible light training image or the infrared light training image; A high-frequency feature extraction is performed on the target high-frequency layer image to obtain a first target feature, and a low-frequency feature extraction is performed on the target low-frequency layer image to obtain a second target feature.

5. The fusion method according to claim 4, It is characterized in that Decomposing the target image to obtain a target high-frequency layer image and a target low-frequency layer image includes: Performing pixel position convolution on the target image to obtain the target high-frequency layer image; According to the target high-frequency layer image, image difference calculation is performed on the target image to obtain the target low-frequency layer image.

6. The fusion method according to claim 1, It is characterized in that The step of performing attention adversarial fusion on the first infrared light feature and the second visible light feature according to the first visible light feature and the second visible light feature to obtain an adversarial fusion image includes: According to the first visible light feature, high-frequency attention fusion is performed on the first infrared light feature to obtain a high-frequency fused feature image; According to the second visible light feature, performing low-frequency attention fusion on the second visible light feature to obtain a low-frequency fusion feature image; According to the low-frequency fusion feature image, the high-frequency fusion feature image is generatively fused to obtain the adversarial fusion image.

7. The fusion method according to claim 6, It is characterized in that The step of updating parameters of the initialized image fusion model according to the adversarial fusion image to obtain the trained image fusion model includes: Obtaining a recognition probability value corresponding to the adversarial fusion image; According to the discrimination probability value, performing adversarial loss calculation on the adversarial fusion image to obtain a first sub-loss value; According to the high-frequency fusion feature image and the low-frequency fusion feature image, performing content loss calculation on the adversarial fusion image to obtain a second sub-loss value; According to the visible light training image and the infrared light training image, a fusion loss is calculated for the adversarial fusion image to obtain a third sub-loss value; Determining a target loss value based on the first sub-loss value, the second sub-loss value, and the third sub-loss value; The initialized image fusion model is trained according to the target loss value to obtain the trained image fusion model.

8. A visible and infrared image fusion system, It is characterized in that include: An acquisition unit, used for acquiring a visible light image and an infrared light image to be fused; A processing unit, used for inputting the visible light image and the infrared light image into a trained image fusion model to obtain a fused image; The trained image fusion model is obtained by training through the following steps: Acquire original visible light collection images and infrared light collection images; Performing a first preprocessing on the visible light collected image to obtain a visible light training image, and performing a second preprocessing on the infrared light collected image to obtain an infrared light training image; Performing a first feature extraction on the visible light training image to obtain a first visible light feature and a second visible light feature, wherein the first visible light feature is used to characterize an image feature of the visible light training image at a high frequency layer, and the second visible light feature is used to characterize an image feature of the visible light training image at a low frequency layer; Performing a second feature extraction on the infrared light training image to obtain a first infrared light feature and a second infrared light feature, wherein the first infrared light feature is used to characterize an image feature of the infrared light training image at a high frequency layer, and the second infrared light feature is used to characterize an image feature of the infrared light training image at a low frequency layer; According to the first visible light feature and the second visible light feature, performing attention adversarial fusion on the first infrared light feature and the second visible light feature to obtain an adversarial fusion image; According to the adversarial fusion image, the parameters of the initialized image fusion model are updated to obtain the trained image fusion model.

9. An electronic device, It is characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program executable by a processor stored therein, It is characterized in that The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images