A method and device for modeling image fusion model based on visible light and infrared

Abstract

The present application discloses a modeling method and device for an image fusion model based on visible light and infrared. The technical solution provided by the present application first fuses a visible light source image and an infrared source image through a preset image fusion model to obtain a fused image, and then decomposes the fused image into a first decomposition image and a second decomposition image through an image decomposition model. Since the quality of the decomposition result directly depends on the fused image, by comparing the decomposed image with the source image, if the decomposed image is closer to the corresponding source image, it means that more useful information of the source image is retained in the fused image. Based on the comparison result, the model parameters of the image fusion model are cyclically iteratively optimized. Through multiple parameter optimization iterations, the degree of retention of useful information of the source image in the fused image output by the image fusion model is improved. Finally, the optimized image fusion model is used to obtain a fused image with better comprehensive performance.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for modeling an image fusion model based on visible light and infrared. Background Art

[0002] Image fusion is a technology encompassing the field of computer vision that combines the advantages of images from different spectral sources into a single image. Infrared images can capture the thermal radiation emitted by objects, highlighting them, but they often overlook textural details in the environment. Conversely, visible light images contain rich scene texture details, but perform poorly in conditions such as inclement weather, low light levels, at night, and under occlusion. Therefore, the goal of image fusion is to leverage the strengths of different spectral images to produce a fused image with superior overall performance.

[0003] Infrared and visible light images are often combined in substation equipment inspection methods. Infrared images can capture the thermal radiation of equipment and highlight its temperature signature, but their sensitivity to changes in environmental texture details is limited. Visible light images, on the other hand, are more responsive to changes in scene texture detail and offer high image clarity, but they cannot accurately reflect the temperature distribution of the equipment. Therefore, fusing infrared and visible light images to leverage their combined information can more effectively locate potential faults in power equipment.

[0004] At present, the more common image fusion method is the fusion method based on deep learning. However, this type of method is prone to generate a large amount of redundant and inefficient information during the fusion process, reducing the proportion of valuable information, thereby reducing the quality of the fused image. Summary of the Invention

[0005] The present application provides a method and device for modeling an image fusion model based on visible light and infrared, which is used to solve the technical problem of low quality of fused images in existing image fusion methods.

[0006] To solve the above technical problems, the first aspect of the present application provides a method for modeling a visible light and infrared image fusion model, comprising:

[0007] Acquire a fused sample source image, where the fused sample source image includes a visible light source image and an infrared source image corresponding to each other;

[0008] Performing image fusion on the visible light source image and the infrared source image using a preset image fusion model to obtain a fused image;

[0009] Inputting the fused image into a preset image decomposition model to decompose the fused image into a first decomposition image and a second decomposition image through the image decomposition model, wherein the first decomposition image is a visible light image decomposed based on the fused image, and the second decomposition image is an infrared image decomposed based on the fused image;

[0010] The first decomposed image, the second decomposed image, the visible light source image, and the infrared source image are compared; based on the comparison results, the model parameters of the image fusion model are cyclically iteratively optimized, and the optimized image fusion model is output.

[0011] Preferably, comparing the first decomposed image, the second decomposed image with the visible light source image and the infrared source image, and iteratively optimizing the model parameters of the image fusion model according to the comparison results, and outputting the optimized image fusion model specifically includes:

[0012] Comparing the first decomposed image and the second decomposed image with the visible light source image and the infrared source image, and calculating the image loss coefficient of the fused image based on the comparison results and in combination with an image loss coefficient calculation formula, wherein the image loss coefficient is the sum of the loss function outputs of the image fusion model and the image decomposition model, and is used to measure the degree of preservation of beneficial information of the source images in the fused image;

[0013] According to the image loss coefficient, the model parameters of the image fusion model are cyclically iterated and optimized, and the optimized image fusion model is output.

[0014] Preferably, the loss function of the image fusion model is:

[0015] L syn =L intensity +αL grad

[0016]

[0017]

[0018] Where, L syn is the loss function of the image fusion model, L intensity is the strength loss, L grad is the gradient loss, H and W are the height and width of the image respectively, i and j represent the pixels in the i-th row and j-th column in the feature map, I f , I ir and I vi are the fusion image, infrared source image and visible light source image respectively, β1 is the hyperparameter that controls the ratio of infrared and visible light image intensity information, is a gradient operator. In this invention, the Laplace operator is used to calculate the gradient, and β2 is a hyperparameter that controls the ratio of gradient information of infrared and visible light images.

[0019] Preferably, the loss function of the image decomposition model is:

[0020]

[0021] Where H and W are the height and width of the image respectively, i and j represent the pixel in the i-th row and j-th column of the image, and I ir and I vi are infrared source images and visible light source images respectively, I videc and I irdec They are the first decomposed image and the second decomposed image respectively.

[0022] Preferably, the performing image fusion on the visible light source image and the infrared source image by using a preset image fusion model to obtain a fused image specifically includes:

[0023] The image fusion is performed on the visible light source image and the infrared source image through a preset image fusion model and L1 normalization and soft-max joint fusion logic included in the image fusion model to obtain a fused image.

[0024] The second aspect of the present application provides a device for modeling a visible light and infrared image fusion model, comprising:

[0025] A source image acquisition unit, configured to acquire a fused sample source image, wherein the fused sample source image includes a visible light source image and an infrared source image corresponding to each other;

[0026] an image fusion unit, configured to fuse the visible light source image and the infrared source image using a preset image fusion model to obtain a fused image;

[0027] an image decomposition unit, configured to input the fused image into a preset image decomposition model, so as to decompose the fused image into a first decomposition image and a second decomposition image through the image decomposition model, wherein the first decomposition image is a visible light image decomposed based on the fused image, and the second decomposition image is an infrared image decomposed based on the fused image;

[0028] The fusion model optimization unit is used to compare the first decomposed image, the second decomposed image with the visible light source image and the infrared source image, and according to the comparison result, iteratively optimize the model parameters of the image fusion model and output the optimized image fusion model.

[0029] Preferably, the fusion model optimization unit is specifically used to:

[0030] Comparing the first decomposed image and the second decomposed image with the visible light source image and the infrared source image, and calculating the image loss coefficient of the fused image based on the comparison results and in combination with an image loss coefficient calculation formula, wherein the image loss coefficient is the sum of the loss function outputs of the image fusion model and the image decomposition model, and is used to measure the degree of preservation of beneficial information of the source images in the fused image;

[0031] According to the image loss coefficient, the model parameters of the image fusion model are cyclically iterated and optimized, and the optimized image fusion model is output.

[0032] Preferably, the image fusion unit specifically includes:

[0033] The image fusion is performed on the visible light source image and the infrared source image through a preset image fusion model and L1 normalization and soft-max joint fusion logic included in the image fusion model to obtain a fused image.

[0034] A third aspect of the present application further provides a modeling terminal for an image fusion model based on visible light and infrared, comprising: a memory and a processor;

[0035] The memory is used to store program code, and the program code corresponds to the image fusion modeling method based on visible light and infrared provided in the first aspect of the present application;

[0036] The processor is configured to execute the program code.

[0037] The fourth aspect of the present application provides a computer-readable storage medium, which stores program code corresponding to the image fusion modeling method based on visible light and infrared provided in the first aspect of the present application.

[0038] It can be seen from the above technical solutions that this application has the following advantages:

[0039] The technical solution provided in the present application first performs image fusion on the visible light source image and the infrared source image through a preset image fusion model to obtain a fused image, and then uses an image decomposition model to decompose the fused image into a first decomposition image and a second decomposition image. Since the quality of the decomposition result directly depends on the fused image, by comparing the decomposition image with the source image, if the decomposition image is closer to the corresponding source image, it means that more useful information of the source image is retained in the fused image. Based on the comparison result, the model parameters of the image fusion model are cyclically iteratively optimized. Through multiple parameter optimization iterations, the degree of retention of useful information of the source image in the fused image output by the image fusion model is improved. Finally, the optimized image fusion model is used to fuse the visible light image and the infrared image to obtain a fused image with better comprehensive performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 This is a flow chart of an embodiment of a method for modeling an image fusion model based on visible light and infrared provided in this application.

[0042] Figure 2 This is an overall framework diagram of a modeling method for an image fusion model based on visible light and infrared provided in this application.

[0043] Figure 3 This is a network structure diagram of a modeling method for an image fusion model based on visible light and infrared provided in this application.

[0044] Figure 4 This is a schematic diagram of the framework of the image fusion model provided in this application.

[0045] Figure 5 A comparison chart of the fusion effects of the image fusion model provided in this application and other image fusion algorithms.

[0046] Figure 6 This is a schematic diagram of an embodiment of a device for modeling an image fusion model based on visible light and infrared provided in this application.

[0047] Figure 7 The present application provides a schematic diagram of a device embodiment of a modeling terminal based on a visible light and infrared image fusion model. DETAILED DESCRIPTION

[0048] A major challenge in applying deep learning to image fusion is the lack of a standard ground truth for supervised learning. Some methods attempt to overcome this difficulty by manually constructing the ground truth for the fusion results. However, this approach is often inaccurate and can cause image fusion neural networks to generate a large amount of redundant and inefficient information during the fusion process, reducing the proportion of valuable information and, to a certain extent, limiting the learning potential of the neural network.

[0049] In view of this, an embodiment of the present application provides a method and device for modeling an image fusion model based on visible light and infrared, which is used to solve the technical problem of low quality of fused images in existing image fusion methods.

[0050] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] First, a detailed description of an embodiment of a method for modeling a visible light and infrared image fusion model provided by this application is as follows:

[0052] See also Figure 1 This embodiment provides a method for modeling a visible light and infrared image fusion model, including:

[0053] Step 101: Acquire a fused sample source image, where the fused sample source image includes a visible light source image and an infrared source image corresponding to each other.

[0054] Step 102: Perform image fusion on the visible light source image and the infrared source image using a preset image fusion model to obtain a fused image.

[0055] It should be noted that, according to steps 101 and 102, the solution provided in this embodiment first obtains several sets of fused sample source images, each set of fused sample source images including corresponding visible light source images and infrared source images. In this embodiment, the corresponding visible light source images and infrared source images refer to visible light images and infrared images containing the same photographic target. The image fusion model is then used to fuse the visible light source images and infrared source images to produce a fused image.

[0056] Step 103: Input the fused image into a preset image decomposition model, so as to decompose the fused image into a first decomposition image and a second decomposition image through the image decomposition model.

[0057] The first decomposed image is a visible light image obtained by decomposing the fused image, and the second decomposed image is an infrared image obtained by decomposing the fused image.

[0058] Step 104: Compare the first decomposed image, the second decomposed image, the visible light source image, and the infrared source image. Based on the comparison results, perform iterative optimization on the model parameters of the image fusion model, and output the optimized image fusion model.

[0059] It should be noted that, based on the fused image obtained in step 102, the fused image is input into a preset image decomposition model to decompose the fused image into a first decomposition image and a second decomposition image through the image decomposition model. The quality of the decomposition result directly depends on the fused image, and the closer the decomposition result is to the initial source image, the more the fused image output by the image fusion model can retain the beneficial information of the source image. Therefore, after obtaining the decomposed image, this embodiment compares the first decomposition image and the second decomposition image with the visible light source image and the infrared source image. According to the comparison result, the model parameters of the image fusion model are cyclically iteratively optimized, and under the guidance of the loss function, back propagation is continuously performed to update the learning content and parameter weights. After several iterations, a lower loss value is obtained, and the decomposition result of the fused image is continuously made closer to the source image, and finally a trained image fusion network is obtained, so that the fusion network can obtain a fused image with better performance.

[0060] The above is a detailed description of a basic embodiment of a method for modeling a visible light and infrared image fusion model provided by this application. Based on the above embodiment, the embodiment of the method for modeling a visible light and infrared image fusion model provided by this application may also include the following technical contents:

[0061] In some embodiments, step 104 may include:

[0062] The first decomposed image and the second decomposed image are compared with the visible light source image and the infrared source image. According to the comparison results and in combination with the image loss coefficient calculation formula, the image loss coefficient of the fused image is calculated.

[0063] The image loss coefficient is the sum of the loss function outputs of the image fusion model and the image decomposition model, and is used to measure the degree of preservation of beneficial information of the source images in the fused image.

[0064] According to the image loss coefficient, the model parameters of the image fusion model are optimized iteratively and the optimized image fusion model is output.

[0065] It should be noted that the image loss coefficient calculation formula of this embodiment is based on the synthetic network loss function L synAnd decomposition network loss function L dec It consists of two parts:

[0066] L=L syn +L dec

[0067] In the synthetic network loss function, it is divided into intensity loss and gradient loss:

[0068] L syn =L intensity +αL grad

[0069] Among them, α is a hyperparameter that controls the loss balance in different regions. In this application, α is set to 60.

[0070] Strength loss L intensity The guided fused image retains the beneficial information represented by pixel intensity and can improve the fidelity of the fused image, which is defined as follows:

[0071]

[0072] Where H and W are the height and width of the image, i and j represent the pixels in the i-th row and j-th column of the feature map, and I f , I ir and I vi are the fused image, infrared source image and visible light source image respectively. β1 is a hyperparameter that controls the ratio of the intensity information of infrared and visible light images. In this application, β1 is set to 0.5.

[0073] Gradient loss L grad The fused image contains richer texture information, which is defined as follows:

[0074]

[0075] in, is a gradient operator. The invention adopts the Laplace operator to calculate the gradient. β2 is a hyperparameter that controls the ratio of gradient information of infrared and visible light images. In this application, β2 is set to 0.5.

[0076] Decomposition network loss function L dec The decomposed image obtained by decomposing the fused image is made as close as possible to the original source image, which is defined as follows:

[0077]

[0078] Where H and W are the height and width of the image respectively, i and j represent the pixel in the i-th row and j-th column of the image, and I ir and I vi are infrared source images and visible light source images respectively, Ividec and I irdec They are the first decomposed image and the second decomposed image respectively.

[0079] In some embodiments, step 102 of this embodiment includes the following steps:

[0080] Through the preset image fusion model and the L1 normalization and soft-max joint fusion logic contained in the image fusion model, the visible light source image and the infrared source image are fused to obtain a fused image.

[0081] It should be noted that when using the fusion strategy to select significant features, the effect of using the ordinary addition strategy is relatively general and can only roughly fuse features. Therefore, this application combines L1 normalization with soft-max to select features. This fusion strategy can better calculate the weights of each feature map and obtain better fusion results.

[0082] The image multi-processing fusion framework based on the synthesis and decomposition network proposed in this application, such as Figure 2 As shown, its specific image fusion network architecture includes image fusion model and image decomposition model. The specific logic block diagrams of the two networks are shown in Figure 3 As shown in Figure 3, the two complement each other and make the fused image have better performance.

[0083] To better process the diverse spectral images captured by different sensors, the synthesis subnetwork employs a multi-branch structure to perform multivariate processing on the source image. Four convolutional layers are used in each of the two branches for processing the source image's private modality. Except for the first convolutional layer, which has a kernel size of 5×5, all other layers have a kernel size of 3×3. Dense connections are used between the third and fourth convolutional layers to minimize information loss, and the activation function is a leaky rectified linear unit (LReLU). In each of the two branches for processing the source image's shared modality, three convolutional layers are used for feature extraction. Except for the first convolutional layer, which has a kernel size of 5×5, all other layers have a kernel size of 3×3, and the activation function is also set to LReLU. The features of the two branches are then combined into a single shared modality branch. Finally, the features of the two branches of the private modality are combined with the features of the single branch of the shared modality using a specific fusion strategy based on a combination of L1 norm and soft-max. Finally, a 1×1 convolutional layer with a hyperbolic tangent (Tanh) activation function is used to obtain the fused image.

[0084] Specifically, the fusion strategy is as follows Figure 4 As shown, represents the feature map obtained by the encoder from the input image, Represents the activity level diagram, Calculated by the L1 norm and the block-based average operator, f m Represents the fused feature map. Initial activity level map C i Calculated by the following formula:

[0085]

[0086] Among them, ||||1 represents the L1 norm, M represents the number of feature maps, and (x, y) represents the corresponding position in the feature map and the fused feature map.

[0087] Final activity level graph Computed using the block-based averaging operator:

[0088]

[0089] Wherein, r represents the block size, which is set to 1 in this application.

[0090] After obtaining the final activity level map, the weights of each feature map can be calculated:

[0091]

[0092] Finally, the fused feature map is obtained:

[0093]

[0094] The final fused image will be reconstructed by the decoder, where the fused feature map f m As input, according to the above strategy, the two branch features of the common modality of the source image are first integrated into one, and the integrated result is then strategically fused with the single branch of the private modality of the source image to obtain the initial image fusion result.

[0095] After obtaining the fused image through the synthesis sub-network, this embodiment relatively decomposes it using a decomposition sub-network, allowing it to be closer to the multi-spectral source image under the guidance of the loss function, thereby making the performance of the fused image more excellent. The decomposition sub-network first decomposes the fused image into an infrared branch and a visible light branch by a common convolution layer with a convolution kernel size of 1*1, and the number of feature map channels is 128. Then, the decomposition result is generated by a dual-branch network structure with three convolution layers with a convolution kernel size of 3*3. The number of output feature map channels is 128, 16, and 4 respectively. Except for the last convolution layer that uses the Tanh activation function, the remaining convolution layers all use LReLU as the activation function. It is worth mentioning that this image fusion network uses LReLU as the activation function. Compared with the ReLU function commonly used in the field of deep learning, the LReLU function does not ignore negative activations, which makes the information better retained and more suitable for the field of image fusion.

[0096] This application also conducts a quantitative analysis of four image evaluation indicators, namely information entropy, interaction information, spatial frequency and visual fidelity, on the power equipment dataset.

[0097] Table 1 Quantitative index evaluation data of different fusion methods

[0098] Methods / Indicators Information entropy Interaction Information Spatial frequency Visual fidelity FusionGAN 6.282 1.115 7.633 0.149 U2Fusion 6.505 1.103 16.237 0.218 Densefuse 6.449 1.273 14.230 0.288 PMGI 6.620 1.193 15.282 0.308 MDLatLrr 6.286 1.355 17.337 0.273 This application 6.814 1.297 18.532 0.383

[0099] As shown in the quantitative index evaluation data in Table 1, the method of the present application has significant advantages in the three image quality evaluation indicators of information entropy, spatial frequency and visual fidelity. For the mutual information indicator, it is second only to MDLatLrr. In information theory, information entropy mainly measures the amount of information contained in the image. The larger the information entropy value, the more information the image contains, which shows that the fused image of the method of the present application has a large amount of information. The mutual information measures the amount of information transferred from the source image to the fused image. Although the present application does not have the highest mutual information value, the second highest mutual information value shows that the present application has transferred enough information from the source image to the fused image and has a good fusion result. Spatial frequency measures the spatial frequency information contained in the fused image through the row frequency and column frequency of the image, reflecting the grayscale change rate of the image. The present application has the largest spatial frequency value, which shows that the fusion result of the present application has the best gradient information and fusion quality, so that the fused image presents a clearer effect. As shown Figure 5 As shown, the image visual fidelity measures the information fidelity of the fused image, which is consistent with the human visual system. This application also has the largest visual fidelity value, which is sufficient to show that the image fusion result of this application is more consistent with human visual perception and has a better fusion effect.

[0100] The above is a detailed description of an embodiment of an image fusion model modeling method based on visible light and infrared provided by this application. The following is a detailed description of an embodiment of an image fusion model modeling device based on visible light and infrared provided by this application.

[0101] See also Figure 6 In a second aspect, the present application provides a device for modeling a visible light and infrared image fusion model, comprising:

[0102] The source image acquisition unit 201 is used to acquire a fused sample source image, where the fused sample source image includes a visible light source image and an infrared source image corresponding to each other;

[0103] An image fusion unit 202 is configured to fuse the visible light source image and the infrared source image using a preset image fusion model to obtain a fused image;

[0104] An image decomposition unit 203 is configured to input the fused image into a preset image decomposition model to decompose the fused image into a first decomposition image and a second decomposition image using the image decomposition model, wherein the first decomposition image is a visible light image decomposed based on the fused image, and the second decomposition image is an infrared image decomposed based on the fused image;

[0105] The fusion model optimization unit 204 is used to compare the first decomposed image, the second decomposed image with the visible light source image and the infrared source image, and according to the comparison results, iteratively optimize the model parameters of the image fusion model and output the optimized image fusion model.

[0106] Furthermore, the fusion model optimization unit 204 is specifically configured to:

[0107] Compare the first decomposed image and the second decomposed image with the visible light source image and the infrared source image. Based on the comparison results and the image loss coefficient calculation formula, calculate the image loss coefficient of the fused image. The image loss coefficient is the sum of the loss function outputs of the image fusion model and the image decomposition model, and is used to measure the degree of preservation of beneficial information of the source images in the fused image.

[0108] According to the image loss coefficient, the model parameters of the image fusion model are optimized iteratively and the optimized image fusion model is output.

[0109] Furthermore, the loss function of the image fusion model is:

[0110] L syn =L intensity +αL grad

[0111]

[0112]

[0113] Where, L syn is the loss function of the image fusion model, L intensity is the strength loss, L grad is the gradient loss, H and W are the height and width of the image respectively, i and j represent the pixels in the i-th row and j-th column in the feature map, I f , I ir and I vi are the fusion image, infrared source image and visible light source image respectively, β1 is the hyperparameter that controls the ratio of infrared and visible light image intensity information, is a gradient operator. In this invention, the Laplace operator is used to calculate the gradient, and β2 is a hyperparameter that controls the ratio of gradient information of infrared and visible light images.

[0114] Furthermore, the loss function of the image decomposition model is:

[0115]

[0116] Where H and W are the height and width of the image respectively, i and j represent the pixel in the i-th row and j-th column of the image, and I ir and I vi are infrared source images and visible light source images respectively, I videc and I irdec They are the first decomposed image and the second decomposed image respectively.

[0117] Furthermore, the image fusion unit 202 specifically includes:

[0118] Through the preset image fusion model and the L1 normalization and soft-max joint fusion logic contained in the image fusion model, the visible light source image and the infrared source image are fused to obtain a fused image.

[0119] In addition, the present application also provides a specific description of embodiments such as a modeling terminal based on a visible light and infrared image fusion model and a computer-readable storage medium.

[0120] like Figure 7 As shown, an embodiment of the present application provides a modeling terminal based on a visible light and infrared image fusion model. Terminal types include but are not limited to: personal computers, industrial computers, servers, and embedded intelligent terminals. The terminal mainly includes: a memory 33 and a processor 31. The memory 33 and the processor 31 can be connected via a communication bus 34.

[0121] The memory 33 is used to store program codes, which correspond to the image fusion modeling method based on visible light and infrared provided in the above embodiment;

[0122] The processor 31 is configured to execute the program code to implement the image fusion modeling method based on visible light and infrared provided in the above embodiment.

[0123] An embodiment of the present application provides a computer-readable storage medium, in which program codes corresponding to the image fusion modeling method based on visible light and infrared as in the above embodiment are stored.

[0124] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0126] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0127] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0128] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0131] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for modeling a visible light and infrared image fusion model, characterized in that: include: Acquire a fused sample source image, where the fused sample source image includes a visible light source image and an infrared source image corresponding to each other; The visible light source image and the infrared source image are fused by a preset image fusion model to obtain a fused image; wherein, the image fusion model adopts a multi-branch convolutional layer structure, the activation function is the LReLU function, and the image fusion model is configured as follows: comprising two private modal processing branches and two shared modal processing branches, wherein each private modal processing branch comprises four convolutional layers for feature extraction, and a dense connection is adopted between the third and fourth convolutional layers; each shared modal processing branch comprises three convolutional layers for feature extraction; the features of the two shared modal processing branches are integrated into a single shared modal branch, the two branch features of the private modality and the single branch features of the shared modality are integrated through a fusion strategy based on the combined use of L1 norm and soft-max, and then processed through a convolutional layer with the activation function being the hyperbolic tangent to obtain a fused image; Inputting the fused image into a preset image decomposition model to decompose the fused image into a first decomposition image and a second decomposition image through the image decomposition model, wherein the first decomposition image is a visible light image decomposed based on the fused image, and the second decomposition image is an infrared image decomposed based on the fused image; The first decomposed image, the second decomposed image, the visible light source image, and the infrared source image are compared; based on the comparison results, the model parameters of the image fusion model are cyclically iteratively optimized, and the optimized image fusion model is output.

2. The method for modeling a visible light and infrared image fusion model according to claim 1, characterized in that: The step of comparing the first decomposed image, the second decomposed image, the visible light source image, and the infrared source image, and iteratively optimizing the model parameters of the image fusion model according to the comparison results, and outputting the optimized image fusion model specifically includes: Comparing the first decomposed image and the second decomposed image with the visible light source image and the infrared source image, and calculating the image loss coefficient of the fused image based on the comparison results and in combination with an image loss coefficient calculation formula, wherein the image loss coefficient is the sum of the loss function outputs of the image fusion model and the image decomposition model, and is used to measure the degree of preservation of beneficial information of the source images in the fused image; According to the image loss coefficient, the model parameters of the image fusion model are cyclically iterated and optimized, and the optimized image fusion model is output.

3. The method for modeling a visible light and infrared image fusion model according to claim 2, characterized in that: The loss function of the image fusion model is: L syn =L intensity +αL grad Where, L syn is the loss function of the image fusion model, L intensity is the strength loss, L grad is the gradient loss, H and W are the height and width of the image respectively, i and j represent the pixels in the i-th row and j-th column in the feature map, I f , I ir and I vi are the fusion image, infrared source image and visible light source image respectively, β1 is the hyperparameter that controls the ratio of infrared and visible light image intensity information, is the gradient operator, the Laplace operator is used to calculate the gradient, and β2 is a hyperparameter that controls the ratio of gradient information of infrared and visible light images.

4. The method for modeling a visible light and infrared image fusion model according to claim 2, characterized in that: The loss function of the image decomposition model is: Where H and W are the height and width of the image respectively, i and j represent the pixel in the i-th row and j-th column of the image, and I ir and I vi are infrared source images and visible light source images respectively, I videc and I irdec They are the first decomposed image and the second decomposed image respectively.

5. A modeling device for image fusion model based on visible light and infrared, characterized in that: include: A source image acquisition unit, configured to acquire a fused sample source image, wherein the fused sample source image includes a visible light source image and an infrared source image corresponding to each other; An image fusion unit is configured to perform image fusion on the visible light source image and the infrared source image through a preset image fusion model to obtain a fused image, wherein the image fusion model adopts a multi-branch convolutional layer structure, the activation function is an LReLU function, and the image fusion model is configured to: include two private modality processing branches and two shared modality processing branches, wherein each private modality processing branch includes four convolutional layers for feature extraction, and a dense connection is adopted between the third and fourth convolutional layers; each shared modality processing branch includes three convolutional layers for feature extraction, and then the features of the two shared modality processing branches are integrated into a single shared modality branch, the two branch features of the private modality and the single branch feature of the shared modality are integrated through a specific fusion strategy based on the combined use of L1 norm and soft-max, and then processed through a convolutional layer with a hyperbolic tangent activation function to obtain a fused image; an image decomposition unit, configured to input the fused image into a preset image decomposition model, so as to decompose the fused image into a first decomposition image and a second decomposition image through the image decomposition model, wherein the first decomposition image is a visible light image decomposed based on the fused image, and the second decomposition image is an infrared image decomposed based on the fused image; The fusion model optimization unit is used to compare the first decomposed image, the second decomposed image with the visible light source image and the infrared source image, and according to the comparison result, iteratively optimize the model parameters of the image fusion model and output the optimized image fusion model.

6. The device for modeling a visible light and infrared image fusion model according to claim 5, characterized in that: The fusion model optimization unit is specifically used for: Comparing the first decomposed image and the second decomposed image with the visible light source image and the infrared source image, and calculating the image loss coefficient of the fused image based on the comparison results and in combination with an image loss coefficient calculation formula, wherein the image loss coefficient is the sum of the loss function outputs of the image fusion model and the image decomposition model, and is used to measure the degree of preservation of beneficial information of the source images in the fused image; According to the image loss coefficient, the model parameters of the image fusion model are cyclically iterated and optimized, and the optimized image fusion model is output.

7. A modeling terminal based on a visible light and infrared image fusion model, characterized in that: include: memory and processor; The memory is used to store program code, and the program code corresponds to the image fusion modeling method based on visible light and infrared according to any one of claims 1 to 4; The processor is configured to execute the program code.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code corresponding to the image fusion modeling method based on visible light and infrared according to any one of claims 1 to 4.

Images