Light-weight network design method for infrared and visible light image fusion

Abstract

The invention relates to the technical field of infrared and visible light image fusion, in particular to an infrared and visible light image fusion-oriented lightweight network design method, which comprises the following steps of: introducing an SE module, channel shuffling and residual connection on the basis of deep convolution and point-by-point convolution, and combining linear transformation and channel dimension splicing, so as to obtain a lightweight network; jointly constructing a lightweight convolution module; respectively constructing an encoder and a decoder based on the lightweight convolution module, and forming an auto-encoder network based on the encoder and the decoder; iteratively training the auto-encoder network, and updating parameters of an encoder and a decoder by taking a total loss function in a segmented form based on the number of training rounds, pixel loss and SSIM loss as guidance to obtain the trained auto-encoder network; and inserting a fusion module into the trained auto-encoder network to form a fusion model. According to the method, the parameter quantity and the calculation quantity of the fusion model can be reduced and the image fusion speed can be improved under the condition of ensuring the image fusion quality.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of infrared and visible light image fusion, and in particular to a lightweight network design method for infrared and visible light image fusion. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, deep neural networks have become a research hotspot and have been widely applied in the fields of machine vision, speech recognition, natural language processing, etc. and have achieved great success in many challenges, obtaining excellent results beyond traditional methods. However, the great success of deep neural networks cannot be achieved without complex network structures, large number of parameters and operations, which seriously hinders the practical application of various deep neural network-based models. How to remove redundant parameters and compress model structures while ensuring excellent model performance to obtain lightweight networks for practical application has become the focus of attention, and various model compression and lightweight methods have emerged.

[0003] With the development of hardware devices and the increase of system complexity, in order to meet the requirements of model deployment, compact structure design has become the most vigorous method in lightweight methods. Compact structure design method, also known as lightweight convolutional neural network design, replaces standard convolution with more compact and efficient convolution structure to realize the compression of network model. This method optimizes the performance of the model from the structure level, without relying too much on the optimization steps in the later stage, and can well adapt to the limitations of hardware, providing conditions for the model to be deployed on devices with limited computing power. Iandola et al. proposed to replace the traditional convolution structure with Fire Module containing Squeeze layer and Expand layer in 2016, and designed SqueezeNet based on this, which can reduce the parameter amount to 1 / 50 of AlexNet while maintaining the accuracy. Ma et al. proposed a series of ShuffleNet models and put forward more practical guiding principles for lightweight convolutional neural network design, pointing the way for subsequent researchers. Thereafter, researchers have also proposed a series of lightweight models such as Ghost model, which has greatly enriched the research of lightweight network design.

[0004] At present, the lightweight network design described above has been combined with a variety of different network models, and good results have been achieved in target identification and tracking and other fields. However, the lightweight method described above is mainly aimed at image recognition detection and other fields, and there is less research in the field of image fusion, especially in the field of infrared and visible light image fusion. The commonly used lightweight method in the field of image fusion is to replace the larger convolution kernel in the convolution layer with a smaller convolution kernel, and use pruning and quantization technology to reduce redundant weights and storage requirements. Although this method can reduce the parameter quantity and calculation amount to a certain extent, the reduction is limited, and it will cause the loss of detailed information, which cannot meet the high requirements of special fields for real-time performance and robustness. SUMMARY

[0005] In order to solve the above technical problems, the embodiment of the present application proposes a lightweight network design method for infrared and visible light image fusion, which can reduce the parameter quantity and calculation amount of the fusion model as much as possible under the condition of guaranteeing the quality of image fusion, thereby effectively improving the speed of image fusion, to meet the high requirements of special fields for real-time performance and robustness.

[0006] In order to achieve the above purpose, the embodiment of the present application proposes a lightweight network design method for infrared and visible light image fusion, which comprises: based on depth convolution and point-by-point convolution, introducing an SE module to improve attention to important channels, introducing channel mixing to force interaction between channels, introducing residual connection to make up for the lack of bottom information, and combining linear transformation and channel dimension splicing to jointly build a lightweight convolution module; based on the lightweight convolution module, the standard convolution module, the BN normalization layer and the LeakyReLU activation function layer, an encoder is constructed, based on the lightweight convolution module, the standard convolution module and the Tanh activation function layer, a decoder is constructed, and based on the encoder and the decoder, a self-encoder network is composed; the pre-constructed training set is used to iteratively train the self-encoder network, the total loss function in the form of segmentation based on the training round number, pixel loss and SSIM loss is used as a guide to update the parameters of the encoder and the decoder until the preset convergence condition is met, and the trained self-encoder network is obtained; a fusion module is inserted into the trained self-encoder network to form a fusion model, and the pre-constructed test set is used to test the fusion model.

[0007] To achieve the above object, the embodiment of the present application also proposes a lightweight network design system for infrared and visible light image fusion, which comprises a lightweight convolution construction module, a self-encoder network construction module, a self-encoder network training module and a fusion model construction and test module; the lightweight convolution construction module is used to introduce an SE module to improve the attention to important channels, introduce channel mixing to force interaction between channels, introduce residual connection to make up for the lack of bottom information, and combine linear transformation and channel dimension splicing to jointly construct a lightweight convolution module based on depth convolution and pointwise convolution; the self-encoder network construction module is used to construct an encoder based on the lightweight convolution module, a standard convolution module, a BN normalization layer and a LeakyReLU activation function layer, construct a decoder based on the lightweight convolution module, the standard convolution module and a Tanh activation function layer, and compose a self-encoder network based on the encoder and the decoder; the self-encoder network training module is used to iteratively train the self-encoder network using a pre-constructed training set, use a segmented total loss function constructed based on the training number, pixel loss and SSIM loss as a guide to update the parameters of the encoder and the decoder until a preset convergence condition is met, and obtain a trained self-encoder network; the fusion model construction and test module is used to insert a fusion module into the trained self-encoder network to compose a fusion model and test the fusion model using a pre-constructed test set.

[0008] To achieve the above object, the embodiment of the present application also proposes an electronic device, which comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the lightweight network design method for infrared and visible light image fusion as described above.

[0009] To achieve the above object, the embodiment of the present application also proposes a computer readable storage medium storing a computer program, and the computer program is executable by a processor to implement the lightweight network design method for infrared and visible light image fusion as described above.

[0010] This application proposes a lightweight network design method for infrared and visible light image fusion. It fully leverages the significant reduction in parameter and computational costs associated with depthwise convolution and pointwise convolution. Based on depthwise and pointwise convolution, an SE module is introduced to enhance attention to important channels, strengthen effective channels, suppress ineffective channels, force channel shuffling for interaction between channels, and introduce residual connections to compensate for missing low-level information. These are combined with linear transformations and channel dimension concatenation to construct a lightweight convolution module. Compared to standard convolution modules, this lightweight module significantly reduces the number of parameters, effectively lowering the computational cost during processing. Based on this lightweight module, an encoder and decoder are constructed, forming an autoencoder network. This further reduces parameter and computational costs, enabling faster and better feature extraction. Furthermore, to better match the specific task requirements of the fusion model, a piecewise total loss function based on the number of training epochs, pixel loss, and SSIM loss is used as a guide during autoencoder network training, allowing for faster and better convergence. Finally, a fusion module is inserted into the trained autoencoder network to form a fusion model. The fusion model is then tested using a test set. The fusion model that passes the test can complete image fusion tasks faster and can be smoothly deployed to resource-constrained scenarios, thus improving the versatility of the fusion model.

[0011] Optionally, the lightweight convolutional module consists of a sequentially connected deep convolutional layer, a pointwise convolutional layer, an SE module, a linear transformation layer, and a channel shuffling module. A residual connection is established between the input of the deep convolutional layer and the output of the SE module. A channel-dimensional concatenation is set between the input of the linear transformation layer and the output of the channel shuffling module. The concatenation result is the final output of the lightweight convolutional module. For the lightweight convolutional module, after the input image is processed by the deep convolutional layer, the pointwise convolutional layer, and the SE module, an intermediate feature image with half the number of channels as the final output of the lightweight convolutional module is obtained. The intermediate feature image and the input image are residually connected to preserve the original information, resulting in a summed image. The summed image is then subjected to linear transformation and channel shuffling through the linear transformation layer and the channel shuffling module to enhance feature interaction. Finally, the channel-shuffled image and the summed image are concatenated along the channel dimension to obtain the final output image as the final output of the lightweight convolutional module.

[0012] Optionally, the final output of the lightweight convolutional module can be expressed by the formula: ; ; ; in, This represents the input image for the lightweight convolutional module. a process representing a depth convolution, a process representing a point-wise convolution, a process representing SE module processing, a channel shuffle operation with a group number of a convolution kernel used in a cubic convolution, a size of a size of a final output of the lightweight convolution module.

[0013] Optionally, a size of a size of a size of a convolution kernel used in a depth convolution is a size of a convolution kernel used in a point-wise convolution is a size of a convolution kernel used in a linear transformation is a size of a convolution kernel used in a normal standard convolution is a compression factor of an SE module is ; a compression ratio of the lightweight convolution module, expressed by a formula as: ; ; ; ; ; ; wherein, a parameter amount of a standard convolution, a parameter amount of a depth convolution, a parameter amount of a point-wise convolution, a parameter amount of a linear transformation, a parameter amount of an SE module, a compression ratio of the lightweight convolution module.

[0014] Optionally, the encoder is composed of a standard convolution module, a BN normalization layer, a LeakyReLU activation function layer and three lightweight convolution modules connected in sequence, and the decoder is composed of three lightweight convolution modules, a standard convolution module and a Tanh activation function layer connected in sequence; ​​​​​​​​When iteratively training the autoencoder network, visible light sample images from the training set are used. The input is fed into an autoencoder network, and the encoder... Feature extraction is performed to obtain visible light features. Then the decoder... Perform the restoration to obtain the restored image. ; When testing the fusion model, pairs of infrared sample images are used in the test set. and visible light sample images They are input together into the fusion model, and the encoder processes them. and Feature extraction was performed separately to obtain infrared features. and visible light characteristics Then the fusion module performs... and Perform feature fusion to obtain fused features Finally, the decoder... The image is restored to its original state to obtain the fused image. .

[0015] Optionally, the piecewise total loss function, constructed based on the number of training epochs, pixel loss, and SSIM loss, is expressed by the following formula: ; ; ; ; in, Indicates pixel loss, This indicates the calculation of the Euclidean distance. Indicates SSIM loss, This indicates the calculation of structural similarity. Represents the piecewise coefficient. Indicates the number of training rounds. This represents the total loss function.

[0016] Optionally, the MS-COCO dataset is used as the training set to iteratively train the autoencoder network. The learning rate during training is set to 0.00001, the batch size is 8, the maximum number of training epochs is 300, and the preset convergence condition is to reach the maximum number of training epochs. After obtaining the fusion model, the LLVIP dataset was used as a test set to perform performance tests on the fusion model. After all performance tests were passed, the fusion model that had completed the performance tests was deployed. Attached Figure Description

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the following drawings are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. These drawings described herein are only used to explain the present application and should not be used to limit the present application.

[0018] Figure 1 is a flowchart of a lightweight network design method for infrared and visible image fusion provided in an embodiment of the present application; Figure 2 is a structural schematic diagram of a lightweight convolution module provided in an embodiment of the present application; Figure 3 is a structural schematic diagram of an encoder and a decoder provided in an embodiment of the present application; Figure 4 is a structural schematic diagram of a fusion model provided in an embodiment of the present application; Figure 5 is a result diagram of a simulation experiment provided in an embodiment of the present application; Figure 6 is a structural schematic diagram of a lightweight network design system for infrared and visible image fusion provided in another embodiment of the present application; Figure 7 is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings to describe the embodiments of the present application in detail. Those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to make the readers better understand. However, even without these technical details and based on various changes and modifications of the following embodiments, the technical solutions claimed by the present application can be implemented. The following embodiments are divided for the convenience of description, and should not constitute any limitation on the specific implementation of the present application. The following embodiments can be combined and referenced each other without contradiction.

[0020] One embodiment of the present application proposes a lightweight network design method for infrared and visible light image fusion, which is applied to an electronic device. The electronic device can be a terminal or a server. The following embodiments are described taking the server as an example. The following describes the implementation details of the lightweight network design method for infrared and visible light image fusion proposed in this embodiment. The following implementation details are provided for easy understanding and are not essential to implement the present application.

[0021] The specific process of the lightweight network design method for infrared and visible light image fusion proposed in this embodiment can be as shown in Figure 1 The specific process of the lightweight network design method for infrared and visible light image fusion proposed in this embodiment can be as shown in Step 11. Based on depth convolution and point-wise convolution, SE modules are introduced to improve attention to important channels, channel shuffle is introduced to force interaction between channels, residual connection is introduced to make up for the lack of bottom information, and linear transformation and channel dimension splicing are combined to jointly build a lightweight convolution module.

[0022] In a specific implementation, the basis of lightweight network design is to design a lightweight convolution module. As the most commonly used operation in the encoder and decoder, the reduction of convolution parameters compresses the parameter quantity of the fusion model from the source. Based on depth convolution and point-wise convolution, this embodiment greatly reduces the parameter quantity and calculation quantity of the model. However, compared with standard convolution, such design may have problems such as poor channel feature selection ability, channel information isolation, and lack of bottom information in the calculation process. Based on this, this embodiment introduces SE modules to improve attention to important channels, introduces channel shuffle to force interaction between channels, and introduces residual connection to enhance the expression of bottom information to make up for the lack of bottom information. In addition, linear transformation and channel dimension splicing are combined to jointly build a lightweight convolution module. The lightweight convolution module constructed in this way can effectively reduce the parameter quantity and calculation quantity, and also effectively improve the efficiency of the convolution process.

[0023] In one example, the specific structure of the lightweight convolution module can be as shown in Figure 2 The lightweight convolution module is specifically composed of a depth convolution layer, a point-wise convolution layer, an SE module, a linear transformation layer, and a channel shuffle module connected in sequence. The input of the depth convolution layer and the output of the SE module are connected by a residual connection, the input of the linear transformation layer and the output of the channel shuffle module are connected by channel dimension splicing, and the splicing result is the final output of the lightweight convolution module.

[0024] For the lightweight convolution module, the input image is processed by the depth convolution layer, the point-wise convolution layer, and the SE module, to obtain an intermediate feature image with a channel number being half of the final output of the lightweight convolution module, the intermediate feature image is connected with the input image in residual connection to maintain the original information, to obtain an added image, the added image is linearly transformed and channel shuffled by the linear transformation layer and the channel shuffle module to enhance feature interaction, and then the channel shuffled image is spliced with the added image in the channel dimension to obtain a final output image as the final output of the lightweight convolution module.

[0025] In one example, the final output of the lightweight convolution module is expressed by the formula: ; ; ; wherein, represents the input image of the lightweight convolution module, represents the process of depth convolution, represents the process of point-wise convolution, represents the process of SE module processing, represents the channel shuffle operation with the group number being , , , represents the convolution kernel used by the third convolution, and have a size of , have a size of , represents the final output of the lightweight convolution module.

[0026] It should be noted that, , that is, has a size, , , respectively represent the channel number, height and width of , , , , respectively represent the channel number, height and width of , , .

[0027] In one example, it is assumed that has a size of , has a size of , the size of the convolution kernel used in the depth convolution process is , the size of the convolution kernel used in the point-wise convolution process is , the size of the convolution kernel used in the linear transformation process is , the size of the convolution kernel used in the ordinary standard convolution is , the compression factor of the SE module is , the compression ratio of the lightweight convolution module can be expressed by the formula: ; ; ; ; ; ; wherein, represents the parameter amount of the standard convolution, represents the parameter amount of the depth convolution, represents the parameter amount of the point-wise convolution, represents the parameter amount of the linear transformation, represents the parameter amount of the SE module, represents the compression ratio of the lightweight convolution module.

[0028] Step 12, constructing an encoder based on the lightweight convolution module, the standard convolution module, the BN normalization layer and the LeakyReLU activation function layer, and constructing a decoder based on the lightweight convolution module, the standard convolution module and the Tanh activation function layer, and composing an autoencoder network based on the encoder and the decoder.

[0029] In a specific implementation, after the construction of the lightweight convolution module is completed, an encoder can be constructed based on the lightweight convolution module, the standard convolution module, the BN normalization layer and the LeakyReLU activation function layer, and a decoder can be constructed based on the lightweight convolution module, the standard convolution module and the Tanh activation function layer, and an autoencoder network can be composed based on the encoder and the decoder. The network complexity of such an autoencoder network is greatly reduced, and it has good practicability.

[0030] In one example, as shown in Figure 3 , the encoder is composed of a standard convolution module, a BN normalization layer, a LeakyReLU activation function layer and three lightweight convolution modules connected in sequence, and the decoder is composed of three lightweight convolution modules, a standard convolution module and a Tanh activation function layer connected in sequence.

[0031] In one example, as shown in Figure 4As shown, the finally obtained fusion model includes an encoder, a fusion module and a decoder.

[0032] Step 13, iteratively train the auto-encoder network using the pre-constructed training set, use the total loss function in segmented form constructed based on the training round number, pixel loss and SSIM loss as guidance, update the parameters of the encoder and the decoder, until the preset convergence condition is met, and obtain the trained auto-encoder network.

[0033] In a specific implementation, after the construction of the auto-encoder network is completed, the auto-encoder network also needs to be trained in a targeted manner before it can be put into use. In this embodiment, the pre-constructed training set (which can also be directly used as an open source data set on the Internet) is used to iteratively train the auto-encoder network, the total loss function in segmented form constructed based on the training round number, pixel loss and SSIM loss is used as guidance to update the parameters of the encoder and the decoder of the auto-encoder network, and after the preset convergence condition is met, the trained auto-encoder network is obtained.

[0034] In one example, when iteratively training the auto-encoder network, the visible light sample images in the training set are input into the auto-encoder network, the features of the visible light sample images are extracted by the encoder, the visible light features are obtained, and the visible light features are restored by the decoder to obtain the restored images.

[0035] In one example, the total loss function in segmented form constructed based on the training round number, pixel loss and SSIM loss can be expressed by the formula: ; ; ; ; wherein, pixel loss is represented by L, Euclidean distance is represented by d, SSIM loss is represented by LSSIM, structure similarity is represented by ssim, segmentation coefficient is represented by a, training round number is represented by n, and total loss function is represented by L.

[0036] ​​​​​​It should be noted that the total loss function is designed in a segmented form because there is a difference in orders of magnitude between the pixel loss and the SSIM loss, and the segmented design can eliminate this difference, thereby better training the autoencoder network.

[0037] In one example, the MS-COCO dataset can be used as the training set to iteratively train the autoencoder network, and the learning rate during training is set to 0.00001, the training batch size is 8, the maximum number of training rounds is 300, and the preset convergence condition is to reach the maximum number of training rounds.

[0038] Step 14, inserting a fusion module into the trained autoencoder network to form a fusion model, and testing the fusion model using a pre-built test set.

[0039] In a specific implementation, after obtaining the trained autoencoder network, a fusion module can be inserted into the trained autoencoder network to form a fusion model as shown in FIG. 13, and the fusion model can be tested using a pre-built test set. Figure 4

[0040] In one example, when testing the fusion model, a pair of infrared sample images and visible light sample images in the test set are input into the fusion model, and the encoder extracts features from and respectively to obtain infrared features and visible light features , and the fusion module fuses the features of and to obtain fusion features , and finally the decoder restores to obtain a fusion image .

[0041] In one example, the training and testing of the fusion model are both performed on a device running the Windows 10 operating system, the device also has a NVIDIA 3060 Ti graphics card, the algorithm compilation language is Python 3.9, and a deep learning environment based on GPU is built.

[0042] ​The embodiment proposes a lightweight network design method for infrared and visible light image fusion, which fully utilizes the characteristics of deep convolution and point-by-point convolution that can greatly reduce the parameter quantity and calculation quantity of the model. Based on deep convolution and point-by-point convolution, the SE module is introduced to improve the attention to important channels, enhance effective channels, and suppress ineffective channels. The channel mixing is introduced to force the interaction between channels. The residual connection is introduced to make up for the lack of bottom information. In combination with linear transformation and channel dimension splicing, the lightweight convolution module is constructed. Compared with the ordinary standard convolution module, the lightweight convolution module greatly compresses the parameter quantity and effectively reduces the calculation quantity required in the processing process. Based on the lightweight convolution module, the encoder and the decoder are constructed respectively, and the auto-encoder network is composed based on the encoder and the decoder, which well compresses the parameter quantity and the calculation quantity and can extract features faster and better. In addition, in order to better match the specific task requirements of the fusion model, when training the auto-encoder network, the total loss function in the segmented form based on the training round number, pixel loss and SSIM loss is selected as the guidance, so that the auto-encoder network can converge faster and better. Finally, the fusion module is inserted into the trained auto-encoder network to form the fusion model, and the fusion model is tested by using the test set. The tested fusion model can complete the image fusion task faster and can be smoothly deployed in the resource-limited scene, which improves the universality of the fusion model.

[0043] The step division of the above various methods is only for the purpose of clear description, and in implementation, one step can be combined or some steps can be decomposed into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the present application. Adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0044] In one embodiment, in order to illustrate the effectiveness of the lightweight network design method for infrared and visible light image fusion proposed in the present application, we compared the fusion model designed in the present application (referred to as OURS) and the DenseFuse model on the LLVIP dataset. We use EN, VIF, MI and other indicators as evaluation criteria. The parameter indicators of the two models on the LLVIP dataset are shown in Table 1, and the output fusion images are shown in Figure 5

[0045] Table 1: Parameter indicators of two models on LLVIP dataset

[0046] ​As can be seen from Table 1, OURS obtains better fusion effect on multiple indicators, which shows that OURS can better maintain performance when applied to image fusion, and the parameter amount of OURS is reduced by 61% compared with DenseFuse, and the calculation amount is reduced by 67%, which has great improvement.

[0047] As shown in Figure 5 In addition to the effectiveness of the objective indicators, observing the fusion results of OURS and DenseFuse can also show that OURS can well fuse infrared images and visible light images, and even has better effect in some parts, and obtain ideal fusion results.

[0048] In summary, through the comparison experiment, the subjective and objective comparisons show that OURS can reduce the complexity of the model as much as possible while ensuring the quality of the fused image. The above advantages prove that OURS has extremely important application value and broad market prospect in the field of infrared image and visible light image fusion.

[0049] Another embodiment of the present application proposes a lightweight network design system for infrared and visible light image fusion. The details of the lightweight network design system for infrared and visible light image fusion proposed in this embodiment will be specifically described below. The following content is only provided for the implementation details for easy understanding, and is not necessary for implementing this embodiment. Figure 6 is a structural schematic diagram of the lightweight network design system for infrared and visible light image fusion proposed in this embodiment, which includes a lightweight convolution construction module 21, a self-encoder network construction module 22, a self-encoder network training module 23, and a fusion model construction and pre-test module 24.

[0050] The lightweight convolution construction module 21 is used to introduce SE modules to improve the attention to important channels, introduce channel shuffling to force interaction between channels, and introduce residual connections to make up for the lack of bottom information, in combination with linear transformation and channel dimension splicing, to jointly construct a lightweight convolution module based on depth convolution and pointwise convolution.

[0051] The self-encoder network construction module 22 is used to construct an encoder based on a lightweight convolution module, a standard convolution module, a BN normalization layer, and a LeakyReLU activation function layer, construct a decoder based on a lightweight convolution module, a standard convolution module, and a Tanh activation function layer, and construct a self-encoder network based on the encoder and the decoder.

[0052] The auto-encoder network training module 23 is configured to iteratively train the auto-encoder network by using a pre-constructed training set, and to update the parameters of the encoder and the decoder by using a total loss function in a segmented form constructed based on the training round number, the pixel loss and the SSIM loss as a guide until a preset convergence condition is met, so as to obtain the trained auto-encoder network.

[0053] The fusion model construction and test module 24 is configured to insert a fusion module into the trained auto-encoder network to form a fusion model, and to test the fusion model by using a pre-constructed test set.

[0054] It can be found that the embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and the embodiment can be implemented in cooperation with the above-mentioned method embodiment. The related technical details and technical effects mentioned in the above-mentioned method embodiment are still valid in the embodiment. In order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the embodiment can also be applied to the above-mentioned method embodiment.

[0055] It is worth mentioning that each module and module involved in the embodiment is a logical module. In actual application, a logical unit can be a physical unit, or a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the embodiment, but this does not mean that there are no other units in the embodiment.

[0056] Another embodiment of the present application provides an electronic device, as shown in the figure, comprising: at least one processor 31; and a memory 32 connected in communication with the at least one processor 31; wherein the memory 32 stores instructions executable by the at least one processor 31, and the instructions are executed by the at least one processor 31 to enable the at least one processor 31 to perform a lightweight network design method for infrared and visible light image fusion as described in the above-mentioned method embodiment. Figure 7

[0057] ​The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0058] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0059] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a lightweight network design method for infrared and visible light image fusion as described in the above method embodiments.

[0060] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0061] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A lightweight network design method for infrared and visible light image fusion, characterized in that, The method includes: Based on depthwise convolution and pointwise convolution, a SE module is introduced to enhance attention to important channels, channel shuffling is introduced to force interaction between channels, residual connections are introduced to make up for the lack of underlying information, and then combined with linear transformation and channel dimension splicing to jointly construct a lightweight convolution module. An encoder is constructed based on a lightweight convolutional module, a standard convolutional module, a BN normalization layer, and a LeakyReLU activation function layer. A decoder is constructed based on a lightweight convolutional module, a standard convolutional module, and a Tanh activation function layer. An autoencoder network is formed based on the encoder and the decoder. The autoencoder network is iteratively trained using a pre-built training set. The total loss function, which is constructed based on the number of training epochs, pixel loss, and SSIM loss, is used as a guide to update the parameters of the encoder and decoder until the preset convergence condition is met, thus obtaining the trained autoencoder network. A fusion module is inserted into the trained autoencoder network to form a fusion model, and the fusion model is tested using a pre-built test set.

2. The lightweight network design method for infrared and visible light image fusion according to claim 1, characterized in that, The lightweight convolution module consists of a deep convolutional layer, a pointwise convolutional layer, an SE module, a linear transformation layer, and a channel shuffling module connected in sequence. A residual connection is established between the input of the deep convolutional layer and the output of the SE module. The input of the linear transformation layer and the output of the channel shuffling module are concatenated along the channel dimension. The concatenation result is the final output of the lightweight convolution module. For the lightweight convolutional module, the input image is processed by a deep convolutional layer, a pointwise convolutional layer, and an SE module to obtain an intermediate feature image with half the number of channels as the final output of the lightweight convolutional module. The intermediate feature image is residually connected to the input image to preserve the original information, resulting in a summed image. The summed image is then subjected to linear transformation and channel shuffling through a linear transformation layer and a channel shuffling module to enhance feature interaction. Finally, the channel shuffling image is concatenated with the summed image along the channel dimension to obtain the final output image as the final output of the lightweight convolutional module.

3. The lightweight network design method for infrared and visible light image fusion according to claim 2, characterized in that, The final output of the lightweight convolutional module is expressed by the formula: ； ； ； in, This represents the input image for the lightweight convolutional module. This represents the process of depthwise convolution. This represents the pointwise convolution process. This indicates the processing procedure of the SE module. Indicates the number of groups is Channel mixed washing operation, , , This indicates the convolution kernel used in the third convolution. and The size is , The size is , This represents the final output of the lightweight convolution module.

4. The lightweight network design method for infrared and visible light image fusion according to claim 3, characterized in that, set up The size is , The size is The kernel size used in depthwise convolution is The size of the convolution kernel used in the pointwise convolution process is The sizes of the convolution kernels used in the linear transformation process are respectively The kernel size used in standard convolution is 1. The compression factor of the SE module is ; The compression ratio of the lightweight convolutional module can be expressed by the formula: ； ； ； ； ； ； in, The number of parameters representing a standard convolution. The number of parameters representing a depthwise convolution. This represents the number of parameters in a pointwise convolution. The number of parameters representing a linear transformation. This indicates the number of parameters in the SE module. This indicates the compression ratio of the lightweight convolutional module.

5. A lightweight network design method for infrared and visible light image fusion according to claim 2, characterized in that, The encoder consists of a standard convolutional module, a BN normalization layer, a LeakyReLU activation function layer, and three lightweight convolutional modules connected in sequence. The decoder consists of three lightweight convolutional modules, a standard convolutional module, and a Tanh activation function layer connected in sequence. When iteratively training the autoencoder network, visible light sample images from the training set are used. The input is fed into an autoencoder network, and the encoder... Feature extraction is performed to obtain visible light features. Then the decoder... Perform the restoration to obtain the restored image. ; When testing the fusion model, pairs of infrared sample images are used in the test set. and visible light sample images They are input together into the fusion model, and the encoder processes them. and Feature extraction was performed separately to obtain infrared features. and visible light characteristics Then the fusion module performs... and Perform feature fusion to obtain fused features Finally, the decoder... The image is restored to its original state to obtain the fused image. .

6. A lightweight network design method for infrared and visible light image fusion according to claim 5, characterized in that, The piecewise total loss function, constructed based on the number of training epochs, pixel loss, and SSIM loss, is expressed by the following formula: ； ； ； ； in, Indicates pixel loss, This indicates the calculation of the Euclidean distance. Indicates SSIM loss, This indicates the calculation of structural similarity. Represents the piecewise coefficient. Indicates the number of training rounds. This represents the total loss function.

7. A lightweight network design method for infrared and visible light image fusion according to any one of claims 1 to 6, characterized in that, The MS-COCO dataset was used as the training set to iteratively train the autoencoder network. The learning rate during training was set to 0.00001, the batch size was 8, the maximum number of training epochs was 300, and the preset convergence condition was to reach the maximum number of training epochs. After obtaining the fusion model, the LLVIP dataset was used as a test set to perform performance tests on the fusion model. After all performance tests were passed, the fusion model that had completed the performance tests was deployed.

8. A lightweight network design system for infrared and visible light image fusion, characterized in that, The system includes a lightweight convolutional model building module, a fusion model building module, and a fusion model training module; The lightweight convolution module is built upon depthwise convolution and pointwise convolution. It introduces the SE module to enhance attention to important channels, introduces channel shuffling to force interaction between channels, introduces residual connections to compensate for the lack of underlying information, and combines linear transformation and channel dimension concatenation to jointly build a lightweight convolution module. An autoencoder network construction module is used to build an encoder based on a lightweight convolutional module, a standard convolutional module, a BN normalization layer and a LeakyReLU activation function layer, and to build a decoder based on a lightweight convolutional module, a standard convolutional module and a Tanh activation function layer. The encoder and decoder together form an autoencoder network. The autoencoder network training module is used to iteratively train the autoencoder network using a pre-built training set. It uses a piecewise total loss function based on the number of training epochs, pixel loss, and SSIM loss as guidance to update the parameters of the encoder and decoder until the preset convergence condition is met, thus obtaining the trained autoencoder network. The fusion model building and testing module is used to insert fusion modules into the trained autoencoder network to form a fusion model, and to test the fusion model using a pre-built test set.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which enable the at least one processor to perform a lightweight network design method for infrared and visible light image fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a lightweight network design method for infrared and visible light image fusion as described in any one of claims 1 to 7.

Images