Infrared and visible light fusion network model based on lightweight convolution module
By designing a lightweight convolution module in infrared and visible light fusion network models, combining deep convolution, point-by-point convolution and linear transformation parts, the existing model has complex structure, large parameters and large calculations, and more efficient calculations and better adaptability are achieved.
Patent Information
- Application Number
- CN202411974195.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
AI Technical Summary
The existing infrared and visible light fusion network models have complex structures, large parameters and large calculations, which leads to problems such as overfitting and insufficient computing resources during training.
An infrared and visible light fusion network model based on a lightweight convolution module is designed. The parameter quantity and calculation quantity of the model are reduced through the combination of deep convolution, point-by-point convolution, linear transformation part and the second fusion layer.
Under the condition of ensuring the quality of the fusion image, the parameters and calculation amount of the fusion model are significantly reduced, and the calculation speed and adaptability of the model are improved.
Smart Images
Figure CN119887542A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an infrared and visible light fusion network model based on a lightweight convolution module. Background Art
[0002] In recent years, with the continuous development of deep learning technology, the field of infrared and visible light fusion has also been making continuous progress. Generative adversarial networks represented by FusionGAN have made great progress in this field. They use the generator to generate preliminary infrared and visible light fusion images, and then use the discriminator and the generator to play adversarial games, constantly adding important detail information to the preliminary fusion image until the ideal fusion image is finally obtained. In order to better supplement the detail information, researchers have continuously improved the GAN network and proposed a series of effective measures such as dual discriminators. However, with the improvement of the effect, the model structure of the network has become more and more complex, and the model parameters have continued to increase. In the training process, it is easy to have problems of overfitting and insufficient computing resources. To this end, it is considered to design a lightweight convolution module, which can effectively reduce the number of model parameters and the amount of calculation while ensuring the quality of feature extraction, thereby speeding up the model calculation speed.
[0003] In recent years, deep neural networks have made breakthrough progress in many fields. These achievements are due to the complex network structure and a large number of parameter operations. However, this high computational overhead and storage requirements have greatly limited the widespread application of deep neural networks in practical scenarios. In this context, researchers have proposed lightweight modules such as Fire Module, Depthwise Separable Convolution, and Ghost Module. Lightweight modules optimize the performance of the model system from a structural level, but do not need to rely too much on the later optimization steps. They can better adapt to the limitations of hardware and provide conditions for the model to be deployed on devices with computing power. Depthwise separable convolution is to split the convolution process of traditional standard convolution into two parts-depthwise convolution and pointwise convolution. Depthwise convolution focuses on extracting the spatial information of the image, while pointwise convolution focuses on the information interaction between channels. This separation design greatly reduces the number of parameters and calculations in the convolution process while retaining the key information of the image. The Ghost module starts with the feature map. Considering the existence of redundant information in the feature extraction process, the final output channel is obtained from two parts, one part is obtained by normal convolution, and the other part is obtained by linearly changing the result of the previous part. Both of the above methods provide a new convolution idea, which ensures that the complexity of the model is significantly reduced while the indicators remain basically unchanged, and provides the possibility for the application expansion of deep neural networks in resource-constrained scenarios.
[0004] The above-mentioned lightweight modules have been combined with a variety of different network models and have achieved good results in the fields of target recognition and tracking. However, the current lightweight methods are mainly aimed at fields such as image recognition and detection, and there is little research in the field of image fusion, especially in the field of infrared and visible light image fusion. Most lightweight methods in the field of image fusion replace the larger convolution kernels in the convolution layer with smaller convolution kernels, and use pruning and quantization techniques to reduce redundant weights and storage requirements. Although this method reduces the number of parameters and calculations to a certain extent, the reduction is limited and sometimes leads to the loss of information such as details. To solve the above problems, the present invention designs a lightweight convolution module and applies it to the field of infrared and visible light image fusion, while ensuring the image quality of the fused image, and minimizes the number of parameters and the amount of calculation of the fusion model. Summary of the invention
[0005] The embodiment of the present invention provides an infrared and visible light fusion network model based on a lightweight convolution module, so as to at least solve the technical problems of the infrared and visible light fusion network having a complex structure, a large number of parameters and a large amount of calculation in the prior art.
[0006] According to one aspect of an embodiment of the present invention, an infrared and visible light fusion network model based on a lightweight convolution module is provided. The model may include: an input layer, a first fusion layer, a target generator and an output layer, wherein the target generator includes 4 lightweight convolution modules, a 1×1 convolution layer and an activation function; an input layer, used to input the test set image into the first fusion layer, wherein the test set image includes a number of infrared images and a number of visible light images; a first fusion layer, used to perform channel splicing of each infrared image in the test set with each corresponding visible light image to obtain a number of first fusion images, and input the first fusion images into the target generator; a target generator, used to process the first fusion images to obtain a target fusion image corresponding to each first fusion image, and output the target fusion images to the output layer, wherein the target generator is obtained by training the initial generator using the training set images; an output layer, used to output the target fusion images.
[0007] Optionally, the target generator is obtained by training the initial generator using the training set images, and the process is: obtaining the training set images; wherein the training set images include several infrared images and several visible light images; channel-joining each infrared image in the training set with each corresponding visible light image to obtain several second fused images; processing the several second fused images by the initial generator to obtain the corresponding third fused images after the end of Epoch; processing the third fused image and the visible light image corresponding to the third fused image by the discriminator, and when the discriminator loss function under multiple consecutive Epochs tends to be stable, the network training is completed to obtain the target generator.
[0008] Optionally, each lightweight convolution module includes: a depth convolution part, a point-by-point convolution part, a linear transformation part and a second fusion layer; wherein the depth convolution part is connected to the point-by-point convolution part, the point-by-point convolution part is respectively connected to the linear transformation part and the second fusion layer, and the linear transformation part is connected to the second fusion layer; the depth convolution part is used to process each first fused image to obtain multiple first feature maps; the point-by-point convolution part is used to process each first feature map to obtain multiple second feature maps; the linear transformation part is used to process each second feature map to obtain multiple third feature maps; the second fusion layer is used to perform channel splicing of each second feature map and each third feature map to obtain several fourth fused images.
[0009] Optionally, the width × height of the convolution kernel of the depthwise convolution is , the number of convolution kernels is .
[0010] Optionally, the width × height of the convolution kernel of the point-by-point convolution part is , the number of convolution kernels is .
[0011] Optionally, the width × height of the linear transformation part convolution kernel is , the number of convolution kernels is .
[0012] Optionally, the loss function of the initial generator is expressed as:
[0013] in, represents the total loss function of the initial generator, Represents the adversarial loss between the initial generator and the discriminator, G represents the initial generator, represents the content loss of the initial generator, Used to control the balance between adversarial loss and content loss.
[0014] Optionally, the expression of the adversarial loss between the initial generator and the discriminator is:
[0015] in, Indicates The third fused images are L in total. Indicates The classification result of the third fused image, Indicates the pseudo label corresponding to the third fused image.
[0016] Optionally, the expression of the content loss of the initial generator is:
[0017] in, is the height of any third fused image, is the width of any third fused image, is any third fused image, is the infrared image corresponding to the third fused image, is the visible light image corresponding to the third fused image, is the gradient of any third fused image, is the gradient of the visible light image corresponding to the third fused image, represents the matrix Frobenius norm, 2 represents the square of the matrix Frobenius norm, It is used to control and A positive parameter that balances .
[0018] Optionally, the loss function of the discriminator is expressed as:
[0019] in, represents the loss function of the discriminator, Represents the label of any third fused image, Indicates the label of the visible light image corresponding to any third fused image, represents the classification result of the visible light image corresponding to the third fused image, represents the classification result of any third fused image. The number of visible light images corresponding to the third fused image is , the number of the third fused images is .
[0020] Beneficial effects of the present invention: The present invention proposes an infrared and visible light fusion network model based on a lightweight convolution module. The generator of the model proposed in the present invention includes: 4 lightweight convolution modules, a 1×1 convolution layer and an activation function. Each lightweight convolution module includes: a depth convolution part, a point-by-point convolution part, a linear transformation part and a second fusion layer. The sizes and numbers of the convolution kernels of the depth convolution part, the point-by-point convolution part and the linear transformation part are designed, so that the network model can reduce the number of parameters and the amount of calculation of the fusion model while ensuring the image quality of the fused image during the convolution process. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a schematic diagram of an infrared and visible light fusion network model based on a lightweight convolution module according to an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of an initial generator according to an embodiment of the present invention; Figure 3 3 is a result comparison diagram of different network models according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0024] Example 1 According to an embodiment of the present invention, an infrared and visible light fusion network model based on a lightweight convolution module is provided. It should be noted that the portion shown in the schematic diagram of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions.
[0025] Figure 1 is a schematic diagram of an infrared and visible light fusion network model based on a lightweight convolution module according to an embodiment of the present invention, such as Figure 1 As shown, the model may include: an input layer, a first fusion layer, a target generator and an output layer, and the target generator includes 4 lightweight convolution modules, a 1×1 convolution layer and an activation function.
[0026] The input layer is used to input the test set images into the first fusion layer, wherein the test set images include a plurality of infrared images and a plurality of visible light images.
[0027] In this embodiment, if Figure 1 In the test network shown, the input layer inputs several infrared images and several visible light images in the test set images (images of the selected TNO test set) into the first fusion layer ( Figure 1 Test network ).
[0028] The first fusion layer is used to perform channel splicing on each infrared image in the test set and each corresponding visible light image to obtain a number of first fused images, and input the first fused images into the target generator.
[0029] In this embodiment, each infrared image in the test set is channel-joined with each corresponding visible light image through the first fusion layer to obtain a plurality of first fused images, and the plurality of first fused images are input into the target generator.
[0030] A target generator is used to process several first fused images to obtain a target fused image corresponding to each first fused image, and output the several target fused images to an output layer, wherein the target generator is obtained by training an initial generator using training set images.
[0031] In this embodiment, through the target generator ( Figure 1 The generator is trained in the process (the generator is trained in the process), and several first fused images are processed to obtain a target fused image corresponding to each first fused image.
[0032] The output layer is used to output several target fusion images.
[0033] The above model of this embodiment is further introduced below.
[0034] As an optional implementation method, the process of obtaining the target generator by training the initial generator using the training set images is: obtaining the training set images; wherein the training set images include several infrared images and several visible light images; channel-joining each infrared image in the training set with each corresponding visible light image to obtain several second fused images; processing the several second fused images by the initial generator to obtain the corresponding third fused images after the end of Epoch; processing the third fused image and the visible light image corresponding to the third fused image by the discriminator, and when the discriminator loss function under multiple consecutive Epochs tends to be stable, the network training is completed to obtain the target generator.
[0035] In this embodiment, Figure 1 The training set images include several infrared images ( Figure 1 IR) and several visible light images ( Figure 1 VI); through Figure 1 In Each infrared image in the training set is channel-joined with each corresponding visible light image to obtain several second fused images; through the initial generator ( Figure 1 The generator in the network processes several second fused images to obtain the corresponding third fused images under each Epoch; the third fused image and the visible light image corresponding to the third fused image are processed by the discriminator. When the discriminator loss function under multiple consecutive Epochs is stable, the network training ends and the target generator is obtained. The initial generator includes: 4 lightweight convolution modules, 1×1 convolution layers and activation functions. Figure 2 is a schematic diagram of the initial generator structure according to an embodiment of the present invention, such as Figure 2 As shown, channel splicing is performed before the initial generator, and the structure of the trained generator in the test network is consistent with that in the initial generator, as shown in Figure 2 As shown in the figure, when the first lightweight convolution module inputs a second fused image, it outputs 256 feature maps; the second lightweight convolution module inputs 256 feature maps and outputs 128 feature maps; the third lightweight convolution module inputs 128 feature maps and outputs 64 feature maps; the fourth lightweight convolution module inputs 64 feature maps and outputs 32 feature maps; the last conv, tanh module inputs 32 feature maps and outputs a third fused image.
[0036] As an optional implementation method, each lightweight convolution module includes: a depth convolution part, a point-by-point convolution part, a linear transformation part and a second fusion layer; wherein the depth convolution part is connected to the point-by-point convolution part, the point-by-point convolution part is respectively connected to the linear transformation part and the second fusion layer, and the linear transformation part is connected to the second fusion layer; the depth convolution part is used to process each first fused image to obtain multiple first feature maps; the point-by-point convolution part is used to process each first feature map to obtain multiple second feature maps; the linear transformation part is used to process each second feature map to obtain multiple third feature maps; the second fusion layer is used to channel-join each second feature map with each third feature map to obtain several fourth fused images.
[0037] In this embodiment, if Figure 2 As shown, each lightweight convolution module includes: a depth convolution part, a point-by-point convolution part, a linear transformation part and a second fusion layer, wherein the depth convolution part includes: 1 convolution layer; the point-by-point convolution part includes: 1 convolution layer, 1 BN layer, 1 activation function; the linear transformation part includes: 1 convolution layer, 1 BN layer, 1 activation function.
[0038] Assume that the size of several first fused images is , is the number of first fused images, is the height of each first fused image, is the width of each first fused image.
[0039] The size of the output of several fourth fused images is , is the number of the fourth fused images, is the height of each fourth fused image, is the width of each fourth fused image.
[0040] The convolution kernel sizes in the depth convolution part, point-by-point convolution part, and linear transformation part are: , , .
[0041] The compression ratio of ordinary convolution and lightweight convolution modules is expressed as:
[0042] in, is the final parameter compression ratio, The number of parameters of standard traditional convolution, Represents the number of parameters of the depth convolution part, Represents the number of parameters of the point-by-point convolution part, Represents the parameter quantity of the linear transformation part.
[0043] : is the number of input images in ordinary convolution, is the width / height of the convolution kernel in ordinary convolution, is the number of convolution kernels in ordinary convolution, where the number of input images in ordinary convolution is the same as the number of first fused images.
[0044] : is the width / height of the convolution kernel in the depth convolution part, is the number of convolution kernels in the depth convolution part, where The number of first fused images Equal in size.
[0045] : is the number of input images in the point-by-point convolution part, is the width / height of the convolution kernel in the point-by-point convolution part, is the number of convolution kernels in the point-by-point convolution part, where the number of input images in the point-by-point convolution part The number of first fused images , the number of convolution kernels in the depth convolution part Same size.
[0046] : is the width / height of the linear transformation part of the convolution kernel, is the number of convolution kernels in the linear transformation part. The N in the number of convolution kernels in the linear transformation part is the same as the number of convolution kernels in ordinary convolution.
[0047] The compression ratio of the lightweight convolution module is:
[0048] It should be noted that the same characters above have the same size but different meanings in order to simplify the compression ratio formula of the lightweight convolution module; among them, the processing process of the grower in the training network is the same as that of the generator in the test network, and the structure is also the same.
[0049] As an optional embodiment, the width × height of the convolution kernel of the depth convolution part is , the number of convolution kernels is .
[0050] As an optional implementation method, the width × height of the convolution kernel of the point-by-point convolution part is , the number of convolution kernels is .
[0051] As an optional implementation mode, the width×height of the linear transformation part convolution kernel is , the number of convolution kernels is .
[0052] As an optional implementation mode, the loss function of the initial generator is expressed as:
[0053] in, represents the total loss function of the initial generator, Represents the adversarial loss between the initial generator and the discriminator, G represents the initial generator, represents the content loss of the initial generator, Used to control the balance between adversarial loss and content loss.
[0054] As an optional embodiment, the expression of the adversarial loss between the initial generator and the discriminator is:
[0055] in, Indicates The third fused images are L in total. Indicates The classification result of the third fused image, Indicates the pseudo label corresponding to the third fused image.
[0056] In this embodiment, A value representing the fake data that the initial generator wants the discriminator to believe.
[0057] As an optional implementation mode, the expression of the content loss of the initial generator is:
[0058] in, is the height of any third fused image, is the width of any third fused image, is any third fused image, is the infrared image corresponding to the third fused image, is the visible light image corresponding to the third fused image, is the gradient of any third fused image, is the gradient of the visible light image corresponding to the third fused image, represents the matrix Frobenius norm, 2 represents the square of the matrix Frobenius norm, It is used to control and A positive parameter that balances .
[0059] As an optional embodiment, the loss function of the discriminator is expressed as:
[0060] in, represents the loss function of the discriminator, Represents the label of any third fused image, Indicates the label of the visible light image corresponding to any third fused image, represents the classification result of the visible light image corresponding to the third fused image, represents the classification result of any third fused image. The number of visible light images corresponding to the third fused image is , the number of the third fused images is .
[0061] Experimental part Model training. The present invention uses the TNO dataset for training, where TNO is used as the training set and a randomly selected part is used as the test set, which does not participate in the training. During the training process, all infrared images and visible light images are input into the generator, and the visible light images are additionally input into the discriminator. The learning rate, batch size, and epoch of the model are set to 1e-4, 16, and 5, respectively. The network model is as follows: Figure 1 shown.
[0062] In the present invention, unless otherwise specified, the operating system of the training and testing devices is Windows 10, equipped with an NVIDIA 3060 Ti graphics card, the algorithm compilation language is Python 3.9, and a GPU-based deep learning environment is built.
[0063] Test verification: In order to verify the effectiveness of the present invention, the following experiments are conducted to illustrate the technical effects of the present invention.
[0064] In this test, an ablation experiment was conducted, and image evaluation indicators such as EN, MI, and SSIM were used to evaluate the fusion results before and after the introduction of the lightweight module, and the parameters of the calculation model were compared, as shown in Table 1: Table 1 Parameter indicators before and after the introduction of lightweight convolution module
[0065] As shown in Table 1, the method of the present invention achieves the best fusion effect in terms of EN, MI, and VIF indicators.abf Suboptimal performance was achieved in terms of indicators, indicating that the lightweight convolution module designed in this invention can better maintain the performance of the network when applied to image fusion. On the other hand, the number of parameters of the improved network has dropped by about 67% compared to before the improvement, and the amount of calculation has been reduced by 95%, which is a great improvement. Through the ablation experiment, it can be seen that the lightweight convolution module designed in this paper combines the advantages of the two modules. It has the characteristics of high compression rate of deep separable convolution and the ability of Ghost module to retain detail information. The comprehensive use of the two modules finally achieved an effective improvement in the fusion indicator - the number of parameters. Figure 3 is a result comparison diagram of different network models according to an embodiment of the present invention, Figure 3 (a) is the infrared image, Figure 3 (b) is the visible light image, Figure 3 (c) is the FusionGAN fusion image. Figure 3 (d) Introducing Ghost fusion images for FusionGAN, Figure 3 (e) Introducing deep separable convolutional images for FusionGAN, Figure 3 (f) FusionGAN introduces the lightweight convolution module designed in this paper to fuse images.
[0066] like Figure 3 As shown in the figure, in addition to the effectiveness in objective indicators, by observing and comparing the fusion results of the proposed method before improvement, using only the Ghost module, and using only the depthwise separable convolution module, it can be found that the proposed lightweight convolution module can be well integrated with the information in the infrared image and the visible light image when applied to the fusion network, and a more ideal fusion result can be obtained.
[0067] In summary, through ablation experiments, subjective and objective comparisons show that the proposed infrared and visible light image fusion network based on lightweight convolution modules can reduce the complexity of the model as much as possible while ensuring the quality of the fused image. The above advantages prove that the present invention has extremely important application value and broad market prospects in the field of infrared and visible light image fusion.
[0068] In the embodiment of the present invention, through the input layer, the first fusion layer, the target generator and the output layer, the target generator includes 4 lightweight convolution modules, a 1×1 convolution layer and an activation function; the input layer is used to input the test set image into the first fusion layer, wherein the test set image includes a number of infrared images and a number of visible light images; the first fusion layer is used to channel-join each infrared image in the test set with each corresponding visible light image to obtain a number of first fused images, and input the several first fused images into the target generator; the target generator is used to perform Processing is performed to obtain a target fused image corresponding to each first fused image, and several target fused images are output to the output layer, wherein the target generator is obtained by training the initial generator with the training set images; the output layer is used to output several target fused images, which solves the technical problems of complex infrared and visible light fusion network structure, large number of parameters and large amount of calculation in the prior art, and achieves the technical effect of reducing the number of parameters and the amount of calculation of the fusion model while ensuring the image quality of the fused image by designing a lightweight convolution module and applying it to the field of infrared and visible light image fusion.
[0069] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0070] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0071] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
Claims
1. An infrared and visible light fusion network model based on a lightweight convolution module, characterized in that: include: Input layer, first fusion layer, target generator and output layer. The target generator includes 4 lightweight convolution modules, 1×1 convolution layer and activation function; An input layer, used to input the test set images into the first fusion layer, wherein the test set images include a plurality of infrared images and a plurality of visible light images; The first fusion layer is used to perform channel splicing on each infrared image in the test set and each corresponding visible light image to obtain a number of first fused images, and input the first fused images into the target generator; A target generator is used to process the plurality of first fused images to obtain a target fused image corresponding to each first fused image, and output the plurality of target fused images to an output layer, wherein the target generator is obtained by training an initial generator using training set images; The output layer is used to output several target fusion images.
2. The model according to claim 1, characterized in that The target generator is obtained by training the initial generator using the training set images: Acquire training set images; wherein the training set images include a plurality of infrared images and a plurality of visible light images; Perform channel stitching on each infrared image in the training set and each corresponding visible light image to obtain a number of second fused images; Process several second fused images through the initial generator to obtain the corresponding third fused image after the Epoch ends; The third fused image and the visible light image corresponding to the third fused image are processed by the discriminator. When the discriminator loss function under multiple consecutive Epochs tends to be stable, the network training is completed and the target generator is obtained.
3. The model according to claim 1, characterized in that Each lightweight convolution module includes: a depth convolution part, a point-by-point convolution part, a linear transformation part and a second fusion layer; wherein the depth convolution part is connected to the point-by-point convolution part, the point-by-point convolution part is respectively connected to the linear transformation part and the second fusion layer, and the linear transformation part is connected to the second fusion layer; A deep convolution part, used for processing each first fused image to obtain multiple first feature maps; The point-by-point convolution part is used to process each first feature map to obtain multiple second feature maps; A linear transformation part, used for processing each second feature map to obtain multiple third feature maps; The second fusion layer is used to perform channel splicing on each second feature map and each third feature map to obtain a plurality of fourth fused images.
4. The model according to claim 3, characterized in that The width × height of the convolution kernel of the depth convolution part is , the number of convolution kernels is .
5. The model according to claim 3, characterized in that The width × height of the convolution kernel of the point-by-point convolution part is , the number of convolution kernels is .
6. The model according to claim 3, characterized in that The width × height of the linear transformation part of the convolution kernel is , the number of convolution kernels is .
7. The model according to claim 2, characterized in that The loss function of the initial generator is expressed as: in, represents the total loss function of the initial generator, Represents the adversarial loss between the initial generator and the discriminator, G represents the initial generator, represents the content loss of the initial generator, Used to control the balance between adversarial loss and content loss.
8. The model according to claim 7, characterized in that The expression of the adversarial loss between the initial generator and the discriminator is: in, Indicates The third fused images are L in total. Indicates The classification result of the third fused image, Indicates the pseudo label corresponding to the third fused image.
9. The model according to claim 7, characterized in that The expression of the content loss of the initial generator is: in, is the height of any third fused image, is the width of any third fused image, is any third fused image, is the infrared image corresponding to the third fused image, is the visible light image corresponding to the third fused image, is the gradient of any third fused image, is the gradient of the visible light image corresponding to the third fused image, represents the matrix Frobenius norm, 2 represents the square of the matrix Frobenius norm, It is used to control and A positive parameter that balances .
10. The model according to claim 9, characterized in that The expression of the loss function of the discriminator is: in, represents the loss function of the discriminator, Represents the label of any third fused image, Indicates the label of the visible light image corresponding to any third fused image, represents the classification result of the visible light image corresponding to the third fused image, represents the classification result of any third fused image. The number of visible light images corresponding to the third fused image is , the number of the third fused images is .