Method and device for determining fused image and computer program product
Through convolutional neural network, feature fusion, channel information enhancement, layered compression and feature reconstruction processing of infrared and visible images, the problem of insufficient fusion quality of infrared and visible images in the prior art is solved, and higher quality images are achieved Fusion effect.
Patent Information
- Application Number
- CN202411258680.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-26
AI Technical Summary
The existing infrared image and visible image fusion methods have problems such as manual design and decomposition algorithms that are time-consuming and labor-intensive, ignore image-specific features, and poor feature extraction effects, resulting in insufficient quality of the fusion image.
Image processing is performed using convolutional neural network, including feature fusion processing, channel information enhancement processing, layered compression processing and feature reconstruction processing. The infrared and visible light images are processed through encoder, layered channel enhancement network and decoder to generate high Quality fusion images.
The fusion image quality of infrared images and visible light images is improved, the richness and recognition of feature representations are enhanced, information loss is reduced, and the fusion effect is improved.
Smart Images

Figure CN119027327A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device and computer program product for determining a fused image. Background Art
[0002] Image fusion technology is to process and integrate multi-source images from different sensors to generate a composite image that contains the advantages of each source image. This technology plays an important role in improving human visual perception and computer image processing capabilities. Usually, a single imaging modality has limited information due to the limitations of its sensor technology. For example, infrared images are very good at identifying target objects through thermal radiation, but they are insufficient in expressing background structural features; on the contrary, visible light images mainly focus on texture details in the scene, but they are easy to ignore important target information in complex environments. Therefore, infrared and visible light image fusion technology can effectively integrate specific features of different modalities into a single fused image, which is widely used in pedestrian segmentation, target detection, target tracking and other fields.
[0003] Existing image fusion methods have several limitations. First, manually designing decomposition algorithms and fusion rules is time-consuming and laborious. Second, the same decomposition representation is used for all source images, thus ignoring the unique features of the images. In addition, to ensure the feasibility of subsequent feature fusion, existing image fusion methods usually use the same transformation to extract features from different source images, ignoring the inherent feature differences between them, resulting in poor feature extraction results, which in turn affects the quality of the fused image.
[0004] Therefore, in the related art, there is a problem of how to improve the quality of the fused image of the infrared image and the visible light image.
[0005] Regarding the problem of how to improve the quality of a fused image of an infrared image and a visible light image in the related art, no effective solution has been proposed yet.
[0006] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention
[0007] The embodiments of the present application provide a method, device and computer program product for determining a fused image, so as to at least solve the problem in the related art of how to improve the quality of a fused image of an infrared image and a visible light image.
[0008] According to one aspect of an embodiment of the present application, a method for determining a fused image is provided, comprising: acquiring an infrared image through an infrared sensor; acquiring a visible light image through a visible light imaging sensor; performing image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0009] In an exemplary embodiment, the convolutional neural network includes: an encoder, a hierarchical channel enhancement network, and a decoder. The hierarchical channel enhancement network includes a channel information enhancement module and a hierarchical compression module. The convolutional neural network is used to perform image processing on the infrared image and the visible light image to obtain a fused image, including: performing the feature fusion processing on the infrared image and the visible light image by the encoder to obtain a fused feature vector; performing the channel information enhancement processing on the fused feature vector by the channel information enhancement module to obtain an enhanced feature vector; performing the hierarchical compression processing on the enhanced feature vector by the hierarchical compression module to obtain a hierarchical compressed feature vector; performing the feature reconstruction processing on the hierarchical compressed feature vector by the decoder to generate a fused image.
[0010] In an exemplary embodiment, the encoder includes multiple first convolutional layers, and the encoder performs the feature fusion processing on the infrared image and the visible light image to obtain a fused feature vector, including: extracting features from the infrared image through the target first convolutional layer to obtain an infrared light feature vector; extracting features from the visible light image through the target first convolutional layer to obtain a visible light feature vector; and performing cross-correlation operation on the infrared light feature vector and the visible light feature vector through the target first convolutional layer to obtain the fused feature vector.
[0011] In an exemplary embodiment, the channel information enhancement module performs the channel information enhancement processing on the fused feature vector to obtain an enhanced feature vector, including: obtaining a first fused feature vector output by the first channel, and obtaining a second fused feature vector output by the second channel, wherein the first channel and the second channel correspond to different first convolutional layers; determining a weight matrix by the first fused feature vector and the second fused feature vector, wherein the weight matrix is used to determine the weights of different channels; performing the channel information enhancement processing on the fused feature vector by the following formula to obtain the enhanced feature vector:
[0012]
[0013] Among them, E represents the enhanced feature vector, F represents the fused feature vector, W represents the weight matrix, i represents the i-th channel, j represents the j-th channel, and c represents the number of channels.
[0014] In an exemplary embodiment, determining a weight matrix by using the first fused feature vector and the second fused feature vector includes: performing a bilinear calculation on the first fused feature vector and the second fused feature vector to obtain a bilinear matrix; and performing a normalized calculation on the bilinear matrix by using the following formula to obtain the weight matrix:
[0015]
[0016] Among them, B ij It represents the bilinear matrix calculated when the first channel is the i-th channel and the second channel is the j-th channel.
[0017] In an exemplary embodiment, performing the hierarchical compression process on the enhanced feature vector by the hierarchical compression module to obtain the hierarchical compressed feature vector includes: performing the hierarchical compression process on the enhanced feature vector by the following formula:
[0018] E1=Conv 1×1 (E0), E m+1 =Conv 1×1 (E m );
[0019] Among them, Conv 1×1 represents a 1×1 convolution, E0 represents the enhanced feature vector, E1 represents the compressed feature vector output by the first convolution layer, and E m represents the compressed feature vector of the mth convolution output, E m+1 represents the compressed feature vector output by the (m+1)th convolution layer, where m is a positive integer; and the compressed feature vectors output by all convolution layers are determined as the hierarchical compressed feature vector.
[0020] In an exemplary embodiment, the decoder includes N second convolutional layers, and the decoder performs the feature reconstruction processing on the hierarchical compressed feature vector to generate a fused image, including: inputting the first layer compressed feature vector in the hierarchical compressed feature vector into the first second convolutional layer for feature reconstruction to obtain the reconstructed feature vector output by the first second convolutional layer; inputting the n+1th layer compressed feature vector in the hierarchical compressed feature vector and the reconstructed feature vector output by the nth second convolutional layer into the n+1th second convolutional layer for feature reconstruction to obtain the reconstructed feature vector output by the n+1th second convolutional layer, wherein n is a positive integer less than N, and N is a positive integer greater than 1; performing a convolution operation on the reconstructed feature vector output by the Nth second convolutional layer to obtain a fusion result, and generating the fused image according to the fusion result.
[0021] According to another aspect of an embodiment of the present application, a device for determining a fused image is also provided, including: a first acquisition module, used to acquire an infrared image through an infrared sensor; a second acquisition module, used to acquire a visible light image through a visible light imaging sensor; an image processing module, used to perform image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0022] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining a fused image when running.
[0023] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for determining the fused image through the computer program.
[0024] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.
[0025] Through the present application, a convolutional neural network can be used to perform feature fusion processing, channel information enhancement processing, layered compression processing, feature reconstruction processing and other image processing on the acquired infrared image and the visible light image to obtain a fused image. This solves the problem of how to improve the quality of the fused image of the infrared image and the visible light image in the related art, and achieves the effect of improving the quality of the fused image of the infrared image and the visible light image. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0028] Figure 1 It is a hardware structure block diagram of a computer terminal of a method for determining a fused image according to an embodiment of the present application;
[0029] Figure 2 is a flow chart of a method for determining a fused image according to an embodiment of the present application;
[0030] Figure 3 is a schematic diagram of a method for determining a fused image according to an embodiment of the present application;
[0031] Figure 4 It is a structural block diagram of a device for determining a fused image according to an embodiment of the present application. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer terminal for determining a fused image according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Central Processing Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the fused image in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0036] A wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.
[0037] In this embodiment, a method for determining a fused image is provided. Figure 2is a flow chart of a method for determining a fused image according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0038] Step S202, acquiring an infrared image through an infrared sensor;
[0039] Step S204, acquiring a visible light image through a visible light imaging sensor;
[0040] Step S206, using a convolutional neural network to perform image processing on the infrared image and the visible light image to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0041] Through the above steps, the convolutional neural network can be used to perform feature fusion processing, channel information enhancement processing, layered compression processing, feature reconstruction processing and other image processing on the acquired infrared image and the visible light image to obtain a fused image. This solves the problem of how to improve the quality of the fused image of the infrared image and the visible light image in the related art, and achieves the effect of improving the quality of the fused image of the infrared image and the visible light image.
[0042] In an exemplary embodiment, the convolutional neural network includes: an encoder, a hierarchical channel enhancement network, and a decoder. The hierarchical channel enhancement network includes a channel information enhancement module and a hierarchical compression module. The convolutional neural network is used to perform image processing on the infrared image and the visible light image to obtain a fused image, including: performing the feature fusion processing on the infrared image and the visible light image by the encoder to obtain a fused feature vector; performing the channel information enhancement processing on the fused feature vector by the channel information enhancement module to obtain an enhanced feature vector; performing the hierarchical compression processing on the enhanced feature vector by the hierarchical compression module to obtain a hierarchical compressed feature vector; performing the feature reconstruction processing on the hierarchical compressed feature vector by the decoder to generate a fused image.
[0043] In an alternative embodiment, Figure 3 The structure of the above convolutional neural network is shown in Figure 2. Figure 3 As shown in the figure, the input of the convolutional neural network is a visible light image and an infrared image, and the output of the convolutional neural network is a fused image. The convolutional neural network consists of three parts: an encoder, an HCE (Hierarchical Channel Enhanced Network), and a decoder. The HCE specifically also includes a channel information enhancement module and a hierarchical compression module.
[0044] In an exemplary embodiment, the encoder includes multiple first convolutional layers, and the encoder performs the feature fusion processing on the infrared image and the visible light image to obtain a fused feature vector, including: extracting features from the infrared image through the target first convolutional layer to obtain an infrared light feature vector; extracting features from the visible light image through the target first convolutional layer to obtain a visible light feature vector; and performing cross-correlation operation on the infrared light feature vector and the visible light feature vector through the target first convolutional layer to obtain the fused feature vector.
[0045] In an optional embodiment, if Figure 3 As shown, the encoder includes four convolutional layers E1, E2, E3, and E4 (equivalent to the above-mentioned multiple first convolutional layers). In this embodiment, each convolutional layer contains a 3×3 convolutional block and a ReLU (Rectified Linear Unit) activation function. The convolutional layer extracts the features of the image layer by layer to form a fused feature map containing features of different modalities. Shallow features contain rich image information, but also contain more noise; deep convolutional layers capture more semantic details, less noise, and higher spatial structure and content information of image features. The specific fusion formula is as follows:
[0046]
[0047] Among them, F n represents the fused feature vector, I represents the infrared feature vector, V represents the visible light feature vector, n represents the number of convolution kernels in the first convolution layer, and Figure 3 For example, the number of convolution kernels in E1, E2, E3, and E4 are 32, 64, 128, and 256 respectively.
[0048] In an exemplary embodiment, the channel information enhancement module performs the channel information enhancement processing on the fused feature vector to obtain an enhanced feature vector, including: obtaining a first fused feature vector output by the first channel, and obtaining a second fused feature vector output by the second channel, wherein the first channel and the second channel correspond to different first convolutional layers; determining a weight matrix by the first fused feature vector and the second fused feature vector, wherein the weight matrix is used to determine the weights of different channels; performing the channel information enhancement processing on the fused feature vector by the following formula to obtain the enhanced feature vector:
[0049]
[0050] Among them, E represents the enhanced feature vector, F represents the fused feature vector, W represents the weight matrix, i represents the i-th channel, j represents the j-th channel, and c represents the number of channels.
[0051] In an exemplary embodiment, determining a weight matrix by using the first fused feature vector and the second fused feature vector includes: performing a bilinear calculation on the first fused feature vector and the second fused feature vector to obtain a bilinear matrix; and performing a normalized calculation on the bilinear matrix by using the following formula to obtain the weight matrix:
[0052]
[0053] Among them, B ij It represents the bilinear matrix calculated when the first channel is the i-th channel and the second channel is the j-th channel.
[0054] In an optional embodiment, if Figure 3 As shown, HCE can receive a total of F 32 , F 64 , F 128 , F 256 The fused feature vector of the four channels, the channel information enhancement module in HCE can enhance the discriminative ability of channel features and improve the richness and recognition of the overall feature representation by fusing the complementary information of different channels.
[0055] Here are the steps:
[0056] 1. Extract the feature maps of any two channels (the feature maps include multiple fused feature vectors) F1 and F2, then perform shape format processing on the feature maps to obtain F1' and F2', and calculate the bilinear matrix B through bilinear operation. The formula is as follows:
[0057] B=F1'·(F2') T ;
[0058] 2. Use the Softmax function (a common activation function) to normalize the bilinear matrix B and generate the weight matrix W. The formula is as follows:
[0059]
[0060] B ij It represents the bilinear matrix calculated when F1 comes from the i-th channel and F comes from the j-th channel.
[0061] 3. The feature information of different channels is integrated through the weight matrix W to obtain the enhanced feature vector E. The formula is as follows:
[0062]
[0063] Among them, E represents the enhanced feature vector, F represents the fused feature vector, W represents the weight matrix, i represents the i-th channel, j represents the j-th channel, and c represents the number of channels.
[0064] Optionally, in the above embodiment, the enhanced feature vector E may be added to the initial fused feature vector F to generate a final enhanced feature vector E', and the formula is as follows:
[0065] E' = ε(E) + F;
[0066] Among them, ε is a convolutional block and Leak ReLU (leaky linear rectifier unit) activation function.
[0067] In an exemplary embodiment, performing the hierarchical compression process on the enhanced feature vector by the hierarchical compression module to obtain the hierarchical compressed feature vector includes: performing the hierarchical compression process on the enhanced feature vector by the following formula:
[0068] E1=Conv 1×1 (E0), E m+1 =Conv 1×1 (E m );
[0069] Among them, Conv 1×1 represents a 1×1 convolution, E0 represents the enhanced feature vector, E1 represents the compressed feature vector output by the first convolution layer, and E m represents the compressed feature vector of the mth convolution output, E m+1 represents the compressed feature vector output by the (m+1)th convolution layer, where m is a positive integer; and the compressed feature vectors output by all convolution layers are determined as the hierarchical compressed feature vector.
[0070] Optionally, in the above embodiment, the hierarchical compression module is intended to compress the number of channels of the feature map layer by layer, reduce information loss, and improve the fusion effect. The specific process is as follows: the input enhanced feature vector E0 is compressed layer by layer through a 1×1 convolution operation, and the output feature vector of each layer is used as the input of the next layer of convolution, and the number of channels is compressed layer by layer, and finally a hierarchical compressed feature vector is generated. The formula is as follows:
[0071] E1=Conv 1×1 (E0), E m+1 =Conv 1×1 (E m );
[0072] Among them, Conv 1×1 represents a 1×1 convolution, E0 represents the enhanced feature vector, E1 represents the compressed feature vector output by the first convolution layer, and E mrepresents the compressed feature vector of the mth convolution output, E m+1 represents the compressed feature vector output by the (m+1)th convolution layer, where m is a positive integer; and the compressed feature vectors output by all convolution layers are determined as the hierarchical compressed feature vector.
[0073] Optionally, in the above embodiment, if m=3, E0 is input into the first layer 1×1 convolution to obtain the compressed feature vector E1, E1 is input into the second layer 1×1 convolution to obtain the compressed feature vector E2, and E2 is input into the third layer 1×1 convolution to obtain the compressed feature vector E m Hierarchical compression feature vector H = E0 + E1 + E2 + E m .
[0074] Through the synergistic effect of the above-mentioned channel information enhancement module and hierarchical compression module, the HCE module can effectively enhance the representation ability of the feature map, reduce the loss of feature information in the fusion process, and improve the quality of the fused image.
[0075] In an exemplary embodiment, the decoder includes N second convolutional layers, and the decoder performs the feature reconstruction processing on the hierarchical compressed feature vector to generate a fused image, including: inputting the Nth layer compressed feature vector in the hierarchical compressed feature vector into the first second convolutional layer for feature reconstruction to obtain the reconstructed feature vector output by the first second convolutional layer; inputting the N-1th layer compressed feature vector in the hierarchical compressed feature vector and the reconstructed feature vector output by the nth second convolutional layer into the n+1th second convolutional layer for feature reconstruction to obtain the reconstructed feature vector output by the n+1th second convolutional layer, wherein n is a positive integer less than N and N is a positive integer greater than 1; performing a convolution operation on the reconstructed feature vector output by the Nth second convolutional layer to obtain a fusion result, and generating the fused image according to the fusion result.
[0076] In an optional embodiment, the decoder starts from the deep features, performs feature reconstruction and fusion layer by layer, and finally generates a fused image. Figure 3 As shown in the figure, the decoder consists of five convolutional layers: D1, D2, D3, D4, and CONY. Each convolutional layer includes a 3×3 convolutional block and a Relu function. H4, H3, H2, and H1 represent the 4th layer compressed feature vector, the 3rd layer compressed feature vector, the 2nd layer compressed feature vector, and the 1st layer compressed feature vector from deep features to shallow features in the hierarchical compressed feature vector, which are used to supplement the information such as brightness and texture details lost in feature reconstruction. H4 obtains the reconstructed feature vector d4 through the convolutional layer D1. At this time, before d4 is input into the D2 convolutional layer, the H3 feature is guided to d4 for feature reconstruction, and the features of the deeper and shallower layers are fused layer by layer to minimize the information loss caused by direct fusion.
[0077] The final fused image F IV It is expressed as:
[0078] F IV =Con(d1);
[0079] d4=D1(H4);
[0080]
[0081] Among them, Con (equivalent to Figure 3 The CONY in ( ) consists of four 3×3 convolutional blocks. DX stands for Figure 3 For each layer except D1, the relationship is X=4-i+1.
[0082] Through the above embodiments, the method of the present invention can effectively fuse infrared and visible light images, improve the image fusion quality and information retention capability, and provide efficient support for downstream target detection and recognition tasks.
[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.
[0084] In this embodiment, a device for determining a fused image is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0085] Figure 4 is a structural block diagram of a device for determining a fused image according to an embodiment of the present application, the device comprising:
[0086] A first acquisition module 42, used to acquire an infrared image through an infrared sensor;
[0087] A second acquisition module 44 is used to acquire a visible light image through a visible light imaging sensor;
[0088] The image processing module 46 is used to perform image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0089] Through the above device, the acquired infrared image and the visible light image can be subjected to feature fusion processing, channel information enhancement processing, layered compression processing, feature reconstruction processing and other image processing by using a convolutional neural network to obtain a fused image. This solves the problem of how to improve the quality of the fused image of the infrared image and the visible light image in the related art, and achieves the effect of improving the quality of the fused image of the infrared image and the visible light image.
[0090] In an exemplary embodiment, the convolutional neural network includes: an encoder, a hierarchical channel enhancement network, and a decoder. The hierarchical channel enhancement network includes a channel information enhancement module and a hierarchical compression module. The above-mentioned image processing module 46 is also used to: perform the feature fusion processing on the infrared image and the visible light image through the encoder to obtain a fused feature vector; perform the channel information enhancement processing on the fused feature vector through the channel information enhancement module to obtain an enhanced feature vector; perform the hierarchical compression processing on the enhanced feature vector through the hierarchical compression module to obtain a hierarchical compressed feature vector; perform the feature reconstruction processing on the hierarchical compressed feature vector through the decoder to generate a fused image.
[0091] In an exemplary embodiment, the encoder includes multiple first convolutional layers, and the above-mentioned image processing module 46 is also used to: perform feature extraction on the infrared image through the target first convolutional layer to obtain an infrared light feature vector; perform feature extraction on the visible light image through the target first convolutional layer to obtain a visible light feature vector; perform cross-correlation operation on the infrared light feature vector and the visible light feature vector through the target first convolutional layer to obtain the fused feature vector.
[0092] In an exemplary embodiment, the image processing module 46 is further used to: obtain a first fused feature vector output by a first channel, and obtain a second fused feature vector output by a second channel, wherein the first channel and the second channel correspond to different first convolutional layers; determine a weight matrix by using the first fused feature vector and the second fused feature vector, wherein the weight matrix is used to determine the weights of different channels; perform the channel information enhancement processing on the fused feature vector by using the following formula to obtain the enhanced feature vector:
[0093]
[0094] Among them, E represents the enhanced feature vector, F represents the fused feature vector, W represents the weight matrix, i represents the i-th channel, j represents the j-th channel, and c represents the number of channels.
[0095] In an exemplary embodiment, the image processing module 46 is further used to: perform bilinear calculation on the first fused feature vector and the second fused feature vector to obtain a bilinear matrix; and perform normalization calculation on the bilinear matrix using the following formula to obtain the weight matrix:
[0096]
[0097] Among them, B ij It represents the bilinear matrix calculated when the first channel is the i-th channel and the second channel is the j-th channel.
[0098] In an exemplary embodiment, the image processing module 46 is further configured to perform the hierarchical compression process on the enhanced feature vector using the following formula:
[0099] E1=Conv 1×1 (E0), E m+1 =Conv 1×1 (E m );
[0100] Among them, Conv 1×1 represents a 1×1 convolution, E0 represents the enhanced feature vector, E1 represents the compressed feature vector output by the first convolution layer, and E m represents the compressed feature vector of the mth convolution output, E m+1 represents the compressed feature vector output by the (m+1)th convolution layer, where m is a positive integer; and the compressed feature vectors output by all convolution layers are determined as the hierarchical compressed feature vector.
[0101] In an exemplary embodiment, the decoder includes N second convolutional layers, and the above-mentioned image processing module 46 is further used to: input the first layer compressed feature vector in the layered compressed feature vector into the first second convolutional layer for feature reconstruction, and obtain the reconstructed feature vector output by the first second convolutional layer; input the n+1th layer compressed feature vector in the layered compressed feature vector and the reconstructed feature vector output by the nth second convolutional layer into the n+1th second convolutional layer for feature reconstruction, and obtain the reconstructed feature vector output by the n+1th second convolutional layer, wherein n is a positive integer less than N, and N is a positive integer greater than 1; perform a convolution operation on the reconstructed feature vector output by the Nth second convolutional layer to obtain a fusion result, and generate the fused image according to the fusion result.
[0102] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0103] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:
[0104] S1, obtains infrared images through infrared sensors;
[0105] S2, acquiring a visible light image through a visible light imaging sensor;
[0106] S3, performing image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0107] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0108] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0109] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0110] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0111] S1, obtains infrared images through infrared sensors;
[0112] S2, acquiring a visible light image through a visible light imaging sensor;
[0113] S3, performing image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0114] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0115] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.
[0116] Optionally, in this embodiment, the above computer program may be configured to implement the following steps when executed by a processor:
[0117] S1, obtains infrared images through infrared sensors;
[0118] S2, acquiring a visible light image through a visible light imaging sensor;
[0119] S3, performing image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
[0120] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0121] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
[0122] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for determining a fused image, characterized in that: include: Acquire infrared images through infrared sensors; Acquire a visible light image by using a visible light imaging sensor; The infrared image and the visible light image are processed by using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
2. The method according to claim 1, characterized in that The convolutional neural network includes: an encoder, a layered channel enhancement network, and a decoder. The layered channel enhancement network includes a channel information enhancement module and a layered compression module. The convolutional neural network is used to perform image processing on the infrared image and the visible light image to obtain a fused image, including: Performing the feature fusion processing on the infrared image and the visible light image by the encoder to obtain a fused feature vector; Performing the channel information enhancement processing on the fused feature vector by the channel information enhancement module to obtain an enhanced feature vector; Performing the hierarchical compression processing on the enhanced feature vector by the hierarchical compression module to obtain a hierarchical compressed feature vector; The decoder performs the feature reconstruction process on the hierarchical compressed feature vector to generate a fused image.
3. The method according to claim 2, characterized in that The encoder includes a plurality of first convolutional layers, and the encoder performs the feature fusion processing on the infrared image and the visible light image to obtain a fused feature vector, including: Performing feature extraction on the infrared image through the target first convolution layer to obtain an infrared light feature vector; performing feature extraction on the visible light image through the target first convolution layer to obtain a visible light feature vector; The infrared light feature vector and the visible light feature vector are cross-correlated by the target first convolution layer to obtain the fused feature vector.
4. The method according to claim 3, characterized in that The channel information enhancement module performs the channel information enhancement processing on the fused feature vector to obtain an enhanced feature vector, including: Obtaining a first fused feature vector output by a first channel, and obtaining a second fused feature vector output by a second channel, wherein the first channel and the second channel correspond to different first convolutional layers; Determine a weight matrix by using the first fused feature vector and the second fused feature vector, wherein the weight matrix is used to determine weights of different channels; The channel information enhancement process is performed on the fused feature vector by the following formula to obtain the enhanced feature vector: Among them, E represents the enhanced feature vector, F represents the fused feature vector, W represents the weight matrix, i represents the i-th channel, j represents the j-th channel, and c represents the number of channels.
5. The method according to claim 4, characterized in that Determining a weight matrix by using the first fused feature vector and the second fused feature vector includes: Performing bilinear calculation on the first fused feature vector and the second fused feature vector to obtain a bilinear matrix; The weight matrix is obtained by normalizing the bilinear matrix using the following formula: Among them, B ij It represents the bilinear matrix calculated when the first channel is the i-th channel and the second channel is the j-th channel.
6. The method according to claim 2, characterized in that The layered compression module performs the layered compression processing on the enhanced feature vector to obtain a layered compressed feature vector, including: The hierarchical compression process is performed on the enhanced feature vector by the following formula: E1=Conv 1×1 (E0),E m+1 =Conv 1×1 (E m ); Among them, Conv 1×1 represents a 1×1 convolution, E0 represents the enhanced feature vector, E1 represents the compressed feature vector output by the first convolution layer, and E m represents the compressed feature vector of the mth convolution output, E m+1 Represents the compressed feature vector of the m+1th convolution output, where m is a positive integer; The compressed feature vectors of all convolution outputs are determined as the hierarchical compressed feature vectors.
7. The method according to claim 6, characterized in that The decoder includes N second convolutional layers, and the decoder performs the feature reconstruction process on the hierarchical compressed feature vector to generate a fused image, including: Inputting the first layer compressed feature vector in the layered compressed feature vector into the first second convolution layer for feature reconstruction, to obtain a reconstructed feature vector output by the first second convolution layer; Inputting the n+1th layer compressed feature vector in the layered compressed feature vector and the reconstructed feature vector output by the nth second convolutional layer into the n+1th second convolutional layer for feature reconstruction, to obtain the reconstructed feature vector output by the n+1th second convolutional layer, wherein n is a positive integer less than N, and N is a positive integer greater than 1; A convolution operation is performed on the reconstructed feature vector output by the Nth second convolutional layer to obtain a fusion result, and the fused image is generated according to the fusion result.
8. A device for determining a fused image, characterized in that: include: A first acquisition module, used for acquiring an infrared image through an infrared sensor; A second acquisition module, used for acquiring a visible light image through a visible light imaging sensor; The image processing module is used to perform image processing on the infrared image and the visible light image using a convolutional neural network to obtain a fused image, wherein the image processing includes feature fusion processing, channel information enhancement processing, layered compression processing, and feature reconstruction processing.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when executed.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Bill information extraction method and device and electronic equipment
CN119888746A
