Image fusion method, device and electronic equipment

Abstract

This application provides an image fusion method, apparatus, and electronic device, applied in the field of image processing technology. The image fusion method includes acquiring a first infrared image and a first visible light image; inputting the first infrared image into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and inputting the first visible light image into a visible light image quantum coding network for feature extraction to obtain a visible light feature image; inputting the infrared and visible light feature images into a quantum feature fusion network, and fusing the infrared and visible light feature images using quantum entanglement properties to obtain a fused feature image; constructing a feature vector based on the mean and variance of the fused feature image; and inputting the feature vector into a quantum decoding network to obtain the fused image. Compared with traditional classical neural networks, quantum neural networks can learn more information in the feature space, thereby improving the image fusion effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image fusion method, apparatus and electronic device. Background Technology

[0002] Infrared and visible light images are complementary. Fusing them yields richer and more comprehensive information, resulting in higher accuracy and reliability in target detection and recognition scenarios. However, existing classical neural network techniques suffer from poor feature extraction capabilities in tasks involving the fusion of infrared and visible light images, leading to suboptimal fusion outcomes. Summary of the Invention

[0003] This application provides an image fusion method, apparatus, and electronic device to solve the problem of poor image fusion effect in existing image fusion methods.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide an image fusion method, which includes:

[0006] Acquire the first infrared image and the first visible light image;

[0007] The first infrared image is input into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and the first visible light image is input into a visible light image quantum coding network for feature extraction to obtain a visible light feature image.

[0008] The infrared feature image and the visible light feature image are input into a quantum feature fusion network, and the infrared feature image and the visible light feature image are fused using the quantum entanglement property to obtain a fused feature image;

[0009] Calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image;

[0010] The feature vectors are input into the quantum decoding network to obtain the fused image.

[0011] Optionally, the quantum feature fusion network includes controlled NOT gates and controlled P gates;

[0012] The step of inputting the infrared feature image and the visible light feature image into a quantum feature fusion network, and fusing the infrared feature image and the visible light feature image using quantum entanglement properties to obtain a fused feature image includes:

[0013] The corresponding bit information of the infrared feature image and the visible light feature image is entangled using the control NOT gate;

[0014] The controlled P gate is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image respectively, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, thereby obtaining the fused quantum state feature image;

[0015] The fused quantum state feature image is obtained by performing quantum measurements on the fused quantum state feature image.

[0016] Optionally, acquiring the first infrared image and the first visible light image includes:

[0017] Acquire the second infrared image and the second visible light image;

[0018] The second infrared image is segmented to obtain multiple infrared slice images, and the second visible light image is segmented to obtain multiple visible light slice images;

[0019] The multiple infrared image segments are compressed separately, and the compressed infrared image segments are stitched together to obtain the first infrared image.

[0020] The multiple visible light image segments are compressed separately, and the compressed visible light image segments are stitched together to obtain the first visible light image.

[0021] Optionally, the compression processing of the plurality of infrared slice images includes:

[0022] The multiple infrared slice images are input into the infrared image quantum coding network for feature encoding to obtain multiple infrared slice feature images;

[0023] The compression processing of the plurality of visible light slice images includes:

[0024] The multiple visible light slice images are input into the visible light image quantum coding network for feature encoding to obtain multiple visible light slice feature images.

[0025] Optionally, the infrared image quantum coding network includes a first quantum coding layer, a first quantum convolution layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence, and the visible light image quantum coding network includes a second quantum coding layer, a second quantum convolution layer, a second quantum pooling layer, and a second quantum measurement layer connected in sequence, wherein the convolution kernel of the first quantum convolution layer is larger than the convolution kernel of the second quantum convolution layer, and the number of layers of the second quantum convolution layer is greater than the number of layers of the first quantum convolution layer.

[0026] Optionally, the quantum decoding network includes multiple quantum decoding modules;

[0027] The construction of feature vectors based on the mean and variance of the fused feature images includes:

[0028] The mean and variance of the fused feature image are converted into a one-dimensional latent vector;

[0029] The step of inputting the feature vector into the quantum decoding network to obtain the fused image includes:

[0030] The one-dimensional potential vector is input into the multiple quantum decoding modules for decoding to obtain multiple fused sub-images. Different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image.

[0031] The multiple fused sub-images are stitched together to obtain a fused image.

[0032] Optionally, the method further includes:

[0033] The reconstruction loss value is calculated based on the difference between the fused feature image and the fused image;

[0034] The perceptual loss value is calculated based on the difference between the fused image and the first infrared image;

[0035] Based on the difference between the fused image and the first visible light image, the content loss value is calculated;

[0036] Based on the reconstruction loss value, the perception loss value, and the content loss value, the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network are adjusted to optimize the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

[0037] Secondly, embodiments of this application also provide an image fusion apparatus, which includes:

[0038] The first acquisition module is used to acquire a first infrared image and a first visible light image;

[0039] The first extraction module is used to input the first infrared image into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and to input the first visible light image into a visible light image quantum coding network for feature extraction to obtain a visible light feature image.

[0040] The first fusion module is used to input the infrared feature image and the visible light feature image into the quantum feature fusion network, and use the quantum entanglement property to fuse the infrared feature image and the visible light feature image to obtain a fused feature image;

[0041] The first construction module is used to calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image;

[0042] The first processing module is used to input the feature vector into the quantum decoding network to obtain the fused image.

[0043] Optionally, the quantum feature fusion network includes controlled NOT gates and controlled P gates;

[0044] The first fusion module includes:

[0045] The first processing unit is used to entangle the corresponding bit information of the infrared feature image and the visible light feature image using the control NOT gate;

[0046] The second processing unit is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image using the controlled P gate, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, and obtain the fused quantum state feature image.

[0047] The third processing unit is used to perform quantum measurements on the fused quantum state feature image to obtain the fused feature image.

[0048] Optionally, the first acquisition module includes:

[0049] The first acquisition unit is used to acquire the second infrared image and the second visible light image;

[0050] The first segmentation unit is used to segment the second infrared image to obtain multiple infrared slice images, and to segment the second visible light image to obtain multiple visible light slice images.

[0051] The fourth processing unit is used to compress the multiple infrared slice images respectively, and to stitch the compressed multiple infrared slice images together to obtain the first infrared image.

[0052] The fifth processing unit is used to compress the plurality of visible light slice images respectively, and to stitch the compressed plurality of visible light slice images together to obtain the first visible light image.

[0053] Optionally, the fourth processing unit includes:

[0054] The first encoding subunit is used to input the multiple infrared slice images into the infrared image quantum coding network for feature encoding to obtain multiple infrared slice feature images;

[0055] The fifth processing unit includes:

[0056] The second encoding subunit is used to input the multiple visible light slice images into the visible light image quantum encoding network for feature encoding, thereby obtaining multiple visible light slice feature images.

[0057] Optionally, the infrared image quantum coding network includes a first quantum coding layer, a first quantum convolution layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence, and the visible light image quantum coding network includes a second quantum coding layer, a second quantum convolution layer, a second quantum pooling layer, and a second quantum measurement layer connected in sequence, wherein the convolution kernel of the first quantum convolution layer is larger than the convolution kernel of the second quantum convolution layer, and the number of layers of the second quantum convolution layer is greater than the number of layers of the first quantum convolution layer.

[0058] Optionally, the quantum decoding network includes multiple quantum decoding modules;

[0059] The first building module includes:

[0060] The first conversion unit is used to convert the mean and variance of the fused feature image into a one-dimensional latent vector;

[0061] The first processing module includes:

[0062] The first decoding unit is used to input the one-dimensional potential vector into the plurality of quantum decoding modules for decoding processing to obtain a plurality of fused sub-images. The different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image.

[0063] The first stitching unit is used to stitch together the multiple fused sub-images to obtain a fused image.

[0064] Optionally, the device further includes:

[0065] The first calculation module is used to calculate the reconstruction loss value based on the difference between the fused feature image and the fused image;

[0066] The second calculation module is used to calculate the perception loss value based on the difference between the fused image and the first infrared image;

[0067] The third calculation module is used to calculate the content loss value based on the difference between the fused image and the first visible light image;

[0068] The first adjustment module is used to adjust the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network according to the reconstruction loss value, the perception loss value, and the content loss value, so as to optimize the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

[0069] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor, the processor being used for:

[0070] Acquire the first infrared image and the first visible light image;

[0071] The first infrared image is input into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and the first visible light image is input into a visible light image quantum coding network for feature extraction to obtain a visible light feature image.

[0072] The infrared feature image and the visible light feature image are input into a quantum feature fusion network, and the infrared feature image and the visible light feature image are fused using the quantum entanglement property to obtain a fused feature image;

[0073] Calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image;

[0074] The feature vectors are input into the quantum decoding network to obtain the fused image.

[0075] Fourthly, embodiments of this application also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the image fusion method described above.

[0076] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image fusion method described above.

[0077] In a sixth aspect, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the image fusion method as described in the first aspect.

[0078] The image fusion method of this application includes acquiring a first infrared image and a first visible light image; inputting the first infrared image into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and inputting the first visible light image into a visible light image quantum coding network for feature extraction to obtain a visible light feature image; inputting the infrared feature image and the visible light feature image into a quantum feature fusion network, and fusing the infrared feature image and the visible light feature image using quantum entanglement properties to obtain a fused feature image; calculating the mean and variance of the fused feature image, and constructing a feature vector based on the mean and variance of the fused feature image; and inputting the feature vector into a quantum decoding network to obtain the fused image. This method leverages the unique data expression and processing capabilities of quantum neural networks, utilizes different quantum coding networks to extract features from the first infrared image and the first visible light image, and utilizes the quantum entanglement characteristics of the quantum feature fusion network to fuse the infrared feature image and the visible light feature image. Compared to traditional classical neural networks, quantum neural networks can learn more information in the feature space, thereby improving the image fusion effect. Attached Figure Description

[0079] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This is one of the flowcharts of the image fusion method provided in the embodiments of this application;

[0081] Figure 2 This is a quantum feature fusion circuit diagram of the quantum feature fusion network provided in the embodiments of this application;

[0082] Figure 3 This is a schematic diagram of the quantum coding layer provided in an embodiment of this application;

[0083] Figure 4 This is a schematic diagram of the quantum convolutional layer and quantum pooling layer provided in the embodiments of this application;

[0084] Figure 5 This is a schematic diagram of the quantum circuit provided in an embodiment of this application;

[0085] Figure 6 This is the second flowchart of the image fusion method provided in the embodiments of this application;

[0086] Figure 7 This is a schematic diagram of the image fusion method provided in the embodiments of this application;

[0087] Figure 8 This is a structural diagram of the image fusion apparatus provided in the embodiments of this application;

[0088] Figure 9 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0089] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0090] Infrared and visible light images are image data from two different wavelength ranges. Infrared images can capture the thermal radiation information of objects and are unaffected by illumination, but often lose detailed texture information. Visible light images reflect visual information such as the shape, details, and texture of objects, but are easily affected by illumination, resulting in poor imaging quality in conditions such as darkness. Therefore, the two are complementary. Fusing infrared and visible light images can obtain richer and more comprehensive information, resulting in higher accuracy and reliability in scenarios such as target detection and target recognition. The image fusion method in this application is mainly based on quantum neural networks. Quantum neural networks utilize quantum mechanical phenomena such as quantum superposition and quantum entanglement, and can learn features beyond those of classical neural networks in certain fields, achieving better simulation of the distribution patterns between objects. Quantum neural networks have already shown some potential advantages in the field of image processing, such as quantum image processing, image compression, and image generation. This application breaks through classical neural network technology and designs an image fusion method based on quantum neural networks.

[0091] This application provides an image fusion method. See also... Figure 1 , Figure 1 This is a flowchart of the image fusion method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0092] Step 101: Acquire the first infrared image and the first visible light image;

[0093] In this step, an initial infrared image is first acquired using an infrared camera, and then the initial infrared image undergoes image preprocessing. To prevent the image from being too large to directly quantum encode all pixels, the preprocessed initial infrared image needs to be compressed to obtain the first infrared image. Similarly, the first visible light image is obtained.

[0094] Step 102: Input the first infrared image into the infrared image quantum coding network for feature extraction to obtain an infrared feature image, and input the first visible light image into the visible light image quantum coding network for feature extraction to obtain a visible light feature image;

[0095] In this step, since the first infrared image and the first visible light image contain different feature information, two quantum coding networks are constructed respectively: an infrared image quantum coding network and a visible light image quantum coding network. Features are extracted from the first infrared image using the infrared image quantum coding network to obtain an infrared feature image; similarly, features are extracted from the first visible light image using the visible light image quantum coding network to obtain a visible light feature image.

[0096] Step 103: Input the infrared feature image and the visible light feature image into the quantum feature fusion network, and use the quantum entanglement property to fuse the infrared feature image and the visible light feature image to obtain a fused feature image;

[0097] Although infrared and visible light feature images contain different types of image feature information, they are correlated because these features originate from the same object, only from different perspectives (one visible light, the other infrared). This step inputs the infrared and visible light feature images into a quantum feature fusion network, using two quantum coding circuits and auxiliary bit circuits within the network to fuse the two feature images, resulting in a fused feature image.

[0098] Step 104: Calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image;

[0099] In this step, the mean and variance of the fused feature image are calculated. Specifically, the mean of the fused feature image is calculated by dividing the sum of the pixel values ​​of all pixels in the fused feature image by the total number of pixels in the fused feature image; the variance of the fused feature image is calculated by squared the difference between the pixel value of each pixel in the fused feature image and the mean, then summing the squared differences of all pixels and dividing the sum by the total number of pixels. After calculating the mean and variance of the feature values, the mean and variance of the fused feature image are transformed into feature vectors using the reparameterization technique.

[0100] Step 105: Input the feature vector into the quantum decoding network to obtain the fused image.

[0101] To utilize quantum neural networks for image fusion, starting from step 102, the first infrared image is input into the infrared image quantum encoding network, and the first visible light image is input into the visible light image quantum encoding network, thereby converting classical image data into quantum state data. Since the final fused image data should be classical image data, the feature vector obtained in step 104 needs to be input into the quantum decoding network to convert the quantum state image data into classical image data.

[0102] In one implementation, a first infrared image and a first visible light image are first acquired. Then, based on the differences in feature information contained in the first infrared image and the first visible light image, features are extracted from the first infrared image using an infrared image quantum coding network to obtain an infrared feature image, and features are extracted from the first visible light image using a visible light image quantum coding network to obtain a visible light feature image. The obtained infrared and visible light feature images are then input into a quantum feature fusion circuit, where quantum entanglement is used to fuse the infrared and visible light feature images to obtain a fused feature image. A feature vector is constructed based on the mean and variance of the fused feature image. Finally, the feature vector is input into a quantum decoding network to obtain the fused image.

[0103] This implementation method leverages the unique data representation and processing capabilities of quantum neural networks. It utilizes different quantum coding networks to extract features from the first infrared and first visible light images, and then employs the quantum entanglement features of a quantum feature fusion network to fuse the infrared and visible light feature images. Compared to traditional classical neural networks, quantum neural networks can learn more information in the feature space, thereby improving the image fusion effect. Furthermore, quantum mechanics possesses the ability to process data rapidly; using quantum neural networks for image fusion further enhances the speed of image fusion.

[0104] Optionally, the quantum feature fusion network includes controlled NOT gates and controlled P gates;

[0105] The step of inputting the infrared feature image and the visible light feature image into a quantum feature fusion network, and fusing the infrared feature image and the visible light feature image using quantum entanglement properties to obtain a fused feature image includes:

[0106] The corresponding bit information of the infrared feature image and the visible light feature image is entangled using the control NOT gate;

[0107] The controlled P gate is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image respectively, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, thereby obtaining the fused quantum state feature image;

[0108] The fused quantum state feature image is obtained by performing quantum measurements on the fused quantum state feature image.

[0109] In one implementation, see Figure 2 , Figure 2 This diagram illustrates the quantum feature fusion circuitry of a quantum feature fusion network. The circuitry includes auxiliary circuits, infrared feature circuits, and visible light feature circuits. First, the infrared feature image is encoded onto the infrared feature circuit and subjected to convolutional feature extraction. Similarly, the visible light feature image is encoded onto the visible light feature image and subjected to convolutional feature extraction. Then, a controlled NOT gate is used to entangle the corresponding bit information of the two feature images, achieving strong entanglement at the feature space level. Next, a controlled P gate is used to perform quantum deposition on the entangled infrared and visible light feature images, transferring the bit information from each entangled feature image to its corresponding auxiliary bit, resulting in a fused quantum state feature image. Finally, quantum measurement is performed on the fused quantum state feature image to convert the quantum state image data into classical image data, yielding the fused feature image.

[0110] This implementation method utilizes different feature lines in a quantum feature fusion network to fuse infrared feature images and visible light feature images, thereby improving the image fusion effect.

[0111] Optionally, acquiring the first infrared image and the first visible light image includes:

[0112] Acquire the second infrared image and the second visible light image;

[0113] The second infrared image is segmented to obtain multiple infrared slice images, and the second visible light image is segmented to obtain multiple visible light slice images;

[0114] The multiple infrared image segments are compressed separately, and the compressed infrared image segments are stitched together to obtain the first infrared image.

[0115] The multiple visible light image segments are compressed separately, and the compressed visible light image segments are stitched together to obtain the first visible light image.

[0116] In one embodiment, the second infrared image is an image obtained after image preprocessing of an initial infrared image acquired by an infrared camera, and the second visible light image is an image obtained after image preprocessing of an initial visible light image acquired by a visible light camera. Specifically, the image preprocessing process includes:

[0117] (1) The initial infrared image and the initial visible light image are registered at the pixel level using an image registration algorithm;

[0118] (2) Since the initial infrared image often contains a lot of noise, it is necessary to filter and reduce the noise of the initial infrared image. At the same time, the image size of the initial infrared image and the image size of the initial visible light image after noise reduction are uniformly bilinearly interpolated to 64*64 to obtain the second infrared image and the second visible light image.

[0119] To address the quantum circuit size limitations in the era of Noisy Intermediate-Scale Quantum (NISQ), directly using excessively large image information as feature vectors for quantum encoding would exceed the quantum circuit size limit. Therefore, it is necessary to segment the second infrared image to obtain multiple infrared slice images, and to segment the second visible light image to obtain multiple visible light images.

[0120] Each infrared slice image from multiple infrared image segments is compressed, and then the compressed infrared slice images are stitched together to obtain a first visible light image; similarly, each visible light slice image from multiple visible light image segments is compressed, and then the compressed visible light slice images are stitched together to obtain a first infrared image. The first infrared image is smaller in size than the second infrared image, and the first visible light image is smaller in size than the second visible light image, which facilitates subsequent quantum encoding of the first infrared image and the first visible light image.

[0121] Optionally, the compression processing of the plurality of infrared slice images includes:

[0122] The multiple infrared slice images are input into the infrared image quantum coding network for feature encoding to obtain multiple infrared slice feature images;

[0123] The compression processing of the plurality of visible light slice images includes:

[0124] The multiple visible light slice images are input into the visible light image quantum coding network for feature encoding to obtain multiple visible light slice feature images.

[0125] In one embodiment, an infrared image quantum coding network is used to compress multiple infrared slice images. Specifically, multiple infrared slice images are input into the infrared image quantum coding network, which includes a first quantum coding layer, a first quantum convolutional layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence. First, each infrared slice image is input into the first quantum coding layer to encode classical image data into a quantum system, forming quantum state image data. Then, the quantum state image data is input into the first quantum convolutional layer, and the convolution kernel of the quantum convolutional layer is used to extract features from the quantum state image data to obtain a quantum state feature image. Next, the quantum state feature image is input into the quantum pooling layer to reduce the size of the feature image. Finally, the reduced quantum state feature image is input into the quantum measurement layer to convert the quantum state image data into classical image data, resulting in multiple infrared slice feature images. Based on the same processing principle, multiple visible light slice images are input into a visible light image quantum coding network for feature encoding to obtain multiple visible light slice feature images; the specific process is not described in detail here.

[0126] In this embodiment, inputting multiple infrared slice images into an infrared image quantum coding network for compression processing, and inputting multiple visible light images into a visible light image quantum coding network for compression processing, is beneficial for obtaining multiple infrared slice feature images and multiple visible light slice feature images of suitable image size.

[0127] Optionally, the infrared image quantum coding network includes a first quantum coding layer, a first quantum convolution layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence, and the visible light image quantum coding network includes a second quantum coding layer, a second quantum convolution layer, a second quantum pooling layer, and a second quantum measurement layer connected in sequence, wherein the convolution kernel of the first quantum convolution layer is larger than the convolution kernel of the second quantum convolution layer, and the number of layers of the second quantum convolution layer is greater than the number of layers of the first quantum convolution layer.

[0128] In one implementation, see Figure 3 , Figure 3 This diagram illustrates a quantum coding layer (including a first quantum coding layer and a second quantum coding layer), used to encode classical image data into quantum state image data. This application employs a probabilistic amplitude encoder to achieve quantum coding, which can exponentially reduce the number of qubits required for encoding. First, 2... n The classical features are normalized and then encoded into the quantum circuit of the image quantum coding network by a quantum encoder, achieving encoding of n qubits into 2. n Classical features, Figure 3 The image shows a 4-dimensional classical feature encoded by 2 qubits.

[0129] See Figure 4 , Figure 4 This diagram illustrates quantum convolutional layers (including first and second quantum convolutional layers) and quantum pooling layers (including first and second quantum pooling layers). Quantum convolutional layers can achieve convolution kernels of different sizes through varying parameter values, utilizing parameterized quantum circuits to perform matrix operations on the quantum state image data of the quantum coding layer. Quantum convolution possesses unique advantages not found in classical convolution. The convolution kernels of quantum convolution are orthogonal, thus providing better feature regularization and reducing the likelihood of gradient explosion. Therefore, the number of convolutions can be appropriately increased to extract richer features.

[0130] Because visible light images highlight edges and details more effectively, a 2x2 convolutional kernel with a small receptive field is used to extract details, further refined through deeper convolutions. Infrared images, possessing more semantic information and global features, are extracted using a 3x3 convolutional kernel with a large receptive field, further refined through fewer convolutions. By varying the kernel size, two types of quantum circuits can be constructed (one for infrared images and one for visible light images), allowing for feature extraction on different images. Furthermore, it should be noted that the quantum circuit for infrared image feature extraction uses two quantum convolutional layers, while the quantum circuit for visible light image feature extraction uses four convolutional layers, thus achieving different feature extraction methods.

[0131] The purpose of quantum pooling layers is to reduce the size of the feature image, thereby reducing the number of parameters. Quantum has a natural advantage in pooling; it only needs to ignore the middle and last qubits in the quantum circuit to reduce the number of qubits, achieving a 2x2 pooling dimensionality reduction operation with a stride of 2. Here, ignoring refers to ignoring the middle and last qubits in the quantum circuit. For example, if the current circuit is an 8-qubit quantum circuit, a maximum of 16x16(2) pooling can be achieved. 8 Image encoding, specifically pooling, is a form of dimensionality reduction. Therefore, after pooling, the image should be 8x8, or 2x2. 6 That is, 6 bits. Therefore, for any bit quantum circuit, only the middle and last bits need to be ignored, which is equivalent to performing one pooling operation.

[0132] The quantum measurement layer (both the first and second quantum measurement layers can be understood by referring to the explanation of quantum measurement layers) provides classical information about the expectation, variance, or probability of the quantum system. For the quantum expectation of state-based measurements, see [link to relevant documentation]. Figure 5 The desired measurement of a quantum state can be achieved through the Pauli Z-gate.

[0133] Optionally, the quantum decoding network includes multiple quantum decoding modules;

[0134] The construction of feature vectors based on the mean and variance of the fused feature images includes:

[0135] The mean and variance of the fused feature image are converted into a one-dimensional latent vector;

[0136] The step of inputting the feature vector into the quantum decoding network to obtain the fused image includes:

[0137] The one-dimensional potential vector is input into the multiple quantum decoding modules for decoding to obtain multiple fused sub-images. Different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image.

[0138] The multiple fused sub-images are stitched together to obtain a fused image.

[0139] In one implementation, after calculating the mean and variance of the fused feature image, a reparameterization technique is used to transform the mean and variance of the fused feature image into a one-dimensional latent vector. Since the fused feature image has a large noise space, this implementation uses a quantum decoding network composed of multiple quantum decoding modules. Different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image, which are then stitched together to form the final fused image. By introducing different quantum decoding modules, the quantum decoding network becomes more robust and has a better ability to adapt to noise and variations, thereby improving the quality and clarity of the resulting fused image.

[0140] It should be noted that the quantum decoding network also includes a quantum coding layer, a quantum convolutional layer, and a quantum measurement layer, but it does not have a quantum pooling layer.

[0141] Optionally, the method further includes:

[0142] The reconstruction loss value is calculated based on the difference between the fused feature image and the fused image;

[0143] The perceptual loss value is calculated based on the difference between the fused image and the first infrared image;

[0144] Based on the difference between the fused image and the first visible light image, the content loss value is calculated;

[0145] Based on the reconstruction loss value, the perception loss value, and the content loss value, the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network are adjusted to optimize the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

[0146] In one implementation, to reduce the difference between the distribution of the fused feature image and the distribution of the fused image, and to improve the clarity of the resulting fused image, a reconstruction loss value can be calculated based on the difference between the fused feature image and the fused image to optimize the structural parameters of the quantum neural network. The specific calculation of the reconstruction loss value can be found in the following formula:

[0147]

[0148] Where, μ j Let σ represent the mean of the j-th fused feature map. j Let represent the variance of the j-th fused feature map.

[0149] Since the first infrared image contains more advanced semantic information, including shape and structure, the structural parameters of the quantum neural network can be optimized by calculating the perceptual loss value based on the differences between the fused image and the first infrared image. The specific calculation formula for the perceptual loss value is as follows:

[0150] L2(x,y)=E x,y [||Vgg(y)-Vgg(x)||1]

[0151] Where x represents the first infrared image, y represents the fused image, Vgg(x) represents the features of x calculated by the Vgg network, and Vgg(y) represents the features of y calculated by the Vgg network.

[0152] Since the first visible light image contains rich information such as brightness, contrast, edges, and details, the content loss value can be calculated based on the difference between the fused image and the first visible light image to optimize the structural parameters of the quantum neural network. The calculation of the content loss value is based on the following formula:

[0153]

[0154] Where X represents the first visible light image, Y represents the fused image, and μ x Let μ be the mean of x. y Let σ be the mean of y. xy Let σ be the covariance of x and y. x 2 and σ y 2 Let x and y be the variances, and c1 and c2 be constants.

[0155] In this implementation, the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network can be adjusted by reconstructing the loss value, the perception loss value, and the content loss value, thereby optimizing the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

[0156] In one implementation, see Figure 6 and Figure 7 The acquired initial infrared and visible light images are preprocessed to obtain corresponding second infrared and visible light images. The second infrared image is segmented to obtain multiple infrared and visible light image slices. These slices are then compressed using an infrared image feature encoding network and stitched together to obtain the first infrared image. Similarly, the visible light image slices are compressed using a visible light image feature encoding network and stitched together to obtain the first visible light image. The resulting first infrared and visible light images are now large enough to be quantum-encoded into a quantum circuit.

[0157] The first infrared image is input into an infrared image quantum coding network for feature extraction to obtain an infrared feature image. The first visible light image is then input into a visible light image quantum coding network for feature extraction to obtain a visible light feature image. The infrared and visible light feature images are then input into a quantum feature fusion network, where quantum entanglement is used to fuse them, resulting in a fused feature image. The mean and variance of the fused feature image are calculated and converted into a one-dimensional latent vector Z. This one-dimensional latent vector Z and noise are input into a quantum decoding network, where different quantum decoding modules decode different parts of the fused feature image, generating different portions of the fused image.

[0158] See Figure 8 , Figure 8 This is a structural diagram of an image fusion apparatus provided in an embodiment of this application. Figure 8 As shown, the image fusion apparatus 800 includes:

[0159] The first acquisition module 801 is used to acquire a first infrared image and a first visible light image;

[0160] The first extraction module 802 is used to input the first infrared image into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and to input the first visible light image into a visible light image quantum coding network for feature extraction to obtain a visible light feature image.

[0161] The first fusion module 803 is used to input the infrared feature image and the visible light feature image into the quantum feature fusion network, and use the quantum entanglement property to fuse the infrared feature image and the visible light feature image to obtain a fused feature image;

[0162] The first construction module 804 is used to calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image;

[0163] The first processing module 805 is used to input the feature vector into the quantum decoding network to obtain the fused image.

[0164] Optionally, the quantum feature fusion network includes controlled NOT gates and controlled P gates;

[0165] The first fusion module includes:

[0166] The first processing unit is used to entangle the corresponding bit information of the infrared feature image and the visible light feature image using the control NOT gate;

[0167] The second processing unit is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image using the controlled P gate, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, and obtain the fused quantum state feature image.

[0168] The third processing unit is used to perform quantum measurements on the fused quantum state feature image to obtain the fused feature image.

[0169] Optionally, the first acquisition module includes:

[0170] The first acquisition unit is used to acquire the second infrared image and the second visible light image;

[0171] The first segmentation unit is used to segment the second infrared image to obtain multiple infrared slice images, and to segment the second visible light image to obtain multiple visible light slice images.

[0172] The fourth processing unit is used to compress the multiple infrared slice images respectively, and to stitch the compressed multiple infrared slice images together to obtain the first infrared image.

[0173] The fifth processing unit is used to compress the plurality of visible light slice images respectively, and to stitch the compressed plurality of visible light slice images together to obtain the first visible light image.

[0174] Optionally, the fourth processing unit includes:

[0175] The first encoding subunit is used to input the multiple infrared slice images into the infrared image quantum coding network for feature encoding to obtain multiple infrared slice feature images;

[0176] The fifth processing unit includes:

[0177] The second encoding subunit is used to input the multiple visible light slice images into the visible light image quantum encoding network for feature encoding, thereby obtaining multiple visible light slice feature images.

[0178] Optionally, the infrared image quantum coding network includes a first quantum coding layer, a first quantum convolution layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence, and the visible light image quantum coding network includes a second quantum coding layer, a second quantum convolution layer, a second quantum pooling layer, and a second quantum measurement layer connected in sequence, wherein the convolution kernel of the first quantum convolution layer is larger than the convolution kernel of the second quantum convolution layer, and the number of layers of the second quantum convolution layer is greater than the number of layers of the first quantum convolution layer.

[0179] Optionally, the quantum decoding network includes multiple quantum decoding modules;

[0180] The first building module includes:

[0181] The first conversion unit is used to convert the mean and variance of the fused feature image into a one-dimensional latent vector;

[0182] The first processing module includes:

[0183] The first decoding unit is used to input the one-dimensional potential vector into the plurality of quantum decoding modules for decoding processing to obtain a plurality of fused sub-images. The different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image.

[0184] The first stitching unit is used to stitch together the multiple fused sub-images to obtain a fused image.

[0185] Optionally, the device further includes:

[0186] The first calculation module is used to calculate the reconstruction loss value based on the difference between the fused feature image and the fused image;

[0187] The second calculation module is used to calculate the perception loss value based on the difference between the fused image and the first infrared image;

[0188] The third calculation module is used to calculate the content loss value based on the difference between the fused image and the first visible light image;

[0189] The first adjustment module is used to adjust the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network according to the reconstruction loss value, the perception loss value, and the content loss value, so as to optimize the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

[0190] It should be noted that this embodiment is as a comparison with... Figure 1 For a detailed implementation of the corresponding device-side embodiment, please refer to [link / reference]. Figure 1 The related descriptions of the embodiments shown in the figure will not be repeated in this embodiment to avoid repetition, and the same beneficial effects can be achieved.

[0191] This application also provides an electronic device. Since the principle by which the electronic device solves the problem is similar to the image fusion method in this application, the implementation of this electronic device can be found in the implementation of the method, and repeated details will not be described again. Figure 9 As shown, the electronic device according to an embodiment of this application includes: a processor 900, configured to read a program from a memory 920 and execute the following processes:

[0192] Acquire the first infrared image and the first visible light image;

[0193] The first infrared image is input into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and the first visible light image is input into a visible light image quantum coding network for feature extraction to obtain a visible light feature image.

[0194] The infrared feature image and the visible light feature image are input into a quantum feature fusion network, and the infrared feature image and the visible light feature image are fused using the quantum entanglement property to obtain a fused feature image;

[0195] Calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image;

[0196] The feature vectors are input into the quantum decoding network to obtain the fused image.

[0197] Among them, Figure 9In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 900) and memory (memory 920). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 900 is responsible for managing the bus architecture and general processing, and memory 920 can store data used by processor 900 during operation.

[0198] Optionally, the quantum feature fusion network includes controlled NOT gates and controlled P gates;

[0199] The processor 900 is used to read the program in the memory 920 and execute the following processes:

[0200] The corresponding bit information of the infrared feature image and the visible light feature image is entangled using the control NOT gate;

[0201] The controlled P gate is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image respectively, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, thereby obtaining the fused quantum state feature image;

[0202] The fused quantum state feature image is obtained by performing quantum measurements on the fused quantum state feature image.

[0203] Optionally, the processor 900 is configured to read the program from the memory 920 and execute the following processes:

[0204] Acquire the second infrared image and the second visible light image;

[0205] The second infrared image is segmented to obtain multiple infrared slice images, and the second visible light image is segmented to obtain multiple visible light slice images;

[0206] The multiple infrared image segments are compressed separately, and the compressed infrared image segments are stitched together to obtain the first infrared image.

[0207] The multiple visible light image segments are compressed separately, and the compressed visible light image segments are stitched together to obtain the first visible light image.

[0208] Optionally, the processor 900 is configured to read the program from the memory 920 and execute the following processes:

[0209] The multiple infrared slice images are input into the infrared image quantum coding network for feature encoding to obtain multiple infrared slice feature images;

[0210] The compression processing of the plurality of visible light slice images includes:

[0211] The multiple visible light slice images are input into the visible light image quantum coding network for feature encoding to obtain multiple visible light slice feature images.

[0212] Optionally, the infrared image quantum coding network includes a first quantum coding layer, a first quantum convolution layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence, and the visible light image quantum coding network includes a second quantum coding layer, a second quantum convolution layer, a second quantum pooling layer, and a second quantum measurement layer connected in sequence, wherein the convolution kernel of the first quantum convolution layer is larger than the convolution kernel of the second quantum convolution layer, and the number of layers of the second quantum convolution layer is greater than the number of layers of the first quantum convolution layer.

[0213] Optionally, the quantum decoding network includes multiple quantum decoding modules;

[0214] The processor 900 is used to read the program in the memory 920 and execute the following processes:

[0215] The mean and variance of the fused feature image are converted into a one-dimensional latent vector;

[0216] The processor 900 is used to read the program in the memory 920 and execute the following processes:

[0217] The one-dimensional potential vector is input into the multiple quantum decoding modules for decoding to obtain multiple fused sub-images. Different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image.

[0218] The multiple fused sub-images are stitched together to obtain a fused image.

[0219] Optionally, the processor 900 is configured to read the program from the memory 920 and execute the following processes:

[0220] The reconstruction loss value is calculated based on the difference between the fused feature image and the fused image;

[0221] The perceptual loss value is calculated based on the difference between the fused image and the first infrared image;

[0222] Based on the difference between the fused image and the first visible light image, the content loss value is calculated;

[0223] Based on the reconstruction loss value, the perception loss value, and the content loss value, the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network are adjusted to optimize the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

[0224] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described image fusion method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0225] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0226] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0228] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image fusion method, characterized in that, include: Acquire the first infrared image and the first visible light image; The first infrared image is input into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and the first visible light image is input into a visible light image quantum coding network for feature extraction to obtain a visible light feature image. The corresponding bit information of the infrared feature image and the visible light feature image is entangled using the control NOT gate in the quantum feature fusion network; The controlled P gate in the quantum feature fusion network is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image respectively, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, thereby obtaining the fused quantum state feature image; Quantum measurements are performed on the fused quantum state feature image to obtain the fused feature image; Calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image; The feature vectors are input into the quantum decoding network to obtain the fused image.

2. The image fusion method according to claim 1, characterized in that, The acquisition of the first infrared image and the first visible light image includes: Acquire the second infrared image and the second visible light image; The second infrared image is segmented to obtain multiple infrared slice images, and the second visible light image is segmented to obtain multiple visible light slice images; The multiple infrared image segments are compressed separately, and the compressed infrared image segments are stitched together to obtain the first infrared image. The multiple visible light image segments are compressed separately, and the compressed visible light image segments are stitched together to obtain the first visible light image.

3. The image fusion method according to claim 2, characterized in that, The compression processing of the multiple infrared slice images includes: The multiple infrared slice images are input into the infrared image quantum coding network for feature encoding to obtain multiple infrared slice feature images; The compression processing of the plurality of visible light slice images includes: The multiple visible light slice images are input into the visible light image quantum coding network for feature encoding to obtain multiple visible light slice feature images.

4. The image fusion method according to any one of claims 1 to 3, characterized in that, The infrared image quantum coding network includes a first quantum coding layer, a first quantum convolutional layer, a first quantum pooling layer, and a first quantum measurement layer connected in sequence. The visible light image quantum coding network includes a second quantum coding layer, a second quantum convolutional layer, a second quantum pooling layer, and a second quantum measurement layer connected in sequence. The convolution kernel of the first quantum convolutional layer is larger than the convolution kernel of the second quantum convolutional layer, and the number of layers in the second quantum convolutional layer is greater than the number of layers in the first quantum convolutional layer.

5. The image fusion method according to claim 1, characterized in that, The quantum decoding network includes multiple quantum decoding modules; The construction of feature vectors based on the mean and variance of the fused feature images includes: The mean and variance of the fused feature image are converted into a one-dimensional latent vector; The step of inputting the feature vector into the quantum decoding network to obtain the fused image includes: The one-dimensional potential vector is input into the multiple quantum decoding modules for decoding to obtain multiple fused sub-images. Different quantum decoding modules decode different parts of the fused feature image to generate different parts of the fused image. The multiple fused sub-images are stitched together to obtain a fused image.

6. The image fusion method according to claim 1, characterized in that, The method further includes: The reconstruction loss value is calculated based on the difference between the fused feature image and the fused image; The perceptual loss value is calculated based on the difference between the fused image and the first infrared image; Based on the difference between the fused image and the first visible light image, the content loss value is calculated; Based on the reconstruction loss value, the perception loss value, and the content loss value, the structural parameters of the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network are adjusted to optimize the infrared image quantum coding network, the visible light image quantum coding network, and the quantum decoding network.

7. An image fusion apparatus, characterized in that, The image fusion device includes: The first acquisition module is used to acquire a first infrared image and a first visible light image; The first extraction module is used to input the first infrared image into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and to input the first visible light image into a visible light image quantum coding network for feature extraction to obtain a visible light feature image. The first fusion module is used to input the infrared feature image and the visible light feature image into the quantum feature fusion network, and use the quantum entanglement property to fuse the infrared feature image and the visible light feature image to obtain a fused feature image; The first construction module is used to calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image; The first processing module is used to input the feature vector into the quantum decoding network to obtain the fused image; The quantum feature fusion network includes controlled NOT gates and controlled P gates; the first fusion module includes: The first processing unit is used to entangle the corresponding bit information of the infrared feature image and the visible light feature image using the control NOT gate; The second processing unit is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image using the controlled P gate, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, and obtain the fused quantum state feature image. The third processing unit is used to perform quantum measurements on the fused quantum state feature image to obtain the fused feature image.

8. An electronic device, characterized in that, Includes a processor, the processor being used for: Acquire the first infrared image and the first visible light image; The first infrared image is input into an infrared image quantum coding network for feature extraction to obtain an infrared feature image, and the first visible light image is input into a visible light image quantum coding network for feature extraction to obtain a visible light feature image. The corresponding bit information of the infrared feature image and the visible light feature image is entangled using the control NOT gate in the quantum feature fusion network; The controlled P gate in the quantum feature fusion network is used to perform quantum deposition on the entangled infrared feature image and the visible light feature image respectively, so as to transfer the bit information of each bit in the entangled infrared feature image and the visible light feature image to the corresponding auxiliary bit, thereby obtaining the fused quantum state feature image; Quantum measurements are performed on the fused quantum state feature image to obtain the fused feature image; Calculate the mean and variance of the fused feature image, and construct a feature vector based on the mean and variance of the fused feature image; The feature vectors are input into the quantum decoding network to obtain the fused image.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the image fusion method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image fusion method as described in any one of claims 1 to 6.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the image fusion method as described in any one of claims 1 to 6.

Images